[jwrosewell]

Closes #11. This brings the client-side JavaScript and JSON endpoints of the web integration, served by both the servlet filter and the Spring interceptor, in line with the Cache header detail section of the web integration specification, quoted below at 6570727d, the head of the specifications main branch. That section was written by 51Degrees/specifications#53, #81, #115, #121 and #136, all merged.

The problem

On rc at b78810ff the endpoints have these faults.

1. The ETag is not an entity tag. It is the exclusive or of the Evidence values' Java hash codes, sent as a bare number such as 661586230. RFC 9110 writes an entity tag in double quotes, so the number is neither a weak nor a strong entity tag.
2. The ETag changes on every request. It is worked out from the Evidence after processing. The Sequence Element adds a new query.session-id to every request that has none, and the Pipeline's Evidence key filter holds that key, so the JavaScript endpoint sends a different ETag every time and no revalidation ever gets a 304.
3. If-None-Match is compared as one whole string. A list of tags, a weak tag, * or a second If-None-Match line never matches.
4. A failed response is cached like a good one. Every 200 gets Cache-Control: private,max-age=1800 and the ETag, even when processing recorded errors, so a browser keeps a response built without the failed element's data for 30 minutes and then revalidates it into a 304. This is Client-side responses to a request whose processing recorded errors are cached with their ETag #11.
5. A matching If-None-Match gets a bare 304 whatever the method. The 304 carries no headers, and a POST to the JSON endpoint gets a 304 too.
6. A matching request is processed first. The servlet filter and the Spring interceptor run the Pipeline on every request before the endpoint compares If-None-Match.
7. A missing builder result throws. When the builder element is the one that failed, the content is null, and response.getWriter().write(content) throws a NullPointerException after the headers are set.
8. Content-Length counts characters rather than bytes. When an application writes its responses in UTF-8, content holding a character outside ASCII, such as a city name with an accent, is longer than the header says, and the servlet container stops at the length the header gives.
9. The max-age cannot be changed. It is fixed at 1800 seconds.

What the specification says

The table gives Cache-Control max-age=1800 private, Vary "all headers in Pipeline Evidence key filter", ETag "Weak entity tag W/"<hash>" of the Evidence values in the Evidence key filter, as the request supplied them", Content-Type application/x-javascript or application/JSON, and Content-Length "Content length in bytes". The section then says the following.

• Lines 242 to 245 say "The ETag depends on the Evidence alone, as the request supplied it (see below), so the endpoint can compare the 'If-None-Match' header with the ETag before the Pipeline processes the request. It MUST do so, and MUST NOT process a request whose 'If-None-Match' header matches."
• Lines 255 to 257 say "The ETag MUST be calculated from the Evidence values in the Pipeline's Evidence key filter exactly as the request supplied them, before any Flow Element has added or replaced a value."
• Line 266 says "The same Evidence MUST give the same ETag in every process running the same build of the Pipeline API."
• Line 273 says "The ETag MUST be sent as a weak entity tag, W/"...", because it is not a strong validator."
• Lines 289 to 293 say that a response to a request whose processing recorded errors "MUST carry Cache-Control: private, no-cache and MUST NOT carry an ETag."
• Lines 304 to 310 say that a request whose If-None-Match matches gets "a 304 status code (Not Modified) to a GET or HEAD request and a 412 status code (Precondition Failed) to any other method", whether or not processing would record errors.
• Lines 313 to 319 say that a 304 "MUST carry each of the Cache-Control, Vary and ETag headers, and the Content-Location, Date and Expires headers, that a 200 response to the same request would carry", and "Because a matching request is not processed, its 304 carries the headers in the table above".
• Line 332 says "Max-age SHOULD be configurable".

RFC 9110 and RFC 9111 are both Internet Standards, and the rfc-editor.org records list nothing that obsoletes or updates either, checked on 13 November 2026.

• Section 8.8.3 gives entity-tag = [ weak ] opaque-tag, weak = %s"W/", opaque-tag = DQUOTE *etagc DQUOTE and etagc = %x21 / %x23-7E / obs-text.
• Section 8.8.3.2 says that with the weak comparison "two entity tags are equivalent if their opaque-tags match character-by-character, regardless of either or both being tagged as "weak"".
• Section 13.1.2 defines If-None-Match = "*" / #entity-tag and says "A recipient MUST use the weak comparison function when comparing entity tags for If-None-Match". A false condition gets 304 for GET or HEAD and 412 for any other method, and a list holding * and other values "is syntactically invalid".
• Section 5.2 makes several lines of one field one list, section 5.6.1.2 has a recipient accept empty list elements, and section 5.6.3 allows spaces and tabs around the commas.
• Section 13.2.1 says a server "MUST evaluate received request preconditions ... just before it would process the request content (if any) or perform the action associated with the request method".

What changed

Line numbers in items 1 to 10 are at 5d419ebc, and in item 11 at this branch's head, a9ab91d9, whose commit changed only the lines item 11 names.

1. Compared before processing. PipelineFilter.doFilter and FiftyOneInterceptor.preHandle first call the new FiftyOneJSServiceCore.serveNotModified. For a request that carries If-None-Match, to the JavaScript or the JSON path, with the client-side features on, it asks the result service for the request's FlowData without processing it, through the new PipelineResultServiceCore.createFlowData, which process now builds on. So the Evidence is the integration's own, including the web context root that the Spring evidence service adds and the JavaScript builder accepts. serveJavascriptNotModified and serveJsonNotModified then compare If-None-Match with the ETag of that Evidence. A match is answered and the FlowData closed, and nothing is processed and the rest of the chain is not called. Anything else is processed as before. A request with no If-None-Match, or to another path, creates nothing extra.
2. One answer to a match. answerMatch gives a GET or HEAD a 304 with no content carrying Vary, Cache-Control: private,max-age=<max-age>, the ETag and Access-Control-Allow-Origin, the headers of a good 200, and any other method but CONNECT, OPTIONS and TRACE (item 11) a 412 with the CORS header, so a script on another origin can read it. serveContent uses it too, so a 304 sent after processing, which only a caller of serveJavascript or serveJson that processed first can reach, carries the same headers. The 200 sends no Content-Location or Expires, so neither does the 304, and Date is the servlet container's to add.
3. A weak ETag of the supplied Evidence. getETag returns W/"<SHA-256 in lower case hex>". The values come from the WebRequestEvidence record that the web integration adds to Evidence as server.http-request, which holds each value as the request supplied it, before any Flow Element runs. Only values whose keys are in the Pipeline's Evidence key filter count. Keys are lower cased and sorted, and each key and value is written as its length in four bytes then its UTF-8 bytes, so the same values give the same ETag in every process.
4. If-None-Match evaluated as section 13.1.2 says. ifNoneMatchMatches joins every If-None-Match line into one list. * matches. Otherwise the field has to be a comma separated list of entity tags, with spaces, tabs and empty elements allowed around the commas, and it matches when one tag's quoted part equals the ETag's, with or without W/ on either side.
5. A failed response. A 200 whose flow data holds errors carries Cache-Control: private, no-cache and no ETag. Vary, Content-Type, Content-Length and the CORS header are set as before.
6. No content. A null content is not written, and the response is sent with a length of 0.
7. Vary. #86 made init read any filter that is an EvidenceKeyFilterWhitelist or derives from one, so that a parallel group's headers are in Vary, and added getHeadersAffectingJavaScript, which works the headers out again until every element has loaded. This draft adds the test of a parallel group, JavaScript_VaryHeader_ParallelElements, which passes on rc too, and setCachingHeaders leaves Vary out when no header affects the response, rather than sending it empty.
8. Content-Length. setHeaders sets the length of the content in bytes in the response's character encoding, which the servlet specification makes ISO-8859-1 when none is set.
9. A configurable max-age. ClientsidePropertyServiceCore.Default.setCacheMaxAge sets the max-age in seconds and refuses a negative number. The default stays 1800 seconds, DEFAULT_CLIENTSIDE_CACHE_MAX_AGE_SECONDS in fiftyone.pipeline.web.Constants. The servlet filter reads the clientside-cache-max-age init parameter and fails to start on a value that is not a whole number of seconds, 0 or more. The Spring integration reads FiftyOneInterceptorConfig.getClientsideCacheMaxAge(), a default method that returns 1800, which FiftyOneInterceptorConfigDefault.setClientsideCacheMaxAge sets and checks.
10. A missing builder. A Pipeline without the JavaScriptBuilderElement or JsonBuilderElement fails with a PipelineConfigurationException on every request for that endpoint, a matching one included. The check before processing leaves such a request to be processed so that the error is reported.
11. CONNECT, OPTIONS and TRACE. ifNoneMatchApplies never matches for those three methods, and serveContent (line 353) and serveNotModified (line 419) ask it in place of ifNoneMatchMatches. So such a request is processed and answered as if it carried no If-None-Match, where before it got a 412. RFC 9110 section 13.2.1 says "a server MUST ignore the conditional request header fields defined by this specification when received with a request method that does not involve the selection or modification of a selected representation, such as CONNECT, OPTIONS, or TRACE."

The new interface methods are default methods that answer nothing, so an application's own implementation of PipelineResultServiceCore, FiftyOneJSServiceCore or ClientsidePropertyServiceCore still compiles, and is then processed first as before.

Merges with rc

rc was merged in with a merge commit at 5ffe4940, bringing 70 commits, among them #80 (paths and element switches for the web integration) and #86 (the Vary headers worked out again). Seven files conflicted, and each kept both sides. Constants, FiftyOneInterceptorConfig, FiftyOneInterceptorConfigDefault and the imports of PipelineFilter hold #80's paths and switches as well as the max-age. FiftyOneInterceptorTests is rc's file with the max-age test added. ClientsidePropertyServiceCore reads Vary through #86's method and leaves an empty Vary out. Earlier merges brought #68.

Tests

Tests first. 7b0cd655 adds or changes four tests on the merged code, and 5d419ebc holds the fix.

• PipelineFilterTests.PipelineFilter_MatchingIfNoneMatch_304_NotProcessed and PipelineFilter_MatchingIfNoneMatch_Post_412_NotProcessed send If-None-Match: * to the real filter and check that nothing is stored as the request's FlowData, the rest of the chain is not called, and the answer is a 304 with private,max-age=1800 and a weak ETag, or a 412 for a POST to the JSON path. PipelineFilter_OtherIfNoneMatch_Processed checks that a tag that does not match is processed and gets a 200.
• FiftyOneInterceptorTests.FiftyOneInterceptor_MatchingIfNoneMatch_304_NotProcessed does the same through the interceptor with the real Spring services.
• ClientsidePropertyServiceTests.JavaScript_ProcessingErrors_MatchingIfNoneMatch_304 now expects the headers of a good response on a 304 while processing records errors, as lines 313 to 319 of the specification ask.

Red, run 43477586444 on the throwaway branch tmp/java-67-red (7b0cd65 with a temporary workflow that runs the web integration's test classes on ubuntu-latest with the runner's own Java and Maven). The four failed and nothing else did, 3 of 76 in pipeline.web and 1 of 13 in pipeline.web.mvc. The three that check processing failed with Mockito's "Never wanted here" for setAttribute("fiftyonedegrees.flowdata", <any>), and the service test with Cache-Control "private, no-cache" where "private,max-age=1800" was wanted.

The fix commit also adds nine tests of the new methods, which call methods that do not exist before it. ClientsidePropertyServiceTests has JavaScript_NotModified_Matching_304, Json_NotModified_Post_412, JavaScript_NotModified_OtherTag_NotAnswered, Json_NotModified_MissingBuilder_NotAnswered and NotModified_NoPipeline_NotAnswered. FiftyOneJSServiceCoreTests has NotModified_NoIfNoneMatch_NothingCreated, NotModified_OtherPath_NothingCreated, NotModified_ClientSideDisabled_NothingCreated and NotModified_JavaScriptAndJson_AskTheirEndpoint, the last of which also checks that the FlowData is closed.

Green, run 43481115579 on tmp/java-67-green (5d419eb with the same temporary workflow). Every test passed, 85 of 85 in pipeline.web and 13 of 13 in pipeline.web.mvc, with OpenJDK 17.0.21 and Maven 3.10.0 as the runner has them. Two earlier attempts on the same branch, 43478134998 and 43479426796, stopped before the tests ran. The first met a raw type, which the build's -Werror refuses, and the second met the repository's PMD rule that every FlowData is created in a try-with-resources statement, which the four tests that stub createFlowData on a mock now set aside because a mock creates nothing to close. Both are fixed in 5d419eb.

Item 11, from the review of 23:03 UTC on 13 November. d03a796d adds ClientsidePropertyServiceTests.Json_MatchingIfNoneMatch_Options_Ignored, which sends an OPTIONS with a matching If-None-Match to the JSON endpoint and expects a 200, and NotModified_IgnoredMethods_NotAnswered, which checks that serveJavascriptNotModified and serveJsonNotModified answer none of CONNECT, OPTIONS and TRACE. a9ab91d9 holds the fix. Red, run 43528259987 on tmp/java-67-red-b (d03a796 with the same temporary workflow), failed those two and nothing else, 2 of 87 in pipeline.web, with 13 of 13 passing in pipeline.web.mvc. Green, run 43528275058 on tmp/java-67-green-b (a9ab91d), passed 87 of 87 and 13 of 13.

Earlier commits on this branch were checked the same way, with red and green runs 42394657601 and 42394664423 (the failed response and the null content), 42416471804 and 42416575313 (the 412 and the 304's headers), and 42726888694 (the missing builder, the empty Vary and the negative max-age). JavaScript_ETag_SameInEveryProcess and Json_ETag_SameAcrossPipelines compare the ETag with SHA-256 values worked out apart from the Java code, with Python's hashlib over the same encoding.

Full build

Run 43530753981 built a9ab91d, this head, merged with rc (30e4886) on nine configurations, Ubuntu x64 with Java 8, 11, 17 and 21, Ubuntu arm64 with Java 17, and Windows with Java 8, 11, 17 and 21, and every job passed. Each of its ten Setting 'PullRequestSha' lines names a9ab91d, and its Configure job printed "Left out 6 macOS entries". rc has moved to 6e03a25 since (#127, #128 and #130), and this branch merges with it without conflict.

macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

For the release notes

• The client-side JavaScript and JSON endpoints answer a revalidation whose If-None-Match matches without running the Pipeline, in the servlet filter and the Spring interceptor, with a 304 to a GET or HEAD and a 412 to any other method but CONNECT, OPTIONS and TRACE, which are answered as if they carried no If-None-Match. The 304 carries the Vary, Cache-Control, ETag and CORS headers of a good response, so a browser that holds a good response keeps it while processing fails.
• The ETag is a weak entity tag, W/"<SHA-256 in hex>", worked out from the Evidence the request supplied. A browser that revalidates the JavaScript now gets a 304, where before the new session id added to every request changed the ETag each time. An ETag kept from an earlier version does not match, so each browser gets one full response after the upgrade.
• If-None-Match accepts *, a list of entity tags, several If-None-Match lines and weak tags.
• A response built while processing recorded errors sends Cache-Control: private, no-cache and no ETag.
• The Vary header is left out when no header affects the response, where it was sent empty.
• Content-Length is the length of the content in bytes.
• The max-age can be set with the clientside-cache-max-age init parameter of the servlet filter or FiftyOneInterceptorConfigDefault.setClientsideCacheMaxAge for Spring. The default stays 1800 seconds.
• A response whose builder element failed is sent with no content rather than throwing a NullPointerException.
• New default methods PipelineResultServiceCore.createFlowData, FiftyOneJSServiceCore.serveNotModified, ClientsidePropertyServiceCore.serveJavascriptNotModified and serveJsonNotModified. An application that wires the web integration itself, rather than through PipelineFilter or FiftyOneInterceptor, calls serveNotModified before processing to answer revalidations the same way.

What remains

• 51Degrees/specifications#136 lists .NET, Node.js and Go as comparing If-None-Match after processing, to change in a later release, and PHP as comparing first. 51degrees-python#59 made Python compare first.
• The 412 carries the CORS header and no other. The specification says nothing about a 412's headers.
• The specification's lines 304 to 310 give "a 412 status code (Precondition Failed) to any other method" and do not name the CONNECT, OPTIONS and TRACE exception of RFC 9110 section 13.2.1, which item 11 follows. 51Degrees/specifications#148 asks for the specification to name them.
• The JSON and the JavaScript are written in the response's character encoding, which the servlet API makes ISO-8859-1 when the application sets none, as on rc (line 281 at 30e48867). RFC 8259 section 8.1 says "JSON text exchanged between systems that are not part of a closed ecosystem MUST be encoded using UTF-8", so in an application that sets no encoding a value with a character outside Latin-1 reaches the browser as bytes that are not UTF-8. Content-Length now counts the bytes actually written, in either encoding. Which encoding the JavaScript should be sent in, and whether it names one in Content-Type, is a separate choice, so both are left to #149.