[jwrosewell]

The Pipeline builder added the web integration elements again when the caller had already added them, and the Set Header Element misread two valid property names. This draft fixes both, with tests for three web integration rules the code already meets.

Problem

1. Web elements added twice. PipelineBuilder.build() joined the Sequence Element, the caller's elements, the JSON Builder and JavaScript Builder, and the Set Headers Element without looking at what the caller had added (pipelinebuilder.py lines 191 to 195 at cf904f8e). A caller's own Sequence Element ran twice, so the sequence moved on by two on every request, and a caller's own JavaScript Builder ran before the JSON Builder the builder added after it.
2. Set Header names. get_response_header_name refused an identifier whose first letter is lower case, and removed every copy of the identifier, so SetHeaderAcceptAccept-CH gave -CH (setheaderelement.py lines 192 to 233 at cf904f8e). SetHeader with nothing after it raised IndexError. get_setheader_properties_pipeline took any property with setheader anywhere in its name (line 105), so a name holding it later on would raise while processing. The names in use (SetHeaderBrowserAccept-CH, SetHeaderHardwareAccept-CH, SetHeaderPlatformAccept-CH) were read correctly.

What the specification says

web-integration.md lines 33 to 45 at 706de1f0:

• The Sequence Element MUST be present. If not, it MUST be added as the first element in the Pipeline.
• The JSON Builder Element MUST be present. If not, it MUST be added as the penultimate element. (Immediately before the JavaScript Builder Element).
• The JavaScript Builder Element MUST be present. If not, it MUST be added after all other elements.

...

• The Set Headers Element MUST be present. If not, it MUST be added after all other elements.

set-headers-element.md lines 10 to 18 at 706de1f0 give the format SetHeader[Identifier][HeaderName], where "The SetHeader prefix indicates that this Property is intended for setting HTTP response headers" and the identifier "MUST NOT have upper-case characters after the first character".

The three rules tested here and already met are web-integration.md lines 242 to 245 ("It MUST do so, and MUST NOT process a request whose 'If-None-Match' header matches"), sequence-element.md lines 39 to 44 ("A query.sequence value that is not a whole number, or a string made only of the digits 0 to 9 ... is not present") and javascript-builder.md line 348 (the endpoint "MUST return JSON data in the same form as the JSON embedded within the JavaScript").

What changed

All in pipeline/fiftyone_pipeline_core.

1. PipelineBuilder.build() adds each web integration element its settings ask for only when the caller's elements hold none with the same data key, in any letters, because Flow Data keeps each element's results under its data key in lower case and the JavaScript Builder and web.py read these elements by that key. The Sequence Element still goes first and the others last, and a JSON Builder goes immediately before a JavaScript Builder the caller added. The switches (add_javascript_builder, add_sequence_element, add_json_builder_element, add_javascript_builder_element, use_setheader_properties, add_set_header_element) work as before, and build_from_configuration is unchanged.
2. get_response_header_name takes the header name from the first upper-case letter after the identifier's first character, so the first character may be either case and the header name is kept whole. A name with no identifier or no header name raises the format message, which now names [Identifier] as the specification does. get_setheader_properties_pipeline takes only properties whose names start with SetHeader.
3. Tests only. A request whose If-None-Match matches gets 304 without the Pipeline processing it (an element counts its process calls). query.sequence given as the numbers -5 and 1.5 is not present, so the sequence is 1. The JSON endpoint's body is the JSON Builder's JSON for the same request. And a new test of BasicListEvidenceKeyFilter, which no core test covered, checks it keeps keys in any letters.

How it was verified

• Red, run 43615923409 on 9c07964, being aee0797 (the tests on rc cf904f8) plus only a throwaway workflow. The seven CallerAddedWebElementTests failed (a caller's Sequence Element ran twice and the sequence was 2, a caller's JSON Builder, JavaScript Builder and Set Headers Element were each added again, and the JSON Builder went after a caller's JavaScript Builder), and so did SetHeaderbrowserAccept-CH, SetHeaderAcceptAccept-CH and SetHeader, on Ubuntu and Windows.
• Green, run 43615953250 on 19ca5a3, being 3fc38de plus the same workflow. Every test passed.
• Red for the Set Header prefix, run 43616228099 on 58fb35d, being a3a6863 plus the workflow. Only test_get_setheader_properties_pipeline_needs_the_prefix failed ("1 failed, 504 passed" in the core tests).
• Green, run 43616259177 on 1c1382e, being the head 94914bc plus the workflow. Every test passed.
• Full build 43618020992 on the head 94914bc, from rc 6ad5ae2, is green on its second attempt. Its ten Build and Test jobs, Python 3.10 to 3.14 on Ubuntu and Windows, each passed, the 11 Setting 'PullRequestSha' lines all name 94914bc, and Configure printed "Left out 5 macOS entries". On the first attempt only Ubuntu_Python_3.14 failed, in the shared Selenium contract tests against both Device Detection examples, at "Driver service process exited unexpectedly before initialization completed" and then a WebDriver session that timed out. That is the browser driver failing to start in the shared tests, which this change does not touch, and selenium-api-tests#28 starts Chrome and Firefox once before the first test that uses them so that it stops. That job was run again and passed. macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

What it releases

fiftyone_pipeline_core and its dependants, which Monday's release plan releases already.

What remains

1. Whether the "MUST be present" rules bind a Pipeline built from a configuration file is an open question for the specification, asked in specifications#154, so build_from_configuration adds none of these elements, as before.
2. A Sequence Element the caller put somewhere other than first is left where the caller put it, as the specification says only where to add a missing one.
3. Searching the basic list filter's keys in a set would only help once each engine keeps its filter. The on-premise Device Detection and IP Intelligence engines and the Cloud Request Engine build a new filter on every call, once per Evidence key, so that change is left for next week, tracked in #156.