Repository navigation
Release candidate - #114
Merged
Merged
Conversation
* Point the documentation build at the example folders and package List the cloud request engine examples and the device detection examples source folder in EXAMPLE_PATH, so Doxygen finds the files the examples name. Include each client hints example's template from its own folder, and read the location package from src/fiftyone_location. * Keep the cloud web example doc comment unbroken and name its view by folder * Keep the onpremise web example doc comment unbroken and name its view by folder
…25) * Point links to the archived repositories at where the code now lives The examples, readmes and example web pages linked to the archived Python repositories, and the IP Intelligence readmes linked to the archived IP Intelligence C repository. Each link now points at the same file or folder in 51degrees-python, or at the IP Intelligence readme in 51degrees-cxx, so readers land on code that is still maintained. Links to a file follow the file to its folder in this repository, for example the Device Detection examples now sit under device-detection/fiftyone_devicedetection_examples/src. Links that used the old version/4.5 branch now use main. The match metrics example closed its Markdown link with a square bracket instead of a parenthesis. The link now closes with a parenthesis. The cloud examples readme changes into the cloned folder after cloning, so its cd line now follows the new clone address. The history table in the root README still names the earlier repositories, because their tags and releases stay there. The links to the data repositories are unchanged, because those repositories are still current. * Describe the IP Intelligence performance test gap without old repository names
* Add a temporary workflow that generates the SWIG wrappers for 51degrees-cxx 4.5.102 * Name the SWIG modules in the temporary workflow instead of reading them * Generate both SWIG wrappers again for 51degrees-cxx 4.5.102 Between 4.5.100 and 4.5.102, 51degrees-cxx removed the two EngineHash process overloads and the three EngineIpi process overloads that take required property indexes. The committed wrappers still called them, so neither on-premise package compiled against 4.5.102, which the nightly submodule update brings in #26. This moves the cxx submodule from 4110d681, the 4.5.100 release, to 1bec7b57, the 4.5.102 release, and generates hash_python_wrap.cxx, DeviceDetectionHashEngineModule.py, ipi_python_wrap.cxx and IpIntelligenceEngineModule.py again with SWIG 4.4.1 built from its v4.4.1 tag. Run against 4110d681, the same command gives files byte for byte the same as the ones committed before, so every difference here comes from cxx. Both engines call only process(evidence), which cxx keeps. * Remove the temporary SWIG wrapper workflow
Co-authored-by: Automation51D <51DCI@51Degrees.com>
…he tests and examples (#36)
…erprise data file (#43)
…1Did workflow (#58) * Fail the cloud tests when no resource key is set, and run the Cloud 51Did workflow Tests that exist to call the live cloud service skipped when no resource key reached them, and the 51Did live tests also skipped when the answer lacked what they checked, so a run with no key or with a key lacking a product reported success without the cloud being called. - Each test folder that calls the live service has the same live_cloud.py, which reads _51DEGREES_RESOURCE_KEY, then _51DEGREES_RESOURCE_KEY_BESPOKE, then resource_key, and gives the failure messages. Every skip for a missing key in the 51Did, Device Detection cloud, Device Detection example, IP Intelligence example and pipeline example tests is now a failure naming the variables, the product the key must carry and how to see what a key carries. - Where the 51Did answer lacks an identifier, the failure gives the cloud's null reason and the product. A host without the creator context fails. Without a licence key the redeem tests check the sealed result redeems as unreadable. - test_did_tier_live.py checks what a Basic key gets, being a 51Did for non-marketing and a refusal naming CloudV5FODiD for standard and personalized. - The live tests are marked, the unit steps deselect them through PYTEST_ADDOPTS, and the pipeline integration step now runs the 51Did live tests. The integration scripts fail when no TestResourceKey is passed, and the Device Detection one keeps the cloud package in the run when 51Degrees.csv is missing. - cloud-51did.yml moves to the root .github/workflows, where GitHub reads it, with its paths corrected. It fails when no key reaches it, and runs the Basic key only through the tier test. The licence-key tests in the pipeline examples and test_licensekey_live.py still skip, because CI has no cloud licence key yet (#13). * Wrap the tox.ini comment about the cloud test keys
…de (#65) - The IP Intelligence data file update example now names the LowMemory profile for the pipeline that downloads and reloads the Enterprise data file, rather than taking the engine's default. - The Device Detection getting started example's comment said it uses the low memory profile, but the code sets MaxPerformance. The comment now says MaxPerformance, and why it suits the free Lite data file. - The IP Intelligence performance example's header said worker processes share a memory mapped data file under LowMemory, as if that were the default. It now says the default MaxPerformance loads the whole file into each worker, and that -p LowMemory lets the workers share the operating system's file cache. - The IP Intelligence package readme named the Lite data file 51Degrees-LiteIpiV41.ipi, which is CI's name for it. The data repository's scripts save it as 51Degrees-LiteV41.ipi.
) The IP Intelligence on-premise specification gives Balanced as the default performance profile, and the C library's default IP Intelligence configuration is the Balanced one, but the Python engine defaulted to LowMemory. The engine now defaults to "Balanced", which its profile switch already maps to the native Balanced configuration, and the docstrings in the engine and both builders say so. Balanced loads some collections into memory and caches others, where LowMemory reads everything from the data file, so a caller relying on the default uses more memory. Naming "LowMemory" keeps the previous behaviour. A new test, on the free ASN data file, compares the collection settings of the configuration an engine built at its defaults gives the native code with those of each profile, because the engine keeps no name for the profile in use.
… as text (#64) * Accept the missing translation behavior as text, and refuse an unknown one TranslationEngineBase and Translator stored the behavior they were given without checking it, and compared it with the MissingTranslationBehavior members. A configuration file can only give text, so a configuration naming "EmptyString" or "flow_error" matched no member and a missing translation kept its original value, with nothing to say the setting had been ignored. MissingTranslationBehavior.from_value now returns a member as it is, and turns text naming a member into that member. It takes the specification's names (Original, EmptyString, FlowError) and the members' own names and values, ignoring case and underscores, and raises ValueError for anything else. The engine and the translator use it, so an unknown value stops the engine being built. The parameter is still named behavior. * Accept the specification's name for the missing translation behavior The specification names the Translation Engine's options SourceElementDataKey and MissingTranslationBehavior. Python already took source_element_data_key, but named the second option behavior. TranslationEngineBase and Translator now take missing_translation_behavior. behavior still works as a keyword-only deprecated alias that raises a DeprecationWarning, and when both are given missing_translation_behavior wins. Neither given means Original, as before. A value given by position is taken as missing_translation_behavior, as it was taken as behavior. The IP Intelligence translation engines now pass every option by its name, so they need a fiftyone_pipeline_translation that takes the new name. Their package requires fiftyone_pipeline_translation 4.5.43 or later, the version the release plan gives this commit. * Leave the IP Intelligence translation engines as they were on rc The IP Intelligence translation engines pass the missing translation behavior by position. A value passed by position is taken as missing_translation_behavior by the new translation package and as behavior by the old one, so the engines work with either release and raise no deprecation warning without any change. Passing it by name needed a minimum version of fiftyone_pipeline_translation that cannot be known until the release that carries the new name, so the engines and their package's requirements are back to their rc content. * Take None as not given for the missing translation behavior rc treated None as Original, because the translator took any value that was not EMPTY_STRING or FLOW_ERROR as Original. Refusing it would stop a configuration holding null, or code passing an unset option on as None, from building after a patch release. The repository's other engines use None to mean not set, so the translation engine and Translator now do the same. Both parameters default to None, None under either name gives Original without a warning, and anything else that names no member is still refused.
…or (#68) Name the competitor brands people search for in the remaining package text. Text only, no code change.
* Check the callback URL as the specification's six rules say Refuse an IPv6 zone identifier, an '@', '?' or '#' anywhere in the host value, and a name ending in a number unless it is an IPv4 address in dotted decimal form. The logged URL shows *** in place of everything in the host before its last '@'. * Test that a port which is present is not empty * Refuse a port that is present but empty A host such as https://example.com otherwise reads as the host https with an empty port. RFC 3986 section 3.2.3 allows an empty port, but section 6.2.3 treats it as equal to none, so the callback URL does not carry one.
Each hidden character is an asterisk, and a key of four or fewer characters is hidden entirely. The update service masks the URL it logs and the message of a failed request, and the Cloud Request Engine masks the URL in its no-data message.
…and give it Element Data (#70) * Give each web integration element its own switch and put Sequence first The Sequence, JSON Builder, JavaScript Builder and Set Headers elements each have a switch that defaults to on, and add_javascript_builder and use_setheader_properties work as before. The Sequence Element is the first element of the pipeline and also writes the session-id and sequence Element Data, as well as its Evidence. * Include the Sequence Element in the JSON, declare its properties and say the Lite file has few Properties The JSON Builder specification requires a top-level entry for each Element Data and excludes only Set Headers, the Cloud Request Engine and Usage Sharing, so the JSON Builder no longer skips the Sequence Element. The Sequence Element declares session-id and sequence as the Sequence Element specification lists them. The Device Detection example comments now say what the required examples specification says of the Lite data file. * Write an empty session id to the Sequence Element Data when the script cannot be given it The Sequence Element's session id now goes into the JSON that is written into the script, so an id the script cannot take is empty there, as it is in the script's own sessionId. * Expect the incremented sequence in the usage sharing record now that Sequence is first * Give the native engines evidence as text, as the Sequence Element now adds a number before them * Read the session id and the sequence independently in the Sequence Element A sequence with no session id now counts up with a new id. A session id that is not a non-empty string is absent, a sequence in text must be decimal digits, and the Evidence and the Element Data are written from the same values. The JSON written into the script escapes the characters that could end the script. * Check the escaped script end in the session id test * Take a sequence of 0 to 2147483646, as the specification says The Sequence Element took a sequence of 1 to 2147483647 and added one, so 2147483647 gave an output of 2147483648, which leaves a 32 bit integer. The specification sets the top at 2,147,483,646 so that adding one stays within one, which is the range 51degrees-node and 51degrees-java take. * Say the Lite data file has a small number of Properties in the last example too datafileupdate_console.py still said "limited capabilities", which is the wording 51degrees-node also replaced in its own update example. * Say plainly in the docstring what add_set_header_element does
…in every process and none after errors (#59) * Test the cache headers of the client-side endpoints * Set the client-side cache headers, with an ETag that is the same in every process and none after errors * Test that a POST with a matching ETag gets 412, and a GET or HEAD 304 * Answer a POST whose If-None-Match matches with 412, as RFC 9110 section 13.1.2 asks * Test that the web request value added for usage sharing is not in the ETag * Leave the web request value out of the ETag * Test that the client-side ETag is weak and If-None-Match is read as RFC 9110 says * Send the client-side ETag as a weak entity tag, and match If-None-Match only when it is "*" or a list of entity tags * Test that the client-side max age can be set * Let the caller set the client-side max age, which the specification says should be configurable * Read the session id from the minified script in the ETag test * Name every header in the evidence key filter in Vary, as the other languages do The Vary header named only the headers the request carried, so a cache would not re-fetch when a header the pipeline accepts appeared for the first time. The web integration specification's cache header detail says Vary carries all headers in the Pipeline Evidence key filter, and 51degrees-node, 51degrees-php and 51degrees-go all build it from each element's filter list. _vary now reads each element's evidence key filter list, and asks a filter that cannot list its keys about the request's own headers one by one, which is what PHP does. A key that is not a field name is left out, and the revalidation headers stay out as before. Three tests cover it. * Name Host in the Vary header expectations, which the JavaScript Builder adds The three new Vary tests expected only the headers their own element accepts. Run 43175480017 showed Host beside them, which is right, because the JavaScript Builder's evidence key filter accepts header.host to build the callback URL the client script posts to, and its filter is not a list so the request's own headers are asked of it one by one. * Test that Content-Length is the length of what was sent
…cence key (#89) * Test that requested properties beside a resource key are sent, with or without a licence key * Send requested properties beside a resource key, with or without a licence key The engine refused requested_properties whenever a resource key was set, because the cloud service was thought to ignore the list. The service adds the listed properties to those the resource key carries, so the engine now builds and sends the list as values, and the licence key too where there is one. It still refuses no key at all, and a licence key with no resource key or list, which the service refuses on every request. The message for properties an answer leaves out now says they are not covered by the keys in use, as a resource key can be the only one. * Test that a name left out beside a resource key alone is reported as not a property * Report a name left out beside a resource key alone as not a property, and drop history from two comments * Say the keys in use in the properties not covered warning, and drop a stale claim about discovery .NET, Java, Node.js and PHP all word the warning as not covered by the keys in use, because the licence behind a resource key and a licence key beside it both decide what comes back. The name not known warning gains the same closing line and the spelling advice those languages give. The discovery comment no longer says a licence key beside a resource key widens the answer, which the corrected Accepted combinations table replaced. Tests assert both wordings. * Test an answer that is not an object, and the message .NET and Java write The report compares the answer with the list it asked for, and an answer that parses as JSON but is not an object has nothing to compare, so the test pins that the request still succeeds and nothing is written. The warning for a name that is not a property is asserted as .NET and Java word it, both of them carrying the same sentence. * Guard the report against an answer that is not an object, and take .NET and Java's wording An answer that parses as JSON but is not an object has no sections to look in, so the report returned an AttributeError out of processing and failed a request that had succeeded. It now leaves such an answer alone, with the same guard and the same wider catch that is_cacheable uses on the same text. The warning for a name that is not a property is now word for word the sentence .NET and Java both write, so Python's two warnings match theirs exactly.
#98) * Align the data update service to the specification Send If-Modified-Since as an HTTP date, wait for the data update expected time before the first poll, add update_from_memory, and log the actions with the messages and levels in the Logging section of the data updates specification. * Treat a data file that is already current as a finished update in the update examples With If-Modified-Since sent as an HTTP date, the Distributor answers 304 when the file is current, which the examples reported as a failure. * Test that no update message holds the licence key, the watcher needs automatic updates, a file data file is written, and content checks carry on * Register the watched data files in the watcher tests with automatic updates on * Mask the licence key in every update message, create the watcher only with automatic updates, write update_from_memory's bytes to a data file kept on disk, and carry on without an MD5 header or with content that is not compressed * Test that a callback that raises does not stop the checks, and a connection failure keeps polling * Log a callback that raises, so the other callbacks and the next check still run * Test that the watcher refresh is routed as the polling refresh is * Route the watcher refresh through the polling refresh * Test the data update messages and levels against .NET * Align the data update messages and levels to .NET Each event in the data update service now carries the message and level of the .NET data update service, which the Errors table of data-updates.md follows. A failed connection, a 429, a 403, any other status, a failed integrity check and a failed refresh move from Warning to Error and take .NET wording. A failed write to the data file becomes a Warning and the engine is still refreshed from the data in memory. Every check is reported to the update complete callbacks once it ends. A watcher event with no newer file reports not needed, and one that arrived whilst another was being applied reports in progress. * Test that a callback's error is logged with .NET's message for any other error DataUpdateService.cs logs an error from an update complete callback with MessageAutoUpdateUnhandledError at Error level. The test now asserts that text, so it fails while Python writes its own sentence. * Test that a callback that raises stops the callbacks after it data-updates.md at e89679e9 says the callbacks after the one that raised are not called. On update polling the error is logged with the last row's message, which is .NET's MessageAutoUpdateUnhandledError, and on a programmatic check it passes to the caller. The tests cover polling, a programmatic check and an update from memory, and that polling still schedules its next check. This commit is the tests alone, so they fail. * Test that a callback that raises is logged and stops the callbacks after it data-updates.md at e89679e9 says the callbacks after the one that raised are not called, and that on update polling and the file watcher the error is logged with the last row's message, .NET's MessageAutoUpdateUnhandledError. Passing the error to the caller of a programmatic check, an update from memory or the update on start-up waits for next week, so on those routes it is logged in the same way for now. This commit is the tests alone, so they fail. * Stop at the first callback that raises, and log it with .NET's message DataUpdateService.cs wraps the callback loop in one try, so the first callback that raises stops the rest, and LogUpdateException logs it with MessageAutoUpdateUnhandledError, "An unhandled error occurred while checking for automatic updates for engine '{0}'", at Error. data-updates.md says the same. Python now does both, on every route, and the next check is still scheduled. * Test that an error from the errors table is logged as .NET logs it data-updates.md at e89679e9, lines 476 to 485, says an error from a row at Error level is logged at Error as "Error during check for data update" with the error attached, and the error carries the row's text. An error of any other kind, a callback's included, is logged with the last row's message, also with the error attached. Python's logger takes a string, so the error follows the entry text, its cause after it as "Error detail:". This commit is the tests alone, so they fail. * Log an error from the errors table as .NET does, with its cause An error from a row at Error level is now logged as "Error during check for data update", with the error after it, being the row's text and then its cause, as .NET logs a DataUpdateException with the exception attached. Any other error, a callback's included, is logged with the last row's message and the error after it. Python's logger takes a string, so the error follows as "Error detail: <cause>", as the other Python messages put a cause. The cause of a failed connection is masked, as it can name the URL with its licence key. * Test that the error entry is written exactly as 51degrees-node writes it 51degrees-node writes an error from a row of the errors table as "Error during check for data update. Error detail: <row text>", with ". Error detail: <cause>" after it when there is a cause, and nothing more when there is none. The entry constant now holds that text, and a new test writes out two entries in full, one without a cause and one with. This commit is the tests alone, so they fail. * Write the error entry as 51degrees-node does An error from a row of the errors table is now logged as "Error during check for data update. Error detail: <row text>", followed by ". Error detail: <cause>" when there is a cause, which is the text 51degrees-node writes at f471d65d (checkErrorText and withCause in dataFileUpdateService.js), so the two languages' logs match character for character. * Test the start-up, refresh and integrity texts as 51degrees-node writes them 51degrees-node at f471d65d writes "Updating on start-up", as the data-updates.md table at e89679e9 does, names the one try in a refresh after a check ("... after 1 tries.") and ends a refresh from memory with a full stop, and writes the integrity row as the table does, with no space before "data update". The tests now expect those texts, so they fail. * Write the start-up, refresh and integrity texts as 51degrees-node does "Updating on start-up" takes the hyphen the data-updates.md table at e89679e9 and 51degrees-node write. A failed refresh after a check or a watched change names its one try ("... after 1 tries."), and one from update_from_memory ends with a full stop, as the table's two rows and 51degrees-node's refreshDataFile do. The integrity row has no space before "data update", as the table writes it. * Test that a failed write ends the update and that an untouched watched file is not reported A data file kept on disk is read back by its engine, so the tests now use a data file whose refresh reads the file, as the on-premise engines do. They cover a download whose write fails, a temporary copy that cannot be written, an update from memory whose write fails, and a watched file nobody has written to. * End the update when the data file cannot be written, and report only the changes the watcher finds The engine of a data file kept on disk reads that file, so letting a failed write through reported a success with the old data still in use. Both routes now end with the failure, under the errors table row for the copy that replaces the file, or the row for any other error when the temporary copy is the one that cannot be written, which is what 51degrees-node writes. The watcher looks every ten seconds where .NET and 51degrees-node are told by the operating system, so a look that finds the file untouched is not an event and is not reported. The statuses take the numbers .NET gives them. * Attach the error a callback raised to the log entry The errors table in the data updates specification says an error of any other kind is logged with the error attached, and that a callback's error is logged in the same way. __emit wrote the message and bound no variable, so the error the callback raised was lost. It is now written after the message, which is what .NET, Java, Node.js and Go each do. The new test file takes the licence header every other file in the repository carries. * Name only the languages that attach a callback's error The comment on the expected message said .NET, Node.js and Go write the error the callback raised, and Java as well. Java's data update service calls each onUpdateComplete in a bare loop with no catch (DataUpdateServiceDefault.java on its rc, the loops after the update and after the file watcher's refresh), so a callback that raises there is not caught and nothing is logged. The comment now names the three that do. * One constant for the unhandled error entry, with the cause written once The merge of the data update alignment brought a second constant for the same entry. This branch already built the text with the cause attached, so appending the cause again made the expectation read it twice. The one constant carries the whole entry, and the comment saying why the cause is attached stays with it. * Test that a callback's error is masked in the log entry * Mask every licence key in a callback's error before logging it * Pair a failed temporary copy with the status its message names The errors table pairs the any other error message with AUTO_UPDATE_UNKNOWN_ERROR, and a temporary copy that cannot be written has no row of its own, so it is that error. The test asserted the copy-over-the-existing-file status instead. * A failed temporary copy reports the unknown error status The write failure logged the any other error message and returned the copy-over-the-existing-file status, which the errors table pairs with a different message. Each branch now returns the status its own row gives.
…ender 1 and no session id without one (#94) * Test that the JavaScript Builder renders sequence 1 and no session id with no Sequence Element * Render sequence 1 and an empty session id with no Sequence Element, as step 5 says * Pin both sides of the branch, and say in the older tests where the no Sequence Element case is covered * Take the session id and the sequence from the Sequence Element Step 5 of the JavaScript Builder's process reads both values from the Sequence Element's Element Data, and renders 1 and an empty session id where there is no Sequence Element. This draft had the second half and read both values from Evidence, which is the half PHP, .NET and Java already have. The Sequence Element now publishes the pair it writes to Evidence as its own Element Data, and the builder reads it from there. The Evidence writes stay, because the JSON Builder and the Sequence Element's own evidence key filter read them. The case-folded Evidence lookup the builder used for these two keys goes with them, having no other reader. A key given in other letters is still read, because Evidence.get folds the key it is given and the Sequence Element reads through it.
…eprecated, starting with FOD_CLOUD_API_URL (#108) * Test that the old cloud endpoint variable name is still read The cloud request engine, the 51Did client and the 51Did creator context example read the cloud endpoint from _51DEGREES_CLOUD_ENDPOINT first and, when that is unset or empty, from FOD_CLOUD_API_URL, reporting once that the old name is deprecated. Constants.FOD_CLOUD_API_URL and did_client.ENDPOINT_VARIABLE hold "FOD_CLOUD_API_URL" again and warn that they are deprecated when read. The environment variable name check leaves the old endpoint name alone, so its own tests move those lines from the names it finds to the names it leaves. Tests only. The code follows in the next commit. * Read FOD_CLOUD_API_URL when _51DEGREES_CLOUD_ENDPOINT is not set, and bring back the constants that held the old name The cloud request engine, the 51Did client and the 51Did creator context example read the cloud endpoint from _51DEGREES_CLOUD_ENDPOINT first. When that is unset or empty they read FOD_CLOUD_API_URL, the name the variable had in 4.5.43, and warn once with a FutureWarning that the old name is deprecated and which name replaces it. When both are set the new name wins and nothing is reported. Constants.FOD_CLOUD_API_URL is back in the cloud request engine, holding "FOD_CLOUD_API_URL", and warns with a DeprecationWarning naming Constants.CLOUD_ENDPOINT when read. did_client.ENDPOINT_VARIABLE holds "FOD_CLOUD_API_URL" again and warns in the same way, naming the new did_client.CLOUD_ENDPOINT_VARIABLE. The environment variable name check no longer fails on the old endpoint name, because the name is read again. * Test that the old 51Did endpoint constant is a plain module attribute did_client.ENDPOINT_VARIABLE has to be found every way it was before, so the tests check dir(), the module's dictionary and a star import as well as a read by name, and that the client reads the old variable through the constant, as it did in 4.5.43. Tests only. The code follows in the next commit. * Make the old 51Did endpoint constant a plain module attribute again did_client.ENDPOINT_VARIABLE was served from the module's __getattr__ so that reading it warned, which took it out of the module's dictionary, so dir() and a star import no longer gave it. It is a plain attribute again, holding "FOD_CLOUD_API_URL" and marked deprecated in its comment, which points at CLOUD_ENDPOINT_VARIABLE. The client reads the old variable through the constant, as it did in 4.5.43. * Test that every renamed environment variable is still read under its old name Every environment variable name a package, a test, an example or a CI script reads is a public name, so a name that changed to its _51DEGREES_ form is still read under the old name when the new one is unset or empty, with one message that the old name is deprecated. The tests cover the helper each test folder holds (live_cloud.py), the three example helpers and the constants they hold, the two Location cloud examples, the 51Did creator context example's two keys, the IP Intelligence test data file, and the upper case read in the two key helpers. The name check's tests say an old name is left alone in a deprecated read and in the files that may name one. Tests only. The code follows in the next commit. * Read every renamed environment variable under its old name when the new one is not set resource_key, license_key, cloud_endpoint and cloud_request_origin read as variables, 51DEGREES_DD_PATH, IPINTELLIGENCEDATAFILE and run_performance_tests are read again by the tests and examples that read them in 4.5.43, each only when its _51DEGREES_ name is unset or empty, and each printing once that the old name is deprecated. RESOURCE_KEY and LICENSE_KEY are read again by the 51Did creator context example. KeyUtils.get_named_key in fiftyone_devicedetection_shared and fiftyone_ipintelligence_shared reads the upper case form of a name again when the name as given is not set, with a FutureWarning. The constants in the example helpers that held an old name hold it again, marked deprecated, beside a constant for each new name. Each tox.ini passes the old names to the tests beside the new ones. The environment variable name check leaves an old name alone where it is the argument straight after its _51DEGREES_ name, and in the files that hold the deprecated constants, their tests and the tox lists. * Test the upper case reads in the IP Intelligence example helper, a switch read by presence, and the name check file by file The IP Intelligence example helper read its resource key and its cloud endpoint under the old names in upper case as well, so a test sets only RESOURCE_KEY and only CLOUD_ENDPOINT. A switch such as the one that turns the performance examples on is on when its variable is set, whatever it holds, so the test helper gains tests of that under both names. The name check's tests say a listed file may hold the old names it is listed with and no others, and that any other name starting with a digit still fails there. Tests only. The code follows in the next commit. * Read the IP Intelligence examples' old names in upper case, keep the performance switch a presence test, and allow each file only its own old names The IP Intelligence example helper reads RESOURCE_KEY and CLOUD_ENDPOINT again, after resource_key and cloud_endpoint, as it did in 4.5.43. The test helper live_cloud.py gains is_set, which says whether a variable is set under its new name or its old one, whatever it holds. The performance examples' switch uses it, so setting the variable to an empty value still turns them on. The environment variable name check no longer leaves a listed file out whole. Each file is listed with the old names it may hold, those names alone are left alone there, and every other old name and every other name that starts with a digit still fails. * Test that the engine reads the old endpoint variable through its constant Constants.FOD_CLOUD_API_URL has to be a plain attribute of its class, so every way of reading it gives the old name, and the engine has to read the old variable through it, as it did in 4.5.43, so that code which points the constant at another variable still decides what is read. The tests that reset a record of what has been reported no longer create the attribute when it is missing, so a rename of one would fail the tests. Tests only. The code follows in the next commit. * Make the engine's old endpoint constant a plain attribute that the engine reads Constants.FOD_CLOUD_API_URL was a descriptor that warned when read, and the engine read the old name from a private copy. So the class dictionary held an object where 4.5.43 held the name, and pointing the constant at another variable no longer changed what the engine read. It is a plain attribute again, holding "FOD_CLOUD_API_URL" and marked deprecated in its comment, which points at Constants.CLOUD_ENDPOINT, and the engine reads the old variable through it. The 51Did client already does the same.
#91) * Test that the JSON Builder leaves out the Cloud Request Engine's data * Leave the Cloud Request Engine's data out of the JSON, as the specification says
* Test that the JSON Builder writes delayexecution only for a property with a value * Write delayexecution only for a property with a value, as the specification says
* Test that the file system watcher runs on its own setting testTheWatcherRunsOnItsOwnSetting registers a data file in three cases, being the watcher on with automatic updates on, the watcher on with them off, and the watcher off with them on. A watcher is created, logged and loads a replaced file exactly when the watcher's own setting is on. The five watcher tests in FileSystemWatcherTests no longer turn automatic updates on through register_watched, which existed only to get a watcher, and testFileSystemWatcherIsLogged no longer sets auto_update. testNoFileSystemWatcherWithoutAutomaticUpdates, which held the watcher to the automatic update setting, is removed. * Run the file system watcher on its own setting register_data_file creates the watcher when file_system_watcher is on, whether or not auto_update is, as it did on main. Watching a file downloads nothing, so it needs no licence key, and the on-premise engines, which turn auto_update off without a key, keep watching the data file. Polling, the update on start-up and the licence key handling are unchanged.
Each commit main holds that rc does not is the squashed merge of an rc into main release whose content rc already holds, so this merge keeps rc's tree unchanged and only records main as a parent. Without it, merging main into a later release head conflicts in every file rc has changed since that release.
Merge main into rc
Co-authored-by: Automation51D <51DCI@51Degrees.com>
…copies (#112) The publish's Test job runs each package's tox pre-publish environment against the packages just built, listed in package/pre-publish-requirements.txt. location/tox.ini built that environment from {[testenv]deps}, which since the packages started testing against this checkout also holds editable copies of the pipeline packages. pip will not install a built and an editable copy of the same package together, so location's pre-publish stopped with ResolutionImpossible and every Test job of the publish failed. The other products keep their test tools in a [common] section and build pre-publish from that alone. location now does the same, and its py environment is unchanged.
jwrosewell
marked this pull request as ready for review
November 13, 2026 06:22
This was referenced Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Releases what has collected on
rcsince the last published release, #39, which merged on 4 November 2026 at 23:05 UTC.The last rc into main release, #67, merged on 12 November 2026 at 23:39 UTC and was not published. Its Publish run, 43331052911, failed in its Test jobs, where pip could not install a built package beside the editable copy of the same package, so its Package job was skipped and no package from #67 reached PyPI. This release publishes all of it, with these changes on top.
LICENSEfile and a test (#113).The release note below is the note of #67 with the C library version changed for #113.
This release moves Python toward one behavior across every 51Degrees language. Each element it changes now follows the published 51Degrees specifications, which in turn follow the RFCs they cite, so that the same input gives the same answer in .NET, Java, Node.js, Python, PHP, Go, Rust and C. The On-premise Engines take the C library release 51degrees-cxx 4.5.106, whose C code is that of 4.5.105. Each change names the package it is in.
Before you upgrade
ShareUsageshared every Evidence value, including every cookie, form field and query string parameter. It now shares a cookie only when its name starts51D_, and a parameter only when its name starts51D_or is onquery_whitelist. It never sharesid.email,id.salt,request-query,request-pathor theCookieheader. Passshare_all=Trueto share everything apart from those five (#50, #61,fiftyone_pipeline_engines_fiftyone).requested_package_size=10to keep the old size (#103,fiftyone_pipeline_engines_fiftyone).FlowElement.get_evidence_key_filter()accepted every key by default and now accepts none. Afterprocess(),flowdata.evidence.get()andget_all()give only the Evidence that an element in the Pipeline accepts, and the rest is in the newEvidence.get_unaccepted(). Overrideget_evidence_key_filterin your element to name the keys it reads (#63,fiftyone_pipeline_core).JSONBundlerElementraises anExceptionwith the message "The JSON Builder needs a sequence number. Place a Sequence Element before the JSON Builder in the Pipeline." when there is none. A Pipeline built withPipelineBuilder.build()already has one. A configuration file that listsJSONBundlerElementneeds aSequenceElementbefore it (#76, #101,fiftyone_pipeline_core).javascriptProperties. A WKT value, such as an IP Intelligence area, is written as{"value": "<text>"}, where it was the text alone. A Property whose value is null, and which was not given "no value", has no entry, where it was written as null. It gains asequenceentry that holdssession-idandsequence(#77, #82, #85, #70,fiftyone_pipeline_core).performance_profileunset uses more memory. Passperformance_profile="LowMemory"to keep the old profile. Device Detection also leaves Update Matched User Agent on. Passupdate_matched_useragent=Falseto turn it off (#57,fiftyone_devicedetection_onpremise, #62,fiftyone_ipintelligence_onpremise).fiftyone_ipintelligence_onpremise).IpandIpV6report the one address that was looked up. At most one of them has a value, and an IPv4-mapped address such as::ffff:8.8.8.8is reported inIpas8.8.8.8. The Engine reads the Data File's client IP keys only, so an address under any other Evidence key is no longer reported (#90,fiftyone_ipintelligence_onpremise).ValueErrorwhen the Pipeline is built. These are a callback protocol other thanhttporhttps(#40), a JavaScript object name that the script cannot declare, such aswindowordocument(#73), and an unknown missing translation behavior (#64)."EmptyString"or"FlowError"was ignored. The option ismissing_translation_behavior, and the old namebehaviorstill works with aDeprecationWarning(#64,fiftyone_pipeline_translation).InvalidBase64.FodId.to_standard_base64returnsNonefor a text it refuses, so check the result before using it (#56,fiftyone_pipeline_did).PipelineBuilder.build()puts the Sequence Element before your own elements, where it put it after them, and the Sequence Element writessession-idandsequenceas Element Data undersequenceas well as the Evidencequery.session-idandquery.sequence. So Usage Sharing records the sequence after the Sequence Element has added one to it, andFlowData.get_where("type", "int")gives one more entry.get_javascript_elements()gives the JSON Builder and the JavaScript Builder only, and the newget_sequence_elements()gives the Sequence Element (#70,fiftyone_pipeline_core).fiftyone_pipeline_engines)._51DEGREES_CLOUD_ENDPOINTreplacesFOD_CLOUD_API_URL, which still works in this release. The Cloud Request Engine and the 51Did client read_51DEGREES_CLOUD_ENDPOINT, and readFOD_CLOUD_API_URLonly when the new name is unset or empty, with one warning throughwarnings.warnthe first time each uses the old name.Constants.FOD_CLOUD_API_URLanddid_client.ENDPOINT_VARIABLEstill hold"FOD_CLOUD_API_URL"and are deprecated, andConstants.CLOUD_ENDPOINTand the newdid_client.CLOUD_ENDPOINT_VARIABLEhold the new name. Thecloud_endpointsetting still comes first. Set_51DEGREES_CLOUD_ENDPOINTto the same value as the old name (#100, #108,fiftyone_pipeline_cloudrequestengine,fiftyone_pipeline_did).cloudentry, which held the Cloud Request Engine's Element Data, being the cloud service's whole answer, as the specification asks. Client code that readcloudfrom the JSON needs to read the Properties of each Engine instead (#91,fiftyone_pipeline_core).What changes
Web integration and client-side caching
fiftyone_pipeline_core.webgainsjavascript_response()andjson_response(), which answer the client-side JavaScript and JSON endpoints of a Flask application withCache-Control,Vary,Content-Type,Content-Lengthand an ETag. The examples use them.max_age_secondssets the max age, which is 1,800 seconds unless given (#59,fiftyone_pipeline_core).W/"...", andIf-None-Matchis compared weakly, as RFC 9110 section 8.8.3 and section 13.1.2 ask. A request whoseIf-None-Matchmatches gets 304 for GET and HEAD, and 412 for any other method. A response to a request whose processing failed hasCache-Control: private, no-cacheand no ETag. The same Evidence gives the same ETag in every process, so a browser's cached response stays valid behind a load balancer (#59,fiftyone_pipeline_core).PipelineBuildergains the settingsadd_sequence_element,add_json_builder_element,add_javascript_builder_elementandadd_set_header_element, each true by default, so each element can be left out on its own.add_javascript_builderanduse_setheader_propertieskeep their meaning (#70,fiftyone_pipeline_core).JavaScript Builder
httporhttps, and the host must be ASCII labels or an IPv6 address in brackets, with no user information, query, fragment or zone identifier, following RFC 3986. A URL from the request that fails is left out of the script and logged (#40, #66,fiftyone_pipeline_core).fiftyone_pipeline_core).<,>and&as\u003c,\u003eand\u0026, so a value cannot end the script early (#70,fiftyone_pipeline_core).JSON Builder
profiles, is written as a list of objects, and an empty list as[]. Both were left out (#85,fiftyone_pipeline_core).delayexecutiononly for a Property that has a value, as the specification and .NET, Java and Go do (#92,fiftyone_pipeline_core).Evidence
fiftyone_pipeline_core).webevidence()adds each session value assession.<name>Evidence, and a read-only view of the web request asserver.http-request(#61,fiftyone_pipeline_core).header.User-Agentandheader.user-agentare one key.get_all()gives each key in the letters it was first added with (#95,fiftyone_pipeline_core).Cloud
resource,license,licenceandvalues, which it writes itself. The name sent is everything after the first dot of the key, in the key's own letter case. Where two keys give the same name, thequeryvalue is sent, thenheader,cookie,server,fiftyoneandlocation(#79, #99,fiftyone_pipeline_cloudrequestengine).fiftyone_pipeline_cloudrequestengine).*(#78,fiftyone_pipeline_core).requested_propertiescan be given beside a Resource Key, and the Cloud Request Engine sends the list, where it refused it. A name the cloud service does not know is reported with a message that asks you to check its spelling (#89,fiftyone_pipeline_cloudrequestengine).Engine.is_cacheable(data)decides, and is true unless an Engine says otherwise (#83,fiftyone_pipeline_engines,fiftyone_pipeline_cloudrequestengine).Usage Sharing
fiftyone_pipeline_engines_fiftyone).escaped="true", where it wasreplaced="true", and cuts a value at 10,000 characters, where it was 512 (#71,fiftyone_pipeline_engines_fiftyone).track_sessionoption tells sessions apart (#53, #50,fiftyone_pipeline_engines_fiftyone).Data updates
If-Modified-Sinceis sent as an HTTP date (#69,fiftyone_pipeline_engines).DataFileUpdateService.update_from_memory(data_file, data)refreshes a data file kept in memory from bytes you give it (#69,fiftyone_pipeline_engines).file_system_watchersetting, whether automatic updates are on or not, and needs no License Key, as in the published packages (#69, #110,fiftyone_pipeline_engines).UpdateStatusgainsAUTO_UPDATE_TEMP_PATH_NOT_SETandAUTO_UPDATE_UNKNOWN_ERROR, numbered 12 and 13 as in .NET. A download that cannot be written to the data file's path ends withAUTO_UPDATE_NEW_FILE_CANT_RENAME, and the Engine keeps the data it has (#98,fiftyone_pipeline_engines).Device Detection and IP Intelligence
IsMobileby position took 10.1% to 16.2% less time than a read by name in each of the 10 jobs of run 43056841063 (#93, #102, both on-premise packages).fiftyone_ipintelligence_onpremise).C library
Examples and documentation
LICENSEfile at its root, the EUPL 1.2 text, and every source file carries the standard 51Degrees header (#104).Environment variables
Each name below has a new name that starts
_51DEGREES_, which every shell can set. The Cloud Request Engine and the 51Did client read the cloud endpoint,KeyUtils.get_named_keyinfiftyone_devicedetection_sharedandfiftyone_ipintelligence_sharedreads the upper case form of the name it is given, and the examples and tests read the rest. Every old name is still read in this release, but only when its new name is unset or empty. Using an old name is reported once as deprecated, withwarnings.warnin the Cloud Request Engine and the 51Did client, with aFutureWarninginKeyUtils.get_named_key, and on the error stream in the examples. The Pipeline settingsresource_key,license_key,cloud_endpointandcloud_request_originare unchanged (#97, #100, #108).FOD_CLOUD_API_URL,cloud_endpoint,CLOUD_ENDPOINT_51DEGREES_CLOUD_ENDPOINTcloud_request_origin_51DEGREES_CLOUD_REQUEST_ORIGINresource_key,RESOURCE_KEY_51DEGREES_RESOURCE_KEYlicense_key,LICENSE_KEY_51DEGREES_LICENSE_KEY51DEGREES_DD_PATH_51DEGREES_DD_PATHIPINTELLIGENCEDATAFILE_51DEGREES_IPI_PATHrun_performance_tests_51DEGREES_RUN_PERFORMANCE_TESTSKeyUtils.get_named_keyFixed
AttributeErrorfor every cookie whose name did not start51d_, oncecookie,query_whitelistorheader_blacklistwas set (#50,fiftyone_pipeline_engines_fiftyone).Evidence.filter()raisedRuntimeError: dictionary changed size during iterationwhen it removed a key that no element accepted (#50,fiftyone_pipeline_core).<,&or"made a usage sharing message that did not parse, so the whole message was lost (#71,fiftyone_pipeline_engines_fiftyone).fiftyone_pipeline_engines_fiftyone).fiftyone_pipeline_core).IsMobilerefusedflowdata.device.ismobile(#93,fiftyone_pipeline_engines).fiftyone_ipintelligence_onpremise).content-md5header whenverify_md5was on, content that was not gzip whendecompresswas on, or a callback that raised (#69,fiftyone_pipeline_engines).If-Modified-Sincewas not sent as an HTTP date, so a server ignored it (#69,fiftyone_pipeline_engines).PipelineBuilder.build()twice added the Sequence Element, the JSON Builder and the JavaScript Builder twice (#70,fiftyone_pipeline_core).Query.Session-Id, was set aside, so the Sequence Element started a new session (#95,fiftyone_pipeline_core).Issues this closes
None. The five issues that this work fixes closed when #67 merged into
main.Build
Full build 43362884264 is green, with macOS. It ran from 04:02 to 05:51 UTC on 13 November 2026 on
tmp/pr-matrix-114rel. All 16Setting 'PullRequestSha'lines, being Configure and the 15 Build and Test jobs, read55e486acc6d403f62ad7bb73e54361d1929c7df0, which is this pull request's head and the head ofrc, so every job built this tree and nothing else. Every job is a success, being Configure, Build and Test on Python 3.10 to 3.14 on each of Ubuntu, Windows and macOS, Compare Performance and Complete. Configure printed no "Left out N macOS entries" line. Report Compare Performance was skipped because Compare Performance passed.Compare Performance (job 131718177579) passed for all 15. The floor is the average of the last six results less the larger of two standard deviations and 10 per cent of the average. The figures are detections per second.
The changes go both ways, from 27.7 per cent down to 39.5 per cent up, and their median is 4.9 per cent down. The lowest is macOS Python 3.11, 1.85 standard deviations below its average and above its floor, at 19,964 against 19,331. Every result is inside the gate that Compare Performance applies.
Against
main, this head differs from #67's merge commit6b789d72only by #111, which changes no file, #112, which changes onlylocation/tox.ini, and #113, which moves the C library to a release with the same C code and the JavaScript templates by aLICENSEfile and a test. So no Python source, C source or JavaScript template differs frommain, and the full build of #67, run 43310906727 onc69d00fe, was green with macOS on that same source.Only the patch number moves.