[jwrosewell]

Python had no element for the derived properties that derived-properties defines, so a Python pipeline could not offer HumanConfidence or IsGdpr, whilst .NET, Rust and PHP can. This adds a pipeline/fiftyone_pipeline_derivedproperty package whose DerivedPropertyElement runs every script in the scripts folder of derived-properties, and tests that run every case file in its tests folder, the scripts under tests/invalid included.

Closes #105.

What the specification says

All quotes are from derived-properties at cc306386, the head of its main and the commit the new submodule pins.

docs/format-1.md, lines 4 to 6:

Every language implementation of the element (.NET, Node, Java, Python and PHP) is tested against the behaviour described here, so this document is the contract rather than a summary of one.

The one rule, lines 35 to 39:

Where every one of those properties is available, the checks and the rules run and a value is chosen. Where any one of them is not available, the script produces no value, and the message names every property that was missing along with what its source element said about each one.

When a property is absent, lines 451 to 455:

Nothing is evaluated. The output property is present in the element data with no value, and reading it raises the language's existing no value error, which is AspectPropertyValue with a NoValueMessage in .NET and the matching mechanism in each other language. The message names every absent property rather than only the first one found.

Lines 491 to 497, on an element added after a source it reads:

Where an element runs is not judged at build, since a pipeline holding elements that run in parallel reports them in an order that does not say what ran before what. An element added after a source it reads therefore builds, and produces no value on every request, with this message naming the property. An implementation should log that once rather than on every request, because it will otherwise repeat for the life of the process.

Writing YAML, lines 562 to 570:

YAML 1.1 parsers, which several languages still ship, read yes, no, on and off as booleans ... The reference tools in this repository use the YAML core schema, where yes stays the string "yes"

Two places where the languages are allowed to differ, lines 590 to 605, being the line a fault names, and how strictly a JSON script is read:

Text that is broken is reported as broken either way and only the wording of the fault differs, so no script is valid in one language and invalid in another.

docs/testing.md, lines 3 to 6:

every 51Degrees language implementation runs the same case files, so a case written here becomes a test in each language as that language is built, starting with .NET and followed by Node, Java, Python and PHP.

Lines 177 and 196 to 197:

Every property that could not be read has to be listed, so a case naming one of two unreadable properties fails

tests/invalid/ holds scripts that every implementation must refuse, along with what each refusal has to say.

docs/authoring.md, lines 365 to 384 asks for one information line per script at build naming "the script, its version, the format number, where the script came from (a built in name, a file path or code) and the output property", and at debug level "the compiled model as canonical JSON". Lines 441 to 442 add a warning for a deprecated script. Lines 452 to 464 describe the check when the element is added, where "a source property that no earlier element supplies" and "two elements in one pipeline producing the same derived property name" fail the pipeline build. The word "earlier" there is at odds with format-1.md lines 491 to 494 above, and .NET, Rust and PHP accept an element that supplies the property wherever it sits, so Python follows the format reference, which calls itself the contract. derived-properties#19 asks for authoring.md to say the same.

README.md, lines 80 to 109 names the build parameters Scripts and ScriptFiles, each "one string with commas between the names", where "a name that holds a comma is wrapped in double quotes".

What changed

The reference implementations read were .NET, pipeline/DerivedProperty of 51degrees-dotnet, Rust, the derived-properties crate of 51degrees-rust, and PHP, pipeline/derivedproperty of 51degrees-php. The Python code ports the reference tools the specification follows, tools/parse.mjs, validate.mjs, evaluate.mjs and canonical.mjs, function for function, with the same fault paths and messages. Files are linked at the head, 9a8bd25d.

1. The submodule. .gitmodules adds derived-properties as pipeline/fiftyone_pipeline_derivedproperty/src/fiftyone_pipeline_derivedproperty/derived-properties with the relative URL ../derived-properties, pinned at cc306386. It sits inside the package's source folder so that the package reads the built in scripts from it directly, with no copy to keep in step.
2. The package carries the scripts. MANIFEST.in and include_package_data in setup.py put the submodule's scripts/*.yaml in the source distribution and the wheel, and nothing else from the submodule.
3. Reading YAML. The package requires PyYAML, which follows YAML 1.1, where yes, no, on and off are booleans and a date is a date. parse.py uses a loader that keeps only the core schema's null, boolean, integer and float forms, so yes stays the string "yes" as the reference tools read it, and it records the line of each mapping so a fault names its line. JSON scripts are read with the json module, the strict reading the specification allows. jsvalues.py holds the JavaScript rules for numbers, equality and text that the reference tools rely on, so the evaluator gives the same answers.
4. The element. DerivedPropertyElement writes one AspectPropertyValue per script under derived, through DerivedPropertyData, where names are matched without regard to case. A script with a source property missing has no value, and reading value() raises the message naming every missing property, which is the Python pipeline's no value error. It reads a source as Python elements hold it, carrying the source's own no value message, or the message of the exception an element raises for a property it does not have, and turning a weighted list into the Value and Weight shape the evaluator takes. A script reads what an earlier script of the element wrote. A request with no element data at all under a key a script reads is logged once for each script, as format-1.md lines 491 to 497 ask.
5. Configuration. The constructor takes the scripts and script_files settings, which a pipeline configuration file gives, matched without regard to case or underscores, so Scripts and ScriptFiles work as the README names them. DerivedPropertyElementBuilder offers add_built_in_script, add_script_file and add_script in code, as .NET's builder does. Every fault in every script is reported in one ScriptValidationError. A setting the element does not have is refused, so a setting written wrongly cannot leave its scripts out without a word, and an element with no script is refused with a message saying how to give one.
6. When the pipeline is built. on_registration refuses the pipeline with the same messages as .NET, Rust and PHP where nothing supplies a source property or another element writes the same derived property, then logs the line and the canonical JSON authoring.md asks for, and a warning for a deprecated script. Where an element sits is not judged, and an element that lists no properties may supply any, because a cloud engine learns its properties from its first response.
7. Metadata. Each property's metadata has the keys the other Python elements give, being name, type, category and description, followed by the rest of the script's Output block.
8. Wiring. ci/packages.json lists the package with "Publish": false, depending on fiftyone_pipeline_core, and pipeline/ci/build-project.ps1 and pipeline/ci/run-unit-tests.ps1 add it to the pipeline packages, so every build lints it and runs its tests, and Publish leaves it out. AGENTS.md, the README and pipeline/readme.md describe the package and the submodule.

The tests, in pipeline/fiftyone_pipeline_derivedproperty/tests, 224 in all:

• test_shared_cases.py runs the 69 value cases (57 for HumanConfidence and 12 for IsGdpr) twice, once with the source values as AspectPropertyValue and once as plain values, each through a real pipeline with a source element giving the case's properties and the script the package carries, and the 29 rejection cases, each refused with a fault at every expected path and mentioning every expected fragment. A missing value must name exactly the properties the case lists.
• test_built_in_scripts.py checks that the built in scripts are the submodule's scripts folder and builds one element with every one of them.
• test_pipeline_check.py follows the Rust crate's tests/pipeline_check.rs.
• test_element.py and test_scripts.py cover reading the values Python elements give, logging, settings, metadata, the JSON bundler, YAML and JSON, type inference and conversion.

Runs

Each throwaway run uses a workflow that runs the package's tox environments, as every build does and as Publish does, and pylint with the package's .pylintrc, on Ubuntu with Python 3.10 and 3.14 and on Windows with Python 3.14. A fourth job builds the wheel and the source distribution, lists both, installs the wheel into an empty virtual environment with fiftyone_pipeline_core from PyPI, and runs the built in scripts.

• Red, run 43530309792 on b11e1fc, failure. Its parent is the test commit 8c9fe53, which holds the submodule, the package's setup.py, tox.ini and MANIFEST.in, and the tests, and none of the package's code. Every test job stopped with five errors while collecting its five test files, four of them ImportError: cannot import name 'DerivedPropertyElement' (or 'DerivedPropertyElementBuilder') from 'fiftyone_pipeline_derivedproperty' and one ModuleNotFoundError: No module named 'fiftyone_pipeline_derivedproperty.canonical', and the build job failed with the same ImportError for DerivedPropertyElementBuilder.
• Green, run 43530376507 on a1855dc, success. Its parent is 55bf271, the code commit, and nothing else changes. Each test job ran "224 passed", and pylint rated the package 10.00/10. The build job listed fiftyone_pipeline_derivedproperty/derived-properties/scripts/HumanConfidence.yaml and IsGdpr.yaml in the wheel, installed fiftyone_pipeline_core 4.5.43 from PyPI beside it, built both scripts as HumanConfidence 0.8.0 from built in HumanConfidence and IsGdpr 0.1.0 from built in IsGdpr, and answered IsGdpr True for FR.
• Since then rc was merged in twice, at 853446b and at the head 9a8bd25d, each with no conflict and no change to the package.
• Full build, run 43560049158 from tmp/pr-matrix-125b, success. Configure, all ten Build and Test jobs (Ubuntu and Windows, Python 3.10 to 3.14) and Complete are green. All 11 Setting 'PullRequestSha' lines name the head 9a8bd25d, and Configure printed "Left out 5 macOS entries".
• Full build again, run 43666576497 from tmp/pr-matrix-125c, on the same head against rc at cf520fe, which holds this week's other changes, success. Configure, all ten Build and Test jobs (Ubuntu and Windows, Python 3.10 to 3.14) and Complete are green, fiftyone_pipeline_derivedproperty is built and its tests run in each job, all 11 Setting 'PullRequestSha' lines name 9a8bd25, and Configure printed "Left out 5 macOS entries".

macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

What remains

• The package is not published, as it has no FirstVersion and is "Publish": false. A new project on PyPI is hidden from search engines for its first 30 days, so when to publish it is a separate choice, tracked in #154.
• On-premise Device Detection reads a Boolean property through the C library's getValueAsBool, which answers value == "True" (ResultsBase.cpp lines 118 to 140), so a Boolean property stored as Unknown reaches Python as False. So a derived property script that compares a Boolean property with the text "Unknown" never finds it through that engine. PHP found the same and raised it as 51degrees-php#95. It needs a data file to check, so it was not checked here, and #153 tracks it for Python.
• The consumer tests of derived-properties build only .NET, and a row for Python is a change in that repository, as derived-properties#18 is for PHP, tracked for Python in derived-properties#20.
• .NET also lets a script replace a property of another element, by an Output.Name naming that element. Format 1 as published at cc306386 asks Output.Name to match ^[A-Za-z][A-Za-z0-9]*$, so Python follows the format and has no such scripts.