[jwrosewell]

Superseded by #108. The review of #108 asked for the download test to leave the suite every job runs, so #108's branch was moved forward to this draft's head, dd6abbb, and #108 now holds these three commits and describes them. This draft is to be closed as superseded by #108.

Every job of the build matrix now takes the free data files from the run's asset cache, a check fails any job whose tests downloaded one, and the test of the download itself runs once, in a job of its own.

This branch holds the commits of #108 (merged in at a8da3e1 from its head 82f8c1f), because #108 adds the free ASN file to the cache and the download test this draft moves. So it merges after #108. Once #108 is on rc and this branch has taken rc, the diff is the three commits under "What changed".

The rule

James Rosewell set the rule at 22:23 UTC on 14 November 2026.

Try not to get it over and over again. On GH it should come from the asset cache. Locally it can also come from a shared area.

So in GitHub Actions a job takes the free data files (the Device Detection Lite hash file and the IP Intelligence Lite and ASN files) from the run's asset cache, as 51degrees-cxx#153 and 51degrees-go#113 do, and never downloads them in each job. A test of the download itself downloads once, in its own job.

Every place CI could download a free data file

Checked on rc at b01029ae and on #108's head 82f8c1f0, against the download lines of full build 43564508164 of #91's head 6cea14f, whose eight jobs (Ubuntu PHP 7.4 to 8.5 and Windows PHP 8.3) all passed.

1. common-ci's Configure job. The prefetch_assets action runs the root ci/fetch-assets.ps1 when it finds no cache for the day's key, and saves the assets folder. That is the one fetch the rule wants, and it is unchanged. In 43564508164 it found main's cache for the day (Cache hit for: 20261114-D08993235BC59D4245755D798844D1F982A792C9).
2. Each job's Fetch Assets step. Every job runs the root script again (nightly-pull-request.build-and-test.ps1 line 49) after restoring the cache (nightly-pull-request.yml lines 159 to 165), and common-ci's steps/fetch-assets.ps1 line 119 skips an asset the cache holds and downloads one it does not. In 43564508164 each job printed already present in cache, skipping download for every asset its packages list, so this step downloaded nothing.
3. Device Detection's links. device-detection/onpremise/ci/fetch-assets.ps1 line 19 links the cached Lite hash file into cxx/data/device-detection, where the tests and examples look.
4. IP Intelligence's links. On rc, ip-intelligence/ci/fetch-assets.ps1 line 17 lists only the Lite file and links it into ip-intelligence/assets. Run the mixed console example with nothing set, and link every cached free data file where the examples look #108 adds the ASN file and links both free files into cxx/data/ip-intelligence (lines 25 to 30 and 53 to 66 at 82f8c1f0).
5. Device Detection's tests and examples. ExamplesWithoutSettingsTests.php line 49, IniSettingsTests.php line 57, RequestTests.php line 135 and StartWithoutDataFileTests.php line 465 call the examples' ExampleUtils::downloadMissingDataFile for the Lite file in cxx/data/device-detection, and the examples run with nothing set reach it through getDataFile (lines 332 and 347). Each finds the linked file, and no job of 43564508164 printed a download line for it.
6. IP Intelligence's tests and examples. ExamplesWithoutSettingsTests.php line 214, IpIntelligenceOnPremiseTests.php line 936, MixedExamplesTests.php lines 383 and 427, WebExamplesWithoutSettingsTests.php line 101 and the launcher, serve.php line 145, which ServeWebExamplesTests.php runs, call ExampleUtils::downloadMissingDataFile. They take the Lite file from ip-intelligence/assets first, and the Device Detection Lite file from cxx/data/device-detection. The ASN file was not cached, so ExamplesWithoutSettingsTests.php line 100 downloaded it in each of the six jobs that build the extensions (Ubuntu PHP 8.0 to 8.5), each printing 51Degrees-IPIV4AsnIpiV41.ipi is missing, so it is being downloaded by running and Saved ip-intelligence/51Degrees-IPIV4AsnIpiV41.ipi, 6 MB. That is #106, which Run the mixed console example with nothing set, and link every cached free data file where the examples look #108 fixes.
7. Run the mixed console example with nothing set, and link every cached free data file where the examples look #108's download test. DownloadDataFileTests.php downloads the ASN file from nothing on purpose, as a fresh clone does, and Run the mixed console example with nothing set, and link every cached free data file where the examples look #108 put it in the Integration suite (phpunit.xml line 30 at 82f8c1f0), which every matrix job runs.
8. Jobs that build no extension. The Windows and PHP 7.4 jobs skip both on-premise packages (Get-SkipReason in ci/packages.ps1), and neither printed a download line in 43564508164. Device Detection's performance test reads the TAC file and 20000 User Agents.csv from the cached assets folder (run-performance-tests.ps1 lines 12 and 25).
9. Other workflows. None of this repository's other workflows fetches a data file.

What changed

1. ci/check-no-data-file-downloads.ps1 is new, and ci/run-integration-tests.ps1 runs it after every package's unit and integration tests. For each .link file in cxx/data, it fails the job when the file beside the link exists and is not a symbolic link, because Fetch Assets links every cached file there and any other copy was downloaded during the job. It names the file in an error annotation and says to add its asset to the package's ci/fetch-assets.ps1. A test that downloads a free file in every job then fails the build rather than going unnoticed, as the ASN file did.
2. DownloadDataFileTests.php moves from the IP Intelligence Integration suite to a new Download suite (ip-intelligence/phpunit.xml lines 31 to 36). common-ci's php/run-unit-tests.ps1 and php/run-integration-tests.ps1 run only the Unit and Integration suites, so no matrix job runs it.
3. .github/workflows/data-file-download.yml is new and runs the Download suite in one job, for pull requests and for pushes to main and rc. It checks out cxx and its data submodule, writes the package's autoloader with composer dump-autoload --dev without installing anything, and runs PHPUnit 9.6.34, which common-ci gives the matrix.
4. AGENTS.md "CI" says how the jobs take the free files from the cache, what the check does, where the download test runs, and how to run the tests locally from copies already on the machine.

The root ci/fetch-assets.ps1 is unchanged, so common-ci's asset cache key does not move. No environment variable name is added, read, renamed or removed.

Runs

The red and green runs use a throwaway workflow with one job that does what a matrix job does, in the same order and through common-ci's steps/run-script.ps1, with no secrets. It restores main's cache for 14 November (20261114-D08993235BC59D4245755D798844D1F982A792C9), then runs the root fetch-assets, setup-environment, build-project, run-unit-tests and run-integration-tests scripts on Ubuntu 24.04 with PHP 8.3.

• Red, run 43649421249 on tmp/fixer-assets-red f614f72, which is rc b01029a with only the check (934908f, the same change as 7f8e8c8 here) and the throwaway workflow. Fetch Assets took every listed file from the cache. The IP Intelligence integration tests then printed 51Degrees-IPIV4AsnIpiV41.ipi is missing, so it is being downloaded by running, Downloading ip-intelligence/51Degrees-IPIV4AsnIpiV41.ipi and Saved ip-intelligence/51Degrees-IPIV4AsnIpiV41.ipi, 6 MB, every suite passed, and the check failed the job with cxx/data/ip-intelligence/51Degrees-IPIV4AsnIpiV41.ipi was downloaded during this job. Add its asset to the package's ci/fetch-assets.ps1 and link the cached copy there, so that every job takes it from the run's asset cache.
• Green, run 43649421652 on tmp/fixer-assets-green 0c714b4, which is this branch at 461f1f0 with the throwaway workflow. 461f1f0 differs from the head only in the download workflow, which this job does not use. Fetch Assets took the five files the cache held, and because the cache had been filled before Run the mixed console example with nothing set, and link every cached free data file where the examples look #108 added the ASN file to the list, fetched that file itself (Fetching '51Degrees-IPIV4AsnIpiV41.ipi' and Saved ip-intelligence/51Degrees-IPIV4AsnIpiV41.ipi, 6.3 MB) and linked both free files (Linked '51Degrees-LiteIpiV41.ipi' into .../cxx/data/ip-intelligence as '51Degrees-LiteV41.ipi', and the same for the ASN file). No test printed a download line, every suite passed, and the check printed No data file was downloaded during this job.
• Download job, run 43650714015 of the new workflow on tmp/fixer-assets-green b53f862, whose workflow differs from the head's only in one comment and the throwaway branch in its triggers. OK (1 test, 5 assertions) with PHPUnit 9.6.34 on PHP 8.5.12. The test checks that the examples' code wrote 51Degrees-IPIV4AsnIpiV41.ipi is missing, so it is being downloaded and that the file arrived. The first run of the workflow, 43649421675, failed with Class "fiftyone\pipeline\ipintelligence\examples\classes\ExampleConstants" not found, because composer dump-autoload with no earlier install leaves out the autoload-dev classes, so the step now passes --dev.

Overlap with other drafts

• #108. This branch holds its commits and changes none of its lines. The new Download suite starts just before Run the mixed console example with nothing set, and link every cached free data file where the examples look #108's DownloadDataFileTests.php entry, which git shows as unchanged.
• #125 fetches the TAC and CSV files only with a licence. No shared lines, and the check runs in its builds without secrets too.
• #26 changes the Invoke-PackageScript call in ci/run-integration-tests.ps1, just above the lines this draft adds, so the second of the two to merge may need its lines put back together.
• #112, #122, #130, #132 and #137 change ip-intelligence/phpunit.xml. A test file added to the Integration suite goes before the closing tag this draft adds.
• Several drafts change AGENTS.md. This draft adds one paragraph at the end of its "CI" section.

What remains

1. The ASN file in each job until the next nightly. The cache key covers the date, the secrets and the root ci/fetch-assets.ps1 only (action.yml line 35, common-ci#329), so a cache filled from a list without the ASN file keeps being restored that day, and each job fetches the 6.3 MB file in its Fetch Assets step, as the green run did. When no pull request is ready to build, the nightly builds rc itself (nightly-pull-requests.get-prs.ps1 line 23 and steps/get-target-branch.ps1 line 24) and fills main's cache for the day from rc's lists, as the nightly of 14 November did (run 43538727785, whose Configure for pull request 0 ran from 01:28 to 03:03 and saved the cache the runs above restored). So from the first nightly after Run the mixed console example with nothing set, and link every cached free data file where the examples look #108 reaches rc, the cache holds the ASN file and no job fetches it. Changing the root script instead would move the key, so a full build of any branch whose root script differs from the one the day's cache was filled from would miss that cache and fetch every data file again in Configure, the paid IP Intelligence file included, as Run the mixed console example with nothing set, and link every cached free data file where the examples look #108's full build 43579401232 did for 42 minutes.
2. Device Detection's download. No test downloads the Device Detection Lite file from nothing, so CI runs that download code only where it finds the file already there. A Device Detection download test would go in a Download suite in device-detection/onpremise/phpunit.xml and a second step of the new workflow.

Build

The rule for CI-only drafts, from James Rosewell at 11:35 UTC on 12 November, is "rebuild and include, but the change doesn't impact the code so no need to run the full builds." It covers a draft that changes only CI scripts and their notes, which merges into rc on green quick checks and an independent review.

This draft's own three commits change CI scripts, a new workflow, the suite one test sits in, and AGENTS.md, with no package code and no test code. Until #108 is on rc, the draft also carries #108's examples and tests, so the full build has not run yet. Once #108 is on rc and this branch has taken rc, the diff is CI-only and the rule above applies.

The quick checks on head dd6abbb all passed.

• Data file download, run 43651307317, OK (1 test, 5 assertions) with PHPUnit 9.6.34 on PHP 8.5.12. This is the new workflow on the head itself.
• Environment variable names, run 43651307319.
• Only rc into main, run 43651307590.
• UTM link lint, run 43651307345.

[jwrosewell]

Closed as superseded by #108, whose branch holds this draft's commits and whose body describes them.