[jwrosewell]

This is the 51degrees-node step of the run book for moving every SDK to the
data repository,
51Degrees/data#1 (version 2, step 3).

The data repository holds what
device-detection-data
and ip-intelligence-data
held, in the folders device-detection and ip-intelligence. The data files
themselves are not stored there. Each is a <file>.link holding one URL, and
get-data-files.ps1
and get-data-files.sh
download them. 51degrees-cxx includes it as its data submodule, so here it
is cxx/data.

The full build has not passed yet. Each run below starts from a
temporary branch, tmp/pr-matrix-22, which calls common-ci's
nightly-pull-request.yml with dryrun: 'true' and branch: rc.

• 41781590963
  failed on macOS Node 24 in cloudRequestEngine.test.js, at the ETag check
  that Check the Date header in the cloud error test, not ETag #21 replaced with a Date check. The other legs were cancelled.
• 41797475049
  passed on six legs. macOS Node 24 started after Update submodules #20 had moved cxx on
  rc, so it could not merge this branch into rc ("Failed to merge
  submodule cxx"), and macOS Node 22 was cancelled.
• 41853367836,
  on 0626f1e after merging rc, was started at 15:01 UTC on 4 November and is
  waiting for runners.

What changes

The cxx submodule

cxx moves from
1bec7b5,
the cxx 4.5.102 release that rc has, to
f03dbfd,
the head of the cxx change that adds the data submodule,
51degrees-cxx#96.
f03dbfd is nine commits ahead of 1bec7b5 and contains it.

51degrees-cxx#96 was squash merged into cxx rc as
7832155
on 4 November and is due to be released as cxx 4.5.103. Because of the
squash, f03dbfd is not in the history of cxx rc. Its src tree
(e2adedf) and its data submodule commit (e448dcf) are the same as those of
cxx rc at
ac3ba1f,
and its src tree is the same as 4.5.102's. So no SWIG interface changes, and
the committed wrappers stay as they are. cxx moves to the 4.5.103 release
commit before this is merged (see "Still to do").

cxx keeps its old device-detection-data and ip-intelligence-data
submodules for one release, so that every language keeps working until it
moves (run book, "Order, and why"). Nothing here reads them after this change.

Paths

Every cxx/device-detection-data is now cxx/data/device-detection, and every
cxx/ip-intelligence-data is now cxx/data/ip-intelligence, in

• ci/fetch-assets.ps1 and ci/run-performance-tests.ps1
• the default translations folder in
  ip-intelligence/fiftyone.ipintelligence.translation/resources.js, whose
  error message now says to fetch the cxx/data submodule with
  git submodule update --init --recursive or supply translationsPath. Its
  functions and exports are unchanged.
• the device detection and IP Intelligence on-premise tests, their
  testHelper.js files, the smoke scripts and translation.test.js
• the three web examples' 51d.json files
• cloud-performance-test/process.js

Examples download a missing file

This follows "What an example does when its data file is missing" in the run
book. The step is written once in each example helper, exampleUtils.js for
device detection and for IP Intelligence, in ExampleUtils.findFile, which
findDataFile and findDeviceDataFile call.

1. If <file>.link is in cxx/data/device-detection or
   cxx/data/ip-intelligence, the file is read from that folder.
2. If the file is missing there, the helper prints the command and runs the
   data submodule's script for that one file. On Windows that is
   powershell -NoProfile -ExecutionPolicy Bypass -File "<data>\get-data-files.ps1" "<file>" -Source examples-51degrees-node,
   and elsewhere sh "<data>/get-data-files.sh" -s examples-51degrees-node "<file>".
3. It looks again. If the script failed or the file is still missing, it
   prints the command that gets the file and returns undefined, so the
   example fails with its own message.

A file with no link, such as an Enterprise file, is searched for by name as
before. A path given as an argument, or in 51DEGREES_DD_PATH or
51DEGREES_IPI_PATH, still comes first.

One choice here goes further than the run book's wording, which says an
example looks for its file where it looks today. The Node examples found a
file by searching the folder tree by name, so a linked file is now read from
its data folder instead. Node's readdirSync returns names sorted, on
Windows by NTFS and elsewhere by libuv's scandir, and the search takes the
last folder first. Inside cxx it therefore walks device-detection-data
before data. cxx still has that folder, and without Git LFS it holds 132
byte pointer files named 51Degrees-LiteV4.1.hash,
20000 Evidence Records.yml and 20000 User Agents.csv. The search would
return a pointer and never reach the download.

The User-Agent Client Hints web example read its file from a fixed path. It
now finds it with ExampleUtils.findFile, so it downloads the file too.

CI

• ci/fetch-assets.ps1 links the assets into the new folders as before, with
  the Enterprise IP Intelligence file under the Lite name too. It then runs
  cxx/data/get-data-files.ps1 for each folder, and fails if a file is not
  downloaded. The script keeps a file that is already there, so the linked
  files stay. It downloads the Asn file, which was committed in
  ip-intelligence-data and is now a link (run book, decision 2), and which
  ipIntelligenceOnPremise.test.js uses. Without an IP Intelligence license
  key it also downloads the Lite file. In GitHub Actions the script sends
  source=ci-51degrees-node.
• ci/run-performance-tests.ps1 passes both performance examples their data
  file and their evidence file. Before, the IP Intelligence run and the device
  detection evidence relied on the examples' search by name. In CI that search
  would now find the old folder's LFS pointer for 20000 User Agents.csv, and
  the old folder's evidence.csv, which is not the one CI generates. The
  device detection run uses the TAC file when there is one and the Lite file
  otherwise.
• cloud-performance-test/CMakeLists.txt downloaded 20000 User Agents.csv
  from a master branch that device-detection-data no longer has. It now runs
  the data submodule's get-data-files.sh for that file and copies it to
  uas.csv. Nothing in CI runs this project.

Messages, comments and docs

• The examples' messages said to update the old submodules with
  git submodule update --recursive. They now say to fetch the cxx/data
  submodule with git submodule update --init --recursive, because an
  existing clone has not initialised it.
• The data file age warnings and the web examples' pages say to run the
  get-data-files script with -u, or -Update in PowerShell, and link the
  data repository.
• The comments say the Lite file is in cxx/data/device-detection or
  cxx/data/ip-intelligence and is downloaded there if missing.
• README.md, AGENTS.md, device-detection/readme.md,
  ip-intelligence/readme.md, ip-intelligence/run_examples.md and the
  readmes of fiftyone.ipintelligence, fiftyone.ipintelligence.onpremise
  and fiftyone.ipintelligence.translation replace the Git LFS steps with the
  scripts. The README tells an existing clone to delete the old
  cxx/device-detection-data and cxx/ip-intelligence-data folders once cxx
  no longer includes them, because Git leaves a removed submodule's folder on
  disk (run book, check 5).
• The *.node and *.dat Git LFS rules in the three .gitattributes files
  stay. They match no tracked file and are not about data.

Merging rc

rc was merged into this branch on 4 November as
0626f1e.
It brought in the "Releases" section of AGENTS.md (#23) and the submodule
update (#20), which moved pipeline/fiftyone.pipeline.core/javascript-templates
to c773ce2 and cxx to 1bec7b5. The releases merged from main (#18 and #24)
change nothing else. Each file that both sides changed:

• cxx conflicted, with 1bec7b5 on rc and f03dbfd here. The merge keeps
  f03dbfd, which contains 1bec7b5 (see "The cxx submodule").
• AGENTS.md merged without a conflict. It has this branch's "Data files"
  section and rc's new "Releases" section.
• pipeline/fiftyone.pipeline.cloudrequestengine/tests/cloudRequestEngine.test.js
  has the same change on both sides, the Date header check from Check the Date header in the cloud error test, not ETag #21, which
  this branch had already merged.

javascript-templates changed only on rc, so it is rc's commit. The pull
request's diff against rc is now the 60 files described above, cxx
included.

Other open pull requests

Each draft into rc was merged into this branch's head locally to find
overlaps.

• Check the callback URL before the JavaScript builder writes it #25 (callback URL), Say in AGENTS.md how to get the data, build and run the examples #26 (AGENTS.md), Stop the data file watcher and update timer keeping pipelines alive #28 (data file watcher) and Give each package a homepage and fix the links its README carries #29
  (package homepages and links) merge with this branch without a conflict.
  Give each package a homepage and fix the links its README carries #29 changes 17 of the same files, on other lines.
• Say in AGENTS.md how to get the data, build and run the examples #26 describes both layouts, with and without cxx/data, so its "Without
  cxx/data" part can go once this is merged.
• Run the Selenium contract for real and build pull requests on fewer entries #27 (CI) conflicts in AGENTS.md with rc itself, where Describe the rc approach to releases in AGENTS.md #23's "Releases"
  section sits where Run the Selenium contract for real and build pull requests on fewer entries #27 adds its text. This change does not cause it.
• Stop the data file watcher and update timer keeping pipelines alive #28 adds two tests that read
  cxx/device-detection-data/51Degrees-LiteV4.1.hash and
  cxx/ip-intelligence-data/51Degrees-IPIV4AsnIpiV41.ipi. After this change
  ci/fetch-assets.ps1 links the Lite hash file into
  cxx/data/device-detection and downloads the Asn file into
  cxx/data/ip-intelligence, so whichever of the two is merged second needs
  those two paths moved to the cxx/data folders.

How it was checked

• git diff shows only the lines meant to change. Every file keeps its line
  endings, which are LF in the repository.
• A search of the tree finds device-detection-data and
  ip-intelligence-data only where the README and AGENTS.md say nothing reads
  them, and Git LFS only in the .gitattributes rules above.
• An unchanged copy of the original search, run over a scratch tree holding
  cxx/device-detection-data/20000 User Agents.csv and
  cxx/data/device-detection/20000 User Agents.csv, returned the first.
  libuv's uv__fs_scandir_sort in src/unix/fs.c was read to confirm the
  sort on Linux and macOS.
• The new findFile was run with plain node from a scratch copy of the
  layout holding the real data scripts and the real Asn link, on Windows, so
  through Windows PowerShell. With the file missing it printed the command,
  downloaded and unpacked the file (6.3 MB) and returned its path. Asked
  again, it returned the path with no download. Given a link that could not
  be resolved, it printed the command and returned undefined. The new code
  is the same text in both helpers.
• The sh form of the command was run with the real get-data-files.sh. It
  exits 0 for a file that is there, 1 for a download that fails and 2 for a
  name with no link. The script stops reading options at the first file name,
  so -s comes first.
• The edited CI scripts parse in PowerShell 7. The exit code check in
  ci/fetch-assets.ps1 was tried with a script that ends with exit 1 and one
  that does not, after a command that had left a non-zero exit code. Only the
  first failed. The data file choice in ci/run-performance-tests.ps1 was
  tried with the first file missing and with both missing.
• Every edited JavaScript file passes node --check, and the three
  51d.json files parse. Nothing was installed, built or tested locally, so
  the lint, the tests and the examples are checked by the full build.
• common-ci's UTM link lint, scripts/utm-lint.ps1, is clean for this branch.
• common-ci clones with --recurse-submodules and runs
  git submodule update --init --recursive after checking out the pull
  request, so cxx/data is there when ci/fetch-assets.ps1 runs.

Still to do

• Move cxx from f03dbfd to the cxx 4.5.103 release commit once it is
  released, and check that the release commit's src tree and data
  submodule commit are the same as f03dbfd's, as cxx rc's are today.
• The build passes, and no more tests are skipped than on rc before this
  change (run book, "Checks for every pull request", 1). The mixed example
  tests look for a device detection file in the IP Intelligence folders, so
  they skip on rc today and still will. For comparison, the builds of Check the callback URL before the JavaScript builder writes it #25
  on rc report 2 skipped of 667 unit tests (1 of 45 suites) and 4 skipped
  of 59 integration tests (4 of 37 suites) on every leg, Check the callback URL before the JavaScript builder writes it #25 adding unit
  tests of its own. With this change, run 41781590963 reported 2 skipped of
  587 unit tests (1 of 44 suites) on macOS Node 24.
• The Asn file is downloaded in every job that runs the IP Intelligence tests,
  and ipIntelligenceOnPremise.test.js passes with the latest one rather than
  the copy committed on 17 September 2026. Each job downloads it itself until
  common-ci keeps it in the asset cache, and the Distributor allows each file a
  limited number of downloads an hour from one address.
• From a clean git clone --recurse-submodules, the download script followed
  by each getting-started example works, and with the files deleted each
  example downloads them itself (run book, check 3). This needs the native
  modules built.