[jwrosewell]

A data update download whose response ends before the whole file has arrived is now reported as an error accessing the data update service, whichever version of urllib3 is installed, and the data in use is kept.

The problem

The Data Update Service downloads with requests.get and reads the whole body inside that call (datafile_update_service.py lines 152 to 161 at 3f936f8d). With urllib3 2, which requests installs today, a body shorter than its Content-Length raises ChunkedEncodingError, a RequestException, which the service already reports as "Error accessing data update service". requests 2.34 still accepts urllib3 1.26, and that version gives the bytes that arrived with no error, so the service took the cut short file as a successful update and refreshed the engine with it. A chunked response cut short raises with either version.

What the specification says

data-updates.md, Errors, line 500 (specifications#145), puts "Connecting to the data update URL failed, or its response ended before the whole file arrived" in the row with the log level Error, the status AUTO_UPDATE_HTTPS_ERR and the message "Error accessing data update service at '' for engine ''".

What changed

After the status checks, the service compares the bytes received with the Content-Length the response gave and, when fewer arrived, logs the row's message with "The response ended after of the bytes it gave as its Content-Length." as the detail, reports AUTO_UPDATE_HTTPS_ERR and keeps the data in use (datafile_update_service.py, _cut_short). A response with a Content-Encoding is not measured, as its length counts the encoded bytes and requests gives the decoded ones.

Four tests in test_datafile_update_http.py, ShortDownloadTests. Two run a local HTTP server that promises 1,000 bytes and sends 100, once with Content-Length and once in chunks with no last chunk. One gives the service a reply holding 100 bytes with a Content-Length of 1,000, which fails on rc whatever urllib3 is installed. One checks that a reply with a Content-Encoding is not measured.

How it was verified

• urllib3 1.26.20, red run 43561396780 on b7af7c8, which is rc at 3f936f8 with only the two server tests (0a75521) and a temporary workflow that installs urllib3<2. On Ubuntu and on Windows the service took the response shorter than its Content-Length as a successful update ("True is not false"), and the chunked one passed.
• urllib3 2.8.0, run 43561388521 on 5529300, the same commit with the urllib3 that requests 2.34.2 installs. Both server tests passed on both systems, so with urllib3 2 the service already reported both.
• Red, run 43562019424 on e51e117, which is f6f1582 (all four tests, no fix) with the workflow and urllib3 2.8.0. testTheLengthIsCheckedWhicheverHttpLibraryIsInstalled failed on both systems ("True is not false", "1 failed, 45 passed").
• Green with urllib3 2.8.0, run 43562043311 on e68acca, and green with urllib3 1.26.20, run 43562059908 on 17a5c65, each 06fbc1c with the workflow. All 46 tests of the file passed, then the 69 of fiftyone_pipeline_engines, on both systems in both runs.
• Full build 43579396519 on 06fbc1c is green. Its ten Build and Test jobs, Python 3.10 to 3.14 on Ubuntu and Windows, each passed, the 11 Setting 'PullRequestSha' lines all name 06fbc1c, and Configure printed "Left out 5 macOS entries". An earlier build of the same head, 43563210178, hung in Configure's asset download and was cancelled before any test ran. macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

What it releases

fiftyone_pipeline_engines changes, so it is released with the packages that depend on it. Monday's release already holds it and its dependants, as the plan of Publish dry run 43528475739 shows, so this adds no package.

Merge order

#87 changes other parts of the same file and merges in either order.