[jwrosewell]

An update from memory gave its result to the update complete callbacks as well as to its caller, returned a failure as a status, and logged it. It now reports nothing to the callbacks, returns AUTO_UPDATE_SUCCESS, and raises a failure to its caller without logging it, as the data update specification says.

Closes #149.

Problem

DataFileUpdateService.update_from_memory (datafile_update_service.py lines 310 to 363 at 28e5bf2c) passed its status to the update complete callbacks on both of its paths (self.__emit(status, data_file)), returned AUTO_UPDATE_REFRESH_FAILED or the status of a failed write rather than raising, and logged the failure through _refresh and _write_data_file (#149).

What the specification says

data-updates.md lines 372 to 376 at 706de1f0:

Two calls report nothing to onUpdateCallbacks, and give their result to the caller instead. A programmatic check for a data file the Engine has no configuration for returns the no configuration status (AUTO_UPDATE_NO_CONFIGURATION), and an update from memory returns success or fails with an error.

Lines 477 to 479: "A programmatic check, the update on start-up and an update from memory pass the error to their caller instead of logging it."

What changed

All in pipeline/fiftyone_pipeline_engines.

1. cda98342, the tests, in tests/test_datafile_update_http.py. New: testAFailedRefreshRaisesAndIsNeitherReportedNorLogged (a data file kept in memory keeps its old data), testAFailedRefreshOfAFileDataFileRaisesAndIsNeitherReportedNorLogged, testAFailedTemporaryCopyFromMemoryRaisesAsAnyOtherError, testFailedCopyFromMemoryRaisesAnErrorNamingThePath, testNoTemporaryPathFromMemoryRaises and testAnUpdateFromMemoryCallsNoCallback. Changed, as each asserted the old behaviour: testMemoryDataFileIsRefreshedFromTheBytes and testFileDataFileIsWrittenThenRefreshed now expect no callback and only the "Attempting to refresh" log entry, testAFailedWriteFromMemoryEndsTheUpdate expects an error with nothing reported or logged, the earlier testFailedRefreshIsReportedAndLogged and testACallbackErrorOnAnUpdateFromMemoryIsLoggedAndStopsTheCallbacks are replaced by the tests above, and testFailedCopyNamesThePath and testNoTemporaryPathIsReported now run through update_data_file, so the logging and reporting of the check route stay tested.
2. 502cadcb, the change. A new public DataUpdateException(message, status) carries the failure's status, with the error that caused it chained. update_from_memory reports nothing to the callbacks, returns AUTO_UPDATE_SUCCESS once the engine is refreshed, and raises DataUpdateException for a failed refresh, with "An error occurred when applying a data update to engine ''." and AUTO_UPDATE_REFRESH_FAILED, and for a failed write of a data file kept on disk, with that write's message and status. Neither is logged. The engine keeps the data it had. The update lock and the updating flag work as before. On the check route a failed write is logged and reported exactly as before, through _log_failed_write.

Before you upgrade

• update_from_memory no longer calls the update complete callbacks.
• A failed refresh or a failed write raises DataUpdateException, whose status is the status it returned before (AUTO_UPDATE_REFRESH_FAILED, AUTO_UPDATE_NEW_FILE_CANT_RENAME, AUTO_UPDATE_TEMP_PATH_NOT_SET or AUTO_UPDATE_UNKNOWN_ERROR), and it is not logged. Code that checked the returned status for a failure catches the exception instead.
• For a data file whose engine has been closed it still returns AUTO_UPDATE_NOT_NEEDED and does nothing.

How it was verified

• Red, run 43663363084 on 0a4a927, being cda9834 (the tests on rc 1190974) plus only a throwaway workflow. On Ubuntu and Windows the nine new and changed update from memory tests failed, "9 failed, 43 passed" in test_datafile_update_http.py and "9 failed, 85 passed" across the engines tests, and the two check route tests passed.
• Green, run 43663371813 on be7cc95, being the head 502cadc plus the same workflow, "52 passed" and "94 passed" on both.
• Full build 43664180038 on the head 502cadc, from rc 1190974, is green. Its ten Build and Test jobs, Python 3.10 to 3.14 on Ubuntu and Windows, each passed, with test_datafile_update_http.py passing in each. The 11 Setting 'PullRequestSha' lines all name 502cadc, and Configure printed "Left out 5 macOS entries". macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

What it releases

fiftyone_pipeline_engines and its dependants, which Monday's release plan releases already.

What remains

1. The errors table's row for a failed write of data passed in from memory (line 511) says it is a warning and the engine is still refreshed. Here a failed write ends the update with an error, as the engines of data files kept on disk refresh by reading the file again, and refreshing them from the bytes given would need the data file to switch to memory, which is a larger change, tracked in #171.
2. After an update from memory writes a data file kept on disk, the file system watcher's next pass sees the new file and refreshes the engine again, reporting that to the callbacks, because the time of the written file is not recorded, tracked in #170.
3. Whether "returns success or fails with an error" also binds a call on a data file whose engine has been closed is left as it was, and asked in specifications#164.