Skip to content

FIX Avoid poisoned dataset ZIP caches - #2345

Open
Roman Lutz (romanlutz) wants to merge 1 commit into
microsoft:mainfrom
romanlutz:romanlutz-daily-audit-2026-08-07
Open

FIX Avoid poisoned dataset ZIP caches#2345
Roman Lutz (romanlutz) wants to merge 1 commit into
microsoft:mainfrom
romanlutz:romanlutz-daily-audit-2026-08-07

Conversation

@romanlutz

Copy link
Copy Markdown
Contributor

Description

Interrupted streamed ZIP downloads were written directly to the final dataset cache path. A transient connection failure could therefore leave a partial archive that every later call treated as a cache hit, preventing recovery without manual cache cleanup.

This change downloads into a unique temporary file in the cache directory, parses the archive successfully, and only then atomically publishes it to the final cache path. Failed downloads and parses clean up the temporary file so a later call can redownload normally.

Tests and Documentation

Added a deterministic unit test that simulates a partial chunk followed by a connection failure, verifies no poisoned cache remains, then retries successfully and checks the exact network-call and cache-file behavior.

  • 4,667 dataset and Many-shot tests passed
  • 38 focused ZIP loader and Moral Integrity Corpus tests passed
  • Ruff, formatting, ty, async-suffix, pre-commit hooks, and git diff --check passed
  • Documentation not applicable; no public API or behavior contract changed
  • JupyText not applicable

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants