Skip to content

fix: upload artifacts to GCS without gzip encoding - #195

Merged
fedorov merged 1 commit into
mainfrom
fix/gcs-upload-without-gzip
Nov 11, 2026
Merged

fedorov merged 1 commit into
mainfrom
fix/gcs-upload-without-gzip

Conversation

@fedorov

@fedorov fedorov commented Nov 11, 2026

Copy link
Copy Markdown
Member

Summary

Upload release artifacts to GCS with gzip disabled and a 60-second cache lifetime. This is step 1 of the plan in #193.

google-github-actions/upload-cloud-storage gzips by default, so objects were stored with Content-Encoding: gzip. That had three effects:

  • The MD5 GCS reports didn't match the file. GCS hashed the compressed bytes: 73,823,273 bytes for idc_index.parquet, against 74,097,726 bytes actually served. The hash couldn't be used to check a download.
  • Range requests were ignored for clients that don't accept gzip, such as DuckDB, so they downloaded the whole file (see Investigate if/how we could support search without downloading parquet files idc-index#254).
  • A replaced object could be served from cache for up to an hour, under the default max-age=3600.

Parquet is already zstd-compressed, so gzip saved about 0.4%.

Changes

  • cd.yml: gzip: false and cache-control: public, max-age=60 on the GCS upload. current/ is filled by copying from the release folder, which keeps both settings.
  • tests/test_gcs_cors.py: the existing range tests passed in spite of the problem, because requests sends Accept-Encoding: gzip. GCS then applies the range to the compressed bytes, and the 206 body starts with 1f 8b rather than PAR1. The range tests now ask for the bytes as stored. New tests check the Parquet magic number, the stored encoding, that the reported MD5 matches the downloaded file, and that the cache lifetime is at most 60 seconds.

Production state

The objects in gs://idc-index-data-artifacts/current/release_artifacts/ were re-uploaded by hand without gzip, after the script was tested on a copy in a separate bucket. Each Parquet file was checked against its .sha256 sidecar before upload. The re-upload is what lets this PR's CI and the daily GCS check pass now. Earlier versioned folders are unchanged.

Testing

  • tests/test_gcs_cors.py against production: before the re-upload, the 6 range and metadata tests failed and 11 passed. After it, all 17 pass.
  • pre-commit passes on the changed files.

Closes #148

🤖 Generated with Claude Code

The upload-cloud-storage action gzips by default, so objects were stored
with Content-Encoding: gzip. GCS then hashed the compressed bytes, so its
MD5 could not be checked against the downloaded file, and it ignored Range
requests from clients that don't accept gzip, such as DuckDB. Parquet is
already zstd-compressed, so gzip saved about 0.4%.

Upload with gzip disabled and a 60-second cache lifetime, so a replaced
object isn't served from cache for the default hour.

The range tests passed despite this because requests sends
Accept-Encoding: gzip, which makes GCS honour Range against the compressed
bytes. They now request the stored bytes, and new tests check for the
Parquet magic number, the stored encoding, the reported MD5 and the cache
lifetime.

Closes #148

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fedorov
fedorov merged commit b483757 into main Nov 11, 2026
13 checks passed
@fedorov
fedorov deleted the fix/gcs-upload-without-gzip branch November 11, 2026 22:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement gzip: false and cache-control for the parquet artifacts

1 participant