Skip to content

docs: describe prior_versions_index in its schema sidecar - #189

Merged
fedorov merged 1 commit into
mainfrom
docs/prior-versions-descriptions
Oct 20, 2026
Merged

fedorov merged 1 commit into
mainfrom
docs/prior-versions-descriptions

Conversation

@fedorov

@fedorov fedorov commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

prior_versions_index was the only bundled index whose schema JSON had no table or column descriptions. Its SQL builds the column list inside an EXECUTE IMMEDIATE string, so the build skipped description parsing for it. Without descriptions the table is easy to misread, and in ImagingDataCommons/IDC-REST-MCP#52 that misreading led to wrong release diffs.

Changes

  • SQL in two parts. EXECUTE IMMEDIATE now only fills a temp table with every past series version. The final SELECT is static SQL, documented with the usual # table-description: / # description: comments. The bucket mapping is computed once, and the S3 URL is built from it.
  • Build. Drop the prior_versions_index special case in idc_index_data_manager.py.
  • Tests. tests/test_real_sql_parsing.py checks that the table description and all 12 column descriptions parse.
  • Docs. docs/dev/ no longer calls the table undocumented.

What the descriptions say

  • What the table is: one row per series version no longer served, with the range of releases that served it. A SeriesInstanceUID that is also in the main index was revised; one that is not was removed. Its main use is resolving manifests made from earlier releases.
  • Not a record of what is new:
    • It never lists added series.
    • A row ending at a release does not mean that release changed the series. A new row also starts when a version only moved storage bucket. That is the case for 11,338 versions in 25.0.4, mostly at v19→v20, and reading those as revisions turns 590 real v20 revisions into 11,928.
    • For what a release added or revised, the description points at series_init_idc_version / series_revised_idc_version in the main index.

Verification

  • Ran the new script against BigQuery: it returns the same 82,428 rows as the 25.0.4 parquet, with the same columns in the same order and no differing row.
  • The parser tests pass (19), and pre-commit is clean.

Not addressed

No row starts at v1, because the query drops the original idc-open-idc bucket. So series versions that were served in v1 and replaced later are not recorded. This PR does not change that.


🤖 Generated with Claude Code

prior_versions_index was the one bundled index whose schema JSON carried
no table or column descriptions: its SQL builds the column list inside an
EXECUTE IMMEDIATE string, so the build skipped description parsing for it.

Split the script so EXECUTE IMMEDIATE only fills a temp table with every
past series version, and the final SELECT is static SQL documented with
the usual "# table-description:" / "# description:" comments; drop the
special case in idc_index_data_manager.py. The bucket mapping is computed
once and the S3 URL built from it. The output is unchanged: run against
BigQuery, the script returns the same 82,428 rows as the 25.0.4 parquet,
with the same columns in the same order and no differing row.

The descriptions say what the table is not: it is not a record of what is
new in a release (it never lists added series, and a row ending at a
release does not mean that release changed the series; 11,338 versions in
25.0.4 span several rows only because their storage bucket changed), and
point at series_init_idc_version / series_revised_idc_version in the main
index instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fedorov
fedorov merged commit 8be575a into main Oct 20, 2026
13 checks passed
@fedorov
fedorov deleted the docs/prior-versions-descriptions branch October 20, 2026 01:03
fedorov added a commit to ImagingDataCommons/IDC-REST-MCP that referenced this pull request Oct 20, 2026
idc-index-data 25.0.5 ships descriptions for prior_versions_index and
its columns (ImagingDataCommons/idc-index-data#189), so drop the local
fill-ins this branch carried for them, and the fill-in mechanism with
them. The data is unchanged: 179 collections, 1,044,191 series, 99.9 TB,
and the same prior_versions_index rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant