mpd2020.xlsx: record the repoint (hold until the lecture PRs merge) - #41
Merged
Conversation
Flips the tracker once both consuming lectures read this repo, and records them in the manifest's consumers list so a future correction knows what to rebuild. Held as a draft deliberately. The strict audit has no green state for a dataset whose consumers have only partly moved, so this must not merge before QuantEcon/lecture-python-intro#823 and QuantEcon/lecture-wasm#52. Verified both ways: against the lecture repos' current main this fails with "marked repointed but consumers still read via ['own-repo', 'sibling']"; against their repoint branches it passes clean, and the orphan count falls 35 -> 34 as wasm's mirror copy goes. mpd2020 is the first dataset to exercise the two-repo rule end to end. intro's copy could not be deleted while wasm fetched that exact blob by URL, so the pair moved together and this flip trails both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mmcky
marked this pull request as ready for review
August 6, 2026 05:49
Contributor
Author
Both lecture PRs are merged — this is now the green side, and
|
| Branch | Strict audit |
|---|---|
| this PR | ✅ exit 0, orphans 35 → 34 |
main |
❌ mpd2020.xlsx: marked landed but some consumer already reads data-lectures |
So the inconsistency has flipped sides exactly as designed, and main is red until this merges. That is the unavoidable window: with two consuming repos there is no ordering of three PRs that keeps every one of them green, so the choice is which side is briefly wrong. Lectures-first is the right side to be wrong on — a stale tracker entry is a bookkeeping error, whereas flipping the tracker first would have meant published lectures reading a file that had already been deleted.
Worth merging promptly rather than leaving open: while the window is open, any push to main or the Monday 05:17 UTC scheduled run will fail the build, skip the Pages deploy, and open a drift issue via the #29 notifier. Nothing is broken for readers — the published dashboard simply freezes at the last green build — but the alarm will ring for a known, expected reason, which is exactly the noise that trains people to ignore it.
Lecture CI passed on both halves before merge, so the repoint is verified at the level that matters: lecture-wasm build-and-deploy 2m1s, lecture-python-intro test 11m0s and preview 8m21s — both lectures genuinely rendered against the new URL.
There was a problem hiding this comment.
Pull request overview
Records mpd2020.xlsx as fully repointed by updating the migration tracker and the dataset manifest’s consumers, and regenerates the catalog so downstream rebuild targets are discoverable.
Changes:
- Flip
mpd2020.xlsxinmigration.ymlfromlandedtorepointedand record the two lecture PR repoints. - Populate
lectures/mpd2020.xlsx.ymlconsumerswith the two consuming repos / lecture files. - Regenerate
CATALOG.mdto reflect the updated “Used by” column and counts.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| migration.yml | Marks mpd2020.xlsx as repointed and records the repoint PRs for both consuming lecture repos. |
| lectures/mpd2020.xlsx.yml | Adds the two consuming lectures to consumers so future corrections can identify rebuild targets. |
| CATALOG.md | Regenerated catalog reflecting the new mpd2020.xlsx consumer links and updated rollup counts. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Copilot caught mpd2020's Consumers comment still reading "Empty until the Track A repoint lands ... lectures read this file from intro's own copy" while sitting directly above a populated consumers list naming both repos reading it from here. It flatly contradicted the data beneath it. The mistake is structural rather than a typo: the comment described the field's CURRENT VALUE, so it was guaranteed to go stale the moment its own PR landed. The identical comment was in all eight manifests from #38, so it would have gone stale again on repoint sets 2 and 3 and been flagged both times. All eight now describe what the field MEANS, which stays true in either state: every lecture that reads this file from this repo, empty not implying the file is unused since it may still be read from a lecture repo's own copy -- which is exactly what a repoint moves -- and populated in the repoint PR alongside the migration.yml flip. Comments only. Verified: every manifest still parses, consumers values are untouched (mpd2020 two, the other seven zero), CATALOG.md regenerates byte-identical, and the strict audit stays green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 6, 2026
mmcky
added a commit
that referenced
this pull request
Aug 10, 2026
* Guardrail: hash a file whenever its manifest records a hash `consumers: []` short-circuited the check before it reached the sha256 comparison, and this is the repo's only byte-integrity gate. Manifests land ahead of their repoints by convention, so a dataset arrives here with an empty consumer list and is flipped to `repointed` by a later PR in another repo — which made the one PR that introduces new bytes the one PR that never verifies them. `mpd2020.xlsx.yml` landed exactly that way in #38: hash recorded, consumers empty, never hashed until #41 filled them in. The hash check now keys on `integrity.sha256` being recorded rather than on `consumers` being non-empty. The consumer-specific errors are unchanged, so a live lecture whose file goes missing still says so in those terms. No behaviour change on `main`: all 18 manifests carry consumers and a hash today, so the run is identical bar the summary wording. Verified against a scratch tree for the four cases that matter — unconsumed + correct hash passes; unconsumed + drifted bytes fails; unconsumed + hash recorded but file absent fails; unconsumed with no hash stays out of scope. Lands before the `high_dim_data` fold, which introduces six datasets at `consumers: []` — QuantEcon/workspace-lectures#23 step 3, PR B. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Guardrail: fail on a malformed integrity block, and count only real hashes Two fixes from Copilot's review, both confirmed by reproduction. A present-but-non-mapping `integrity` (e.g. `integrity: "sha256: …"`, one missing indent level) read as "no hash recorded", so a manifest with `consumers: []` short-circuited before the byte check and the run was silently green — 0 errors, 0 files hashed, exit 0. That is a hole this PR opened: with a consumer present the same manifest still errored, so it was only the new unconsumed path, which is exactly the shape every new dataset lands in. It is now a hard error either way, and the comment claiming a non-dict integrity "lands in the not recorded error below" goes with it, because after the keying change it did not. `checked` was also incremented before the existence and hash-recorded checks, so the summary could report a file as hash-checked when sha256() was never called. Accurate under the old "consumed file(s) checked" label; wrong under this PR's rename. The counter moves to where the hash is computed. Verified: the four cases in the PR body still behave, plus the two new ones — a string integrity with no consumers now exits 1, and a consumed manifest with no recorded hash reports 0 hash-checked rather than 1. The real tree is unchanged: 18 manifests, 18 hash-checked, exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft on purpose — do not merge before QuantEcon/lecture-python-intro#823 and QuantEcon/lecture-wasm#52.
Flips
mpd2020.xlsxfromlandedtorepointedand records both consuming lectures in the manifest'sconsumerslist, so a future correction knows what to rebuild.Why this is a third PR rather than part of the other two
The strict audit has no green state for a partially-repointed dataset — it fails a record marked
landedonce any consumer reads this repo, and fails one markedrepointedwhile any consumer still does not. With two consuming repos there is no ordering of three PRs that keeps every one of them green on its own, so the tracker update trails the lectures and merges last.Verified in both directions rather than assumed:
main(PRs unmerged)mpd2020.xlsx: marked repointed but consumers still read via ['own-repo', 'sibling']repoint/long-run-growthbranchesThat second row is the state this PR merges into.
What this set proves
mpd2020.xlsxis the first dataset to exercise the two-repo rule end to end.lecture-wasmfetched intro's committed blob by URL, so intro's copy could not be deleted on its own without 404-ing the wasm build — silently, since nothing in intro would have signalled it. The pair therefore moved together and this flip trails both. Both rules are written up under "Repoint rules" inPLAN.md.Sixteen of the thirty-one remaining datasets have this same intro + wasm shape, so the pattern established here is the one the rest of Track A follows.
Part of #8 and #4.