Repository navigation
docs: a notebook for summary-event allocation, and two findings it surfaced (#533) - #538
Open
Polichinel wants to merge 1 commit into
Open
Polichinel wants to merge 1 commit into
Polichinel wants to merge 1 commit into
Conversation
…t surfaced (#533) Draws the #533 defect and the three candidate allocations against the live consolidated store: one cell holding 113,395 state-based deaths in December 2021, the same month on the grid, the monthly series a model would train on, the rank-size head of the whole dataset, and the 13 months since 1989 where one cell held over 70% of the world. Writing the conservation check surfaced two things that were not known: 1. production_parity does not merely mis-place imprecise events, it DROPS them. 43,933 events carrying 785,626 fatalities (+22.6% on what production reports) are excluded by where_prec in (4, 6) and admitted by the new profiles. Anyone comparing totals across a profile switch meets this first. 2. The legacy ceil_split reports 96 deaths across 74 events that appear in no version of the source, because it rounds each month's share up. The apportioned profiles exceed the source on zero events. Both numbers come from the same code path, so the check is not decorative. The naive "all three totals are equal" assertion is therefore wrong and is not what the notebook asserts. It asserts the property that holds and can fail: no allocator may report more deaths for an event than UCDP ever recorded for it in any candidate version. Outputs are committed. First run builds the three viewpoints (~11 min) and caches them under data/viewpoint/allocation_comparison/, which is gitignored. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
notebooks/summary_event_allocation.ipynb, following the convention set byacled_cursor_parity.ipynb: a question as the title, narrative markdown, committed outputs.Unlike that notebook, this one re-runs. Every number is recomputed from the consolidated store, and the three viewpoints come from the same
scripts/build_viewpoint.pythe pipeline uses. First run builds them (~11 min) and caches underdata/viewpoint/allocation_comparison/, which is gitignored.Six figures
Two findings that came out of writing it
Neither was known before. Both are in §3, measured, with the cells that produce them.
1. Production drops imprecise events, it does not merely mis-place them.
production_parityexcludeswhere_prec ∈ {4, 6}. That is 43,933 events carrying 785,626 fatalities — +22.6% on what production reports today. The new profiles admit them, per ADR-049.This is the dominant difference between profiles, it is an addition of recorded deaths rather than a redistribution, and anyone comparing totals across a profile switch will meet it before they meet anything else. It is the single most important fact for planning the switch.
2.
ceil_splitreports deaths that are not in the source.On the events both builds contain, they agree to within 104 out of 3.47M. The disagreement is not rounding noise in both directions: the legacy splitter rounds each month's share up, so a multi-month event gains a death per month.
Measured against the largest value UCDP ever recorded for each event across every candidate version in the store:
production_paritykgi_kti_distributedkgi_kti_activity96 deaths out of 3.47M is small. It is also live in production today, and it is the current path reporting more than the source contains.
What the notebook asserts
The naive check — "all three builds carry the same total" — is wrong, and I wrote it that way first. It fails for reason 1 above, which is intended behaviour.
What the notebook asserts instead is the property that holds and can fail: no allocator may report more deaths for an event than UCDP ever recorded for it. The guard is not decorative — the same code path returns 74/96 for
production_parityand 0/0 for the apportioned builds, so it is seen to fire on real input in the notebook itself.Not in scope here
No profile change, no pipeline change.
production_parityremains whatrefresh_pipeline.shinvokes.🤖 Generated with Claude Code
https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9