Committed code that has not yet been applied to the artifacts. Every entry here is a run someone still has to make before the change it belongs to is real.
This file exists because the pipeline's state is not in git. The verdict store, the
survivor pool, the text overlay and the API caches are all outside the repository, so
git status — the one check every session makes — cannot show that a run is owed. A
committed fix therefore looks finished while the artifact it fixes is still stale.
That is how issue #196's overlay backfill sat unrun: overlay.py::worklist() covered
the missing registrations, the commit landed, and 1,115 admitted OSF works kept
reaching the screen with no registration text in front of them.
How it is enforced. .claude/settings.json reads the open entries below at session
start and before every prompt. It never blocks a command — it states what is owed, so
the work cannot be reported as complete while an entry is open.
How to use it. Add an entry in the same commit as any change whose effect needs a
run. Tick it off (- [x]) in the commit that records the run's result. An entry needs
the command, pasteable from the project root, and what proves it worked.
- Render the admission gate (PR #229) into
data/extracted.csv. First merge the flora-validation change that addsscreen_deferredtoRETIRE_REASONS(csv_to_db.py): until then its retire step refuses the whole manifest. Then refresh the exemption list, check, render, and commit the three files together:.venv/bin/python -m analysis.build_human_judged --apply, then.venv/bin/python -m extract.export --release <newest> --check, then without--check; commitdata/human_judged.csv,data/extracted.csv(+ set-asides) anddata/retired_pairs.csv, push. Expect ~850screen_deferredretirements (857 on release 61f07fd07f19). Done when the next nightly run showsretire_supersededSUCCESS.
-
DONE 2026-10-05 — last batch 2026-10-04 night; 3 open works remain (retries). Timer stopped; final render 9,608 rows. Original entry: Finish the option-B campaign: nightly runs at 01:15 BST (after the midnight-UTC OpenAlex reset), systemd user timers
flora-night-run-4.timer(nightly from 2026-10-02; paused on 2026-10-01 because another project had OpenAlex priority),scripts/night_run.sh 2d767fe03199. First night (2026-09-25/26): screen 17,962 decided — proceed 11,514, discard 6,227, 221 incomplete (re-asked by the next run); extraction ran ~2,575 works until OpenAlex's daily budget ran out at 09:48 UTC; 8,757 left (≈61k OpenAlex credits, so ~3-4 more nights at this budget). Done when the extract dry run offers ~0 works; thensystemctl --user stop flora-night-run-4.timerand re-render (after the import). Start every run under a memory cap:systemd-run --user -p MemoryMax=8G -p MemorySwapMax=0 … scripts/night_run.sh 2d767fe03199(as a timer's service properties, the same two). The 2026-09-27 night grew past 14 GB of the box's 15 and thrashed it until a hard reboot; with the cap a runaway run is killed on its own, its claims lapse, and the next night resumes. -
DONE 2026-09-30 — rendered on release 2d767fe03199: extracted.csv 3,683 → 7,792 rows; 28 retirements appended (0.8% of the imported pairs). Later campaign nights need another render. Original entry: Re-render after the flora-validation import lands (not before: every render moves the removal share the maintainer's one-off
EXTRACTOR_MAX_REMOVAL_PERCENT=17was set for). Owed by6cbdb36(clean_doi re-spelling: 24 pair ids superseded, 1 held) and any later Stage 3 change..venv/bin/python -m extract.export --release <newest> --check, then without--check; commitdata/extracted.csvtogether withdata/retired_pairs.csvand the set-aside CSVs, push. Done when the next nightly run imports it withretire_supersededSUCCESS. -
DONE 2026-09-30 — admin "Sync + report" run cd9302ef at 19:12 UTC imported commit d7d239d (same data as 4de666d): +1,137 / −483 pairs (16.1%, cap 17), retire_superseded SUCCESS: 550 archived and deleted, 10 flagged. Original entry: Import the 2026-09-24 export and retire what it stopped shipping. flora-extractor
4de666d(pushed) shipsdata/extracted.csv(3,683 rows) anddata/retired_pairs.csv. Owed in flora-validation: merge PR #13 (sync baseline recovery by checksum, import pair-reassignment fix, nightly auto-retire; createsretired_recordson startup), set RailwayEXTRACTOR_MAX_REMOVAL_PERCENT=17for one run (removal is 16.1% of the 2,999 pairs imported atd7f55d9), let the nightly run or "Sync + report" import and auto-retire (~540 retired, ~5 flagged), then set it back to 10. Done when the run'sstage_statusshowssync_csvandretire_supersededSUCCESS. -
DONE 2026-09-23 — deleted. HF commit
e284f62removed the 14builds/*/candidates-*.parquet; the remote now lists 2,232 pool files under gated536bc51b9b2. A freshpool_sync --pull --no-overlayinto a scratch dir stampedexpected_files: 2232andpool_fingerprint()returnedd34c277e…, identical to the local pool. Original entry: Decide what to do about the 14candidates-*.parquetfiles in the shared pool repo.lukaswallrich/flora-survivor-poolcarries them in the pool root alongside the 2,232part-*.parquetfiles — 1,363,959 rows of the retiredCANDIDATES_COLScorpus (doi_r,title_r, …). Every pool reader here globs*.parquetover that directory, which is the failure the comment atsearch/snapshot_scan.py:269explains the_-prefixed sidecar to avoid; a pull also stampsexpected_files: 2246, so a straightrouterefuses the pool as partial. Locally they are moved tocache/legacy_candidates/and the sidecar re-stamped to 2,232 with the recorded gated536bc51b9b2. Fixing the REMOTE is a push decision for whoever owns it. Done when a freshpool_sync --pullproduces a directory thatfilter.engine routeaccepts untouched. -
Extract the Observatory works — DONE 2026-09-22. 1,198 of 1,198 have a row; the last 275 ran with
OPENAI_DAILY_TOKEN_BUDGET=0on the command line (~$0.65). The dry run offers only the 27 DOI twins held out under #210. Export 3,040 → 4,141 rows, 908 new works. In-pool agreement with the Observatory: 69% same original, 74% of those same outcome (analysis/mo_observatory/in_pool_agreement.csv). Original entry:(2026-09-22: 773 settled in one day-cap of luna tokens; export 3,163 → 3,885 rows, 594 new works; agreement with the Observatory inanalysis/mo_observatory/in_pool_agreement.csv). Original entry: Status 2026-09-21: #208 merged, rule promoted (eb9b892), routed as releasec048ab6483d3(screen_expensive7,760 → 9,105), screened live — 1,469 works, 1,346 proceed / 123 discard (92% admitted). The extract run settled ~150 works and stopped cleanly onOPENAI_DAILY_TOKEN_BUDGET(resets midnight UTC); the export was rendered (3,040 → 3,163 rows, 101 new works). Resume — under Python 3.13, not.venv(see the handover's Environment section; the dry run must printgeneration 010cf32bb63351e1):```bash PY=/home/lukas/.venvs/flora313/bin/python $PY -m extract.tier --release c048ab6483d3 --only "$(cat analysis/mo_observatory/extract_only_2026-09-21.txt)" # dry run $PY -m extract.tier --run --release c048ab6483d3 --batch-label observatory-2026-09-21 --only "$(cat analysis/mo_observatory/extract_only_2026-09-21.txt)" $PY -m extract.export --release c048ab6483d3 ``` The `--only` list is the worklist MINUS 27 junk OpenAlex records that carry a real paper's DOI under another paper's title (`analysis/mo_observatory/doi_title_mismatch.csv`, 30 of the rule's 1,790 admissions). `doi_in` cannot tell them apart and the screen passed them; extracting one fuses two papers. They stay open until a rule discards them. Expect real spend: the first 150 works used far more gpt-5.6-luna tokens per work than the dry run's per-rung assumption. Done when the dry run above offers 0. Original entry, for the numbers: **Ship the Metascience Observatory rule: merge #208, promote, route, screen, extract.** Step-by-step, with the numbers to check at each step and the environment gotchas: **[`docs/handover-observatory-screen.md`](docs/handover-observatory-screen.md)**. The decision is made — measured, it buys 1,346 works no rule reaches. The scoring half is DONE (2026-09-20, against release `2e31c9543026`, reproduced locally from the pulled pool — same id, so the admission column is the live one). `analysis/mo_observatory/curated-observatory.json` names 2,987 Observatory replication DOIs; 2,396 of them are pool works: | | works | | --- | --- | | already admitted to `screen_expensive` | 579 | | `pending` / `no_filter_matched`, WITH text — what the rule buys | **1,346** | | `pending` / `no_filter_matched`, no text — would land in `no_text` | 164 | | `pending` / `no_text` — a claim rule matched; not rescued by this | 308 | | `discard` (`osf-registration-protocol`) | 1 | So it is worth shipping: 1,346 works no rule reaches, against the 7,760 the release currently admits. Promotion is `git mv analysis/mo_observatory/curated-observatory.json filter/spec/`, then add `"curated-observatory": ("screen_expensive", 745, None, False)` to `EXPECTED` in `tests/test_engine_spec.py` and widen `test_the_expensive_screen_has_exactly_two_routes`, then `.venv/bin/python -m filter.engine route` (mints a release) and `.venv/bin/python -m filter.engine screen --tier screen_expensive --run --release <new>`. Done when the extract dry run (`.venv/bin/python -m extract.tier`, no `--run`) reports the works it opened. **One thing to settle before the screen runs, and it is not code.** `SCREENING_MODEL_2` (`gpt-5.4-mini`) needs `OPENAI_API_KEY`, which is absent from the working `.env`. The cache is no longer a blocker: the shared repo did hold the `llm` shards all along — the manifest had un-published them (#209) and was repaired 2026-09-21, so a pull now reaches ~180,000 LLM answers including the `classifyvote_*` entries, and a re-screen reads them rather than re-buying. Regenerating the list needs the source CSV, which is gitignored (13 MB): `curl -sSL -o analysis/mo_observatory/replications_database_2026_09_04_184008.csv https://raw.githubusercontent.com/delton137/metascience-observatory/main/data/replications_database_2026_09_04_184008.csv` -
Re-screen every screened work under the new voter pair and gate (voter 1 is now
deepseek/deepseek-v4-flash@low, the gate G-unanimous; evidence:analysis/screening_eval/cheap_voter_2026-08.md). The voter swap minted a new screening generation, so every screened work is claimable again and the extract tier's worklist offers nothing until this runs — it holds back every work without a current-generation screen verdict, which makes this a prerequisite of the issue #198 re-extract below..venv/bin/python -m filter.engine screen --tier screen_expensive --run --release 56076eb—--releaseis required, and release8b3drefuses on the rebuilt overlay. If the OSF projects backfill below runs first, its re-route mints a newer release; name THAT one here and in the extract entry instead, or the OSF works whose text it recovers are screened twice. Expect roughly $1.80 of DeepSeek spend for ~10k works (measured $0.070/390 calls at effort low; halved off-peak outside 01:00–04:00 and 06:00–10:00 UTC from 2026-08-16); the gpt-5.4-mini side reads the joint-era cache entries — including the model-less majority shape, lifted by provider — and costs nothing. Before the full run: the 20-work smoke test (… screen --tier screen_expensive --run --limit 20 --mode validation --release 56076eb) must show no missing votes and a sane per-call wall clock — it is what checks the OpenRouter provider pinning (require_parameters+ throughput floor) leaves an eligible host for every model. Done whenfilter.engine statusshows the screen tier settled for the release and the extract worklist is non-empty again. Done 2026-08-14/15, release16d370746b45(minted by the combined re-route below): 7,760 works screened, 880 discard / 6,880 proceed; DeepSeek ≈$2 (9.0M in / 1.8M out). The extract worklist reopened at 5,928 works. -
Reopen the 55 works that shared one OSF identifier (issue #201). Until
681556a,osf_registration_guid()read the path segment in front of a guid as the guid, so every pool work with a download-shaped URL keyedosf:osf.io/download. That key is on disk as a definitive miss: the 55 were asked about once, between them, and all still wear that one answer. The code is fixed; the checkpoint is not..venv/bin/python -c "import shared.config; from shared import abstract_store; print(abstract_store.drop_misses(['osf']))"then re-run the OSF backfill phase as the Done entry below records. Done when no work resolves to a guid ofdownload,preprintsorproject, and the re-run reports a non-zero "Abstracts found". Done 2026-08-13: the misses were dropped and the OSF backfill re-run; the re-route below carried the recovered text. -
Freeze and re-route, carrying the OSF text and the no-text exemption (issue #196). Two committed changes reach nothing until a route reads them:
overlay-0007.parquet(57 project descriptions) is written but not frozen, ande0feb7dexempts titled OSF records from theno_textdowngrade..venv/bin/python -c "import shared.config as c; from filter.engine.overlay import freeze; print(freeze(c.OVERLAY_DIR)['overlay_hash'])"then.venv/bin/python -m filter.engine route. Expect roughly 849 works to move out ofpending/no_textinto a screening pile. Carry the issue #200 aliases in the SAME route if they are ready — each re-route mints a release, and screening between two of them pays twice for works about to be merged. Done when the route report showspendingdown by ~849 and the new release id is recorded in the re-screen entry above. Done 2026-08-14: frozen and re-routed together with the issue #200 aliases; release16d370746b45, overlay hash011bba95582f(9 chunks), 7,760 works inscreen_expensive. -
Deduplicate OpenAlex works that name one OSF record (issue #200). One OSF record ships as several rows: 117 records in
data/extracted.csvare reached by more than one work id, 296 of 2,602 rows (11%). The aliases ARE derived and merged:filter/spec/aliases.jsonholds the 14,744 same-guid entries (analysis/build_osf_aliases.py, commit3fb6efe— adjudication and QC in its message; 405 groups excluded as OpenAlex mis-located works). What remains is the route that applies them — the combined re-route above — and the export check. Done when a re-route has collapsed the duplicates and the export's row count falls by the merged surplus (expected ≈165 rows on the current CSV). Done 2026-08-16: the export of release16d370746b45shows OSF records reached by more than one work id down from 92 (223 rows) to 6 (13 rows). The total row count is not the proof any more: the re-screened gate admitted 928 works no earlier render held, so the main CSV rose to 3,147 rows. -
Re-extract every settled work under the new outcome policy (issue #198). The outcome prompt now codes as the authors report: an overall author verdict decides; otherwise their comparisons to the named original decide; a result with no stated bearing on that original's finding is
cannot_be_determined; and «descriptive» needs the authors' own account of reusing the methods. Editing it minted a new extract generation, so every settled work is already reopened and the shipped CSV still carries verdicts coded under the old policy..venv/bin/python -m extract.tier --release 56076eb --runthen.venv/bin/python -m extract.export --release 56076eb(release8b3drefuses on the rebuilt overlay; if the OSF backfill re-routes first, name its release instead — the export now refuses a stale release too). Before the run:.venv/bin/python -m analysis.purge_osf_docs --apply(re-rank the cached OSF storage files under the plan-file demotion) and.venv/bin/python -m analysis.purge_epmc_retries --apply(unsuppress the fixed Europe PMC tier), plus.venv/bin/python -m shared.cache_sync --pull --parts doi_verify. Both purges print what they would do without--apply. Expect the run to re-parse the ~900 cached documents:TEXT_EXTRACTION_VERSIONis at 2, so every parse entry written under the old page window and section splitter is a miss. That is local compute (~7.5 s per document, ~30 min atEXTRACT_WORKERS=4) and buys no LLM call — the reference extractors are keyed on the prompt, the model and the PDF's content hash. The outcome-family LLM calls are re-bought; the link picks, keyed confirms and search grades are cache hits, and DOI verification is ≈0 OpenAlex credits (its caches carry no generation — checkprint_search_summary()at the end came out near zero). Sandbox-measured on 25 works before commit: 17 of the 20 stratified rows unchanged, nosuccessfulorfailedrow pushed tocannot_be_determined. Done when the export prints NOrows from a superseded generation:line —--checkalone does not test carry-forward — and section 2 ofhandover.htmlis re-read off the new render. Done 2026-08-16: 5,928 of 5,928 works settled (1,567 undergpt-5.4-mini, the rest undergpt-5.6-lunafor $6.12); the dry run reports 0 open works. Rendered with.venv/bin/python -m extract.export --release 16d370746b45 --current-generation-only: 3,147 main rows, 6,434 in all,--checkzero diff. The flag is deliberate: the default render carries 96 works whose only verdict is from a superseded generation and which the worklist never offers — 85 the current screen discards, 11 on the FLoRA skip list — so the done-criterion (nosuperseded generationline) held only with it. The export now drops screen-discarded works itself, and the plain render matches the file (--checkzero diff). -
The OSF projects backfill (issue #196), 2026-08-13. 1,699 targets, 726 descriptions recovered, 57 written to
overlay-0007.parquet. The gap is the two write guards doing their job: 625 refused because the row already had a pool abstract an overlay row would have REPLACED (751 of 752 admitted OSF projects carry one, 297 past 500 chars against a median description of 252), and 17 refused as labels — "Replication", "PCI RR submission" — that the row's own title already carries. An earlier attempt the same day recovered 0 of 1,699: OSF was down, every request failed from the first one, and the circuit breaker stopped the phase at 25 consecutive transient failures without checkpointing any of them. The re-run waited for two consecutive healthy probes and ran atOSF_RATE_SEC=2. Read afterwards, and it changed the plan: 941 targets still have no text, but only 93 of those are admitted and 92 of THOSE already have a pool abstract — so the honest yield is one admitted textless work, plus 57 rows that were unroutable. That is what sent the fix to theno_textpolicy instead of to more text sources. -
Re-route after the OSF overlay backfill (issue #196), 2026-08-13. Release
56076eb48fda, overlay hasha3ccbfb18a38(7 chunks, 3,159 rows). The backfill recovered 427 of 854 OSF records; the rest are projects and components, which the registrations endpoint 404s on and no template line can ever reach.osf-registration-protocolmatched 2,103 → 2,479, andosf-registration-completed523 → 574. Both domain-coverage gaps shrank: 1,685 → 1,348 and 1,162 → 774. Piles moved discard +376, screen_expensive −336. Which release the next campaign names is now a decision, not a formality: 336 works 8b3d admitted are preregistrations56076eb48fdadiscards, so an export against the new release stops shipping their rows — which is the fix, and is also a visible drop indata/extracted.csv. The two open entries above still say--release 8b3d; changing them to56076ebis what applies the re-route to what ships.