Repository navigation
feat(gift-eval): notebook-aligned replication registry - #45
Merged
Merged
Conversation
…flow Add a separate replication config (9 families), S3 prefix, pilot/full entrypoints, and verify/Makefile targets so we can re-run GIFT-Eval with official notebook params without changing the default full-grid configs. Co-authored-by: Cursor <cursoragent@cursor.com>
Use leaderboard-style geomean checks for the full replication grid while keeping strict per-job verify for CI and pilot. Drop Chronos-2 and FlowState from the FF replication matrix and install foundationforecast from PyPI on Modal. Co-authored-by: Cursor <cursoragent@cursor.com>
1 task done
Chronos2Pipeline predict_quantiles can exhaust A10G memory at default batch sizes; retry with a smaller batch before failing the job. Co-authored-by: Cursor <cursoragent@cursor.com>
…i OOM batches Add Chronos-2 to the 8-family grid with batch 16, mount editable foundationforecast on Modal, optional --model-key for targeted backfills, partial-failure dispatch logging, and lower Moirai replication batch sizes for A10G. Co-authored-by: Cursor <cursoragent@cursor.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
… S3 jobs in one walk - Add Chronos Bolt base/small, Chronos-2 small, Toto 2 4M, PatchTST-FM r1, TiRex 1.1, Moirai 1.1 large and T0 alpha to configs/replication with notebook-aligned params; add autogluon--chronos-2-small to models.yaml. - Replace per-job S3 exists() calls with a single fs.find() index (filter_jobs_by_s3_mode / index_s3_job_artifacts) and parse dataset names that contain '/'. - Add --max-containers to run_replication_full and cap Modal fan-out. - Skip aggregate verify for families whose HF reference is not published yet (REPLICATION_PENDING_HF_REFERENCE). - Use google/timesfm-1.0-200m-pytorch repo id for TimesFM 1.0. Co-authored-by: Cursor <cursoragent@cursor.com>
Add tests for parse_job_dir_suffix / filter_jobs_by_s3_mode, the 16-family count, Chronos-2 small and Moirai 1.1 large replication params, and skip the HF alias check for families pending a published reference. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…hPredictor Lift the per-batch predict_quantiles retry closure out of Chronos._predict so the halving logic is a plain callable that can be exercised without a GPU. No behavior change. Co-authored-by: Cursor <cursoragent@cursor.com>
Fake pipeline raises torch.cuda.OutOfMemoryError to check that Chronos-2 halves batch_size until it fits, keeps the reduced size for later batches, re-raises when it cannot halve further, and that non-Chronos-2 pipelines re-raise without a batch_size kwarg. Co-authored-by: Cursor <cursoragent@cursor.com>
verify-replication-full passed the aggregate check but then crashed with HTTP 404 while writing the analysis table: build_replication_table fetched the GIFT-Eval reference for autogluon--chronos-2-small, whose results are not published yet. Skip REPLICATION_PENDING_HF_REFERENCE keys like the aggregate verify does, and treat a remote HTTPError like a missing local reference (warn and skip) instead of aborting the run. Co-authored-by: Cursor <cursoragent@cursor.com>
…_chronos - test_replication_table.py: pending HF reference keys never hit the network; a 404 on the remote reference is skipped instead of raised. - Fold the _QuantileBatchPredictor OOM back-off tests into tests/models/test_chronos.py next to the other Chronos tests. Co-authored-by: Cursor <cursoragent@cursor.com>
Per-model official vs replicated leaderboard aggregates (MASE, CRPS), their relative difference, the per-config standard deviation of ours/official - 1, and A10G wall time / cost. Markdown table plus a collapsible LaTeX source (booktabs) for papers. Co-authored-by: Cursor <cursoragent@cursor.com>
…lication table sigma is now the standard deviation across the 97 configs of (ours - official) on the Seasonal-Naive-normalized scale, i.e. the same units as the aggregate columns, instead of the std of relative differences. Co-authored-by: Cursor <cursoragent@cursor.com>
…F column - Org first, alphabetical, shown once per group; models sorted by FF MASE. - |Delta| (absolute relative difference) instead of signed Delta. - 'Ours' renamed FF (FoundationForecast), defined in the note. - LaTeX uses threeparttable so the note spans the table width, with no 'Table N' caption; note states every model is within 2%. Co-authored-by: Cursor <cursoragent@cursor.com>
Replace the operator-oriented README with the experiment narrative (why, replication table PNG, Pareto PNG, Makefile repro, infra, cite). Figures only — no markdown/LaTeX tables. Co-authored-by: Cursor <cursoragent@cursor.com>
Link the replication table and Pareto PNG via user-attachments URLs instead of committing binaries to the tree. Co-authored-by: Cursor <cursoragent@cursor.com>
Paper-ready tabular (threeparttable) with org grouping, |delta|, FF, and shaded top-3 CRPS cells, under the hosted PNG. Co-authored-by: Cursor <cursoragent@cursor.com>
Replace the HTML-styled PNG and the collapsible source with a rendered booktabs table hosted on GitHub attachments. Co-authored-by: Cursor <cursoragent@cursor.com>
…table TimesFM 3, PatchTST-FM r2, and T0 beta are 1st/2nd/3rd on both metrics for official and FF aggregates. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…phones The landscape artboard shrank to unreadably small type in the GitHub mobile README; the stacked 720px figure fills the column instead. Co-authored-by: Cursor <cursoragent@cursor.com>
Host the 2x log-cost artboard via GitHub attachments instead of the stacked phone crop. Co-authored-by: Cursor <cursoragent@cursor.com>
Refresh the hosted README figure: Inter unchanged, only the callout brand name uses the frontier color. Co-authored-by: Cursor <cursoragent@cursor.com>
AzulGarza
marked this pull request as ready for review
October 8, 2026 19:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
configs/replication/with 9 model families, notebook-traced params (TimesFM-3 context 15360, Moirai2 4000/2048 batch, Chronos-2 uncapped max_length + HF alias, t0-beta slug, etc.), andnotebooks.yamltraceability.replicationregistry through eval, verify, and Modal so defaultconfigs/models.yamland existing S3results/layout stay unchanged.s3://foundationforecast-gift-eval/results/replication/<run_id>/with Modalrun_replication_pilot(9 ×m4_weekly/short) andrun_replication_full(9 × 97), plus Makefile sync/verify targets and README workflow.tests/test_replication_config.py(HF alias match, pilot job count, param snapshots) and--replicationon localrun_model.Test plan
cd experiments/gift-eval && uv run pytest tests/test_replication_config.py -n 0uv run modal run -m src.runners.run_modal::run_replication_pilot --run-id <RUN_ID>make verify-replication-pilot RUN_ID=<RUN_ID>after syncmake verify-replication-full RUN_ID=<RUN_ID>once pilot looks goodMade with Cursor