Tokens-per-Parameter Coverage Is Critical for
Robust LLM Scaling Law Extrapolation

Joshua Shay Kricheli*, Alexander Lawrence Reid*, Soumajyoti Sarkar, Venkata Gandikota, Paulo Shakarian
_{* equal contribution}

---

• What is this?
• Installation
• Quick start
• Code structure
• Full pipeline
• Paper item to command
• Known behaviours
• Testing
• Citing
• Acknowledgments

What is this?

The code and data pipeline behind the NeurIPS 2026 paper Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation.

Scaling laws are fitted on small models and used to predict large ones. Whether such a fit extrapolates depends on the tokens-per-parameter (TPP) ratios D/N that the training runs cover:

Collinear (CO) design	Non-collinear (NC) design
Grows N and D together along a few TPP rays	Spreads the same number of runs over TPP
The law's parameters are badly conditioned	The parameters are well conditioned
Extrapolates poorly	Wins 97.3% of the extrapolation comparisons

We test this on about 1,900 trained models over five corpora and four scaling laws, and give a criterion, computable before any training, that tells which designs are well conditioned.

All the data the analysis needs is public on HuggingFace and needs no account:

• leibnitz-lab/colinear_scaling_models - training metrics, checkpoints and pre-computed outputs of every trained model;
• leibnitz-lab/public_checkpoint_reanalysis - run tables of the ten published studies re-analysed in the camera-ready version.

Installation

uv is the only tool needed. One locked environment (Python 3.10) runs the analysis, the training and the tests.

git clone https://github.com/lab-v2/tpp-coverage.git && cd tpp-coverage
uv sync

Quick start

Reproduce the paper's tables and figures from the public data, no GPU:

uv run python src/tpp_analysis/download_and_extract.py --results-only
uv run python src/tpp_analysis/run_analysis.py all --data-dir extracted_losses/ --n-workers "$(nproc)"
uv run python src/tpp_analysis/generate_appendix_tables.py full_coverage_results/
uv run bash src/tpp_analysis/visualizations/run_appendix_figures.sh

Add --clear-cache to run_analysis.py to refit every law from the loss tables; the refit reproduces the released results bit for bit (about 20 minutes on 50 cores).

Code structure

%%{init: {"theme": "base", "flowchart": {"curve": "basis", "nodeSpacing": 30, "rankSpacing": 50, "padding": 14}, "themeVariables": {"fontFamily": "Inter, -apple-system, Segoe UI, Helvetica, sans-serif", "fontSize": "15px", "lineColor": "#94A3B8", "clusterBkg": "transparent", "clusterBorder": "#94A3B8", "titleColor": "#94A3B8", "edgeLabelBackground": "transparent"}}}%%
flowchart TB
    subgraph TRAIN["TRAIN  ·  GPU"]
        direction LR
        A("Download<br/>5 corpora") --> B("Generate the<br/>N × D grid") --> C("Train<br/>1,900 models")
    end
    subgraph ANALYSE["ANALYSE &nbsp;·&nbsp; CPU"]
        direction LR
        D("Extract<br/>losses") --> E("Fit 4<br/>scaling laws") --> F("Tables<br/>and figures")
    end
    H[("Public data<br/>on HuggingFace")]
    G("Camera-ready<br/>analyses")
    TRAIN ==> ANALYSE
    H -.-> ANALYSE
    H -.-> G
    classDef train fill:#F76900,stroke:#FFB27A,stroke-width:1px,color:#FFFFFF
    classDef analyse fill:#1F2A5C,stroke:#7C8DD6,stroke-width:1px,color:#FFFFFF
    classDef data fill:#334155,stroke:#94A3B8,stroke-width:1px,color:#FFFFFF
    classDef extra fill:#475569,stroke:#94A3B8,stroke-width:1px,color:#FFFFFF
    class A,B,C train
    class D,E,F analyse
    class H data
    class G extra

Loading

Folder	What it holds
src/training_models/	Llama-style models and the distributed trainer
src/tpp_analysis/	Corpora, job generation, loss extraction, law fitting, tables and figures
camera_ready/	The two analyses added for the camera-ready version
docs/	Step-by-step guides
tests/	The pytest suite

A one-line map of every top-level folder is in docs/STRUCTURE.md.

Full pipeline

Step	Command	Guide
1. Download and tokenize the corpora	src/tpp_analysis/dataset_downloaders/download_{wiki,cosmopedia,redpj,peso,c4}.py	TRAINING
2. Write the jobs of the trained grid	src/tpp_analysis/generate_jobs/generate_jobs.py --dataset <name>	TRAINING
3. Train (4 GPUs per run)	src/training_models/training/dist_trainer_main.py under torchrun	TRAINING
4. Extract per-epoch losses	src/tpp_analysis/get_results/get_results.py from_local <metrics dir>	DOWNLOADING
5. Fit the laws, write the tables	run_analysis.py, run_special_analysis.py, run_prelim_bigtpp.py, generate_appendix_tables.py	ANALYSIS
6. Figures	run_appendix_figures.sh, notebooks in visualizations/	VISUALIZATIONS

Steps 1-3 need GPUs (about 3,600 GPU-hours in total). Step 4 can start from the released metrics: download_and_extract.py without --results-only fetches them (~0.5 GB).

Paper item to command

Paths are under src/tpp_analysis/ unless noted.

Paper item	Command
Main win-rate table (97.3%) and R2/RMSE summary	run_analysis.py full_coverage
Appendix metrics, fitted parameters, win rate by holdout split	generate_appendix_tables.py full_coverage_results/
BF16 replication (98.7%)	run_special_analysis.py, then generate_appendix_tables.py special_results/ --special
BigTPP preliminary (63.3%)	run_prelim_bigtpp.py
Regime-A subset table	dispatch_seeds.sh 0 21, subset_analysis/aggregate_seed_runs.py, subset_analysis/per_law_winrate_ci.py --mode first
Figure 1 and appendix three-panel figures	run_analysis.py seed_variance, then visualizations/run_appendix_figures.sh
Architecture table, n-layers figure	utility/audit_architectures.py, then utility/plot_nlayers_distribution.py --mode runs
n-gram KL table	ngram_analysis/run_all_ngrams.sh (needs the tokenized corpora)
Published-design classification (24 designs)	camera_ready/grid_classification/classify.py
Public-checkpoint reanalysis	camera_ready/public_checkpoints/

The theorems and the severity, eigenvalue and R_min tables are analytic; no script computes them.

Known behaviours

These produced the paper's numbers and are kept as they are.

• Padding is scored. The cross-entropy has no ignore_index, so padded positions count toward the loss, and reported losses sit below the loss on real tokens.
• BF16 is a full cast. --use-bf16 calls model.bfloat16(); only the logits are cast to FP32 for the loss.
• Four GPUs per run. The learning-rate schedule follows the per-GPU step count, so another GPU count trains a different model.
• One training seed. Every model is trained with seed 42; the analysis seeds are fitting restarts and subset draws.
• Released runs. C4 has 321 of its 324 runs; epoch-2 and epoch-3 checkpoints exist for all.

Testing

uv run pytest -n auto

The suite checks law recovery on synthetic data, every entry point's --help, that the released tables are reproduced, the camera-ready analyses, the README's links, images, diagram and citation, and that no private host or path is in the tree.

Citing

If you use this work, please cite our paper:

@article{kricheli2026tpp,
  title         = {Tokens-per-Parameter Coverage Is Critical for Robust {LLM} Scaling Law Extrapolation},
  author        = {Kricheli, Joshua Shay and Reid, Alexander Lawrence and Sarkar, Soumajyoti and Gandikota, Venkata and Shakarian, Paulo},
  journal       = {arXiv preprint arXiv:2605.08541},
  year          = {2026},
  eprint        = {2605.08541},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2605.08541},
  note          = {Accepted to the 40th Conference on Neural Information Processing Systems (NeurIPS 2026)}
}

Acknowledgments

This work was done at the Leibniz Lab, Department of Electrical Engineering and Computer Science, Syracuse University. It was funded by ARO grant W911NF-24-1-0007 and NSF ACCESS grant CIS230318. The code is released under the GNU General Public License v2 (LICENSE).