How Rust Projects Use Macros: A Large-Scale Empirical Study

Replication package for the paper submitted to SBCARS 2026 @ CBSoft 2026.

Paper link: (to be added in case of acceptance — camera-ready DOI will appear here)

---

Abstract

Macros are a central feature of the Rust programming language, enabling metaprogramming and compile-time code generation. However, there is limited empirical understanding into how they are used in practice. We analyze macro usage across the 99 most popular open-source Rust repositories on GitHub, comprising 4,001 crates and 2,215,517 total invocations, using a custom TreeSitter-based static analysis pipeline. We address three research questions: (RQ1) how are invocations distributed across macro categories (declarative, derive, attribute, function-like); (RQ2) how concentrated is macro usage around a small set of widely-used ecosystem crates; and (RQ3) how does invocation density vary across project domains. We find that declarative macros dominate macro definitions while function-like and declarative call sites jointly dominate macro invocations; derive and attribute macros from widely-used ecosystem crates account for 31.1% of invocations, concentrated among a small set of providers; and educational and AI/LLM-oriented projects exhibit higher density than general-purpose application code.

---

Repository Organization

.
├── src/               Rust scraper/analyzer pipeline (tree-sitter-based)
├── scripts/           Python scripts for statistics, tables, and visualization
│   ├── generate_numbers.py          populate paper/numbers.tex and paper/tables/
│   ├── generate_report.py           generate interactive HTML report
│   ├── stats_tests.py               Kruskal-Wallis + Mann-Whitney for RQ3
│   ├── score_validation.py          inter-rater κ and per-category precision
│   ├── draw_validation_sample.py    draw stratified validation sample
│   └── generate_domain_labels.py    browser-based domain labeling helper
├── corpus/
│   ├── sbcars-2026/   Frozen corpus for this submission
│   └── latest/        Symlink-equivalent: mirrors sbcars-2026/ now, overwritten by a fresh run
├── data/              Frozen analysis outputs used in the paper
│   ├── results.csv                              per-repo macro counts and LOC
│   ├── data.json                                full structured output (HTML report)
│   ├── domain_labels_final.csv                  domain label for each of the 99 repos
│   ├── validation_sample_answer_key.csv         answer key for the sample
│   ├── validation_sample_pass_author1_labeled.csv   author 1 labels
│   ├── validation_sample_pass_author2_labeled.csv   external researcher labels
│   ├── validation_stats.json                    κ and per-category precision
│   └── stats_tests.json                         Kruskal-Wallis + Mann-Whitney results
├── Cargo.toml
├── requirements.txt
└── README.md          (this file)

---

Requirements

Rust analyzer (src/)

• Rust stable ≥ 1.96 — install via rustup.rs
• cloc ≥ 1.96 on PATH — only needed for the full pipeline re-run
• GitHub personal access token — only needed for Step 1 of the full pipeline (querying the GitHub GraphQL API for the top-100 repo list); not needed when using the pinned corpus in corpus/latest/
• ~20 GB disk space if cloning all repositories

Python scripts (scripts/)

• Python ≥ 3.10
• Install dependencies once:

python3 -m venv .venv
source .venv/bin/activate      # Windows: .venv\Scripts\activate
pip install -r requirements.txt

requirements.txt pins: plotly, pandas, numpy, statsmodels, scipy.

Hardware

All Python scripts run on a standard laptop with no GPU. The full Rust pipeline (clone + analysis of all repos) takes roughly 1–2 hours depending on network speed. Using the frozen data/ outputs — the recommended path for artifact evaluation — completes in under 5 minutes (possibly less).

---

Installation

After setting up the Python environment above, verify it:

source .venv/bin/activate
python scripts/stats_tests.py
# Expected last line: results written to data/stats_tests.json

Verify the Rust build:

cargo build --release 2>&1 | tail -1
# Expected: Finished `release` profile [optimized] target(s) in ...

---

Reproducing Paper Results

All three RQs can be reproduced from the frozen data/ directory without re-running the scraper. The full pipeline re-run is optional and described at the end of this section.

RQ1 — How are macro invocations distributed across categories, and which macros are most frequently invoked?

Primary data: data/results.csv (columns declarative_count, derive_count, user_attr_count, builtin_count) and data/data.json (per-macro top-N counts).

source .venv/bin/activate
python scripts/generate_numbers.py

Outputs written:

File	Contents
paper/numbers.tex	All \newcommand values cited in the paper
paper/tables/top-builtins.tex	Table of top built-in attribute invocations
paper/tables/top-user-attrs.tex	Table of top user-defined attribute invocations
paper/tables/top-fl-decl.tex	Table of top function-like/declarative invocations
paper/tables/top-derive.tex	Table of top derive macro invocations
paper/tables/top-nonstd-fnlike.tex	Top non-std function-like/declarative
paper/tables/top-nonstd-derive.tex	Top non-std derive macros

Key numbers to verify against the paper:

\newcommand	Expected value
\totalRepos	99
\totalCrates	4,001
\totalInvocations	2,215,517

Known limitation: declarative_count == function_like_count in every row of results.csv. Both columns count macro_invocation AST nodes because tree-sitter cannot syntactically distinguish declarative (macro_rules!-defined) from function-like proc macro call sites at the invocation level without macro expansion. This is documented in §3.4 of the paper and quantified by the inter-rater validation (§3.5).

---

RQ2 — How concentrated is macro usage around a small set of widely-used ecosystem crates?

Primary data: data/data.json (per-macro invocation counts aggregated across the corpus, with std_origin flags distinguishing standard-library from ecosystem macros).

source .venv/bin/activate
python scripts/generate_numbers.py   # idempotent — safe to re-run

RQ2 findings are captured in the \topN* and \nonstd* commands written to paper/numbers.tex, and in paper/tables/top-nonstd-*.tex.

---

RQ3 — Does macro invocation density differ significantly across project domains?

Primary data: data/results.csv joined with data/domain_labels_final.csv (99 repos labeled into 6 domains: systems-cli 29, application 20, ai-llm 17, library 17, devtools 11, educational 5).

source .venv/bin/activate
python scripts/stats_tests.py

Output written: data/stats_tests.json

Key numbers to verify against the paper:

Metric	Expected value
Kruskal-Wallis H	25.39
Kruskal-Wallis df	5
Kruskal-Wallis p	0.00012
Significant pairs (Bonferroni-corrected)	3 of 15
Sensitivity H (excl. educational repos)	20.26
Sensitivity p	0.00044
Sensitivity significant pairs	2 of 10

---

Classifier Validation — Inter-rater agreement (κ = 0.917)

Reproduces the inter-rater reliability results reported in §3.5.

Primary data: data/validation_sample_pass_author1_labeled.csv, data/validation_sample_pass_author2_labeled.csv, data/validation_sample_answer_key.csv.

source .venv/bin/activate
python scripts/score_validation.py --sep ";"

Output written: data/validation_stats.json

Key numbers to verify against the paper:

Metric	Expected value
Cohen's κ	0.917 ("Almost perfect")
Precision — derive	100%
Precision — user-defined attribute	75%
Precision — built-in attribute	67.6%
Precision — declarative + function-like	34.1% (conflation artifact — see §3.4)
Items excluded as unsure	9

The 200-item stratified sample was drawn with SEED=20260619 via scripts/draw_validation_sample.py. The sample and both authors' labels are committed to data/ so the scoring step can be reproduced without re-drawing.

---

Interactive HTML Report

Generates a Plotly-based interactive report with all per-repo and per-macro charts from the paper.

source .venv/bin/activate
python scripts/generate_report.py
open data/report.html        # macOS; use xdg-open on Linux

---

Full pipeline re-run (optional)

Re-runs the scraper from scratch against the pinned corpus. The pipeline is resumable: delete data/state.ron to restart, or set individual step timestamps to None to re-run only that step.

export GITHUB_TOKEN=<your-personal-access-token>
cargo run --release

Corpus pinning: corpus/sbcars-2026/ is the authoritative frozen snapshot for this submission — 100 repos and the exact HEAD commit per repo used in the paper. The tool reads from corpus/latest/ at runtime; corpus/latest/ currently mirrors corpus/sbcars-2026/ exactly. A fresh cargo run would overwrite corpus/latest/ with a new GitHub query, so copy corpus/sbcars-2026/ over corpus/latest/ first if you want to reproduce the paper's exact corpus:

cp corpus/sbcars-2026/repos.json    corpus/latest/repos.json
cp corpus/sbcars-2026/snapshot.json corpus/latest/snapshot.json

Pipeline steps (each is skipped if already recorded in data/state.ron):

1. github::get_most_popular_repos — loads from corpus/latest/repos.json
2. github::clone_repos — clones into data/repos/<owner>.<name>/
3. cloc::cloc_repos — runs cloc --include-lang=Rust per repo
4. crate_paths::find_crate_paths — discovers all crate roots (Cargo.toml)
5. analyzis::analyze_crates — tree-sitter AST walk, counts all macro definitions and invocations
6. count_code::count_crates_code — counts source lines per crate
7. Writes data/results.csv and data/data.json

After the pipeline completes, run the Python scripts above to regenerate all derived outputs.

---

License

Released under the MIT License — see LICENSE for the full text.