I test whether language models know how certain to be, and whether the values they state hold together — an economist's experimental toolkit (pre-registration, published human baselines, proper scoring) pointed at model behavior. Dual graduate training in Economics (Iowa State) and Information (Michigan).
Running an ongoing weekly evaluation program. Through Round 10 it asked whether frontier models represent their own uncertainty — probabilistic forecast calibration, interval calibration as question specificity decreases, and cross-level belief consistency. Scored with proper scoring rules (CRPS, log loss, pinball), ECE, and ground-truth-free self-consistency checks; power-analyzed rather than anecdotal (effect sizes, scenarios-needed-for-80%-power, multiple-comparison caveats). Across that stretch: frontier models stay overconfident on rare/underspecified cases, and self-reported confidence is largely uninformative. The anchoring arm closed at Round 10, and the finding it ended on wasn't the anchoring effect itself but a same-turn contradiction — models answer "greater" on turn one, then emit an interval lying entirely on the other side of the anchor. Whether clearer labels fix it turns out to be model-specific, token priming was ruled out with a matched control, and the Round 10 confirmation was predicted in a git-committed file before the runs and held on a new item set.
From Round 12 the question is no longer uncertainty. llm-valuation-evals administers the stated-preference referendum from Vossler et al. (2023, PNAS) and asks whether the dollar figures cohere — scope, distance, spatial scale, income — without treating the published human estimates as an answer key. On the referendum, the two newest models are the ones whose votes fall off with price, and Sonnet 5 is the only one to pass scope ordering, nested dominance, and distance decay in that grid. Haiku 4.5 votes yes at every price through $3,000. The income and basin follow-up was frozen in git before the runs: willingness to pay rises with income wherever both sides can be compared, and the full study region is worth at least as much as the local watershed for Sonnet 5 and gpt-5.6-terra. Sonnet 4.5 and GPT-4o price the larger region below the home watershed in a few cells.
An ongoing weekly program. Rounds 1–10 ask whether LLMs can represent their own uncertainty; from Round 12 the question is whether dollar figures from a stated-preference instrument cohere. Newest first.
| Round | Eval | Question | Builds on |
|---|---|---|---|
| R12– | llm-valuation-evals | Handed the Vossler et al. (2023) water-quality referendum, do a model's dollar figures cohere — scope, distance, income — without a human answer key? | Vossler et al. 2023 |
| R6–R10 | llm-anchoring-evals | Does an irrelevant anchor move a model's estimate and its own interval — and do the two turns agree with each other? | R3 taxon substrate |
| R5 | eval-meta-consistency | In a fresh context, does a model know where its own uncertainty is highest? | R3 p10/p50/p90 CIs |
| R3 | bird-taxonomy-evals | Are independent beliefs about genus/family/order mutually coherent? | R1 species domain |
| R2 | florida-weather-evals | Do intervals widen as questions get less specific? (known unknowns) | R1 design |
| R1 | michigan-bird-evals | Probabilistic calibration against eBird ground truth (unknown unknowns) | — |
The income and basin round was pre-registered — the prediction sits in commit 049d367, committed before those runs, and it held.
R13 (advisor vs household persona) was pre-registered — the prediction sits in commit 19f8d63. The headline was mixed: the persona moved GPT-4o, not the Claude models.
Round 10 was pre-registered — the prediction sits in commit f09a6c2, committed before the confirmatory runs, and it held.
Side branch (R4) — same harness, different question: model output quality and judge bias rather than uncertainty. haiku-evals (5-7-5 form + subject grounding via embedding similarity) → haiku-judge-evals (blind pairwise judging — week 1 finds self-preference under side-by-side comparison; week 2's PRePair protocol, with an out-of-family Gemini judge, takes that signal to about zero)
- plagiarism-detection-ir — multi-signal (BM25 + SBERT + Jaccard) reuse detection, PAN 2011 benchmark
- sandhill-crane-migration-networks — max-flow min-cut migration bottlenecks over 3.2M observations
- ebird-species-prediction — collaborative-filtering and multi-task models over 13.6B observations (SLURM/HPC)
- Research integrity & reproducibility — independent statistical replications and forensic data analysis at Cornell supporting institutional research-integrity investigations; co-launched a cross-disciplinary reproducibility initiative, independently reproducing 15+ peer-reviewed studies across economics, sociology, communication, and veterinary medicine; presented methodology at IASSIST 2019.
- Large-scale applied ML — billion-observation HPC pipelines (SLURM, Great Lakes cluster) for ecological modeling: collaborative-filtering and multi-task neural models over 13.6B observations, directed-network migration bottleneck analysis via max-flow min-cut over 3.2M observations.
- Information retrieval — multi-signal (BM25 + SBERT + Jaccard) plagiarism/reuse detection pipeline evaluated on the PAN 2011 benchmark, addressing a gap standard tools like Turnitin miss.
- Agentic systems — technical lead on an LLM-powered natural-language-to-simulation-code interface for a biological cell-modeling platform (Compucell3D / UMich).
