Repository navigation
Run the local arm for real: local_logits and generative have never left the test suite #3
Description
Why this is the highest-value item in the backlog
local_logits and generative have never run outside the test suite. Both are
exercised against fakes; neither has been pointed at a real checkpoint or a
real chat model on a real dataset.
That matters more for local_logits than it sounds, because the
restricted_softmax class is the null hypothesis of the entire tool.
plumbline's argument is that a vendor's calibration claim has to be measured
rather than believed, and the thing it should be measured against is a plain
softmax over the declared option tokens, read straight out of an open model
with no calibration claim attached. METHODOLOGY states this: the local arm "is
the null hypothesis the calibrated claims have to beat."
Right now that is an assertion. Nobody has run it.
What done looks like
local_logitsruns against a pinned open checkpoint on the vendored public
fixture, and the run completes end to end: artifact written, metrics computed
against their nulls, report rendered.generativeruns against a real chat model and its refusal path is observed
rather than simulated. It declaresnonesemantics, so it contributes
accuracy only; confirm the report actually excludes it from calibration
rather than merely intending to.- The
localextra installs and imports on a machine that has a GPU, and the
revision-pinning guard is exercised against a real HuggingFace revision. - METHODOLOGY's
restricted_softmaxsection is updated with what was measured,
with its n and its null, replacing the current assertion. docs/PLAN.mdrecords the checkpoint, the revision, and the date.
The outcome worth wanting
A result showing the local readout is well calibrated would weaken a claim
this project makes. It would mean a restricted softmax over option tokens,
with no calibration training and no vendor claim, is already doing the job the
calibrated claim is sold on, and that the gap plumbline exists to detect is
smaller than implied.
That is the correct outcome to want. A measurement tool that only produces
results flattering to its own framing is not a measurement tool. If the null is
strong, the honest move is to say so in METHODOLOGY and let readers draw the
conclusion, not to quietly stop reporting it.
Either result is publishable. Only not running it is not.
Activity
fix(local): what running the local arm for real found (#96)
TMHSDigital commentedon Sep 25, 2026
The local half is done (#96).
local_logitsran end to end againstQwen/Qwen2.5-1.5B-Instruct, pinned to commit989aa7980e4cf806f80c7fef2b1adb7bc71aa306, on the vendored fixture. It used a GPU (RTX 4060, CUDA torch 2.14, transformers 5.17, Windows). Thelocalextra installs and imports, the revision pin is enforced, the artifact is written, every figure is read against its null, and the report renders. Two runs gave identical figures.The run found four faults, all fixed in #96:
--deviceoption.Still open here:
restricted_softmaxmeans, so it should be decided before Per-label temperature scaling, so a refused global fit has a remedy rather than only a diagnosis #4 builds on this arm.feat(local): ask a local checkpoint its options by letter (#102)
TMHSDigital commentedon Sep 25, 2026
The multi-token decision is implemented (#102): letter labels, opt-in.
--option-style letterasks the options as A, B, C and reads the letter tokens. On the public fixture it scores all 105 rows, where the default scores 39. The pinned Qwen2.5-1.5B landed at chance accuracy (0.343 against a chance null of 0.342) with an ECE of 0.349 against a floor of 0.087, which is distinguishable from noise: a small model that is confidently wrong. The default still reads each option's own token.The generative half is still open, and it needs a paid run.