professor3333 · Nov 15, 2026 · Nov 15, 2026
diff --git a/README.md b/README.md
@@ -162,10 +162,33 @@ and none was used for training.
 | 1 · Feasibility | Pinned ingestion, corpus statistics, label schema, offset-preserving preprocessing, pilot sample, annotation tooling | ✅ built |
 | 1 · Pilot | 100-document pilot (`pilot-2`: 22 hand-labelled, 78 from model drafts), [decision gate](docs/pilot_gate.md), guidelines v0.2 frozen, [acceptance targets](docs/targets.md) declared | ✅ done |
 | 2 · Baselines | Rules baseline, CRF + logistic-regression attributes and groups, evaluator with bootstrap CIs, evidence viewer | ✅ built · preliminary pilot dev scores |
-| 3 · Neural models | Frozen leakage-safe splits, annotation batches (01–04, 06, 07: model-drafted), DistilBERT and DeBERTa-v3 with span / role / group heads, experiment runner, CRF ablations and learning curve | 🟨 trained on model drafts (run 3) · DeBERTa ≈ CRF on held-out agreement, no clear winner |
+| 3 · Neural models | Frozen leakage-safe splits, annotation batches (01–04, 06, 07: model-drafted), DistilBERT and DeBERTa-v3 with span / role / group heads, experiment runner, CRF ablations and learning curve | 🟨 trained on model drafts (run 3) · dev: DeBERTa ≈ CRF, no clear winner · test: CRF ahead |
 | 4 · Reliability | Calibration and review policy, formatting robustness (rules: 100%), challenge-set and error-catalogue tooling | 🟨 tooling ready · thresholds await labels |
 | 5 · Application | FastAPI API, review UI, SQLite (WAL) with append-only history, exports, Docker Compose stack, release manifests, tested rollback, CI image smoke test | ✅ built |
 | 6 · Production | Benchmark harness, ONNX/INT8 export (FP32 parity exact; INT8 fails the accuracy gate on the trained DeBERTa), local proxy benchmarks and load tests, monitoring; CRF v0.1.0 measured and deployed (`app verify` passed) on the 2 vCPU / 4 GB target host | 🟨 CRF meets every serving target on the target host · transformers not yet measured there |
+| 7 · Validation | Declared held-out test evaluation, fresh-clone reproduction (macOS and Linux), [public demo](https://professor3333.github.io/TriggerMap/), downloadable models | 🟨 test evaluated once (agreement; gates not judgeable with 4 hand-labelled test records) · no prospective batch or usability study |
+
+### Held-out test results (declared release evaluation, 2026-11-15)
+
+Run once, on the owner's declaration, on the locked test split (newest project groups, so it
+measures temporal and project shift). It is logged in [`docs/test-access-log.md`](docs/test-access-log.md).
+The 146 model-drafted test records measure **agreement with the drafting model, not accuracy**.
+95% project-group bootstrap intervals.
+
+| Test split, 146 model-drafted records | Typed-condition micro-F1 | Exact span precision | Exact span recall | False-prerequisite rate | Reversed-meaning rate |
+|---|---:|---:|---:|---:|---:|
+| Rules (`rules-2`) | 0.048 [0.021, 0.082] | 0.101 | 0.053 | 0.646 | 0.043 |
+| CRF (`crf-batches-01-02`, default) | **0.193 [0.154, 0.238]** | **0.263** | 0.219 | **0.482** | 0.037 |
+| DeBERTa-v3-small (run 3) | 0.154 [0.125, 0.186] | 0.114 | **0.393** | 0.650 | **0.005** |
+
+The CRF agrees with the drafts better than DeBERTa (−0.039 typed F1 for DeBERTa, 95% CI
+[−0.080, −0.000], clear but only just). It is clearly more precise and invents clearly fewer
+prerequisites. DeBERTa has clearly higher recall and clearly fewer reversed meanings. It predicts
+8.4 conditions per document where the test drafts mark 2.4. Its dev lead did not transfer. On the
+4 hand-labelled test records (7 conditions), typed F1 is 0.125 (CRF), 0.133 (DeBERTa) and 0.167
+(rules), and no difference is clear. The release gates in [`docs/targets.md`](docs/targets.md)
+are judged on hand-labelled test records only, so they **cannot be judged** with 4. Full report,
+with per-class support: [`reports/test-evaluation/`](reports/test-evaluation/README.md).
 
 ### Preliminary pilot results (dev split only)
 

diff --git a/annotations/README.md b/annotations/README.md
@@ -78,7 +78,8 @@ against these batches are reported as agreement with the drafting model, not acc
 were also drawn for hand labelling and were submitted from model drafts on 2026-11-09, by the
 owner's decision: every record is `model_draft_accepted` (`model-batch-07-guidelines-0.2`,
 `model-batch-04-guidelines-0.2`), with no annotation time recorded. The drafting process read
-the batch-04 test texts; no model was developed, tuned or scored on them. CI checks batch-04
+the batch-04 test texts; no model was developed or tuned on them. They were scored once, in the
+declared release evaluation of 2026-11-15 (`reports/test-evaluation/`). CI checks batch-04
 records for provenance only.
 
 `second-drafts/batch-07.jsonl` (annotation version `batch-07-draft-2`) redrafts the same

diff --git a/docs/dataset_card.md b/docs/dataset_card.md
@@ -131,7 +131,9 @@ Annotated so far, by split:
 | `batch-04` | 0 | 0 | 138 | 0 |
 
 Test records are never read for development. The batch-04 test texts were read once, to draft
-their model labels; nothing was tuned or scored on them, and no test evaluation has been run ([`test-access-log.md`](test-access-log.md) has no entries).
+their model labels; nothing was tuned on them. They were scored once, in the declared release
+evaluation of 2026-11-15 ([`reports/test-evaluation/`](../reports/test-evaluation/README.md),
+logged in [`test-access-log.md`](test-access-log.md)).
 
 ## Sampling
 

diff --git a/docs/evaluation.md b/docs/evaluation.md
@@ -68,7 +68,7 @@ gated.
 |---|---|---|
 | train | training | Never scored as a result. |
 | dev | model selection, thresholds, calibration (on a held-out half) | Every reported number so far is dev. |
-| test | final release evaluation only | Locked. `--split test` refuses to run without `--release-evaluation "<reason>"`, and every such run is appended to [`test-access-log.md`](test-access-log.md) (no entries so far). |
+| test | final release evaluation only | Locked. `--split test` refuses to run without `--release-evaluation "<reason>"`, and every such run is appended to [`test-access-log.md`](test-access-log.md). Used once, in the declared release evaluation of 2026-11-15 ([`reports/test-evaluation/`](../reports/test-evaluation/README.md)). |
 | challenge set, prospective batch | reported separately | Never mixed into test numbers. |
 
 **Gold origin.** Each annotation records how its labels were made (`origin`, see
@@ -85,7 +85,10 @@ on (`training_origins` in `metadata.json`).
 **Current status (2026-11-09).** Only 22 records are `manual` (10 in dev, 4 in test). Every
 batch after the pilot, including the test batch `batch-04`, is model-drafted, by the owner's decision, so new results are reported as
 agreement with the drafting model, labelled as such, and the release gates in
-[`targets.md`](targets.md) cannot be judged as declared.
+[`targets.md`](targets.md) cannot be judged as declared. The owner has confirmed that no hand
+labels will come. The test split was evaluated once on 2026-11-15, on the owner's declaration:
+agreement on 146 model-drafted test records, and accuracy on the 4 hand-labelled ones
+([`reports/test-evaluation/`](../reports/test-evaluation/README.md)).
 
 The model drafts are not stable reference labels. Two model drafting runs of the 100 `batch-07`
 documents agree at 0.375 typed F1 [0.309, 0.444] and 0.431 exact span F1

diff --git a/docs/model_card.md b/docs/model_card.md
@@ -83,6 +83,23 @@ model drafts. `crf-batches-01-02` adds `batch-02` (239 records: 7 `manual`, 232
 Its dev results, as agreement with the model drafts and as accuracy on the 10 hand-labelled
 documents, are in [`reports/agreement/`](../reports/agreement/README.md).
 
+## Held-out test results (declared release evaluation, 2026-11-15)
+
+Run once on the locked test split, which holds the newest project groups (temporal and
+project shift). Agreement with 146 model-drafted test records, not accuracy:
+
+| System | Typed F1 | Span precision | Span recall | False-prerequisite rate | Reversed-meaning rate |
+|---|---:|---:|---:|---:|---:|
+| CRF `crf-batches-01-02` (default) | 0.193 [0.154, 0.238] | 0.263 | 0.219 | 0.482 | 0.037 |
+| DeBERTa-v3-small run 3 | 0.154 [0.125, 0.186] | 0.114 | 0.393 | 0.650 | 0.005 |
+| `rules-2` | 0.048 [0.021, 0.082] | 0.101 | 0.053 | 0.646 | 0.043 |
+
+DeBERTa − CRF: typed F1 −0.039 [−0.080, −0.000], clear but only just. The CRF is clearly more
+precise and invents clearly fewer prerequisites; DeBERTa has clearly higher recall and fewer
+reversed meanings. On the 4 hand-labelled test records (7 conditions), no difference is clear,
+and the release gates cannot be judged. Per-class support and full tables:
+[`reports/test-evaluation/`](../reports/test-evaluation/README.md).
+
 ## Preliminary dev results
 
 These are measured on the **10 hand-labelled dev documents** (17 gold conditions). The intervals

diff --git a/docs/targets.md b/docs/targets.md
@@ -99,3 +99,4 @@ about 512 tokens, ≤ 8 s at about 2,000 tokens, peak memory ≤ 3 GB, throughpu
 | 2026-11-02 | First declaration, after the pilot and before any test results. |
 | 2026-11-05 | Status note added: labels after the pilot are model drafts, so the gates cannot be judged as declared. No target changed; the test-access log still has no entries. |
 | 2026-11-09 | batch-04 (138 test records) and batch-07 (100 dev records) submitted from model drafts by the owner's decision; hand-labelled test records remain 4. No target changed; the test-access log still has no entries. |
+| 2026-11-15 | Declared release evaluation on the test split (owner: "Final held-out evaluation of v0.1.0 (CRF) and the run-3 DeBERTa candidate; no hand labels will come"). With 4 hand-labelled test records the gates are reported as not judgeable; results in `reports/test-evaluation/`. No target changed, and none may change for these releases. |
diff --git a/docs/test-access-log.md b/docs/test-access-log.md
@@ -4,3 +4,13 @@ Every evaluation on the locked test split, appended automatically by `triggermap
 
 | UTC time | Models | Gold | Reason |
 |---|---|---|---|
+| 2026-11-15 02:58 | rules-2 | `annotations/batches/batch-04.jsonl`, `annotations/pilot/pilot-2.jsonl` | Final held-out evaluation of v0.1.0 (CRF) and the run-3 DeBERTa candidate; no hand labels will come. |
+| 2026-11-15 02:58 | rules-2 | `annotations/batches/batch-04.jsonl`, `annotations/pilot/pilot-2.jsonl` | Final held-out evaluation of v0.1.0 (CRF) and the run-3 DeBERTa candidate; no hand labels will come. |
+| 2026-11-15 02:58 | crf-batches-01-02 | `annotations/batches/batch-04.jsonl`, `annotations/pilot/pilot-2.jsonl` | Final held-out evaluation of v0.1.0 (CRF) and the run-3 DeBERTa candidate; no hand labels will come. |
+| 2026-11-15 02:58 | crf-batches-01-02 | `annotations/batches/batch-04.jsonl`, `annotations/pilot/pilot-2.jsonl` | Final held-out evaluation of v0.1.0 (CRF) and the run-3 DeBERTa candidate; no hand labels will come. |
+| 2026-11-15 02:58 | deberta-3-base-bio_weightingsqrt-learning_rate5e-05-seed2 | `annotations/batches/batch-04.jsonl`, `annotations/pilot/pilot-2.jsonl` | Final held-out evaluation of v0.1.0 (CRF) and the run-3 DeBERTa candidate; no hand labels will come. |
+| 2026-11-15 02:58 | deberta-3-base-bio_weightingsqrt-learning_rate5e-05-seed2 | `annotations/batches/batch-04.jsonl`, `annotations/pilot/pilot-2.jsonl` | Final held-out evaluation of v0.1.0 (CRF) and the run-3 DeBERTa candidate; no hand labels will come. |
+| 2026-11-15 02:58 | crf-batches-01-02, deberta-3-base-bio_weightingsqrt-learning_rate5e-05-seed2 | `annotations/batches/batch-04.jsonl`, `annotations/pilot/pilot-2.jsonl` | Final held-out evaluation of v0.1.0 (CRF) and the run-3 DeBERTa candidate; no hand labels will come. |
+| 2026-11-15 02:58 | crf-batches-01-02, deberta-3-base-bio_weightingsqrt-learning_rate5e-05-seed2 | `annotations/batches/batch-04.jsonl`, `annotations/pilot/pilot-2.jsonl` | Final held-out evaluation of v0.1.0 (CRF) and the run-3 DeBERTa candidate; no hand labels will come. |
+| 2026-11-15 02:58 | crf-batches-01-02, rules-2 | `annotations/batches/batch-04.jsonl`, `annotations/pilot/pilot-2.jsonl` | Final held-out evaluation of v0.1.0 (CRF) and the run-3 DeBERTa candidate; no hand labels will come. |
+| 2026-11-15 02:59 | crf-batches-01-02, rules-2 | `annotations/batches/batch-04.jsonl`, `annotations/pilot/pilot-2.jsonl` | Final held-out evaluation of v0.1.0 (CRF) and the run-3 DeBERTa candidate; no hand labels will come. |
diff --git a/reports/test-evaluation/README.md b/reports/test-evaluation/README.md
@@ -0,0 +1,125 @@
+# Held-out test evaluation (declared release evaluation, 2026-11-15)
+
+This is the one declared evaluation on the locked test split of `splits/pool-v1.json`. The owner
+declared it on 2026-11-15 with the reason: *"Final held-out evaluation of v0.1.0 (CRF) and the
+run-3 DeBERTa candidate; no hand labels will come."* Every scoring run is logged in
+[`docs/test-access-log.md`](../../docs/test-access-log.md). Nothing was tuned on these results,
+and no system changes because of them. The test split holds the newest project groups
+(published 2024-04 → 2026-10), so the results measure **temporal and project shift**.
+
+## Systems (frozen)
+
+| System | Source | Conditions predicted (146 docs) |
+|---|---|---:|
+| `rules-2` | deterministic rules | 188 (1.3 per document) |
+| CRF `crf-batches-01-02` (default) | release [v0.1.0](https://github.com/professor3333/TriggerMap/releases/tag/v0.1.0) | 297 (2.0 per document) |
+| DeBERTa-v3-small `deberta-3-…sqrt…5e-05-seed2` | pre-release [model-deberta-3-run3](https://github.com/professor3333/TriggerMap/releases/tag/model-deberta-3-run3), downloaded and checked against its SHA-256 | 1,227 (8.4 per document) |
+
+Before scoring, each system's predictions on the 88 pilot documents outside the test split
+were confirmed identical to its committed dev predictions, so these are the same frozen
+systems that the dev reports describe.
+
+## Test data: two slices
+
+| Slice | Documents | Gold conditions | What it measures |
+|---|---:|---:|---|
+| Model-drafted test records (138 `batch-04` + 8 `pilot-2`) | 146 (139 with conditions, 7 without) | 356 | **Agreement with the drafting model, not accuracy** |
+| Hand-labelled test records (`pilot-2`, `origin: manual`) | 4 (3 with conditions, 1 without) | 7 | Accuracy, but far too few documents to conclude anything |
+
+The `batch-04` labels were drafted by the same session and process as the first batch-07
+draft (2026-11-09). That session read these test texts in order to draft them. No model was
+developed, tuned or scored on them before this evaluation.
+
+## Results: agreement, 146 model-drafted test records
+
+95% project-group bootstrap intervals (1,000 resamples, seed 0).
+
+| Metric | `rules-2` | CRF (default) | DeBERTa run 3 |
+|---|---:|---:|---:|
+| Typed-condition micro-F1 | 0.048 [0.021, 0.082] | **0.193 [0.154, 0.238]** | 0.154 [0.125, 0.186] |
+| Exact span F1 | 0.070 [0.035, 0.110] | **0.239 [0.194, 0.288]** | 0.177 [0.146, 0.207] |
+| Exact span precision | 0.101 [0.051, 0.168] | **0.263 [0.207, 0.335]** | 0.114 [0.092, 0.137] |
+| Exact span recall | 0.053 [0.027, 0.081] | 0.219 [0.176, 0.266] | **0.393 [0.337, 0.444]** |
+| Role macro-F1 | 0.017 [0.008, 0.028] | 0.132 [0.093, 0.179] | 0.125 [0.087, 0.171] |
+| Group exact-match F1 | 0.000 | 0.000 | 0.000 |
+| Document exact match, with conditions (139) | 0.000 | 0.058 [0.016, 0.101] | 0.043 [0.014, 0.075] |
+| Document exact match, without conditions (7) | 0.857 [0.500, 1.000] | 0.857 [0.500, 1.000] | 0.571 [0.167, 1.000] |
+| False-prerequisite rate (lower is better) | 0.646 [0.500, 0.762] | **0.482 [0.400, 0.556]** | 0.650 [0.605, 0.691] |
+| Reversed-meaning rate (lower is better) | 0.043 [0.000, 0.102] | 0.037 [0.008, 0.073] | **0.005 [0.000, 0.014]** |
+
+Per-role exact span + role F1, with gold support:
+
+| Role (support) | CRF | DeBERTa run 3 |
+|---|---:|---:|
+| REQUIRED (299) | 0.240 | 0.202 |
+| EXCEPTION (11) | 0.000 | 0.133 |
+| NOT_REQUIRED (16) | 0.209 | 0.114 |
+| UNCERTAIN (30) | 0.080 | 0.053 |
+
+Per-category exact span + category F1, with gold support:
+
+| Category (support) | CRF | DeBERTa run 3 |
+|---|---:|---:|
+| CONFIGURATION (57) | 0.188 | 0.188 |
+| USAGE (166) | 0.203 | 0.149 |
+| ACCESS (91) | 0.247 | 0.177 |
+| PLATFORM (42) | 0.111 | 0.119 |
+
+Paired differences (system minus CRF):
+
+| Metric | DeBERTa run 3 − CRF | `rules-2` − CRF |
+|---|---:|---:|
+| Typed-condition micro-F1 | −0.039 [−0.080, −0.000] clearly worse (only just) | −0.145 [−0.190, −0.105] clearly worse |
+| Exact span F1 | −0.062 [−0.106, −0.019] clearly worse | −0.169 [−0.221, −0.127] clearly worse |
+| Exact span precision | −0.149 [−0.208, −0.101] clearly worse | −0.162 [−0.225, −0.111] clearly worse |
+| Exact span recall | +0.174 [+0.119, +0.228] clearly better | −0.166 [−0.214, −0.125] clearly worse |
+| False-prerequisite rate | +0.168 [+0.106, +0.239] clearly worse | +0.164 [+0.061, +0.268] clearly worse |
+| Reversed-meaning rate | −0.032 [−0.067, −0.002] clearly better | +0.006 [−0.046, +0.066] not clear |
+
+## Results: accuracy, 4 hand-labelled test records
+
+| Metric | `rules-2` | CRF (default) | DeBERTa run 3 |
+|---|---:|---:|---:|
+| Typed-condition micro-F1 | 0.167 [0.000, 0.286] | 0.125 [0.000, 0.182] | 0.133 [0.000, 0.235] |
+| Exact span F1 | 0.167 [0.000, 0.286] | 0.125 [0.000, 0.182] | 0.200 [0.000, 0.545] |
+| False-prerequisite rate | 0.500 [0.000, 1.000] | 0.667 [0.000, 0.750] | 0.684 [0.500, 0.889] |
+
+No paired difference on this slice is clear. Seven gold conditions in four documents cannot
+separate these systems.
+
+## Release targets
+
+[`docs/targets.md`](../../docs/targets.md) judges the release gates on hand-labelled test
+records only. With 4 such records, **the gates cannot be judged as declared**, and they are
+reported as not judgeable. The targets were not changed after the fact.
+
+## Reading
+
+- **The CRF remains the right default.** On the 146 test records it agrees with the drafts
+  better than DeBERTa (typed F1 0.193 against 0.154; the difference is clear but only just).
+  It is far more precise (0.263 against 0.114) and invents clearly fewer prerequisites (0.482
+  against 0.650). The pre-set rule kept the CRF before any test result was seen, and the test
+  result supports that choice.
+- **DeBERTa's dev lead did not transfer to the test drafts.** On dev it led the CRF without a
+  clear difference (held-out typed F1 0.185 against 0.127). On test it trails. It predicts 8.4
+  conditions per document where the test drafts mark 2.4, so its high recall (0.393) costs
+  precision. This matches batch-07, where DeBERTa's lead was clear only against the denser
+  second draft. Against drafts made the way `batch-04` was made, it had no clear lead there
+  either.
+- **DeBERTa is better at one of the two errors this project weighs most:** reversed meaning
+  (0.005 against 0.037, clear). The CRF is better at the other: invented prerequisites.
+- **Both systems remain weak in absolute terms.** No group is ever matched exactly, the CRF
+  finds no EXCEPTION condition, and fewer than 6% of condition-bearing documents come out
+  fully right. These are agreement scores against one drafting model on newer projects. They
+  are not accuracy, and the 4 hand-labelled documents cannot settle accuracy.
+
+## Reproduce
+
+```bash
+# predictions (frozen systems; DeBERTa from the model-deberta-3-run3 asset)
+uv run triggermap predict --sample annotations/batches/batch-04.json annotations/pilot/pilot_manifest.json --out reports/test-evaluation/predictions/rules-2.jsonl
+uv run triggermap predict --model crf --model-dir artifacts/crf-batches-01-02 --sample annotations/batches/batch-04.json annotations/pilot/pilot_manifest.json --out reports/test-evaluation/predictions/crf-batches-01-02.jsonl
+uv run triggermap predict --model transformer --model-dir deberta-3-base-bio_weightingsqrt-learning_rate5e-05-seed2 --sample annotations/batches/batch-04.json annotations/pilot/pilot_manifest.json --out reports/test-evaluation/predictions/deberta-3-base-bio_weightingsqrt-learning_rate5e-05-seed2.jsonl
+# scoring: a declared release evaluation only (it appends to docs/test-access-log.md)
+bash scripts/experiments/test_evaluation.sh "<declared reason>"
+```