diff --git a/README.md b/README.md index 0843e37..57ccdab 100644 --- a/README.md +++ b/README.md @@ -157,6 +157,8 @@ and none was used for training. ## Project status +Final audit against the Definition of Done, with evidence for each item: [`docs/definition_of_done.md`](docs/definition_of_done.md). + | Phase | Scope | Status | |---|---|:---:| | 1 ยท Feasibility | Pinned ingestion, corpus statistics, label schema, offset-preserving preprocessing, pilot sample, annotation tooling | โœ… built | diff --git a/docs/definition_of_done.md b/docs/definition_of_done.md new file mode 100644 index 0000000..d01d5bf --- /dev/null +++ b/docs/definition_of_done.md @@ -0,0 +1,32 @@ +# Definition of Done: final audit (2026-11-15, v0.2.0) + +Each item of the project's Definition of Done, with its status and the evidence for it. +โœ… done ยท ๐ŸŸจ done within a stated limit ยท โ›” out of scope (the reason is given). + +| # | Item | Status | Evidence | +|---|---|:---:|---| +| 1 | A pinned public-data snapshot can be reconstructed | โœ… | Snapshot commit and output SHA-256 in [`data/manifests/snapshot.json`](../data/manifests/snapshot.json). The fresh-clone check rebuilds it byte for byte on macOS arm64 and Linux x86-64 ([`reproducibility.md`](reproducibility.md)). | +| 2 | Dataset licensing, attribution, filtering and counts are documented | โœ… | [`dataset_card.md`](dataset_card.md) (36,910 scanned โ†’ 6,350 PyPI kept, withdrawn and quarantine counts), [`corpus_stats.md`](corpus_stats.md), CC BY 4.0 layering in [`../annotations/README.md`](../annotations/README.md). | +| 3 | Annotation guidelines and quality checks are published | โœ… | [`annotation_guidelines.md`](annotation_guidelines.md) (v0.2 frozen). Deterministic validator and CI checks over every committed record ([`../tests/test_committed_annotations.py`](../tests/test_committed_annotations.py)). Import reports with origin and drift, and drafter consistency ([`../reports/drafter-consistency/`](../reports/drafter-consistency/README.md)). | +| 4 | โ‰ˆ 800 advisories labelled, with actual counts and exclusions reported | ๐ŸŸจ | 710 advisories annotated, plus a second draft of 100. Only **22 are hand-labelled**; the rest are model drafts, by the owner's decision, and no more hand labels will come. Counts by origin and split are in [`dataset_card.md`](dataset_card.md). | +| 5 | Duplicate-, alias- and project-aware split manifests are reproducible | โœ… | [`../splits/pool-v1.json`](../splits/pool-v1.json) (frozen, with SHA-256), [`splits.md`](splits.md), and leakage and assignment tests in [`../tests/test_splits.py`](../tests/test_splits.py). | +| 6 | Rules, traditional supervised and transformer experiments run from scripts | โœ… | `triggermap predict / train / experiment`, [`../configs/experiments/`](../configs/experiments), [`../scripts/gpu/run_neural.sh`](../scripts/gpu/run_neural.sh), `scripts/experiments/*.sh`. | +| 7 | Fine-tuned weights, tokenizer, labels, calibration and provenance are downloadable | โœ… | CRF in releases v0.1.0 and v0.2.0; run-3 DeBERTa in the pre-release `model-deberta-3-run3`. Checksums in the [model card](model_card.md#downloads). | +| 8 | Held-out results include per-class support, uncertainty and difficult slices | ๐ŸŸจ | Declared test evaluation, run once ([`../reports/test-evaluation/`](../reports/test-evaluation/README.md)): per-role and per-category support, project-group bootstrap intervals, and paired comparisons. It measures **agreement** on 146 model-drafted records; accuracy rests on only 4 hand-labelled records, so the release gates are not judgeable. | +| 9 | Every predicted condition is linked to source evidence | โœ… | Every condition is an exact `[start, end)` span of the canonical text, with an offset map back to the raw text. Every system ends in the deterministic validator ([`model_card.md`](model_card.md)). The API checks each extraction again and returns `invalid_prediction` instead of showing or caching an item without valid evidence (`src/triggermap/serve/app.py`; test `test_invalid_prediction_is_rejected_not_shown`). | +| 10 | Ambiguous structures and unsupported inputs produce visible review states | โœ… | [`../configs/review_policy.yaml`](../configs/review_policy.yaml) routes uncertain, unresolved and unsupported output, and low-confidence items, to *Review required*. "No explicit conditions found" is shown as distinct from "not affected" (the demo's abstention page). | +| 11 | A user can import an advisory, inspect predictions, correct them, record answers and export | โœ… | The review app and API ([`deployment.md`](deployment.md)). `app verify` checks extraction โ†’ review โ†’ export end to end. | +| 12 | Authenticated saved reviews survive a service restart | โœ… | `test_reviews_survive_restart_and_detect_advisory_change` in `tests/test_app.py`. Reviews kept across the rollback drills. | +| 13 | The API and interface run locally through Docker and on a hosted CPU deployment | โœ… | Local Docker Compose. Hosted on a 2 vCPU / 4 GiB x86-64 AWS EC2 instance on 2026-11-09, with `app verify` passing ([`deployment.md`](deployment.md#hosted-deployment-record)). The instance was temporary, and there is no permanent server. | +| 14 | Load tests, optimization comparisons, monitoring and rollback are demonstrated | ๐ŸŸจ | CRF on the target host: p95 0.062 s / 0.145 s, 1,000 of 1,000 load-test requests ([`benchmarks.md`](benchmarks.md)). ONNX FP32 is exact; INT8 fails its gate on the trained DeBERTa ([`../reports/onnx-int8/`](../reports/onnx-int8/README.md)). Prometheus metrics, and a rollback drill v0.1.0 โ†’ v0.2.0 โ†’ v0.1.0. The transformers were not measured on the target host. | +| 15 | A fresh checkout reproduces evaluation within documented tolerances | โœ… | [`reproducibility.md`](reproducibility.md): byte-identical corpus, rules, retrained CRF and reports, on macOS arm64 (208 s) and Linux x86-64 (245 s). | +| 16 | The public demo shows a successful extraction, a difficult linguistic case and a failure or abstention | โœ… | : aiohttp, Flask and Django pages, built by `scripts/demo/build_site.sh`. | +| 17 | Resume claims contain measured results only | โœ… | Kept outside the repository. Every figure is taken from the reports linked here, with its caveat (agreement vs accuracy). | + +## Roadmap items outside the checklist + +| Item | Status | Evidence | +|---|:---:|---| +| Prospective batch | ๐ŸŸจ | 259 advisories published after the snapshot, behaviour only, with no labels ([`../reports/prospective/`](../reports/prospective/README.md)). | +| Constructed challenge set | โ›” | Its labels would need human adjudication. Writing them without it would fabricate gold labels, so the set stays empty (`challenge/`). | +| Preliminary usability study | โ›” | No developer participants are available, now or later. Walkthroughs by AI models are not reported as usability evidence (README, *Limitations*). | diff --git a/tests/test_app.py b/tests/test_app.py index 015f255..7e848e7 100644 --- a/tests/test_app.py +++ b/tests/test_app.py @@ -302,3 +302,45 @@ def test_app_cli_uses_the_server_database(tmp_path: Path, monkeypatch: pytest.Mo monkeypatch.setenv("TRIGGERMAP_DATABASE", str(target)) args = build_parser().parse_args(["app", "create-user", "someone"]) assert args.database == target + + +def test_invalid_prediction_is_rejected_not_shown(db: Path) -> None: + """A model output with a span outside the text is refused (500), never shown or cached.""" + from triggermap.schema import Category, Condition, DocumentStatus, Role + + settings = Settings(database=db, review_policy=ROOT / "configs/review_policy.yaml") + + class BrokenExtractor: + def __init__(self, real: Any) -> None: + self.real = real + + def __getattr__(self, name: str) -> Any: + return getattr(self.real, name) + + def extract(self, doc: Any, advisory_id: str | None = None) -> Any: + extraction = self.real.extract(doc, advisory_id) + outside = Condition( + id="c1", + span=(0, len(doc.text) + 5), + category=Category.USAGE, + role=Role.REQUIRED, + source="model", + ) + return extraction.model_copy( + update={ + "conditions": (outside,), + "groups": (), + "document_status": DocumentStatus.HAS_CONDITIONS, + } + ) + + def broken_service() -> Service: + service = Service(settings) + service.extractor = BrokenExtractor(service.extractor) # type: ignore[assignment] + return service + + with TestClient(create_app(settings, broken_service)) as c: + for _ in range(2): # the refused output is not cached either + response = c.post("/v1/extractions", json={"advisory_id": FLASK}) + assert response.status_code == 500 + assert response.json()["error"]["code"] == "invalid_prediction"