From 2461b2be5a17e24c96451157e6933ff5eaebe83c Mon Sep 17 00:00:00 2001 From: Niroj33 <86530493+professor3333@users.noreply.github.com> Date: Sun, 15 Nov 2026 09:40:55 +0545 Subject: [PATCH] Add a Limitations section to the README States the measured limits in one place: model-drafted labels (22 hand-labelled records, no more coming), unstable references (two drafting runs agree at 0.375 typed F1), a single annotator, residual leakage risk, the narrow scope, temporal and project shift on test, weak rare structure, unmodelled cross-window logic, INT8 failing for DeBERTa, and that no usability study will be run because no developer participants are available. --- README.md | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index b2d7a1e..0843e37 100644 --- a/README.md +++ b/README.md @@ -166,7 +166,7 @@ and none was used for training. | 4 · Reliability | Calibration and review policy, formatting robustness (rules: 100%), challenge-set and error-catalogue tooling | 🟨 tooling ready · thresholds await labels | | 5 · Application | FastAPI API, review UI, SQLite (WAL) with append-only history, exports, Docker Compose stack, release manifests, tested rollback, CI image smoke test | ✅ built | | 6 · Production | Benchmark harness, ONNX/INT8 export (FP32 parity exact; INT8 fails the accuracy gate on the trained DeBERTa), local proxy benchmarks and load tests, monitoring; CRF v0.1.0 measured and deployed (`app verify` passed) on the 2 vCPU / 4 GB target host | 🟨 CRF meets every serving target on the target host · transformers not yet measured there | -| 7 · Validation | Declared held-out test evaluation, fresh-clone reproduction (macOS and Linux), [public demo](https://professor3333.github.io/TriggerMap/), downloadable models | 🟨 test evaluated once (agreement; gates not judgeable with 4 hand-labelled test records) · prospective check on 259 new advisories (behaviour only, no labels) · no usability study | +| 7 · Validation | Declared held-out test evaluation, fresh-clone reproduction (macOS and Linux), [public demo](https://professor3333.github.io/TriggerMap/), downloadable models | 🟨 test evaluated once (agreement; gates not judgeable with 4 hand-labelled test records) · prospective check on 259 new advisories (behaviour only, no labels) · no usability study (no participants available) | ### Held-out test results (declared release evaluation, 2026-11-15) @@ -518,6 +518,36 @@ docs/ guidelines, pilot protocol, corpus statistics tests/ unit, property-based, synthetic end-to-end and real-export regression tests ``` +## Limitations + +- **Most labels are model drafts.** 22 records are hand-labelled (10 dev, 4 test). Every batch + after the pilot was submitted from model drafts, and no more hand labels will come. Most scores + are therefore agreement with a drafting model, not accuracy. The accuracy slices are tiny, and + the release gates in [`docs/targets.md`](docs/targets.md) cannot be judged. +- **The reference labels are unstable.** Two model drafting runs of the same 100 documents agree + at only 0.375 typed F1 ([`reports/drafter-consistency/`](reports/drafter-consistency/README.md)). +- **There is one annotator, and no inter-annotator agreement.** The 22 hand-labelled records come + from a single annotator, and no repeat annotation was done. +- **Independence is not guaranteed.** Advisories are copied across packages, aliases and revisions. + Grouping removes the duplicates it can detect (shared package, repository, alias, identical text, + MinHash near-duplicates), but some leakage risk remains. +- **The scope is narrow:** one source (GitHub-reviewed advisories), the PyPI ecosystem, and English + only. Nothing is claimed about other sources or ecosystems. +- **Writing style varies strongly by project,** and the test split holds the newest project + groups, so test results measure temporal and project shift. +- **Rare structure is weak.** Group exact match is 0 for every system, EXCEPTION and NOT_REQUIRED + conditions are rarely right, and `UNCERTAIN` and `UNRESOLVED` are unsupported by the pilot gate. +- **Nested and cross-window logic is marked unresolved, not modelled.** This matters more over + time: 71% of advisories published after the snapshot need more than one 512-token window + ([`reports/prospective/`](reports/prospective/README.md)). +- **Absence of an extracted condition is not evidence that none exists.** "No explicit conditions + found" is never "not affected". +- **Exported runtimes:** dynamic INT8 fails the accuracy gate for DeBERTa, and the transformers + have not been measured on the target host. +- **No usability study.** No developer participants are available, now or later, so no usability + study will be run. The interface is checked only by automated API and UI tests and the public + demo. Walkthroughs by AI models are not reported as usability evidence. + ## Data and licensing - **Source:** [GitHub Advisory Database](https://github.com/github/advisory-database), reviewed