feat(bench): parity capture and compare tooling, FP16 PE engine gate run (WP-0.1)

[davidamacey]

Closes #208

What

scripts/bench/parity.py with two subcommands, the accuracy gate for every v0.6.0 change (WP-0.1):

• capture ingests the pinned 2,000-image COCO set into a fresh project through the public routes, trains clusters once, and writes capture.json + a float16 arrays.npz (detector boxes/classes/scores, integer crop rectangles, PE crop and whole-frame vectors, cluster id/distance/similarity/core, stored class, COCO ground-truth class, optional VLM labels, MobileCLIP / face / OCR probes). capture.json carries hardware, git sha and dirty flag, container image ids, Triton model configs, the engine record (label, free KEY=VALUE pairs, plan size and sha256), the manifest hash and the sha256 of the NPZ.
• compare pairs two captures by image, box overlap and class name (never by id), measures the gate metrics (box matching, cosine min/mean/p01/median, kNN overlap and near-duplicate decisions, ARI/NMI, nearest-reference-centroid agreement, k-means objective, purity against COCO classes, kNN class precision, label vote, OCR exact match) and applies scripts/bench/parity_gates.json (values from the plan). Exit 0 PASS, 1 FAIL (SKIP fails only with --strict), 2 unusable input. --out writes a small summary JSON and markdown.

Modules: parity_metrics.py (pure maths), parity_store.py (format), parity_assemble.py, parity_gates.py, parity_capture.py (HTTP), parity_env.py, parity.py (CLI). Every file is under 700 lines.

Tests

tests/bench/test_parity_*.py: 87 offline tests on synthetic boxes, vectors, labellings, strings and a fake stack (httpx.MockTransport), no GPU or Docker. Written failing-first (import error before the module). Mutation check: 18 injected mutants (flipped IoU comparison, dropped class-name check, cosine without norm, kNN keeping self, purity min, ARI without expectation, gate operator flipped, skipped sha256 and union checks, ...); 2 survived the first suite and were killed by added tests, none survive.

FP16 PE engine against FP32 (live, isolated compose project, GPU 0, published v0.5.0 images)

FP16 passes every gate in two independent runs (4 and 1 uploads in flight); P-vlm is skipped (no VLM configured). The same-plan FP32 repeat is the noise floor.

Gate	FP16 run 1	FP16 run 2	FP32 repeat 1	FP32 repeat 2
P-box identical images (>= 0.98)	0.9934	0.9934	0.9924	0.9939
unmatched boxes >= 0.5 (<= 0)	0	0	1	0
P-pe crop cosine p01 / median (>= 0.995 / 0.999)	0.9984 / 0.9999	0.9990 / 0.9999	1.0000 / 1.0000	1.0000 / 1.0000
P-pew frame cosine p01 / median	0.9975 / 0.9998	0.9975 / 0.9998	1.0000 / 1.0000	1.0000 / 1.0000
kNN overlap@10 (>= 0.95)	0.9681	0.9706	0.9978	0.9974
cluster centroid agreement (>= 0.98)	0.9977	0.9977	0.9996	0.9996
cluster purity drop, points (<= 0.5)	-0.71	-0.45	-0.40	-0.16
label vote agreement (>= 0.97)	0.979	0.982	0.999	1.000
ARI (recorded)	0.341	0.332	0.351	0.334

Full tables: docs/PERFORMANCE.md ("Wave 0 parity and FP16 gates"), summaries in docs/benchmarks/parity/.

Findings worth knowing

• The detector is not bit-reproducible between identical runs (about 1 % of crop rectangles differ, one unmatched confident box in one FP32 repeat), so the crop-rectangle identity check is recorded, not gated, and the reference should be captured twice.
• The curation API never serves embedding vectors; PE vectors are read by id from the project's OpenSearch indexes. Cluster centroids are likewise derived from member vectors.
• Cluster capture needs the auto-label worker (compose profile curation).

Checks

pre-commit run --from-ref origin/main --to-ref HEAD exit 0, check_docs_vs_code.py clean, pytest tests/bench tests/curation/test_script_api_prefix.py tests/test_no_model_name_literals.py 156 passed.