feat(vlm): optional detector hint for VLM labeling; remove the registry prior (closes #61)

CHANGELOG.md

@@ -13,10 +13,15 @@ history of this codebase and was never published. This release is `[0.4.1]`.
 
 ### Added
 
-- Registry prior for VLM labeling (#61): a prompt pack's `registry_prior_top_k` (0 = off, max 50)
-  adds the top-k registry classes by validated count and the pending proposal names to the
-  open-vocabulary prompt as a hint. Served in the pack schema as `kind: "int"`; a new
-  `pack_field_out_of_range` validation code; the run refuses when the counts are unreadable.
+- Detector hint for VLM labeling (#61): a prompt pack's `detector_hint_min_confidence_pct`
+  (0 = off, max 100) tells the VLM each item's stored detector class name, per image, when the
+  detector confidence is at least that percent. Served in the pack schema as `kind: "int"`; a new
+  `pack_field_out_of_range` validation code. Measured on a public COCO oracle (539 crops, local
+  VLM) it raised overall accuracy from 0.775 to 0.853 (0.7846 to 0.8478 in an arm with ten
+  validated classes seeded) and accuracy when answered from 0.850 to 0.918. Caveat: the detector
+  is COCO-trained and the oracle is COCO ground truth, so the gain may be optimistic for other
+  domains. A registry-prior variant (top validated classes plus pending proposals in the prompt)
+  was measured, showed no gain (and about -1 point with seeded classes) and was removed.
 - Gateway mode (#93): `OP_GATEWAY_SUBPATHS=true` makes Cropwright's nginx serve
   Grafana, Prometheus, OpenSearch Dashboards and MLflow under `/grafana/`,
   `/prometheus/`, `/dashboards/` and `/mlflow/` of its one published port, so the UIs
@@ -36,6 +41,11 @@ history of this codebase and was never published. This release is `[0.4.1]`.
   `POST /pipeline/auto_label` never writes. Every residual-clustering run now records
   itself (`clustering/last_run.py`, called by `cluster_residuals` and
   `assign_only_residuals`) and the stats serve that record.
+- The shipped proposal-name denylist now also covers the scene and quality words a live COCO
+  oracle run produced (#193): `abstract*`, `*_background`, `*_object`, `scene*`, `empty*`,
+  `blank_*`, `*_image`, `shadow*`, `outdoors` and similar. The text-free generic pack and
+  `examples/prompt_packs/vehicle_wheel.json` carry it too, tested against the COCO class names so
+  no glob eats a real class.
 - Cropwright's nginx now answers redirects with relative `Location` headers; the
   `/OpenProcessor` redirect used to name the container port 8080.
 - Config-store writes (prompt packs, region profiles, open-vocabulary sets, VLM endpoints,

contracts/openapi/curation.json

@@ -15703,6 +15703,13 @@
             "title": "Combined User Template",
             "type": "string"
           },
+          "detector_hint_min_confidence_pct": {
+            "default": 0,
+            "maximum": 100.0,
+            "minimum": 0.0,
+            "title": "Detector Hint Min Confidence Pct",
+            "type": "integer"
+          },
           "open_class_system": {
             "default": "",
             "title": "Open Class System",
@@ -15750,13 +15757,6 @@
             "title": "Region Visible User",
             "type": "string"
           },
-          "registry_prior_top_k": {
-            "default": 0,
-            "maximum": 50.0,
-            "minimum": 0.0,
-            "title": "Registry Prior Top K",
-            "type": "integer"
-          },
           "synonyms": {
             "additionalProperties": {
               "type": "string"

docs-site/docs/guides/prompt-packs.mdx

@@ -34,14 +34,16 @@ code. Packs are editable per project, versioned, and can be rolled back.
   (`blurry_*`, `*_scene`) for proposed new-class names that are scene or image-quality
   words rather than classes. A matching proposal is dropped, so the crop counts as
   "nothing fits" and does not reach the new-class queue. It defaults to empty for a
-  stored pack that omits it; the shipped generic pack carries a starter list.
-
-  An optional integer, `registry_prior_top_k` (`0` = off, the default; at most 50), adds a
-  registry prior to open-vocabulary labeling: the prompt lists that many registry classes
-  (most validated items first, ties by name) and the pending new-class proposal names, built
-  from the project's own state. It is a hint, not a constraint: a reply outside the list is
-  still accepted. If the counts cannot be read the run refuses rather than labeling without
-  the prior.
+  stored pack that omits it; the shipped generic packs carry a starter list covering blur,
+  quality, empty and scene words plus `abstract*`, `*_background` and `*_object`.
+
+  An optional integer, `detector_hint_min_confidence_pct` (`0` = off, the default; at most
+  100), adds one line per image naming the item's stored detector class when the detector
+  confidence is at least that percent. It is a hint, not a constraint. On a public COCO oracle
+  (539 crops) it raised overall accuracy from 0.775 to 0.853 (0.7846 to 0.8478 with ten
+  validated classes seeded). The detector is COCO-trained and the oracle is COCO ground truth,
+  so the gain may be optimistic for other domains. A registry-prior variant was measured, gave
+  no gain and was removed.
 - **Placeholders.** Only three fields are formatted with `str.format`:
   `class_user_template` and `open_class_user_template` accept
   `class_names_csv`; `combined_user_template` accepts `class_block` and
@@ -114,9 +116,9 @@ named region profile (default: the active one). The response is
 |---|---|---|
 | `pack_name_invalid`, `pack_name_reserved` | error | Name must match `^[a-z0-9][a-z0-9_.-]{1,63}$`; `off`, `none` and `default` are reserved |
 | `name_conflict` | error | Name already used |
-| `pack_field_missing`, `pack_field_empty` | error | A text field is absent or blank; `proposal_denylist` is not a list of strings, or holds a blank pattern; `registry_prior_top_k` is not an integer |
+| `pack_field_missing`, `pack_field_empty` | error | A text field is absent or blank; `proposal_denylist` is not a list of strings, or holds a blank pattern; `detector_hint_min_confidence_pct` is not an integer |
 | `pack_field_too_long` | error | A field over 20000 characters, a map or `proposal_denylist` over 500 entries, or a denylist pattern over 200 characters |
-| `pack_field_out_of_range` | error | `registry_prior_top_k` is outside 0 to 50 |
+| `pack_field_out_of_range` | error | `detector_hint_min_confidence_pct` is outside 0 to 100 |
 | `pack_template_format_error` | error | The field cannot be formatted (for example a single brace) |
 | `pack_placeholder_missing`, `pack_placeholder_unknown` | error | A formatted field lacks a required placeholder or uses another |
 | `pack_placeholder_in_verbatim_field` | error | A placeholder in a field that is not formatted |

docs/CURATION.md

@@ -501,14 +501,17 @@ Browse and cluster at box level:
 A prompt pack holds the VLM prompt templates and two vocabulary tables
 (`class_descriptions`, `synonyms`) and an optional `proposal_denylist` of
 glob patterns (`blurry_*`, `*_scene`) whose matching new-class proposals are
-dropped. An optional `registry_prior_top_k` (integer, `0` = off, at most 50)
-turns on a registry prior for open-vocabulary VLM labeling: the prompt then lists
-that many registry classes, most validated items first (ties by name), plus the
-pending new-class proposal names, built server-side from the project's own state
-(never from a client list). It is a hint, not a constraint: a reply outside the
-list is still accepted and resolved as usual. If the counts cannot be read the
-run refuses (HTTP 503 on `vlm/label_batch`, a stage error in the auto-label
-pipeline) instead of labeling without the prior. It is separate from the region profile:
+dropped. The shipped default denylist covers blur, quality, empty and scene words plus
+`abstract*`, `*_background` and `*_object`. An optional
+`detector_hint_min_confidence_pct` (integer, `0` = off, at most 100) adds, per
+image, the item's stored detector class name to the open-vocabulary prompt when the
+detector confidence is at least that percent; it is a hint, not a constraint (a reply
+outside it is still accepted). Measured on a 539-crop public COCO oracle with a local
+VLM it raised overall accuracy from 0.775 to 0.853 (0.7846 to 0.8478 with ten validated
+classes seeded). The detector is COCO-trained and the oracle is COCO ground truth, so the
+gain may be optimistic for other domains. A registry-prior variant (top validated
+classes plus pending proposals in the prompt) was measured, gave no gain and was
+removed. It is separate from the region profile:
 a profile says where to look, a pack says what to ask. Either can be used
 without the other.
 

docs/design/README.md

@@ -33,7 +33,7 @@ Rows marked implemented are merged for v0.4.0; the rest are open.
 | #57, #58, #59, #60, #64, #66 | Implemented in the v0.4.0 hardening merge (regions re-verify safety, config-store hardening, test fakes and dev tooling, silent failures, docs, GPU clustering overlay); the issues stay open until the owner closes them |
 | #53, #55 | Implemented (shard budget `capacity` block, combine preview warnings) |
 | #46 | Implemented (region-stage hit-rate gate, pause and resume) |
-| #61 | VLM proposal noise, optional registry prior, label agreement vs cluster purity |
+| #61 | VLM proposal noise, optional detector hint, label agreement vs cluster purity |
 | #62 | Split oversize files, unify job state files, rename the `api` compose service |
 | #63 | Release decisions (`:latest`, control-plane-only) and remaining local VLM catalog verification |
 | #65 | Roadmap: deferred any-domain features |

examples/prompt_packs/vehicle_wheel.json

@@ -17,6 +17,38 @@
   "region_visible_user": "For each numbered crop, answer: is a wheel (tire plus rim) of the car visible anywhere in the image? Count partial, angled, or small wheels as visible; count fully hidden wheels as not visible. Respond as a JSON object whose ``results`` field is an array of per-image verdicts in input order:\n{\"results\": [{\"img\": 1, \"visible\": true|false}, ...]}",
   "class_descriptions": {},
   "synonyms": {},
-  "proposal_denylist": [],
-  "registry_prior_top_k": 0
+  "proposal_denylist": [
+    "abstract*",
+    "background*",
+    "*_background",
+    "object",
+    "objects",
+    "*_object",
+    "blurry*",
+    "blurred*",
+    "*_blurry",
+    "*_blur",
+    "blur_*",
+    "out_of_focus*",
+    "low_quality*",
+    "low_resolution*",
+    "unclear*",
+    "unidentified*",
+    "scene*",
+    "*_scene",
+    "empty*",
+    "unknown*",
+    "*_image",
+    "*_photo",
+    "*_abstract",
+    "blank",
+    "blank_*",
+    "indoor",
+    "indoors",
+    "outdoor",
+    "outdoors",
+    "shadow*",
+    "*_shadow"
+  ],
+  "detector_hint_min_confidence_pct": 0
 }

frontend/CHANGELOG.md

@@ -9,7 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
-- The served pack schema gains a `registry_prior_top_k` row with `kind: "int"` (the optional VLM registry prior, 0 = off). The pack editor shows an unknown kind read-only and saves it unchanged, so the value round-trips; `PackSchemaField.kind` lists `int`.
+- The served pack schema gains a `detector_hint_min_confidence_pct` row with `kind: "int"` (the optional per-item detector-name hint, 0 = off). The pack editor shows an unknown kind read-only and saves it unchanged, so the value round-trips; `PackSchemaField.kind` lists `int`.
 - Gateway mode: with `OP_GATEWAY_SUBPATHS=true` the entrypoint installs
   `nginx-gateway.conf`, which proxies `/grafana/`, `/prometheus/`, `/dashboards/` and
   `/mlflow/` to the compose services (bare service names only, overridable with

frontend/src/lib/test/fixtures/promptPacks.ts

@@ -101,8 +101,8 @@ export function schemaFixture(): PromptPackSchema {
         help: '',
       },
       {
-        field: 'registry_prior_top_k',
-        label: 'Registry prior top k',
+        field: 'detector_hint_min_confidence_pct',
+        label: 'Detector hint min confidence pct',
         group: 'open_classify',
         kind: 'int',
         formatted: false,

frontend/src/lib/types_packs.ts

@@ -21,7 +21,7 @@ import type {
  * A pack body: every `PromptPack` field except `name`, keyed by the
  * schema's `field` ids. `kind: "text"` fields are strings, `kind: "map"`
  * fields (`class_descriptions`, `synonyms`) are string maps, and
- * `kind: "list"` (`proposal_denylist`) is a list of case-insensitive glob strings, and `kind: "int"` (`registry_prior_top_k`, 0 = off) is shown read-only.
+ * `kind: "list"` (`proposal_denylist`) is a list of case-insensitive glob strings, and `kind: "int"` (`detector_hint_min_confidence_pct`, 0 = off) is shown read-only.
  * Kept as a record so a field the backend adds renders from the schema alone.
  */
 export type PackFieldValue = string | number | string[] | Record<string, string>;

scripts/curation/bakeoff/vlm_prior_oracle.py → scripts/curation/bakeoff/vlm_hint_oracle.py

@@ -1,14 +1,14 @@
 #!/usr/bin/env python3
-"""Score the VLM registry prior on/off against an oracle (#61 item 2).
+"""Score the VLM detector hint on/off against an oracle (#61 item 2).
 
 Two steps, run by hand around two labeling runs of the same crops (one with a
-pack whose ``registry_prior_top_k`` is 0, one with it set):
+pack whose ``detector_hint_min_confidence_pct`` is 0, one with it set):
 
     # after each run, dump (crop, oracle name, VLM answer) for that run's pack
-    .venv/bin/python scripts/curation/bakeoff/vlm_prior_oracle.py dump \\
+    .venv/bin/python scripts/curation/bakeoff/vlm_hint_oracle.py dump \\
         --project <slug> --pack <pack-name> --truth-field <field> --out off.jsonl
 
-    .venv/bin/python scripts/curation/bakeoff/vlm_prior_oracle.py compare \\
+    .venv/bin/python scripts/curation/bakeoff/vlm_hint_oracle.py compare \\
         --off off.jsonl --on on.jsonl
 
 ``dump`` reads items whose ``vlm_prompt_pack`` stamp starts with ``--pack`` and

src/routers/curation/_prompt_pack_models.py

@@ -11,7 +11,7 @@
     ActiveRef,
     ValidationReport,
 )
-from src.services.labeling.registry_prior import MAX_REGISTRY_PRIOR_TOP_K
+from src.services.labeling.detector_hint import MAX_DETECTOR_HINT_PCT
 from src.services.labeling.vlm_prompts import REPLY_KEY_CONTRACT
 
 
@@ -52,14 +52,14 @@ class PromptPackBody(BaseModel):
     ]
     # Typed ``Any`` for the same reason: a bool/str/out-of-range value must come
     # back as a validation issue. 0 = off.
-    registry_prior_top_k: Annotated[
+    detector_hint_min_confidence_pct: Annotated[
         Any,
         Field(
             default=0,
             json_schema_extra={
                 'type': 'integer',
                 'minimum': 0,
-                'maximum': MAX_REGISTRY_PRIOR_TOP_K,
+                'maximum': MAX_DETECTOR_HINT_PCT,
             },
         ),
     ]

src/routers/curation/pipeline.py

@@ -152,7 +152,6 @@ async def _run_auto_label(
     from src.services.curation.autolabel.job import with_elapsed_tick
     from src.services.curation.clustering.auto_promote import gated_auto_promote
     from src.services.curation.image_serving import THUMBNAIL_CACHE
-    from src.services.labeling.vlm_models import ItemCrop
 
     reject_detection_profile(detection_profile)
     if isinstance(vlm, str) and not vlm_resolved and not run_vlm:
@@ -397,8 +396,7 @@ async def _run_auto_label(
         summary['final'] = {'unvalidated': 0, 'human_required': 0}
         return summary
 
-    # Reuse the VLM label_batch logic by calling it directly (no HTTP
-    # hop). Build ItemCrops here so we can chunk.
+    # Calls the labeler directly (no HTTP hop); ItemCrops are built here to chunk.
     from src.services.curation.region_class import item_classes
     from src.services.labeling.vlm_class_names import (
         format_class_catalog,
@@ -441,13 +439,10 @@ async def _run_auto_label(
     labeler = _get_vlm_labeler(_labeler_pack, _labeler_revision, endpoint=endpoint)
     class_catalog = format_class_catalog(class_dicts, labeler._pack)
 
-    from src.services.curation.registry_prior_source import prior_or_error
+    from src.services.labeling.detector_hint import DETECTOR_HINT_FIELDS, hinted_crop
     from src.services.labeling.vlm_prompts import prompt_pack_stamp
 
     _pack_stamp = prompt_pack_stamp(labeler._pack, revision=_labeler_revision)
-    registry_prior, prior_error = await prior_or_error(opensearch, labeler._pack, class_names)
-    if prior_error:
-        return {**summary, 'stages_error': prior_error}
 
     # Count how many crops bypass the synonym/fuzzy force-fit because the
     # VLM's confidence is low — those route straight to the raw-label
@@ -479,9 +474,9 @@ async def _run_chunk(ids: list[str]) -> tuple[int, int, list[dict[str, Any]]]:
         docs = await mget_crops(
             opensearch,
             ids,
-            source_includes=['image_path', 'bbox_norm'],
+            source_includes=['image_path', 'bbox_norm', *DETECTOR_HINT_FIELDS],
         )
-        crops: list[ItemCrop] = []
+        crops = []
         for crop_id, doc in docs.items():
             src = doc.get('_source') or {}
             image_path = src.get('image_path', '')
@@ -496,7 +491,7 @@ async def _run_chunk(ids: list[str]) -> tuple[int, int, list[dict[str, Any]]]:
             except Exception as exc:
                 logger.warning('pipeline_thumb_failed', crop_id=crop_id, error=str(exc))
                 continue
-            crops.append(ItemCrop(img_id=crop_id, jpeg_bytes=jpeg))
+            crops.append(hinted_crop(crop_id, jpeg, src, labeler._pack))
         # No item classes yet (a fresh project, or only the region class):
         # nothing to label items as.
         if not crops or not class_names:
@@ -505,7 +500,6 @@ async def _run_chunk(ids: list[str]) -> tuple[int, int, list[dict[str, Any]]]:
             crops,
             class_names,
             class_catalog=class_catalog,
-            registry_prior=registry_prior,
         )
         # Collect per-doc updates, then dispatch via
         # occ_skip_on_conflict_bulk so a concurrent human edit always

src/routers/curation/prompt_packs.py

@@ -50,7 +50,7 @@
     rollback_pack,
     save_pack,
 )
-from src.services.labeling.registry_prior import MAX_REGISTRY_PRIOR_TOP_K
+from src.services.labeling.detector_hint import MAX_DETECTOR_HINT_PCT
 from src.services.labeling.vlm_prompts import FORMATTED_PLACEHOLDERS, REPLY_KEY_CONTRACT
 
 
@@ -81,21 +81,21 @@
     'class_descriptions': 'vocabulary',
     'synonyms': 'vocabulary',
     'proposal_denylist': 'vocabulary',
-    'registry_prior_top_k': 'open_classify',
+    'detector_hint_min_confidence_pct': 'open_classify',
 }
 
 _FIELD_KIND: dict[str, Literal['map', 'list', 'int']] = {
     'class_descriptions': 'map',
     'synonyms': 'map',
     'proposal_denylist': 'list',
-    'registry_prior_top_k': 'int',
+    'detector_hint_min_confidence_pct': 'int',
 }
 
 _FIELD_HELP: dict[str, str] = {
-    'registry_prior_top_k': (
+    'detector_hint_min_confidence_pct': (
         'Optional hint for open-vocabulary labeling: 0 = off; 1 to '
-        f'{MAX_REGISTRY_PRIOR_TOP_K} lists that many registry classes (most validated first) '
-        'and pending proposal names in the prompt. A hint, not a constraint.'
+        f"{MAX_DETECTOR_HINT_PCT} tells the VLM each item's stored detector class name when "
+        'the detector confidence is at least that many percent. A hint, not a constraint.'
     ),
 }