Complete final evidence and policy hardening

README.md

@@ -11,7 +11,7 @@ The submission is a complete local application: a deterministic audit engine, mu
 - **Evidence adapters** for JSON, SQLite-compatible SQL, structured logs, text, PDFs, and screenshots.
 - **Temporal safety** at both evidence-capture and activity-link levels; post-`audit_at` data cannot influence a verdict.
 - **Constraint-based workflow attribution** that resolves anonymous batched targets only when every cutoff-safe one-to-one assignment agrees.
-- **Policy planner** for delete, redact, anonymize, retain, block-restore, and disable-flow actions.
+- **Data-driven policy planner** that loads `deletion_rules.csv` and resolves delete, redact, anonymize, retain, block-restore, and disable-flow actions.
 - **Fail-closed uncertainty**: missing subject-linked proof produces `INCOMPLETE` with a warning instead of an invented `COMPLETE` verdict.
 - **Investigation UI** with search, status filters, case drill-down, evidence lineage, ordered plan, responsive layout, theme support, and keyboard-aware controls.
 - **Zero-runtime-model architecture**: reproducible predictions, no network dependency, no API key, and no per-request inference cost.
@@ -82,7 +82,7 @@ Then open [http://127.0.0.1:8787](http://127.0.0.1:8787). The UI and API are ser
 
 1. **Intent:** multilingual deletion language is accepted; correction, export-only, marketing-only, and explicit non-deletion requests are rejected as `NOT_APPLICABLE`.
 2. **Identity:** the request email is resolved to one stable user, including historical email and phone aliases valid for the request timeline.
-3. **Evidence:** structured sources are parsed semantically; PDFs and images use local extraction with a content-addressable text cache.
+3. **Evidence:** JSON, cloud-envelope, syslog, pipe-delimited, SQL, and structured-text sources are parsed semantically; PDFs and images use local extraction with a content-addressable text cache.
 4. **Time:** evidence captures, source-level event times, and resource-to-subject activity links must exist at or before `audit_at`.
 5. **Attribution:** anonymous workflow targets are solved as a cutoff-aware bipartite assignment; a target is accepted only when it is invariant across every valid assignment.
 6. **State:** the latest eligible observation per system/resource is selected.
@@ -117,7 +117,7 @@ On the 32 participant-visible labeled cases:
 
 GhostTrace reaches every participant-visible sample verdict without reading labels at prediction time. Remaining exact-sequence gaps are concentrated in canonical resource links that are not present in cutoff-eligible participant evidence; the engine does not invent those links. Full methodology and limitations are documented in [`code/evaluation/evaluation_report.md`](code/evaluation/evaluation_report.md).
 
-The final full-data run currently produces 148 `COMPLETE`, 360 `INCOMPLETE`, and 10 `NOT_APPLICABLE` decisions. Its `output.csv` passes the strict validator with 518/518 rows. Step count is bounded after parser hardening (median 5, 95th percentile 12, maximum 14), preventing service instance IDs from being promoted to user resources. The workflow solver is independently checked against 164 participant-evidence subject links with zero mismatches; those links validate the assignments but are not permitted to bypass each case's runtime cutoff.
+The final full-data run currently produces 148 `COMPLETE`, 360 `INCOMPLETE`, and 10 `NOT_APPLICABLE` decisions. Its `output.csv` passes the strict validator with 518/518 rows. Every authorized case now has an evidence-grounded plan containing the status-required phases; no `COMPLETE` or `INCOMPLETE` row is empty. Step count remains bounded after parser hardening (median 6, 95th percentile 13, maximum 14), preventing service instance IDs from being promoted to user resources. The workflow solver is independently checked against 164 participant-evidence subject links with zero mismatches; those links validate the assignments but are not permitted to bypass each case's runtime cutoff.
 
 ## Security and privacy posture
 
@@ -140,6 +140,7 @@ code/
 │   ├── intent.py            multilingual intent gate
 │   ├── evidence.py          multi-format evidence adapters/index
 │   ├── workflow.py          cutoff-safe batch assignment solver
+│   ├── policy.py            deletion_rules.csv policy interpreter
 │   ├── engine.py            policy reducer and plan generator
 │   ├── validation.py        strict output contract validator
 │   └── server.py            local investigation API/server

code/evaluation/evaluation_report.md

@@ -10,7 +10,7 @@ Command:
 python3 code/evaluation/main.py
 ```
 
-Environment: Python 3.14.0, deterministic local execution, 32 public sample cases, 518 full-dataset cases, 94 evidence files, and 2,903 extracted observations. Extraction completed with zero adapter warnings in the measured environment.
+Environment: Python 3.14.6, deterministic local execution, 32 public sample cases, 518 full-dataset cases, 94 evidence files, and 2,984 extracted observations. Extraction completed with zero adapter warnings in the measured environment.
 
 ## Public-sample metrics
 
@@ -37,7 +37,7 @@ The production command emitted 518/518 canonically sorted rows and passed every
 - Empty evidence IDs on future remediation/verification.
 - Canonical phase, operation, system, and resource ordering.
 
-Plan-size distribution after parser hardening: median 5 steps, 95th percentile 12, maximum 14. No plan exceeds 20 steps.
+Plan-size distribution after parser hardening: median 6 steps, 95th percentile 13, maximum 14. No plan exceeds 20 steps, and every authorized case contains a non-empty status-appropriate plan.
 
 ## Additional evaluation cases
 
@@ -77,6 +77,10 @@ Generate the entire 518-row file in a temporary directory and run the strict val
 10. **Evidence integrity.** The validator now rejects missing finding citations and evidence IDs absent from every participant-visible evidence namespace.
 11. **Anonymous batch attribution.** Local nearest-event matching was insufficient when several deletion workflows shared a report time. A cutoff-aware bipartite solver now filters candidates by workflow-result compatibility and ownership conflicts, anchors explicit subject matches, enforces one-to-one assignment, and emits a resource only when every valid assignment agrees. An independent participant-evidence audit validates 164 resolved targets with zero mismatches. Evidence after a target cutoff is used only by this offline regression check, never by runtime prediction.
 12. **Source event-time reconstruction.** Structured text logs previously inherited the file capture time for every line. Leading ISO timestamps are now preserved per observation, so later `NOT_FOUND` states correctly supersede earlier `FOUND` states without violating the case cutoff.
+13. **Multi-dialect trace loss.** Structured JSON lines, cloud-runtime envelopes, RFC-style syslog, and pipe-delimited operations streams placed subject and resource state on different lines under a shared trace identifier. The previous parser recognized only `trace_id=` syntax and left 15 authorized cases without plans. The trace correlator now parses all supplied dialects, preserves terminal resource timestamps, and reduces empty authorized plans from 15 to zero. Against the preceding release, this reconstructs 158 additional contract steps across 53 full-data rows while retaining the public verdict gate.
+14. **Static policy duplication.** Required actions were previously mirrored in source code. The runtime now loads and validates `deletion_rules.csv`; only operational interpretations absent from its action vocabulary—disabling active flows and blocking an unsafe restore while retaining the archive—are derived by the policy interpreter.
+15. **Catalog-to-artifact reconciliation.** Object-storage screenshots expose filenames while the data catalog exposes stable resource IDs. A time-scoped filename alias join now connects those two participant-visible facts, recovering `object_19` for case 61 without deriving an identifier from naming coincidence alone.
+16. **Incomplete-plan validation.** The validator now rejects empty or phase-incomplete `INCOMPLETE` plans and empty `COMPLETE` verification arrays. This makes the brief's status-specific content requirement an executable release gate.
 
 ## Remaining limitations
 
@@ -93,4 +97,4 @@ python3 code/main.py run
 python3 code/main.py validate
 ```
 
-Final regression result: **21/21 tests pass**. The suite includes manifest-completeness, extraction-health, temporal-safety, invariant-assignment, and independent-attribution gates so a checkout cannot silently omit evidence or accept an unsupported workflow mapping.
+Final regression result: **29/29 tests pass**. The suite includes manifest-completeness, extraction-health, temporal-safety, trace-dialect, dynamic-policy, full-plan, alias-validity, invariant-assignment, independent-attribution, and byte-reproducible packaging gates so a checkout cannot silently omit evidence, accept an unsupported mapping, or emit a timestamp-dependent submission archive.

code/ghosttrace/engine.py

@@ -9,6 +9,7 @@
 from .evidence import EvidenceIndex
 from .identity import IdentityIndex
 from .intent import authorizes_deletion
+from .policy import PolicyIndex
 from .workflow import WorkflowAttributor
 
 
@@ -31,39 +32,11 @@
     "VERIFY_RESTORE_BLOCKED": 5,
 }
 
-ACTION_BY_CATEGORY = {
-    "USER_PROFILE": "DELETE",
-    "SECONDARY_IDENTIFIER": "DELETE",
-    "SEARCH_INDEX_ENTRY": "DELETE",
-    "CACHE_ENTRY": "DELETE",
-    "SUPPORT_MESSAGE": "REDACT",
-    "SUPPORT_ATTACHMENT": "DELETE",
-    "ANALYTICS_PROFILE": "ANONYMIZE",
-    "PRODUCT_TELEMETRY": "ANONYMIZE",
-    "USER_EXPORT": "DELETE",
-    "MARKETING_PROFILE": "DELETE",
-    "EMAIL_DELIVERY_LOG": "REDACT",
-    "SECURITY_AUDIT_LOG": "RETAIN",
-    "LEGAL_HOLD_RECORD": "RETAIN",
-    "BACKUP_ARCHIVE": "BLOCK_RESTORE",
-    "ACCOUNT_TOMBSTONE": "REDACT",
-    "DATA_FLOW": "DISABLE_DATA_FLOW",
-}
-
-TARGET_STATE = {
-    "DELETE": "ABSENT",
-    "REDACT": "REDACTED",
-    "ANONYMIZE": "ANONYMIZED",
-    "RETAIN": "PRESENT",
-    "BLOCK_RESTORE": "BLOCKED",
-    "DISABLE_DATA_FLOW": "DISABLED",
-}
-
-
 class GhostTraceEngine:
     def __init__(self, dataset_dir: Path, cache_dir: Path | None = None):
         self.dataset_dir = dataset_dir.resolve()
         self.identities = IdentityIndex(self.dataset_dir)
+        self.policy = PolicyIndex(self.dataset_dir)
         self.evidence = EvidenceIndex(self.dataset_dir, cache_dir=cache_dir)
         self.cases = self._load_cases(self.dataset_dir / "deletion_requests.csv")
         context_cases = list(self.cases)
@@ -113,6 +86,10 @@ def audit(self, case: Case) -> Decision:
             observation
             for observation in self.evidence.eligible(case.audit_at)
             if self.evidence.matches(observation, aliases, case.audit_at)
+            and not (
+                observation.attributes.get("inference") == "catalog_file_alias"
+                and observation.observed_at < case.requested_at
+            )
         ]
         workflow_account = self.workflow.account_resource(
             case
@@ -131,7 +108,9 @@ def audit(self, case: Case) -> Decision:
         ):
             observations.append(anonymization_proof)
         observations = latest_by_resource(observations)
-        observations = [item for item in observations if item.category in ACTION_BY_CATEGORY]
+        observations = [
+            item for item in observations if self.policy.supports(item.category)
+        ]
 
         # Prefer state evidence close to the audit cutoff while retaining one
         # authoritative state per resource. Resource history is still available
@@ -171,10 +150,8 @@ def audit(self, case: Case) -> Decision:
             warnings=warnings,
         )
 
-    @staticmethod
-    def _compliant(observation: Observation) -> bool:
-        action = ACTION_BY_CATEGORY[observation.category]
-        return observation.state == TARGET_STATE[action]
+    def _compliant(self, observation: Observation) -> bool:
+        return observation.state == self.policy.target_state(observation)
 
     def _steps(
         self,
@@ -187,7 +164,7 @@ def _steps(
             for observation in observations:
                 if self._compliant(observation):
                     continue
-                action = ACTION_BY_CATEGORY[observation.category]
+                action = self.policy.action_for(observation)
                 if action == "DELETE":
                     finding = ("REPORT_REMAINING_RECORD", "PRESENT")
                     remediation = ("DELETE_RECORD", "ABSENT")
@@ -210,7 +187,7 @@ def _steps(
             *(planned_verifications or []),
         ]
         for observation in verification_observations:
-            action = ACTION_BY_CATEGORY[observation.category]
+            action = self.policy.action_for(observation)
             operation = {
                 "DELETE": "VERIFY_RECORD_ABSENT",
                 "RETAIN": "VERIFY_RECORD_PRESENT",
@@ -220,7 +197,14 @@ def _steps(
                 "BLOCK_RESTORE": "VERIFY_RESTORE_BLOCKED",
             }[action]
             evidence_id = observation.evidence_id if status == "COMPLETE" else ""
-            raw.append(("VERIFICATION", observation.system_id, observation.resource_id, operation, TARGET_STATE[action], evidence_id))
+            raw.append((
+                "VERIFICATION",
+                observation.system_id,
+                observation.resource_id,
+                operation,
+                self.policy.target_state(observation),
+                evidence_id,
+            ))
 
         raw.sort(key=lambda item: (PHASE_ORDER[item[0]], OPERATION_ORDER[item[3]], item[1], item[2]))
         return [
@@ -232,6 +216,7 @@ def diagnostics(self) -> dict[str, object]:
         return {
             "cases": len(self.cases),
             "identities": len(self.identities.identities),
+            "policy_rules": len(self.policy.rules),
             "evidence_files": len(self.evidence.files),
             "observations": len(self.evidence.observations),
             "extraction_warnings": list(self.evidence.extraction_warnings),