diff --git a/README.md b/README.md index a136b96..4efdfec 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,7 @@ The submission is a complete local application: a deterministic audit engine, mu - **Historical identity graph** across current identity data and audit-log mutations. - **Evidence adapters** for JSON, SQLite-compatible SQL, structured logs, text, PDFs, and screenshots. - **Temporal safety** at both evidence-capture and activity-link levels; post-`audit_at` data cannot influence a verdict. +- **Constraint-based workflow attribution** that resolves anonymous batched targets only when every cutoff-safe one-to-one assignment agrees. - **Policy planner** for delete, redact, anonymize, retain, block-restore, and disable-flow actions. - **Fail-closed uncertainty**: missing subject-linked proof produces `INCOMPLETE` with a warning instead of an invented `COMPLETE` verdict. - **Investigation UI** with search, status filters, case drill-down, evidence lineage, ordered plan, responsive layout, theme support, and keyboard-aware controls. @@ -24,6 +25,8 @@ evidence manifest ── format adapters ── temporal evidence index │ system activity ── cutoff-safe resource/subject links │ +batched workflows ── invariant assignment solver + │ deletion_rules ─────────────── policy reducer + state planner │ strict validator ── output.csv @@ -80,11 +83,12 @@ Then open [http://127.0.0.1:8787](http://127.0.0.1:8787). The UI and API are ser 1. **Intent:** multilingual deletion language is accepted; correction, export-only, marketing-only, and explicit non-deletion requests are rejected as `NOT_APPLICABLE`. 2. **Identity:** the request email is resolved to one stable user, including historical email and phone aliases valid for the request timeline. 3. **Evidence:** structured sources are parsed semantically; PDFs and images use local extraction with a content-addressable text cache. -4. **Time:** both evidence captures and resource-to-subject activity links must exist at or before `audit_at`. -5. **State:** the latest eligible observation per system/resource is selected. -6. **Policy:** the required terminal state is derived by category. -7. **Plan:** violations generate canonically ordered findings, remediation, and verification; compliant cases generate evidence-backed verification only. -8. **Validation:** output is rejected if it violates any part of the challenge contract. +4. **Time:** evidence captures, source-level event times, and resource-to-subject activity links must exist at or before `audit_at`. +5. **Attribution:** anonymous workflow targets are solved as a cutoff-aware bipartite assignment; a target is accepted only when it is invariant across every valid assignment. +6. **State:** the latest eligible observation per system/resource is selected. +7. **Policy:** the required terminal state is derived by category. +8. **Plan:** violations generate canonically ordered findings, remediation, and verification; compliant cases generate evidence-backed verification only. +9. **Validation:** output is rejected if it violates any part of the challenge contract. ## Output contract @@ -107,13 +111,13 @@ On the 32 participant-visible labeled cases: | --- | ---: | | Case status accuracy | 100.000% | | Exact row / exact sequence accuracy | 50.000% | -| Step-field accuracy | 78.792% | -| Step precision | 79.651% | -| Step recall | 70.984% | +| Step-field accuracy | 80.118% | +| Step precision | 81.034% | +| Step recall | 73.057% | GhostTrace reaches every participant-visible sample verdict without reading labels at prediction time. Remaining exact-sequence gaps are concentrated in canonical resource links that are not present in cutoff-eligible participant evidence; the engine does not invent those links. Full methodology and limitations are documented in [`code/evaluation/evaluation_report.md`](code/evaluation/evaluation_report.md). -The final full-data run currently produces 143 `COMPLETE`, 365 `INCOMPLETE`, and 10 `NOT_APPLICABLE` decisions. Its `output.csv` passes the strict validator with 518/518 rows. Step count is bounded after parser hardening (median 5, 95th percentile 12, maximum 14), preventing service instance IDs from being promoted to user resources. +The final full-data run currently produces 148 `COMPLETE`, 360 `INCOMPLETE`, and 10 `NOT_APPLICABLE` decisions. Its `output.csv` passes the strict validator with 518/518 rows. Step count is bounded after parser hardening (median 5, 95th percentile 12, maximum 14), preventing service instance IDs from being promoted to user resources. The workflow solver is independently checked against 164 participant-evidence subject links with zero mismatches; those links validate the assignments but are not permitted to bypass each case's runtime cutoff. ## Security and privacy posture @@ -135,6 +139,7 @@ code/ │ ├── identity.py temporal identity resolution │ ├── intent.py multilingual intent gate │ ├── evidence.py multi-format evidence adapters/index +│ ├── workflow.py cutoff-safe batch assignment solver │ ├── engine.py policy reducer and plan generator │ ├── validation.py strict output contract validator │ └── server.py local investigation API/server diff --git a/code/evaluation/evaluation_report.md b/code/evaluation/evaluation_report.md index 9006452..27cf26e 100644 --- a/code/evaluation/evaluation_report.md +++ b/code/evaluation/evaluation_report.md @@ -18,19 +18,19 @@ Environment: Python 3.14.0, deterministic local execution, 32 public sample case | --- | --- | ---: | | Case accuracy | Correct `deletion_status` | 32/32 = **100.000%** | | Row accuracy | Status and complete ordered step array exactly match | 16/32 = **50.000%** | -| Field accuracy | Micro accuracy across seven aligned step fields; missing/extra steps count as seven misses | **78.792%** | +| Field accuracy | Micro accuracy across seven aligned step fields; missing/extra steps count as seven misses | **80.118%** | | Sequence accuracy | Exact ordered `(phase, system, resource, operation, state)` sequence | 16/32 = **50.000%** | -| Step precision | Exact predicted step objects also present in target | **79.651%** | -| Step recall | Exact target step objects recovered | **70.984%** | +| Step precision | Exact predicted step objects also present in target | **81.034%** | +| Step recall | Exact target step objects recovered | **73.057%** | -Status accuracy is the primary readiness gate because it measures whether GhostTrace reaches the correct deletion conclusion. The accuracy-hardening release raises verdict accuracy from 90.625% to 100%, exact rows from 18.750% to 50%, precision from 62.346% to 79.651%, and recall from 52.332% to 70.984%. Exact sequence remains lower because the participant-visible evidence does not consistently expose every subject-to-resource edge implied by the sample plan. The runtime intentionally refuses to recover those edges from labels. +Status accuracy is the primary readiness gate because it measures whether GhostTrace reaches the correct deletion conclusion. Across the accuracy-hardening releases, verdict accuracy rose from 90.625% to 100%, exact rows from 18.750% to 50%, precision from 62.346% to 81.034%, and recall from 52.332% to 73.057%. Exact sequence remains lower because the participant-visible evidence does not consistently expose every subject-to-resource edge implied by the sample plan. The runtime intentionally refuses to recover those edges from labels. ## Full-dataset contract evaluation The production command emitted 518/518 canonically sorted rows and passed every validator rule: - Exact header and one row per request. -- 143 `COMPLETE`, 365 `INCOMPLETE`, 10 `NOT_APPLICABLE`. +- 148 `COMPLETE`, 360 `INCOMPLETE`, 10 `NOT_APPLICABLE`. - Valid JSON arrays and consecutive step numbering. - Legal status, phase, operation, and state combinations. - Evidence on `COMPLETE` verification and `INCOMPLETE` findings. @@ -75,6 +75,8 @@ Generate the entire 518-row file in a temporary directory and run the strict val 8. **Workflow/readback correlation.** Unique workflow targets can now be joined to independent cutoff-eligible absence or anonymization proof. The join is rejected when concurrent targets make attribution ambiguous. 9. **Negative account proof.** A uniquely attributed account workflow can use a later, cutoff-eligible complete account snapshot as absence proof; activity success alone is never treated as proof. 10. **Evidence integrity.** The validator now rejects missing finding citations and evidence IDs absent from every participant-visible evidence namespace. +11. **Anonymous batch attribution.** Local nearest-event matching was insufficient when several deletion workflows shared a report time. A cutoff-aware bipartite solver now filters candidates by workflow-result compatibility and ownership conflicts, anchors explicit subject matches, enforces one-to-one assignment, and emits a resource only when every valid assignment agrees. An independent participant-evidence audit validates 164 resolved targets with zero mismatches. Evidence after a target cutoff is used only by this offline regression check, never by runtime prediction. +12. **Source event-time reconstruction.** Structured text logs previously inherited the file capture time for every line. Leading ISO timestamps are now preserved per observation, so later `NOT_FOUND` states correctly supersede earlier `FOUND` states without violating the case cutoff. ## Remaining limitations @@ -91,4 +93,4 @@ python3 code/main.py run python3 code/main.py validate ``` -Final regression result: **17/17 tests pass**. The suite includes a manifest-completeness and extraction-health gate so a checkout cannot silently omit participant evidence. +Final regression result: **21/21 tests pass**. The suite includes manifest-completeness, extraction-health, temporal-safety, invariant-assignment, and independent-attribution gates so a checkout cannot silently omit evidence or accept an unsupported workflow mapping. diff --git a/code/ghosttrace/engine.py b/code/ghosttrace/engine.py index 9e91c1a..8e6cea4 100644 --- a/code/ghosttrace/engine.py +++ b/code/ghosttrace/engine.py @@ -9,6 +9,7 @@ from .evidence import EvidenceIndex from .identity import IdentityIndex from .intent import authorizes_deletion +from .workflow import WorkflowAttributor PHASE_ORDER = {"FINDING": 0, "REMEDIATION": 1, "VERIFICATION": 2} @@ -65,6 +66,15 @@ def __init__(self, dataset_dir: Path, cache_dir: Path | None = None): self.identities = IdentityIndex(self.dataset_dir) self.evidence = EvidenceIndex(self.dataset_dir, cache_dir=cache_dir) self.cases = self._load_cases(self.dataset_dir / "deletion_requests.csv") + context_cases = list(self.cases) + sample_path = self.dataset_dir / "sample_deletion_requests.csv" + if sample_path.is_file(): + context_cases.extend(self._load_cases(sample_path)) + self.workflow = WorkflowAttributor( + self.identities, + self.evidence, + context_cases, + ) @staticmethod def _load_cases(path: Path) -> list[Case]: @@ -104,7 +114,9 @@ def audit(self, case: Case) -> Decision: for observation in self.evidence.eligible(case.audit_at) if self.evidence.matches(observation, aliases, case.audit_at) ] - workflow_account = self.evidence.unambiguous_account_resource(case, aliases) + workflow_account = self.workflow.account_resource( + case + ) or self.evidence.unambiguous_account_resource(case, aliases) if workflow_account and not any( item.resource_id == workflow_account for item in observations ): diff --git a/code/ghosttrace/evidence.py b/code/ghosttrace/evidence.py index c119bd1..bead19e 100644 --- a/code/ghosttrace/evidence.py +++ b/code/ghosttrace/evidence.py @@ -21,6 +21,7 @@ USER_RE = re.compile(r"\buser[_\s-]?(\d+)\b", re.I) HASH_RE = re.compile(r"\b[a-f0-9]{64}\b", re.I) KV_RE = re.compile(r"([A-Za-z_][\w.-]*)=(\"[^\"]*\"|'[^']*'|\S+)") +ISO_PREFIX_RE = re.compile(r"^\s*(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?Z)\b") PREFIX_META = { "account": ("account_service", "USER_PROFILE"), @@ -249,9 +250,13 @@ def __init__(self, dataset_dir: Path, cache_dir: Path | None = None): self.resource_links: dict[str, list[tuple[Any, set[str], str]]] = defaultdict(list) self.activity_events: list[dict[str, Any]] = [] self.observations: list[Observation] = [] + self.observations_by_resource: dict[str, list[Observation]] = defaultdict(list) self.extraction_warnings: list[str] = [] + self._captured_at = {item.file_id: item.captured_at for item in self.files} self._load_activity_links() self._extract_all() + for observation in self.observations: + self.observations_by_resource[observation.resource_id].append(observation) def _load_manifest(self) -> list[EvidenceFile]: result: list[EvidenceFile] = [] @@ -430,10 +435,10 @@ def _from_text( ) -> list[Observation]: if not text.strip(): return [] - groups: dict[str, list[str]] = defaultdict(list) + groups: dict[str, list[tuple[int, str]]] = defaultdict(list) free_lines: list[str] = [] result: list[Observation] = [] - for line in text.splitlines(): + for line_no, line in enumerate(text.splitlines(), 1): parts = [part.strip().lstrip("- ") for part in line.split("|")] if len(parts) >= 6 and RESOURCE_RE.search(parts[2]): resource = canonical_resource(parts[2]) @@ -444,6 +449,7 @@ def _from_text( "record_type": parts[3], "result": parts[4], "lookup_value": parts[5], + "_line_no": line_no, } inferred_system, inferred_category = meta_for_text_resource( resource, evidence, record, line @@ -463,9 +469,13 @@ def _from_text( continue match = re.search(r"\btrace_id=([^\s]+)", line) if match: - groups[match.group(1)].append(line) + groups[match.group(1)].append((line_no, line)) elif "resource_id=" in line: record = {key: value.strip("\"'") for key, value in KV_RE.findall(line)} + record["_line_no"] = line_no + leading_time = ISO_PREFIX_RE.search(line) + if leading_time: + record["observed_at"] = leading_time.group(1) resource = canonical_resource(record.get("resource_id", "")) if resource: system_id, category = meta_for_text_resource( @@ -488,9 +498,14 @@ def _from_text( ) else: free_lines.append(line) - for lines in groups.values(): + for entries in groups.values(): + lines = [line for _line_no, line in entries] merged = " ".join(lines) record = {key: value.strip("\"'") for key, value in KV_RE.findall(merged)} + record["_line_no"] = min(line_no for line_no, _line in entries) + leading_time = ISO_PREFIX_RE.search(lines[0]) + if leading_time: + record["observed_at"] = leading_time.group(1) explicit_resource = record.get("resource_id", "") resources = {canonical_resource(explicit_resource)} if explicit_resource else set() if not resources: @@ -543,29 +558,58 @@ def _from_text( return result def eligible(self, audit_at) -> list[Observation]: - captured = {item.file_id: item.captured_at for item in self.files} return [ observation for observation in self.observations - if captured.get(observation.evidence_id, observation.observed_at) <= audit_at + if self._captured_at.get( + observation.evidence_id, observation.observed_at + ) <= audit_at and observation.observed_at <= audit_at ] def matches(self, observation: Observation, aliases: set[str], audit_at) -> bool: """Resolve subject ownership without allowing post-cutoff activity links.""" + relation = self.ownership_relation(observation.resource_id, aliases, audit_at) + if relation != "UNKNOWN": + return relation == "MATCH" + return observation.matches(aliases) + + def ownership_relation( + self, + resource_id: str, + aliases: set[str], + audit_at, + ) -> str: + """Return MATCH, CONFLICT, or UNKNOWN at a decision cutoff. + + Explicit activity-catalog links have precedence over extracted text. + Subject-bearing observations are used only when no authoritative link + exists. Anonymous evidence therefore remains UNKNOWN rather than being + silently assigned. + """ eligible_links = [ subjects for linked_at, subjects, _activity_id in self.resource_links.get( - observation.resource_id, [] + resource_id, [] ) if linked_at <= audit_at ] - # The data catalog is authoritative when it explicitly links a resource. - # This prevents a broad OCR window from assigning a neighboring table - # row's identity to a resource owned by somebody else. if eligible_links: - return any(bool(subjects & aliases) for subjects in eligible_links) - return observation.matches(aliases) + return "MATCH" if any(subjects & aliases for subjects in eligible_links) else "CONFLICT" + subject_observations = [ + item + for item in self.observations_by_resource.get(resource_id, []) + if item.subjects + and self._captured_at.get(item.evidence_id, item.observed_at) <= audit_at + and item.observed_at <= audit_at + ] + if not subject_observations: + return "UNKNOWN" + return ( + "MATCH" + if any(item.subjects & aliases for item in subject_observations) + else "CONFLICT" + ) def unambiguous_account_resource( self, @@ -723,7 +767,9 @@ def verified_unique_anonymization( if normalize(event.get("target_system")) != "analytics_platform": continue resource = canonical_resource(event.get("resource_id", "")) - if resource.startswith("analytics_"): + if resource.startswith("analytics_") and self.ownership_relation( + resource, aliases, case.audit_at + ) != "CONFLICT": candidates.add(resource) if len(candidates) != 1: return None diff --git a/code/ghosttrace/identity.py b/code/ghosttrace/identity.py index 910a119..48cb817 100644 --- a/code/ghosttrace/identity.py +++ b/code/ghosttrace/identity.py @@ -2,6 +2,7 @@ import csv from collections import defaultdict +from datetime import datetime from pathlib import Path from .domain import Case, Identity, normalize, parse_time @@ -42,18 +43,41 @@ def resolve(self, case: Case) -> Identity | None: return None def aliases_at(self, identity: Identity, case: Case) -> set[str]: - aliases = {normalize(case.email), normalize(identity.user_id)} + return self.aliases_at_time( + identity, + case.audit_at, + case.email, + include_all_history=True, + ) + + def aliases_at_time( + self, + identity: Identity, + audit_at: datetime, + request_email: str = "", + *, + include_all_history: bool = False, + ) -> set[str]: + """Return only aliases known by a specific decision cutoff. + + Cross-case workflow attribution uses the target decision's cutoff for + every cohort member. This prevents a later case's evidence window from + leaking future identity changes into an earlier audit. + """ + aliases = {normalize(request_email), normalize(identity.user_id)} updated_at = parse_time(identity.current.get("updated_at")) - if updated_at <= case.audit_at: + if updated_at <= audit_at: aliases.update(normalize(identity.current.get(key)) for key in IDENTITY_FIELDS) for row in self.audit_rows: - if row["user_id"] != identity.user_id or parse_time(row["event_time"]) > case.audit_at: + if row["user_id"] != identity.user_id or parse_time(row["event_time"]) > audit_at: continue if row["field_name"] in IDENTITY_FIELDS: aliases.add(normalize(row["old_value"])) aliases.add(normalize(row["new_value"])) - # Historical deletion requires finding every identifier ever attached to - # the stable subject. The cutoff is enforced on evidence, not aliases. - aliases.update(identity.aliases) + if include_all_history: + # The primary audit path deliberately searches every historical + # alias ever attached to the stable subject. Evidence eligibility + # still enforces the case cutoff. + aliases.update(identity.aliases) return {alias for alias in aliases if alias} diff --git a/code/ghosttrace/workflow.py b/code/ghosttrace/workflow.py new file mode 100644 index 0000000..7fd79a0 --- /dev/null +++ b/code/ghosttrace/workflow.py @@ -0,0 +1,138 @@ +from __future__ import annotations + +from collections import defaultdict +from datetime import datetime +from typing import Iterable + +from .domain import Case, normalize +from .evidence import EvidenceIndex, canonical_resource +from .identity import IdentityIndex +from .intent import authorizes_deletion + + +RESULT_FOR_STATUS = { + "COMPLETE": "SUCCESS", + "PARTIALLY_COMPLETE": "PARTIAL", + "NOT_STARTED": "FAILED", +} + + +class WorkflowAttributor: + """Resolve anonymous workflow targets through cutoff-safe constraints. + + The activity stream batches several deletion workflows around a common + report time. Individual rows may omit the subject, so local nearest-event + matching is unsafe. This solver instead builds candidate sets from workflow + result compatibility, removes resources that conflict with cutoff-eligible + ownership evidence, anchors explicit matches, and accepts a target only if + every valid one-to-one assignment gives that case the same resource. + """ + + def __init__( + self, + identities: IdentityIndex, + evidence: EvidenceIndex, + cases: Iterable[Case], + ) -> None: + self.identities = identities + self.evidence = evidence + self.groups: dict[datetime, list[Case]] = defaultdict(list) + for case in cases: + if authorizes_deletion(case.request_text): + self.groups[case.reported_at].append(case) + self.account_events: dict[tuple[datetime, str], set[str]] = defaultdict(set) + for reported_at in self.groups: + for event in evidence.activity_events: + offset_minutes = ( + event["parsed_time"] - reported_at + ).total_seconds() / 60 + if not (-75 <= offset_minutes <= -30): + continue + if event.get("event_type") != "DELETE_RECORD": + continue + if normalize(event.get("target_system")) != "account_service": + continue + result = normalize(event.get("reported_result")).upper() + resource = canonical_resource(event.get("resource_id", "")) + if resource.startswith("account_"): + self.account_events[(reported_at, result)].add(resource) + self._cache: dict[tuple[str, datetime], str] = {} + + def account_resource(self, target: Case) -> str: + cache_key = (target.case_id, target.audit_at) + if cache_key in self._cache: + return self._cache[cache_key] + members = self.groups.get(target.reported_at, []) + if not members or target not in members: + self._cache[cache_key] = "" + return "" + + options: dict[str, set[str]] = {} + for member in members: + identity = self.identities.resolve(member) + expected_result = RESULT_FOR_STATUS.get(member.reported_status) + if identity is None or expected_result is None: + self._cache[cache_key] = "" + return "" + aliases = self.identities.aliases_at_time( + identity, + target.audit_at, + member.email, + ) + candidates = set( + self.account_events.get((member.reported_at, expected_result), set()) + ) + relationships = { + resource: self.evidence.ownership_relation( + resource, aliases, target.audit_at + ) + for resource in candidates + } + candidates = { + resource + for resource in candidates + if relationships[resource] != "CONFLICT" + } + anchored = { + resource + for resource in candidates + if relationships[resource] == "MATCH" + } + options[member.case_id] = anchored or candidates + if not options[member.case_id]: + self._cache[cache_key] = "" + return "" + + viable: list[str] = [] + for resource in sorted(options[target.case_id]): + remaining = [ + member for member in members if member.case_id != target.case_id + ] + if self._has_matching(remaining, options, {resource}): + viable.append(resource) + resolved = viable[0] if len(viable) == 1 else "" + self._cache[cache_key] = resolved + return resolved + + def _has_matching( + self, + remaining: list[Case], + options: dict[str, set[str]], + used: set[str], + ) -> bool: + if not remaining: + return True + current = min( + remaining, + key=lambda item: len(options[item.case_id] - used), + ) + available = options[current.case_id] - used + if not available: + return False + rest = [ + item for item in remaining if item.case_id != current.case_id + ] + return any( + self._has_matching(rest, options, used | {resource}) + for resource in sorted(available) + ) diff --git a/code/tests/test_engine.py b/code/tests/test_engine.py index 2c326e8..97861c0 100644 --- a/code/tests/test_engine.py +++ b/code/tests/test_engine.py @@ -14,9 +14,11 @@ sys.path.insert(0, str(CODE_DIR)) +from ghosttrace.domain import parse_time # noqa: E402 from ghosttrace.engine import GhostTraceEngine, OPERATION_ORDER, PHASE_ORDER # noqa: E402 from ghosttrace.intent import authorizes_deletion # noqa: E402 from ghosttrace.validation import validate_output # noqa: E402 +from ghosttrace.workflow import RESULT_FOR_STATUS # noqa: E402 from main import write_output # noqa: E402 @@ -107,6 +109,86 @@ def test_unambiguous_workflow_account_is_in_future_verification(self) -> None: self.assertEqual(matching[0].operation, "VERIFY_RECORD_ABSENT") self.assertEqual(matching[0].evidence_id, "") + def test_batch_workflow_solver_resolves_only_invariant_targets(self) -> None: + cases = { + item.case_id: item + for item in self.engine._load_cases( + self.dataset / "sample_deletion_requests.csv" + ) + } + self.assertEqual( + self.engine.workflow.account_resource(cases["case_18"]), + "account_5521", + ) + self.assertEqual( + self.engine.workflow.account_resource(cases["case_20"]), + "account_5037", + ) + for ambiguous in ("case_25", "case_26", "case_28", "case_29", "case_31", "case_32"): + self.assertEqual(self.engine.workflow.account_resource(cases[ambiguous]), "") + + def test_batch_workflow_solver_never_returns_conflicting_owner(self) -> None: + for members in self.engine.workflow.groups.values(): + for case in members: + resource = self.engine.workflow.account_resource(case) + if not resource: + continue + identity = self.engine.identities.resolve(case) + self.assertIsNotNone(identity) + aliases = self.engine.identities.aliases_at_time( + identity, + case.audit_at, + case.email, + ) + self.assertNotEqual( + self.engine.evidence.ownership_relation( + resource, aliases, case.audit_at + ), + "CONFLICT", + case.case_id, + ) + + def test_batch_workflow_solver_matches_independent_subject_evidence(self) -> None: + validated = 0 + for members in self.engine.workflow.groups.values(): + for case in members: + resource = self.engine.workflow.account_resource(case) + identity = self.engine.identities.resolve(case) + expected_result = RESULT_FOR_STATUS.get(case.reported_status) + if not resource or identity is None or expected_result is None: + continue + candidates = self.engine.workflow.account_events.get( + (case.reported_at, expected_result), set() + ) + independently_linked = { + candidate + for candidate in candidates + if any( + observation.subjects & identity.aliases + for observation in self.engine.evidence.observations_by_resource.get( + candidate, [] + ) + ) + } + if len(independently_linked) != 1: + continue + validated += 1 + self.assertEqual(resource, next(iter(independently_linked)), case.case_id) + self.assertGreaterEqual(validated, 150) + + def test_structured_log_preserves_source_event_time(self) -> None: + matching = [ + item + for item in self.engine.evidence.observations + if item.evidence_id == "evidence_94" + and item.resource_id == "account_3610" + ] + self.assertEqual(len(matching), 1) + self.assertEqual( + matching[0].observed_at, + parse_time("2026-09-26T08:37:56Z"), + ) + def test_unique_workflow_target_requires_independent_anonymized_readback(self) -> None: cases = { item.case_id: item diff --git a/log.txt b/log.txt index ff29f1f..f4a9f2d 100644 --- a/log.txt +++ b/log.txt @@ -284,3 +284,47 @@ branch=codex/ghosttrace-accuracy-hardening repo_root=/Users/amoghrb/Documents/ChatGPT/halloween hackathon worktree=/Users/amoghrb/Documents/ChatGPT/halloween hackathon parent_agent=none + +## [2026-11-04T22:11:13+05:30] Realistic scoring headroom assessed + +User Prompt (verbatim, secrets redacted): +is this the maximum we can realistically achieve or is there more + +Agent Response Summary: +Assessed the merged release against the published judging dimensions and measured evaluation results. Concluded that the system is production-complete and already maximizes public verdict accuracy, but is not yet the realistic scoring ceiling because exact row and sequence accuracy remain 50%. Identified generalized cross-case resource assignment and stronger evidence-graph reconciliation as the remaining high-value, integrity-preserving path rather than case hardcoding. + +Actions: +* Reviewed `problem_statement.md`, `WINNING_STRATEGY.md`, and `code/evaluation/evaluation_report.md`. +* Compared the published judging dimensions with the current public metrics and documented evidence gaps. + +Context: +tool=Codex desktop +model=GPT-5 +branch=main +repo_root=/Users/amoghrb/Documents/ChatGPT/halloween hackathon +worktree=/Users/amoghrb/Documents/ChatGPT/halloween hackathon +parent_agent=none + +## [2026-11-04T22:32:22+05:30] Cutoff-safe ultimate attribution hardening completed + +User Prompt (verbatim, secrets redacted): +we must max out everything to win go ultimate god mode and become the elite version of urself and max out everything as much as possible for winning this hackathon proceed with highest engineering standards + +Agent Response Summary: +Implemented the highest-value remaining generalized accuracy improvement without hardcoding public labels or violating the evidence cutoff. GhostTrace now solves anonymous workflow targets through invariant one-to-one batch assignment, preserves source-level log timestamps, and independently validates 164 attributed targets with zero mismatches. The full system was rebuilt and passed its complete contract, evaluation, archive-integrity, and clean-checkout release gates. + +Actions: +* Added a cutoff-aware bipartite workflow attribution solver with conflict rejection, explicit anchors, and invariant-only resolution. +* Added temporal identity views, indexed ownership reconciliation, and source event-time parsing for structured logs. +* Expanded regression coverage to 21 tests, including independent assignment precision and temporal safety gates. +* Improved public step-field accuracy to 80.118%, step precision to 81.034%, and step recall to 73.057% while retaining 100% status accuracy. +* Generated and validated all 518 output rows: 148 COMPLETE, 360 INCOMPLETE, and 10 NOT_APPLICABLE. +* Rebuilt and integrity-tested the dataset-free 20-file code.zip, then reproduced all results from a fresh zero-cache clone. + +Context: +tool=Codex desktop +model=GPT-5 +branch=codex/ghosttrace-ultimate-hardening +repo_root=/Users/amoghrb/Documents/ChatGPT/halloween hackathon +worktree=/Users/amoghrb/Documents/ChatGPT/halloween hackathon +parent_agent=none