perf(status): ~4s on remote store is get_records pulling every full live doc to drift-hash — parallelize + fingerprint (follow-up to #28/#31)

[AusafG5]

Follow-up to #28 + #31 (both shipped, thank you). Re-profiled /api/state on 0.4.4 against a remote store and traced the real residual — it is NOT branch_list/pr_list/recent_history (those are 0–290ms or disabled). It is entirely inside status(), and specifically get_records fetching every full live document to hash it for drift detection.

Timings (0.4.4, remote store, warm client)

/api/state handler calls:

call	time
make_engine (cached after 1st)	396ms first / 1ms after
whoami	0ms
status	3840ms
branch_list / pr_list	disabled here (0ms)
recent_history(50)	287ms

Inside status():

sub-call	time
_all_refs(include_history=True)	880ms
_batch_live_and_heads(refs) (310 refs)	1952ms
↳ get_records(collB, 224 ids)	2065ms
↳ get_heads(collB, 224 ids)	563ms
↳ get_records(collA, 75 ids)	276ms
↳ get_heads(collA, 75 ids)	666ms
↳ list_record_ids(collA)	586ms

Root cause

status() computes hash_doc(live) per record to compare against the stored head oid — so it must fetch the full live doc of every tracked record. One collection has 224 records with large docs (nested config/schema/pricing), so get_records alone transfers a few MB = ~2s. This is inherent to "pull live, hash, compare against head." It's not a forgotten projection (that was #28); it's the drift check itself.

Suggestions

1. Parallelize the per-collection get_records/get_heads. perf(ui): /api/state still ~5s on remote store — serialize-to-parallel the remaining per-collection round-trips (follow-up to #28) #31 added _gather, but within _batch_live_and_heads the per-collection full-doc fetches still run such that the ~2s big-collection transfer isn't overlapped with the others. Fanning the collections out concurrently → wall-clock ≈ max(collection) ≈ ~2s instead of ~4s.
2. Avoid transferring full docs for the drift check (the real win). Persist the content hash on the head record at commit time, and detect drift with a cheap server-side comparison instead of pulling+hashing every live doc client-side — e.g. a stored content_hash field on the live doc updated by cfgit writes (compare hashes, only fetch the full doc when they differ / when the user opens a diff). That takes status from "fetch every doc" (~4s) to "compare N hashes" (~200ms). Bigger change; the payoff is status staying flat as configs grow.
3. Lazy status for the UI: /api/state could return the ref list + head metadata immediately and compute live-vs-head drift per-row on demand (when a row is expanded), so first paint isn't blocked on hashing all N docs.

Net: gets remote-store status from ~4s to sub-second and keeps it flat as the number/size of tracked docs grows.

(Profiled on 0.4.4. Timing script available.)

[AusafMo]

Investigated the suggested fix (option 2: persist a content hash on the live doc and compare stamps). Built it end-to-end, then reverted it — it's unsound for cfgit's core purpose. Recording why so we don't reach for it again.

Why a persisted live-doc stamp is unsound

A stamped __cfg_hash__ on the live doc is a "cfgit last wrote X" marker, not proof the doc currently equals X. The whole reason cfgit exists is to catch out-of-band writes — and a raw admin edit changes a field but leaves the stamp untouched. So:

• raw edit sets severity: low directly, __cfg_hash__ stays stale
• stale stamp still == head.oid
• fast-path concludes clean for a record that actually drifted ❌

A false clean in a drift-detection tool is the worst failure mode. A parity test (fast-path status vs. full fetch+hash) confirmed it: identical when the record is untouched, divergent the moment a raw edit lands — exactly the case that matters. Reverted.

Why the other cheap options don't work either

• Server-side md5(doc) — unsound the other direction: cfgit hashes a stripped/normalized view (drops ignore_fields, secret_fields, ignore_patterns, then normalizes). A raw md5 over the full doc flags ignored churn like updated_at as drift → false positives.
• Projection (fetch only non-ignored fields) — measured on a representative config set: ~21% less transfer (9% on the biggest doc type). The bulk of the payload is in the hashed config/schema/pricing fields, not the ignored metadata, so this is ~4s → ~3s, not sub-second.
• Parallelization — already applied (perf(ui): /api/state still ~5s on remote store — serialize-to-parallel the remaining per-collection round-trips (follow-up to #28) #31). The residual is the inherent full-doc fetch, not serial RTTs.

What would actually be correct (and its cost)

Detecting an out-of-band edit fundamentally requires reading the current live content — or a hash the DB computes over the same stripped/normalized view cfgit uses, so it can be compared to head.oid without transferring the doc. That means reimplementing cfgit's strip_for_hash + _normalize + sha256 as a Mongo $function/aggregation and a Postgres expression, kept byte-for-byte in sync with the Python hashing (normalization of numbers/unicode/key-order/secret-drop). Large, adapter-specific, and fragile — its own multi-PR effort with a strong equivalence test harness, not a quick win.

Status

0.4.4 already took remote /api/state from ~20–37s to ~4–5s (#28 + #31). Leaving this open as "needs a sound server-side fingerprint design" rather than shipping a fast-but-wrong drift check. Happy to scope the server-side-hash approach if the remote-store latency is worth that investment.