list_indexed_repositories hangs indefinitely (reproduces on clean state); separately, 1.0.9 upgrade duplicates every watch entry

[hrabbach]

Version: memtrace 1.0.9 (upgraded 2026-08-10 15:34 local; the prior version had been running since at least 08-04)
Platform: Windows 11 Pro 10.0.26200, Node v24.14.0, 16-core
MCP client: Claude Code (300 s tool idle timeout)
Backend: memdb, workspace store at C:\Users\<user>\Code\.memdb, 12 watched repos

Summary

The 1.0.9 upgrade re-registered every watched repository using Win32 extended-length (\\?\) paths without removing the pre-existing plain-path entries. watches.json now holds 24 entries for 12 repositories — each repo registered twice, under two spellings of the same directory.

After this, list_indexed_repositories never returns. The client aborts it at the 300 s idle timeout. Every per-repo scoped call against the same store in the same daemon returns instantly, so the database itself is fine — it is repository enumeration that is broken.

This is a total-outage failure mode, because list_indexed_repositories is documented as the mandatory first call and the scoped tools refuse to proceed without the repo_id it would return.

Evidence: the duplication is exactly 1:1 and timestamped to the upgrade

~/.memtrace/watches.json:

total watch entries : 24
distinct repo_ids   : 12
plain-path entries  : 12
\\?\-prefixed       : 12

Every one of the 12 repos appears exactly twice. Same repo, same branch, two path spellings:

{ "path": "C:\\Users\\<user>\\Code\\my-repo",
  "repo_id": "my-repo", "branch": "master",
  "registered_at": "2026-08-04T10:20:37.536446200+00:00", "origin": "manual" },

{ "path": "\\\\?\\C:\\Users\\<user>\\Code\\my-repo",
  "repo_id": "my-repo", "branch": "master",
  "registered_at": "2026-08-10T14:35:59.413590300+00:00", "origin": "manual" }

2026-08-10T14:35:59Z is the upgrade, to the second: the 1.0.9 binary has mtime 08-10 15:34:56 local and the daemon started 15:35:36 local (= 14:35:36 UTC). All 12 \\?\ entries were written in a 160 ms burst 23 s after daemon start.

runtime.json corroborates the switch — its workspaceMembers are now exclusively \\?\-prefixed, while the pre-existing watch entries are not.

Symptom 1 — list_indexed_repositories hangs, and never even logs

Task failed: MCP server "memtrace" tool "list_indexed_repositories" sent no response
or progress for 300s; aborting.

Reproduced twice, ~35 min apart, plus once independently by a separate agent process.

memcore-server.log contains no entry for any of these calls. Its last line is 15:19:41; the calls were at ~15:52 and ~16:55. The call does not reach a point where it logs anything — consistent with blocking on lock acquisition before doing any work. If enumeration takes a per-repo lock and the repo list contains the same store twice, a non-reentrant lock self-deadlocks on the second acquisition.

Symptom 2 — same DB, scoped call, instant

Immediately after the aborted call:

get_repository_stats(repo_id="my-repo")   → instant

{"_backend":"memdb","repo_id":"my-repo","branch":"master",
 "total_nodes":114222,"total_edges":234195,"community_count":415,
 "episode_count":655,"symbol_node_count":4039,"indexing_incomplete":false}

mem_diag also returns instantly. So: process-level calls fine, per-repo scoped reads fine, enumeration hangs forever.

Symptom 3 — 12.3 % of a core, sustained, for 25 hours

Daemon	single instance on 127.0.0.1:3030
Uptime	25.4 h
CPU consumed	11,206 s (~12.3 % of one core, continuously)
RSS	493 MB

memcore-server.log shows index maintenance: pruned dead/cold rids from prop_index firing every ~5 minutes for 25 hours without pause, dropping up to 32,307 rids in a single pass:

14:39:16 dropped 500      15:14:39 dropped 1027
14:44:18 dropped 173      15:19:41 dropped 23799
13:58:57 dropped 32307    09:57:32 dropped 16263

24 watchers on 12 directories means every file save is ingested twice, which would produce exactly this churn. Note live_records: 2081697 against total_nodes: 114222 — ~18× more live records than nodes.

Possibly contributing — startup served traffic with structural indexes unbuilt

From the 08-10 boot:

WARN structural index fast-path unavailable on large store; deferring full iter_live repair
     until after startup
     db=memtrace live_records=2081697 wal_high_water_bytes=36637
     rid_uuid=false edge_adj=false count_store=true property_idx=false
WARN deferred structural index repair started; UI/API were already ready
WARN deferred primary record-index checkpoint started; UI/API were already ready
WARN deferred HNSW recovery incomplete vs durable VectorBlobs — repairing the missing vectors
     node_count=4449 live_vectorblobs=0

Three structural indexes reported unavailable while the API was already accepting requests. If enumeration depends on property_idx or rid_uuid and falls back to a full iter_live scan over 2.08 M records — twice, once per duplicate registration — that is another route to the same unbounded hang.

Impact

list_indexed_repositories is documented as:

ALWAYS call this first to discover available repo_ids — most other tools require a repo_id.

and the scoped tools refuse to infer one:

{"error_code":"repo_scope_required","scope_required":true,"scope_unresolvable":true,
 "diagnostic":{"message":"No explicit repo_id/scope was provided, and the current workspace does not identify one safe repository."}}

So an agent following the documented flow is fully blocked: discovery hangs for 300 s, then every downstream tool refuses for want of the repo_id it never got. Observed directly — an agent burned its full 300 s budget here and silently fell back to ripgrep, degrading its results without the user knowing why.

Workaround (confirmed)

Skip discovery; pass repo_id explicitly. It can be read from disk:

// <workspace>/.memdb/.memtrace-store-scope.json
{"version":1,"members":[{"repo_id":"my-repo","path":"c:/Users/<user>/Code/my-repo"}]}

All scoped tools then behave normally. Only usable by someone who knows the file exists.

Suggested fixes

1. Make watch registration idempotent by canonical path. Normalise \\?\-prefixed and plain Windows paths to one canonical form before registering, and de-duplicate on upgrade. This is the root cause.
2. Ship a one-time migration that collapses existing duplicate entries — users upgrading from pre-1.0.9 are already in this state and have no signal that anything is wrong.
3. Bound list_indexed_repositories. It should fail fast with a diagnostic rather than hang unbounded, given it sits behind an "ALWAYS call this first" instruction.
4. Emit progress events during enumeration so MCP idle timeouts do not fire on merely-slow work.
5. Let scoped tools fall back to .memtrace-store-scope.json when enumeration is unavailable. Returning scope_unresolvable while that file names exactly one member turns a recoverable state into a fatal one.

Minor issues noticed while diagnosing

• ~/.memtrace/last-run-version still reads 0.6.20 — never updated across many upgrades, including this one. If anything gates migrations on it, those migrations are not running.
• Stale daemon.pid in the per-repo .memdb directory points at a PID from three days earlier that is no longer running, while the workspace-level .memdb/daemon.pid is correct.
• Index 5 days stale. last_episode_time = 2026-08-06T23:13:40Z and last_indexed = null on 08-11, despite continuous work in a watched repo and 24 active watchers. indexed_node_count is null while total_nodes is populated.

[blocksorg]

Mention Blocks like a regular teammate with your question or request:

@blocks can you add technical details to this issue?
@blocks make the following changes ...
@blocks create an issue from what was mentioned in the following comment ...
@blocks explain the following code ...
@blocks are there any security or performance concerns?

Run @blocks /help for more information.

Workspace settings | Disable this message

[hrabbach]

Correction: the watch duplication is not the cause of the hang

I filed this attributing the hang to the duplicated \\?\ watch entries. That diagnosis is wrong and I want to correct it before it sends anyone down the wrong path.

What I tested

Deduplicated watches.json from 24 entries back to 12 (kept the \\?\ form), deleted all stale pid files and all 25 .lock files with the daemon stopped, then restarted cleanly.

State after restart, verified:

watches.json entries : 12
duplicate repo_ids   : 0
plain-path entries   : 0     (the dedupe held — nothing re-added the old form)
stale pid files      : 0
stale lock files     : 0

The daemon came up clean and finished all deferred work:

16:19:59 INFO  MemDB ready  ready_in_ms=48588  bind=127.0.0.1:50051
16:20:30 INFO  deferred HNSW VectorBlob rebuild completed  seen=9957 inserted=1748 skipped=8209
16:21:41 INFO  deferred primary record-index checkpoint completed  elapsed_ms=101645

Result: still hangs, identically

list_indexed_repositories called after all deferred work completed still never returns — client aborts at the idle timeout, and memcore-server.log again contains no entry for the call at all.

get_repository_stats(repo_id=…) on the same daemon remains instant.

So the discriminator in the original report holds — enumeration hangs, scoped reads are fine — but the cause is something else.

What this means for the report

• The \\?\ duplication is real and worth fixing on its own merits — 24 watchers on 12 directories, and the prior daemon burned ~12.3 % of a core continuously for 25 h. But it is a separate bug, not the cause of this hang.
• The hang reproduces on a completely clean install state: single daemon, 12 unique watches, no stale locks or pids, all indexes rebuilt. That makes it a straightforward bug in the enumeration path rather than a state-corruption artifact, which should make it easier to reproduce on your side.
• The "suggested fixes" list still stands, but items 3–5 (bound the call, emit progress, allow store-scope fallback) are now the relevant ones; items 1–2 (path canonicalisation and migration) address the separate duplication bug.

One more observation that may be relevant

The graph appears to be per-branch, and enumeration behaviour may interact with that. Same repo, before and after the restart:

// before restart — daemon had been up 25 h, reported the branch it started on
{"branch":"master","total_nodes":114222,"total_edges":234195,
 "community_count":415,"episode_count":655,"indexing_incomplete":false}

// after restart — re-detected current HEAD, a feature branch
{"branch":"gsd/phase-238-…","total_nodes":1,"total_edges":0,
 "community_count":0,"episode_count":0,
 "nodes_by_kind":{"Repository":1},
 "indexed_node_count":114433,"queryable_node_count":1,
 "indexing_incomplete":true,"last_indexed":"2026-08-10T17:29:08Z"}

indexed_node_count: 114433 against queryable_node_count: 1 and indexing_incomplete: true. If list_indexed_repositories tries to reconcile or summarise incompletely-indexed branch graphs across all 12 watched repos, that would be a plausible place for it to block — offered as a lead, not a claim, since I got the last one wrong.

Happy to run any specific diagnostic against this instance; the reproduction is stable.

[hrabbach]

Re-test on 1.1.1 after a from-scratch index rebuild — the hang and the CPU burn are two separate bugs

Windows 11, 16 cores, memtrace 1.1.1. I deleted and fully rebuilt the index from scratch before this
session, so none of the below is contaminated by the stale-state theories in the original report.

Headline: the hang is a deadlock, not resource starvation — and the CPU burn is a completely separate,
UI-driven problem. I was able to isolate them with a controlled test.

1. get_repository_stats has regressed — the discriminator from my last comment no longer holds

In my previous comment the key discriminator was: enumeration hangs, scoped reads are instant. I
wrote that get_repository_stats(repo_id=…) "on the same daemon remains instant", and it returned a
full total_nodes: 114222 master graph immediately.

On 1.1.1 that same call hangs. Same repo, same branch=master, freshly rebuilt index.

It is also not collateral of a wedged enumeration: it was issued as the first MCP call of the
session, on a daemon up for 17 minutes with no enumeration ever issued. It hung past 120s.

So the scoped-read path has gone from instant on 1.0.9 to blocked on 1.1.1, and the blast radius of
this bug has grown rather than shrunk.

2. The hang reproduces on a completely idle daemon

This is the important one. After the CPU burn was eliminated (see §3), I re-ran the same call against a
daemon sitting at 0.0% CPU with nothing else in flight. It still hung past 120s.

So it is not lock contention under load, not thread-pool starvation, and not slow work being mistaken
for a hang. It is a blocked call path.

3. The CPU burn is driven by the web UI, and is a different bug

Sustained memcore-server CPU across versions: 12.3% (1.0.9) → ~104% (1.1.0) → 258–271% of one core
(1.1.1). I had assumed this was intrinsic and scaling with store size. It isn't.

Closing every memtrace web UI browser tab took memcore-server from 271% to 0.0% immediately.

The shape while burning:

Signal	Value
memcore-server CPU	258–271% of one core
memcore-server disk I/O	0 reads/s, 0 writes/s
memcore-server RSS	3.0 GB
memtrace (UI process) CPU	4.4%
gRPC connections on :50051	1

Pure in-memory compute with zero disk I/O, commissioned over a single gRPC connection by a UI process
that is itself nearly idle. The UI view I had open was the one that inspects API connections, which
seems the likely trigger, though I haven't narrowed it to a specific panel.

4. The 5-minute maintenance loop is dead too, and stays dead at 0% CPU

memcore-server.log stops at 17:10:43Z mid-session and never writes again — 28 minutes and roughly
five missed ticks of the index maintenance: pruned dead/cold rids from prop_index cycle at the time of
writing. Last lines:

17:05:42.996Z INFO index maintenance: edge delta frozen into CSR   folded_entries=452386 elapsed_ms=168
17:05:43.252Z INFO index maintenance: pruned dead/cold rids from prop_index   dropped=45413
17:10:43.547Z INFO index maintenance: pruned dead/cold rids from prop_index   dropped=1866

Crucially, the maintenance loop did not resume after CPU went to zero. So the freeze is not the
burn starving it.

Combined with §2, my working hypothesis is that something acquires a lock around 17:10:43Z and never
releases it, and that both the periodic maintenance tick and the full-store aggregate calls need that
lock, while the targeted graph lookups in §5 take a different path.

5. What still works — this narrows it a lot

Not everything is wedged. During the exact window in which get_repository_stats was hanging:

• find_symbol (explicit repo_id) — instant, correct file:line, correct blast-radius envelope
• find_code (hybrid search, explicit repo_id) — instant, sane results
• GET :3030 → 200 in 0.28s
• full POST :3030/mcp initialize handshake → 200 (from the earlier 1.1.0 run)
• mem_diag — instant, but it's answered locally by the MCP child (engine: null, reports the child's
  own 196 MB RSS rather than memcore's 3.0 GB), so it is not evidence that the engine is responsive

So the transport is fine and the graph query path is fine. What wedges is specifically the
full-store aggregate/scan family: get_repository_stats and list_indexed_repositories.

Note this reframes the discriminator from my last comment. It is no longer "enumeration vs scoped" —
get_repository_stats takes an explicit repo_id and still hangs. The line now falls between
aggregate/summarise-the-whole-store calls and targeted graph lookups.

Practically this is a real improvement over 1.1.0 for me: with discovery skipped and repo_id always
passed explicitly, code discovery via find_symbol/find_code is usable again.

6. Bonus: the UI process leaks connections to its own port

With every browser tab closed, memtrace (the UI process) held 124 client connections to its own
:3030, all created within a few seconds of each other, alongside 134 server-side Established on the
same port. Chrome accounted for only 12, and the count kept climbing after the tabs were gone (132 → 139).

That looks like a retry/connection storm inside the UI process that is independent of any tab being
open, and it may well be what drives §3.

7. Watch duplication remains fixed

Confirming the one thing that did get better and has stayed better: 12 entries / 12 distinct
repo_ids / 0 plain-path twins after the rebuild.

Suggested next steps

1. A thread/stack dump of memcore-server while get_repository_stats is wedged would probably settle
   §2 and §4 in one shot — happy to capture one if you can tell me how you'd like it taken on Windows.
2. Bounding the aggregate calls with a timeout + progress emission (fixes 3–5 from the original report)
   would at least turn a silent 300s abort into a diagnosable error.
3. §6 looks independently worth a look regardless of the hang.

[hrabbach]

Root cause found: list_indexed_repositories permanently poisons the daemon

Retracting a key claim from my previous comment. One call to list_indexed_repositories wedges the
entire aggregate path until the daemon is restarted — it is not merely a call that hangs, it is a
call that breaks the daemon for everything else.

The reproduction

Restarted memtrace in headless mode (no browser tab auto-opened), waited for all deferred startup
work to finish, then ran four calls in this order:

#	Call	Result
1	get_repository_stats(repo_id=track-family, branch=master)	✅ instant — total_nodes: 41834, total_edges: 44852, indexing_incomplete: false
2	list_indexed_repositories	❌ hung, aborted at 120s
3	get_repository_stats(repo_id=track-family, branch=master) — identical call to #1	❌ hangs
4	find_symbol(repo_id=track-family)	✅ still instant

Step 1 vs step 3 is the whole finding: the same call, same arguments, same daemon, two minutes apart —
instant before the enumeration, hung after it. The daemon does not recover on its own; only a restart
clears it.

This corrects my previous comment

I claimed the hang was independent of enumeration, on the grounds that get_repository_stats hung as
"the first MCP call of the session". That reasoning was wrong. It was the first call I made, but
the web UI had been open for ten minutes before it, so the daemon may already have been poisoned by
something the UI did before my call ever reached it.

Likewise "the hang reproduces on a completely idle daemon at 0.0% CPU" was a true observation but a
misleading one — that daemon was already poisoned, and closing the UI tabs stopped the CPU burn without
releasing whatever is being held.

I could not identify which UI request does it, and I want to be careful not to repeat the mistake of
naming a cause I haven't proven. What I checked:

• The UI bundle contains no reference to list_indexed_repositories or get_repository_stats —
  it talks to a REST surface, not the MCP tools.
• Its repository listing endpoint is GET /api/repos, and that endpoint is healthy: 200 in 0.10s
  returning 7.9 KB, on the already-poisoned daemon. So it is not the culprit and does not share the
  blocked path.

So: a UI session gets poisoned within ~10 minutes while a headless daemon stays clean, which makes the
UI a trigger, but the specific request is unidentified. Other candidates I have not tested are the
heavier insight endpoints the UI calls (/api/insights/api, /api/insights/architecture,
/api/graph/snapshot).

Worth noting for anyone trying to reproduce from the UI side: those insight views require a repository
to be selected in the UI first, so curling the endpoint bare is unlikely to reproduce whatever the
UI actually does — there is view/selection state involved that a single request doesn't capture. That
is part of why I've left this unattributed rather than guessing again.

The MCP list_indexed_repositories path is the only trigger I have actually proven.

What this also explains

• Why the regression looked like it appeared in 1.1.1. On 1.0.9 I reported get_repository_stats
  as "instant" — that may simply have been measured before the daemon was poisoned in that session,
  rather than evidence of a version difference. Treat my earlier regression claim as unproven.
• Why the 5-minute index maintenance: pruned dead/cold rids from prop_index tick dies mid-session
  and never resumes, even at 0% CPU. Consistent with the enumeration taking a lock the maintenance
  loop also needs, and never releasing it.
• Why targeted lookups stay healthy throughout. find_symbol / find_code clearly don't take
  that lock.

On the CPU burn — partially revised

Headless changes the picture but doesn't eliminate it:

	With UI open	Headless
memcore-server CPU	258–271% of one core	~100%
:3030 Established	134 (124 of them the UI process connecting to itself)	3
memcore-server RSS	3.0 GB	1.4 GB

So the self-connection storm is a UI artifact and goes away headless — but a steady ~100% of one core
with zero disk I/O remains on a clean, fully working, headless daemon (matching the ~104% I measured
on 1.1.0). That burn is present while aggregate calls still work, so it is not the deadlock. Still two
bugs, just with the split drawn in a different place than I had it.

Suggested repro on your side

Start a daemon, call get_repository_stats (works), call list_indexed_repositories, then call
get_repository_stats again (hangs). If that reproduces, the fix surface is narrow: whatever lock or
guard the enumeration path acquires before it blocks, it never releases on the way out.

My workaround meanwhile is to run headless and never call list_indexed_repositories, passing
repo_id explicitly everywhere — which leaves the daemon fully usable.