Environment: memtrace 1.0.7, Apple M2 Max (12-core, 64 GB), macOS 25.5.0,
local MemDB, embedder jina-embeddings-v2-base-code/int8 (dim 768, matches
memdb_dim). Private Python repo: ~1,500 files / 26,670 symbols / 5,281 commits
of history. Graph built clean: 27,959 nodes / 123,356 edges at HEAD;
161,979 nodes / 527,020 edges including replayed history.
Repo name, symbol names, and paths below are redacted as <repo>, <ClassA>,
<func_b>, pkg/.... Log excerpts are otherwise verbatim.
Four defects, found during a single clean full-index run. #1 is a correctness
defect and is by far the most serious. #2–#4 share a different root cause:
background phases report no state, so "working", "finished", and "failed" are
indistinguishable from the outside.
1. Relationship queries return WRONG results or hang indefinitely (highest severity)
On a class with four plain from X import Y statements across three files,
verified by grep:
pkg/server/rest.py:33 from pkg.server.executor import <ClassA>
pkg/adapters/ipc/zmq/api_server.py:18 from pkg.server.executor import <ClassA>
pkg/adapters/ipc/zmq/api_server.py:285 from pkg.server.executor import <ClassA>
pkg/frameworks/scheduler/operation_tool.py:204 from pkg.server.executor import <ClassA>
find_symbol("<ClassA>") locates the class correctly at the right file and line,
so the symbol is unambiguously in the graph. Its _pre_edit_check payload says:
"_pre_edit_check": { "complexity_score": 1, "direct_callers": 0,
"risk_level": "low", "warning": "" }analyze_relationships on that same symbol:
query_type |
Result |
imports |
returns count: 0 — wrong, ≥3 files import it |
exporters |
returns count: 0 — wrong |
overrides |
returned instantly once, timed out on a later identical call — non-deterministic |
find_callers |
no response in 280 s |
find_callees |
no response in 60 s |
class_hierarchy |
no response in 60 s |
type_usages |
no response in 60 s |
get_impact (incl. depth: 1, explicit branch) |
no response in 200 s |
The direction semantics of imports vs exporters are ambiguous from the API,
but both return zero, and one of the two directions must be non-zero — so the
import edges are not queryable at all.
Ruled out as causes:
- Not graph scale. A tiny leaf function (
<func_b>, ~1 caller, in a small
module) times out identically on find_callers. If traversal cost scaled with
the target's neighbourhood, this would return instantly.
- Not backend load.
imports/exporters return instantly on the same symbol
at the same moment. An invalid query_type also errors instantly, proving the
server is responsive.
- Not a transient. Reproduced across two separate daemon instances (before and
after a full restart), on a freshly built index.
- Not string-dispatch invisibility. This codebase does use string-based RPC
dispatch that an AST graph legitimately cannot see — but these are direct
import statements, the simplest possible edge.
Impact. This is worse than a hang, because the failing modes are silent. A
class instantiated by both the REST server and the ZMQ API server is reported as
direct_callers: 0, risk_level: low. Tooling and agents that consult Memtrace
before refactoring — which is exactly the documented use case, and what the
bundled UserPromptSubmit hook steers coding agents toward — will conclude the
symbol is unused and safe to change or delete. find_symbol and find_code are
reliable; the relationship layer currently is not, and nothing in the output
signals the difference.
Suggested fix: whatever the root cause, count: 0 should not be returned for
a symbol whose edges failed to resolve — distinguish "no edges" from "edge lookup
failed/unavailable" in the response, so callers can tell absence of evidence from
evidence of absence.
2. embed+replay FAILED: 0/0 successful reported after a SUCCESSFUL run
After ~4 hours (14477588 ms), Phase 2 terminated with:
✗ [1/1] <repo> embed+replay FAILED: 0/0 successful - 870 episodes (14477588 ms)
✗ Bootstrap finished with 1 Phase-2 failure(s) (0 succeeded):
✗ <repo> - 0/0 successful (zero successful embeddings on a non-empty work-set)
The same log shows ~50 pages embedded immediately beforehand, with no errors:
· Embedding 464 symbols (48 trivial skipped) [model=jina-embeddings-v2-base-code/int8, inference_batch=64, producer_pack=64]
· Embedding 502 symbols (10 trivial skipped) [...]
· [1/1] <repo> — embedding 12288/12800 (96%, 0 inference remaining)
Circuit breaker read Closed (never tripped); memory pressure normal.
The work had in fact succeeded. On the next daemon start, the engine itself
reported:
◆ Embedding index incomplete - the engine is repairing it from existing vectors
in the background (resumable, non-blocking; no re-embed, no API stall).
"Repairing from existing vectors" confirms the vectors persisted — only the HNSW
index was left incomplete. Semantic search also worked before that restart:
find_code("rate limit backoff") returned 20 correct, well-ranked results.
Note 0/0 has zero in both numerator and denominator, which suggests the
work-set counter is never incremented, and the "non-empty work-set" guard then
fires on the empty total.
Impact: the message states a 4-hour job failed when it succeeded. The obvious
user response is memtrace start --fresh, which wipes a good store and costs
another 4 hours. We nearly did exactly that.
Suggested fix: report incomplete-index state as a distinct outcome
(PARTIAL — index repair pending), not FAILED; and correct the work-set counter
so the guard cannot fire on a successful run.
3. Dashboard shows "No symbols indexed yet — Run memtrace index ." while indexing is running
Whenever the backend is busy, /api/health nulls out the graph counts:
"graph": { "backend_busy": true, "node_count": null, "edge_count": null,
"repository_count": null }/api/repos, queried at the same instant, returns the truth:
{ "repo_id": "...", "node_count": 34375, "edge_count": 156824, "counts_fresh": false }The dashboard cannot distinguish "counts unavailable" from "counts are zero", so
it renders its empty state and advises memtrace index . — i.e. it tells the user
to start a second indexer while one is already running.
Suggested fix: have the UI branch on backend_busy and render "indexing in
progress" rather than the empty state; or serve the last-known counts with
counts_fresh: false, as /api/repos already does.
4. No progress reporting for any background phase
Phase 1 has excellent staged progress:
scan · Scanned 2262 files · 10%
parse · Parsed 32683 symbols · 25%
resolve · Resolved 92311 edges · 42%
community · Found 1522 modules · 75%
Phase 2 (embed + git-replay) and the post-restart HNSW repair have none:
/api/jobs returns [] during both — the work is not registered as a job/api/health nulls all counts while busymemtrace insight-card <repo> returns empty output while busymemtrace status states outright: "status never opens a local MemDB store"- The log printed nothing for the first ~140 minutes of Phase 2, and nothing at
all for 65+ minutes of HNSW repair
The only way to distinguish progress from a hang was to watch memcore-server
CPU and poll on-disk store size externally. During the HNSW repair even that is
misleading — the store shrinks (14 GB → 8.3 GB, compaction), so a growth-based
heuristic reads as "stalled".
Where progress IS printed, the denominator moves: the embedding counter reads
0/512, then 512/1024, then 1024/1536, i.e. processed / seen-so-far, so the
percentage climbs toward 100% regardless of how much work remains and cannot be
used to estimate completion.
Suggested fix: register background phases in /api/jobs with a phase name and
a completed/total counter. That single change would resolve #3 and #4, and would
have made #2 self-evident.
What worked well
Scoping via .memtraceignore behaved exactly as documented for the code graph:
after excluding a vendored subtree, find_symbol correctly returned nothing for a
class that exists only there, while a re-included subtree's symbols resolved
correctly. The restart path correctly detected a large existing store and skipped
re-indexing (seconds, not hours). find_symbol agreed with grep on every symbol
tested. find_code returned high-quality, well-ranked semantic results.
One scoping caveat: find_code returns files from paths excluded by
.memtraceignore (e.g. node_modules-adjacent package.json files in an excluded
subtree) and from outside the include-list entirely (docs/, config/, tests/).
The code graph honours the ignore file; the full-text layer backing find_code
appears not to. Worth documenting either way, since the two layers currently have
different scopes.
Environment: memtrace 1.0.7, Apple M2 Max (12-core, 64 GB), macOS 25.5.0,
local MemDB, embedder
jina-embeddings-v2-base-code/int8(dim 768, matchesmemdb_dim). Private Python repo: ~1,500 files / 26,670 symbols / 5,281 commitsof history. Graph built clean: 27,959 nodes / 123,356 edges at HEAD;
161,979 nodes / 527,020 edges including replayed history.
Repo name, symbol names, and paths below are redacted as
<repo>,<ClassA>,<func_b>,pkg/.... Log excerpts are otherwise verbatim.Four defects, found during a single clean full-index run. #1 is a correctness
defect and is by far the most serious. #2–#4 share a different root cause:
background phases report no state, so "working", "finished", and "failed" are
indistinguishable from the outside.
1. Relationship queries return WRONG results or hang indefinitely (highest severity)
On a class with four plain
from X import Ystatements across three files,verified by grep:
find_symbol("<ClassA>")locates the class correctly at the right file and line,so the symbol is unambiguously in the graph. Its
_pre_edit_checkpayload says:analyze_relationshipson that same symbol:query_typeimportscount: 0— wrong, ≥3 files import itexporterscount: 0— wrongoverridesfind_callersfind_calleesclass_hierarchytype_usagesget_impact(incl.depth: 1, explicitbranch)The direction semantics of
importsvsexportersare ambiguous from the API,but both return zero, and one of the two directions must be non-zero — so the
import edges are not queryable at all.
Ruled out as causes:
<func_b>, ~1 caller, in a smallmodule) times out identically on
find_callers. If traversal cost scaled withthe target's neighbourhood, this would return instantly.
imports/exportersreturn instantly on the same symbolat the same moment. An invalid
query_typealso errors instantly, proving theserver is responsive.
after a full restart), on a freshly built index.
dispatch that an AST graph legitimately cannot see — but these are direct
importstatements, the simplest possible edge.Impact. This is worse than a hang, because the failing modes are silent. A
class instantiated by both the REST server and the ZMQ API server is reported as
direct_callers: 0, risk_level: low. Tooling and agents that consult Memtracebefore refactoring — which is exactly the documented use case, and what the
bundled
UserPromptSubmithook steers coding agents toward — will conclude thesymbol is unused and safe to change or delete.
find_symbolandfind_codearereliable; the relationship layer currently is not, and nothing in the output
signals the difference.
Suggested fix: whatever the root cause,
count: 0should not be returned fora symbol whose edges failed to resolve — distinguish "no edges" from "edge lookup
failed/unavailable" in the response, so callers can tell absence of evidence from
evidence of absence.
2.
embed+replay FAILED: 0/0 successfulreported after a SUCCESSFUL runAfter ~4 hours (
14477588 ms), Phase 2 terminated with:The same log shows ~50 pages embedded immediately beforehand, with no errors:
Circuit breaker read
Closed(never tripped); memory pressurenormal.The work had in fact succeeded. On the next daemon start, the engine itself
reported:
"Repairing from existing vectors" confirms the vectors persisted — only the HNSW
index was left incomplete. Semantic search also worked before that restart:
find_code("rate limit backoff")returned 20 correct, well-ranked results.Note
0/0has zero in both numerator and denominator, which suggests thework-set counter is never incremented, and the "non-empty work-set" guard then
fires on the empty total.
Impact: the message states a 4-hour job failed when it succeeded. The obvious
user response is
memtrace start --fresh, which wipes a good store and costsanother 4 hours. We nearly did exactly that.
Suggested fix: report incomplete-index state as a distinct outcome
(
PARTIAL — index repair pending), notFAILED; and correct the work-set counterso the guard cannot fire on a successful run.
3. Dashboard shows "No symbols indexed yet — Run
memtrace index ." while indexing is runningWhenever the backend is busy,
/api/healthnulls out the graph counts:/api/repos, queried at the same instant, returns the truth:{ "repo_id": "...", "node_count": 34375, "edge_count": 156824, "counts_fresh": false }The dashboard cannot distinguish "counts unavailable" from "counts are zero", so
it renders its empty state and advises
memtrace index .— i.e. it tells the userto start a second indexer while one is already running.
Suggested fix: have the UI branch on
backend_busyand render "indexing inprogress" rather than the empty state; or serve the last-known counts with
counts_fresh: false, as/api/reposalready does.4. No progress reporting for any background phase
Phase 1 has excellent staged progress:
Phase 2 (embed + git-replay) and the post-restart HNSW repair have none:
/api/jobsreturns[]during both — the work is not registered as a job/api/healthnulls all counts while busymemtrace insight-card <repo>returns empty output while busymemtrace statusstates outright: "status never opens a local MemDB store"all for 65+ minutes of HNSW repair
The only way to distinguish progress from a hang was to watch
memcore-serverCPU and poll on-disk store size externally. During the HNSW repair even that is
misleading — the store shrinks (14 GB → 8.3 GB, compaction), so a growth-based
heuristic reads as "stalled".
Where progress IS printed, the denominator moves: the embedding counter reads
0/512, then512/1024, then1024/1536, i.e.processed / seen-so-far, so thepercentage climbs toward 100% regardless of how much work remains and cannot be
used to estimate completion.
Suggested fix: register background phases in
/api/jobswith a phase name anda completed/total counter. That single change would resolve #3 and #4, and would
have made #2 self-evident.
What worked well
Scoping via
.memtraceignorebehaved exactly as documented for the code graph:after excluding a vendored subtree,
find_symbolcorrectly returned nothing for aclass that exists only there, while a re-included subtree's symbols resolved
correctly. The restart path correctly detected a large existing store and skipped
re-indexing (seconds, not hours).
find_symbolagreed with grep on every symboltested.
find_codereturned high-quality, well-ranked semantic results.One scoping caveat:
find_codereturns files from paths excluded by.memtraceignore(e.g.node_modules-adjacentpackage.jsonfiles in an excludedsubtree) and from outside the include-list entirely (
docs/,config/,tests/).The code graph honours the ignore file; the full-text layer backing
find_codeappears not to. Worth documenting either way, since the two layers currently have
different scopes.