Goal: make a session findable by the file path and the command, not only by their output.
_flatten_content indexes text and tool_result, and drops tool_use inputs. So you can
find a session by what a command printed, but not by the command. Same for a file: by its
contents, not its path.
Fix: index the identifying fields of a tool_use input (file_path, command, pattern,
query, url). About 3.2% of transcript bytes.
Already measured, do not redo
Numbers from a ~740-session corpus, freshly built index, so compare arms with each other and
not with the published gold and held-out figures.
- Tool output is 71.6% of the index. Bash is 55.5% of it, Read 37.8%, WebSearch 1.5%.
- Keep tool output. Dropping it takes tool-content recall to zero (@1 0.283 to 0.000).
- No blanket cap. Capping at 1500 saves 24% of index size and costs 12% of tool-content
recall. Not worth it.
- No per-tool caps. Bash and Read are the biggest, and also the most useful: test
failures, stack traces, and the session that touched a symbol.
- Snippets are fine. 88% of displayed evidence lines already come from typed text.
Bigger win, separate
Per-keystroke cost is mostly reading sessions.tsv (0.091s of 0.246s) and _at_word_start.
Reading the TSV as bytes and counting with a compiled \b regex was estimated at 0.33s to
0.10s, with no change to ranking.
Goal: make a session findable by the file path and the command, not only by their output.
_flatten_contentindexestextandtool_result, and dropstool_useinputs. So you canfind a session by what a command printed, but not by the command. Same for a file: by its
contents, not its path.
Fix: index the identifying fields of a
tool_useinput (file_path,command,pattern,query,url). About 3.2% of transcript bytes.Already measured, do not redo
Numbers from a ~740-session corpus, freshly built index, so compare arms with each other and
not with the published gold and held-out figures.
recall. Not worth it.
failures, stack traces, and the session that touched a symbol.
Bigger win, separate
Per-keystroke cost is mostly reading
sessions.tsv(0.091s of 0.246s) and_at_word_start.Reading the TSV as bytes and counting with a compiled
\bregex was estimated at 0.33s to0.10s, with no change to ranking.