fix(store): split sibling fields back out when they leak into content - #45
Merged
Merged
Conversation
When a caller closes a long `content` argument with the wrong tag (`</content>`, a bare `<summary>`, and in a few cases even after a correct `</parameter>`), the tool-call parser keeps reading: summary / importance / category / tags land inside `content` or `summary` as raw XML, and the real fields fall back to their zod defaults (importance 6, category general). The row looks fine and ranks wrong forever. An instruction telling agents to close tags carefully held for months and the leak kept happening, so the repair now lives at the write path. - tag-leak-repair.mjs: structural detection — a wrong closer followed by a field tag, AND the whole tail must parse as field tags to the end of the string. Backticked mentions and closers followed by prose pass untouched, so a memory that describes this bug is stored verbatim. - Merge rule: fields zod defaults when absent (importance, category, memory_type, memory_level) take the leaked value; summary / tags / supersedes only fill in when the passed value is empty. - store_memory reports 🩹 with the fields it moved back; a leak-shaped tail that doesn't parse cleanly gets⚠️ and no data change. - Unit test for the parser, integration test through the MCP handler; both wired into CI. Co-Authored-By: 千夏 <qianxia@clawgamers.com>
Three problems found in review, each reproduced before fixing: - False positives on markup. The tail parser accepted any closing tag, so content ending in an Atom entry or HTML fragment (`<content>…</content> <summary>…</summary></entry>`) parsed as a leak and was truncated. Now: only field / `parameter` closers and the tool-call envelope are accepted in a tail; a closer that closes an element opened earlier in the text is markup; and text is only cut when at least one valid field is recovered and something of the original is left. - Explicit values overridden. A leaked importance/category replaced the passed value unconditionally. It now only replaces an empty value or one equal to the zod default; when content and summary leak the same field, the content-side value wins. - Quadratic scan. Every leak-shaped start re-parsed the whole remaining tail with fresh slices; 312 KB of repetitive near-leak took 11 s and blocked the event loop before quarantine routing. Tail tokens now match in place with sticky regexes, and a failed parse skips later starts that would hit the same token: the same input takes ~4 ms. Tests: markup cases, explicit-value survival, empty head, content/summary conflict, the remaining field types, a size/time bound, and the quarantine path end to end. Co-Authored-By: 千夏 <qianxia@clawgamers.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A long
store_memoryargument closed with the wrong tag —</content>instead of</parameter>, or a bare<summary>— makes the tool-call parser keep reading. The sibling fields end up insidecontent(orsummary) as raw XML:and the real
summary/importance/categoryarrive empty, so zod fills in its defaults. The write succeeds, the row has no summary line, and an importance-9 decision is stored as a 6. Nothing reports it.A full scan of one production store turned up 47 live rows like this, spread over five months — despite an agent-side instruction to close tags carefully that was in place the whole time. Some were on the
summaryside (…</summary><parameter name="importance">7), and three followed a correct</parameter>, so "be careful with tags" was never going to be enough.What
tag-leak-repair.mjs— a pure function run at the top of thestore_memoryhandler, before quarantine routing or the write.Detection is structural, not substring: a wrong closer immediately followed by a field tag, and the whole tail has to parse as field tags to the end of the string. That distinction matters because the rows most likely to contain these strings are the ones describing the bug:
body</content><parameter name="summary">Sbody, summaryS…a tail like `</content><parameter name="summary">…` then more prose…</content>, which then corrupts importancex</content><parameter name="summary">S</parameter> and more textMerge rule. For fields zod defaults when they're absent (
importance,category,memory_type,memory_level), the passed value is the default standing in for the one that leaked, so the leaked value wins. Fields with no default (summary,tags,supersedes, …) keep a non-empty passed value. Invalid values (unknown category, importance 42) are dropped rather than applied.Feedback. The response gains one line:
Tests
tag-leak-repair.test.mjs: the parser against leak shapes taken from real corrupted rows, plus prose that must pass untouchedtag-leak.integration.test.mjs: spins up the HTTP server on a temp DB, sends a leaked write and a prose write throughstore_memory, and checks the stored rowsExisting rows aren't migrated here. The same function can be run over a store as a one-off (dry-run first, and clear
content_vectoron rows whose content changed so they get re-embedded).🤖 Generated with Claude Code