From d3d334d02c5f12446428cf79168a4e2d1ee2b38f Mon Sep 17 00:00:00 2001 From: anirudhkumar-nanonets Date: Thu, 29 Oct 2026 15:51:24 +0530 Subject: [PATCH 1/2] Stop hook sees Desktop corrections and asks for --personal when it should --- ...e-stop-hook-reads-what-the-anirudhkumar.md | 36 ++++++++ TELEMETRY.md | 1 + src/claude/format.ts | 2 +- src/claude/note-signals.ts | 91 ++++++++++++++----- src/claude/skill-template.ts | 6 +- src/claude/stop.ts | 18 +++- src/cloud/diff.ts | 6 ++ src/telemetry/contract.ts | 10 ++ src/trace/claude.ts | 10 +- test/fixtures/skills/trail/SKILL.md | 6 +- test/telemetry-contract.test.ts | 4 +- test/trace-typed.test.ts | 21 +++++ test/trail-note-ask.test.ts | 60 +++++++++++- 13 files changed, 234 insertions(+), 37 deletions(-) create mode 100644 .trail/learnings/2026-10-29-the-stop-hook-reads-what-the-anirudhkumar.md diff --git a/.trail/learnings/2026-10-29-the-stop-hook-reads-what-the-anirudhkumar.md b/.trail/learnings/2026-10-29-the-stop-hook-reads-what-the-anirudhkumar.md new file mode 100644 index 000000000..056e2ef2c --- /dev/null +++ b/.trail/learnings/2026-10-29-the-stop-hook-reads-what-the-anirudhkumar.md @@ -0,0 +1,36 @@ +--- +title: "The Stop hook reads what the person typed through the shared transcript filter, and counts its asks in note_asked" +author: Anirudhkumar-nanonets +date: 2026-10-29 +branch: claude/fix-save-learning-prompts-84888d +cost: { minutes: 1, tokens: 7385 } +id: l-523d6adc +scope: repo +status: active +src: agent +touches: + - src/claude/note-signals.ts + - src/trace/claude.ts + - src/claude/skills-wiring.ts + - test/trail-words.test.ts + - src/skills/risk.ts + - src/claude/stop.ts + - src/cloud/diff.ts + - src/telemetry/contract.ts +--- + +## Decided +- The save ask's `typed()` (src/claude/note-signals.ts) calls `typedText` from src/trace/claude.ts, the filter every other learning path uses. Its own copy dropped any message starting with `<`, and Claude Desktop puts a `` status block before every message the person types, so in Desktop sessions no correction or option pick was ever seen. +- `ownText` (src/trace/claude.ts) opens an artifact comment ("[Artifact comment sent to Claude]" … `=== BEGIN ARTIFACT COMMENT ===`) to its quoted lines; before, the whole harness wrapper counted as typed. +- New event `note_asked` {kind, saved: repo|personal|no}, sent from `askForNote` on the stop after an ask (`stop_hook_active`). `askAnswer` checks the last "Stop hook feedback" line is trail's ask (it contains the ask's closing text), so another hook's block isn't counted. + +## Tried and ruled out +- A property on `hook_ms` for the Stop hook: it is sent only when learned skills are enabled (src/claude/skills-wiring.ts), so most stops would go uncounted. +- A property on `note_saved` alone: it gives the saves but not how many corrections were asked about, so no share. + +## Watch out +- Agent-facing text (session start, skill, asks) must not say "memory", "notes" or "rules": test/trail-words.test.ts bans them. Say "your own copy". +- `shellEdit` treats a write it can't name as a repo edit. A leading `cd` outside the repo, or full paths that all point outside it, now mean no edit; a bare `python3 - <"\` with the text on stdin, under ## Decided, ## Tried and ruled out and ## Watch out, giving the why. Put files, commands and values in the body under "In the code:". Never for setting up trail, status, or what the diff shows. A summary, never a transcript, and nothing secret. A learning is added to this repo unless it's about how this user likes to work or their own machine: then add --personal and it stays just with them. A learning is followed by default and the user can say otherwise in the moment; only when the user says it must hold even if someone asks otherwise, add --always-follow. One added to this repo goes in .trail/learnings/; commit it with the change so your teammates' Claude starts from it. Close your reply with the line ${cmd('graft note')} prints for it: for one added to this repo, the one for whether it's committed by then (check with git status --porcelain .trail/learnings).\n` + + `Save a learning when the session finds what the code can't tell the next person: a bug's cause after failed attempts, an approach dropped and why, a choice and why, or the user correcting how something is done here. \`${cmd('graft note')} --title ""\` with the text on stdin, under ## Decided, ## Tried and ruled out and ## Watch out, giving the why. Put files, commands and values in the body under "In the code:". Never for setting up trail, status, or what the diff shows. A summary, never a transcript, and nothing secret. A learning is added to this repo unless it's about how this user likes to work, their own machine, or business, growth, sign-up, pricing or competitor reasoning or internal numbers: then add --personal and it stays just with them, even if you also keep your own copy. A learning is followed by default and the user can say otherwise in the moment; only when the user says it must hold even if someone asks otherwise, add --always-follow. One added to this repo goes in .trail/learnings/; commit it with the change so your teammates' Claude starts from it. Close your reply with the line ${cmd('graft note')} prints for it: for one added to this repo, the one for whether it's committed by then (check with git status --porcelain .trail/learnings).\n` + (brand() === 'trail' ? `When the work is ready for review, run \`trail check\` before you commit it, push the branch or open a PR: it compares the branch's changes, committed or not, with this repo's learnings and yours, and exits 1 on a conflict. Fix the conflict, or say why the change is right anyway in the commit message or PR description. If the user asks you to save what changed, save it as a new learning that replaces the old one (\`trail note --replaces \`). A replacement saved on this branch leaves the conflict in place until it's merged, so still say why in the PR. Never edit an existing learning, yours or a teammate's.\n` : '') diff --git a/src/claude/note-signals.ts b/src/claude/note-signals.ts index 79ca55840..42599e5e0 100644 --- a/src/claude/note-signals.ts +++ b/src/claude/note-signals.ts @@ -24,9 +24,11 @@ * Pure, for tests: `noteSignals` takes the transcript's JSONL. */ import { existsSync } from 'node:fs'; +import { homedir } from 'node:os'; import { isAbsolute, relative, resolve, sep } from 'node:path'; import { trailOwnPath } from '../notes/git-facts.js'; import { NOTE_CALL } from '../notes/session-cost.js'; +import { typedText } from '../trace/claude.js'; export type NoteSignalKind = 'fixed' | 'corrected' | 'undone' | 'chose' | 'overrode' | 'shipped' | 'explored'; @@ -77,6 +79,9 @@ export function fileWordRe(platform: NodeJS.Platform = process.platform): RegExp const FILE_WORD = fileWordRe(); /** `cd &&` at the start of a command: where its relative paths start. */ const LEADING_CD = /^\s*cd\s+(["']?)([^"'&;|]+?)\1\s*(?:&&|;)/; +/** A file a command names by its full path, of any kind: `/tmp/pr-body.md`, `~/.claude/…/MEMORY.md`. */ +const FULL_PATH = /(?:^|[\s='"(])((?:~|\/)[^\s'"();&|<>$]*\.[A-Za-z]\w*)(?=$|[\s'"();&|<>])/g; +const home = (p: string) => p.replace(/^~(?=\/|$)/, homedir()); /** * A test, build or type check, as typed: `npm test`, `go test`, `pytest`, @@ -105,9 +110,10 @@ const CONFLICT_FILE = /^(?:- |✗ )([^\s:]+)(?::\d+)?:?/gm; const SETUP = /(?:^|[\s;&|])(?:trail|graft|npx\s+(?:-y\s+)?@(?:trailhq|nanonets)\/[\w-]+)\s|(?:npm|pnpm|yarn|bun)\s+(?:i|install|add|uninstall)\b|\bbrew\s+install\b|\bpip3?\s+install\b/; /** The person correcting the agent: at the start of their message, or a rule stated in it. */ -const CORRECTION_LEAD = /^\s*(?:no|nope|nah|wrong|actually|don't|dont|do not|never|stop|not (?:that|this|like that|quite)|that's (?:not|wrong)|that is (?:not|wrong))\b[\s,.!:;-]/i; +const CORRECTION_LEAD = /^\s*(?:no|nope|nah|na|wrong|actually|don't|dont|do not|never|stop|not (?:that|this|like that|quite)|that's (?:not|wrong)|that is (?:not|wrong))\b[\s,.!:;-]/i; +/** "need not go to repo", "keep it out of the repo", "not in the repo" (not when asked as a question). */ const CORRECTION_RULE = - /\b(?:instead of|instead,|use \S+ instead|we (?:always |usually |never )?(?:use|do|don't|keep|put|run)\b|not like that|that's not how|should(?:n't| not) (?:be|use)|you (?:forgot|missed|broke)\b)/i; + /\b(?:instead of|instead,|use \S+ instead|we (?:always |usually |never )?(?:use|do|don't|keep|put|run)\b|not like that|that's not how|should(?:n't| not) (?:be|use|go)|need(?:n't| not) (?:to )?(?:go|be)\b|keep (?:it|this|that|these|them) out of (?:the )?repo|(?:not|never) (?:in|into|to) (?:the )?repo\b(?![^.!\n]*\?)|you (?:forgot|missed|broke)\b)/i; /** A clarification, not a correction: "no, I mean the other file". */ const CLARIFY = /\bi mean(?:t)?\b|\bi was (?:just )?asking\b|\bjust asking\b/i; @@ -141,20 +147,16 @@ function textOf(content: unknown): string { return ''; } -/** What the person typed, or null: tool results, hook feedback and system lines aren't typed. */ +/** + * What the person typed, or null. The one filter every learning path uses + * (trace/claude.ts): Claude Desktop's `` before the message + * is stripped and the words after it kept, an artifact comment is its quoted + * text, and compaction summaries, task notices, hook feedback and tool results + * aren't typed. + */ function typed(e: any): string | null { - if (e?.type !== 'user' || e?.isMeta) return null; - const content = e?.message?.content; - const text = - typeof content === 'string' - ? content - : Array.isArray(content) && content.some((b: any) => b?.type === 'text') - ? content.filter((b: any) => b?.type === 'text').map((b: any) => String(b.text ?? '')).join('\n') - : null; - if (text === null) return null; - const t = text.trim(); - if (!t || t.startsWith('<') || t.startsWith('[Request interrupted') || t.startsWith('Caveat:') || /^Stop hook feedback/i.test(t)) return null; - return t; + const t = typedText(e); + return t && !/^Stop hook feedback/i.test(t) ? t : null; } /** `"no, use a worktree for…"`: the first line, short, safe inside double quotes. */ @@ -166,9 +168,10 @@ export function quoted(s: string, n = 60): string { /** * Every sign since the session's last saved learning, best first. `repo` is * the folder the session works in, to tell its code from trail's files and - * from files outside it. + * from files outside it. `branchHasLearning` says whether the branch's commits + * already add one to the repo: then what it ships carries a learning. */ -export function noteSignals(jsonl: string, repo: string): NoteSignal[] { +export function noteSignals(jsonl: string, repo: string, branchHasLearning: () => boolean = () => false): NoteSignal[] { const tools: ToolEvent[] = []; const results = new Map(); const prompts: Said[] = []; @@ -223,13 +226,17 @@ export function noteSignals(jsonl: string, repo: string): NoteSignal[] { const p = rel.split(sep).join('/'); return trailOwnPath(p) ? null : p; }; + const outside = (p: string) => { + const rel = relative(repo, p); + return rel.startsWith('..') || isAbsolute(rel); + }; const bash = (t: ToolEvent) => (t.name === 'Bash' && typeof t.input?.command === 'string' ? (t.input.command as string) : null); /** The repo file a shell command wrote, '' when it wrote one this can't name, null when it wrote none. */ const shellEdit = (t: ToolEvent): string | null => { const c = bash(t); if (!c || !BASH_WRITE.test(c) || results.get(t.id)?.error) return null; const cd = LEADING_CD.exec(c)?.[2]?.trim(); - const base = cd ? resolve(t.cwd ?? repo, cd) : (t.cwd ?? repo); + const base = cd ? resolve(t.cwd ?? repo, home(cd)) : (t.cwd ?? repo); let ownOnly = false; for (const w of c.match(FILE_WORD) ?? []) { for (const at of isAbsolute(w) ? [w] : [resolve(base, w), resolve(repo, w)]) { @@ -239,8 +246,11 @@ export function noteSignals(jsonl: string, repo: string): NoteSignal[] { ownOnly = true; } } - // Writing only trail's own files (settings, learnings) is setup, not a fix. - return ownOnly ? null : ''; + // Writing only trail's own files (settings, learnings) is setup, not a fix; writing only outside the + // repo (Claude's memory after `cd ~/.claude/…`, a PR body in a scratch file) doesn't change its code. + if (ownOnly || outside(base)) return null; + const named = [...c.matchAll(FULL_PATH)].map((m) => home(m[1]!)); + return named.length && named.every(outside) ? null : ''; }; const codeEdit = (t: ToolEvent): string | null => (EDIT_TOOLS.has(t.name) ? inRepo(t.input?.file_path ?? t.input?.notebook_path) : shellEdit(t)); /** Work on the repo after `at`: a code edit, or a command that isn't setting tools up. */ @@ -335,12 +345,13 @@ export function noteSignals(jsonl: string, repo: string): NoteSignal[] { } } - // shipped: the work went into a commit, a push or a PR with no learning beside it. + // shipped: the work went into a commit, a push or a PR with no learning beside it, on the branch or since. if (work.some((t) => codeEdit(t) !== null)) { for (const t of [...work].reverse()) { const c = bash(t); const m = c ? SHIP.exec(c) : null; if (!m || results.get(t.id)?.error) continue; + if (branchHasLearning()) break; const verb = m[1]!.startsWith('gh') ? 'opened a PR for' : m[1]!.endsWith('push') ? 'pushed' : 'committed'; out.push({ kind: 'shipped', key: `shipped:${t.uuid}`, what: `You ${verb} this work, and no learning went with it.` }); break; @@ -367,6 +378,9 @@ export function noteSignals(jsonl: string, repo: string): NoteSignal[] { const SAVE = '`trail note --title ""`'; const CLOSE = 'Put files, commands and values in the body under "In the code:". Sections: ## Decided, ## Tried and ruled out, ## Watch out. A summary, never a transcript, nothing secret. Then finish as trail\'s output asks.'; +/** Where a learning goes when it isn't for the repo, even when the agent also keeps its own copy. */ +const PERSONAL = + "Add --personal when it's about how this person likes to work, or about business, growth, sign-up, pricing or competitor reasoning or internal numbers: those never go in the repo, even if you also keep your own copy."; /** What the agent is told: the sign first, since Claude Code folds long hook output to its first lines. */ export function noteAsk(s: NoteSignal): string { @@ -375,7 +389,7 @@ export function noteAsk(s: NoteSignal): string { s.kind === 'fixed' ? `${s.what} If the cause isn't obvious from the diff, save it: ${SAVE}, with the why. ${skip}` : s.kind === 'corrected' - ? `${s.what} Save it as a learning so nobody has to say it again: ${SAVE}; add --personal if it's about how this person likes to work. ${skip}` + ? `${s.what} Save it as a learning so nobody has to say it again: ${SAVE}. ${PERSONAL} ${skip}` : s.kind === 'undone' ? `${s.what} If you dropped an approach, save why under ## Tried and ruled out: ${SAVE}. ${skip}` : s.kind === 'overrode' @@ -384,6 +398,37 @@ export function noteAsk(s: NoteSignal): string { ? `${s.what} If the choice matters to the next person, save it under ## Decided, with the options you dropped and why under ## Tried and ruled out: ${SAVE}. ${skip}` : s.kind === 'explored' ? `${s.what} If it found where something lives or why it works the way it does, and the code won't say so plainly, save it: ${SAVE}. ${skip}` - : `${s.what} If the session found what the diff won't tell the next person (a cause, a dropped approach, how something is done here), save it now so it ships with the change: ${SAVE}. ${skip}`; + : `${s.what} If the session found what the diff won't tell the next person (a cause, a dropped approach, how something is done here), save it now so it ships with the change: ${SAVE}. ${PERSONAL} ${skip}`; return `[trail] ${first}\n${CLOSE}`; } + +/** + * Read on the stop after an ask: whether the agent saved a learning in answer, + * in the repo or `--personal`, or `no`. Null when the last stop hook feedback + * in the transcript isn't trail's ask, so another hook's block isn't counted. + */ +export function askAnswer(jsonl: string): 'repo' | 'personal' | 'no' | null { + let asked = false; + let saved: 'repo' | 'personal' | 'no' = 'no'; + const notes = new Map(); + for (const line of jsonl.split('\n')) { + let e: any; + try { + e = JSON.parse(line); + } catch { + continue; + } + const content = e?.message?.content; + if (e?.type === 'user' && e.isMeta && /^Stop hook feedback/i.test(textOf(content))) { + asked = textOf(content).includes(CLOSE); + saved = 'no'; + continue; + } + for (const b of Array.isArray(content) ? content : []) { + if (b?.type === 'tool_use' && b.name === 'Bash' && typeof b.input?.command === 'string' && NOTE_CALL.test(b.input.command)) notes.set(String(b.id), b.input.command); + const cmd = b?.type === 'tool_result' ? notes.get(String(b.tool_use_id)) : undefined; + if (asked && cmd !== undefined && b.is_error !== true) saved = /\s--personal\b/.test(cmd) ? 'personal' : 'repo'; + } + } + return asked ? saved : null; +} diff --git a/src/claude/skill-template.ts b/src/claude/skill-template.ts index 422a26460..a0448c201 100644 --- a/src/claude/skill-template.ts +++ b/src/claude/skill-template.ts @@ -175,8 +175,10 @@ A learning is added to this repo or kept just for the user: with the change it came from, so your teammates' Claude starts from it. Save it before your final commit when you can; if the work is already committed and not pushed, add it to that commit. -- Just for the user, with \`--personal\`: how this person likes to work, or - their own machine and setup. It stays on their machine, out of the repo. +- Just for the user, with \`--personal\`: how this person likes to work, + their own machine and setup, and business, growth, sign-up, pricing or + competitor reasoning or internal numbers. It stays on their machine, out of + the repo, even if you also keep your own copy. \`trail note\` prints the line to close your reply with. For a learning added to this repo it prints two: one for when the learning is committed by the end diff --git a/src/claude/stop.ts b/src/claude/stop.ts index e5cbd2212..ab6688c6f 100644 --- a/src/claude/stop.ts +++ b/src/claude/stop.ts @@ -26,7 +26,9 @@ import { trailHome } from '../notes/trail-home.js'; import { keepsNotes } from '../notes/home.js'; import { maybeSyncInBackground } from '../cloud/workspace-sync.js'; import { brand } from '../brand.js'; -import { NOTE_ASKS_PER_SESSION, noteAsk, noteSignals } from './note-signals.js'; +import { NOTE_ASKS_PER_SESSION, askAnswer, noteAsk, noteSignals } from './note-signals.js'; +import { branchAddsLearning } from '../cloud/diff.js'; +import { track } from '../telemetry/track.js'; import type { HookOut } from '../skills/types.js'; /** The fields of a Stop payload this handler reads. */ @@ -136,16 +138,22 @@ function turnEnd(input: StopInput, dir: string): string | null { * the stop with the reason. Each sign is asked about once, and a session is * asked at most NOTE_ASKS_PER_SESSION times; a saved learning starts the signs * over. Never blocks twice in a row: Claude Code sets `stop_hook_active` on - * the stop that follows a block. The caller writes it only as the Stop's one - * answer, so a sign is marked asked only when it was. Never throws. + * the stop that follows a block; that stop sends `note_asked`, what was asked + * and whether a learning was saved in answer. The caller writes it only as the + * Stop's one answer, so a sign is marked asked only when it was. Never throws. */ export function askForNote(input: { stop_hook_active?: unknown }, dir: string, id: string, transcript: string | null): HookOut { try { - if (!transcript || brand() !== 'trail' || input.stop_hook_active || !keepsNotes(dir)) return {}; + if (!transcript || brand() !== 'trail' || !keepsNotes(dir)) return {}; const s = readSession(dir, id); const asked = s.noteAsks ?? []; + if (input.stop_hook_active) { + const saved = asked.length ? askAnswer(readFileSync(transcript, 'utf8')) : null; + if (saved) track('note_asked', { kind: asked[asked.length - 1]!.split(':')[0], saved }, { repo: dir }); + return {}; + } if (asked.length >= NOTE_ASKS_PER_SESSION) return {}; - const sign = noteSignals(readFileSync(transcript, 'utf8'), dir).find((x) => !asked.includes(x.key)); + const sign = noteSignals(readFileSync(transcript, 'utf8'), dir, () => branchAddsLearning(dir)).find((x) => !asked.includes(x.key)); if (!sign) return {}; writeSession(dir, id, { ...s, noteAsks: [...asked, sign.key] }); return { block: noteAsk(sign) }; diff --git a/src/cloud/diff.ts b/src/cloud/diff.ts index 67640a6dc..eb7330a03 100644 --- a/src/cloud/diff.ts +++ b/src/cloud/diff.ts @@ -50,6 +50,12 @@ export const WIRING = [ const NOT_WIRING = ["--", ".", ...WIRING.map((p) => `:(exclude)${p}`)]; +/** Whether the branch's commits add a learning to `.trail/learnings/`, so what it pushes carries one. */ +export function branchAddsLearning(repo: string, base = defaultBase(repo)): boolean { + const from = git(repo, ["merge-base", "HEAD", base])?.trim() || base; + return !!git(repo, ["diff", "--name-only", "--diff-filter=A", from, "HEAD", "--", ".trail/learnings"])?.trim(); +} + /** The branch's diff against its base's merge base, working tree included, without trail's own wiring. Null outside a git repo. */ export function branchDiff(repo: string, base = defaultBase(repo), maxBytes = 400 * 1024): BranchDiff | null { const from = git(repo, ["merge-base", "HEAD", base])?.trim() || base; diff --git a/src/telemetry/contract.ts b/src/telemetry/contract.ts index 69a212b89..aac8bfbc9 100644 --- a/src/telemetry/contract.ts +++ b/src/telemetry/contract.ts @@ -68,6 +68,12 @@ export const EVENTS: Record> = { * and the paths never travel. The supply side of the notes loop; the * `notes` property on `query` is the demand side. */ note_saved: new Set(['has_cost', 'touches_bucket', 'scope']), + /** The Stop hook asked the agent to save a learning, read on the stop after: + * the sign that raised it (`kind`), and whether a learning was saved in + * answer (`saved`: in the repo, just for the person, or not). The share of + * `kind=corrected` asks with a learning saved is how many corrections + * become learnings. The transcript and the learning never travel. */ + note_asked: new Set(['kind', 'saved']), /** Once a day per repo, from the session-start hook, for a repo trail runs * in: whether `.trail/` is committed there, how many learnings it holds, * as a bucket, and whether this machine shares it with a workspace. With @@ -139,6 +145,10 @@ export const HOOK_MS_BUCKETS = ['<50ms', '50-100ms', '100-300ms', '300ms-1s', '1 */ export const VALUES: Record>> = { cold_start: Object.fromEntries(INIT_KINDS.flatMap((k) => [`offered_${k}`, `kept_${k}`]).map((key) => [key, new Set(SMALL_COUNTS)])), + note_asked: { + kind: new Set(['fixed', 'corrected', 'undone', 'chose', 'overrode', 'shipped', 'explored']), + saved: new Set(['repo', 'personal', 'no']), + }, skill_offered: { moment: new Set(['asked', 'repeated', 'repaired', 'correction', 'update', 'drift', 'retire', 'promote']) }, skill_answer: { answer: new Set(['yes', 'no', 'show', 'none', 'off', 'retire']) }, skill_changed: { op: new Set(['update', 'extend', 'split', 'replace', 'retire', 'adopted', 'restore-version']) }, diff --git a/src/trace/claude.ts b/src/trace/claude.ts index 531a4f722..5eb1b3122 100644 --- a/src/trace/claude.ts +++ b/src/trace/claude.ts @@ -85,6 +85,13 @@ const NOT_TYPED = [ /^\[Request interrupted/, /^This session is being continued/, ]; +/** + * A comment the person sent from an artifact (Claude Desktop): the harness's + * framing, then the comment quoted with "> " between BEGIN and END lines. Only + * the quoted comment is typed. + */ +const ARTIFACT_COMMENT = /^\[Artifact comment sent to Claude\]/; +const ARTIFACT_QUOTE = /^=== BEGIN ARTIFACT COMMENT (\S+) ===\n([\s\S]*?)\n=== END ARTIFACT COMMENT \1 ===$/m; /** The text parts of a message's content, joined; images, tool parts and the rest left out. */ export function textOf(content: unknown): string { @@ -104,7 +111,8 @@ export function textOf(content: unknown): string { * pastes marked, and "" when what's left is harness text or nothing typed. */ export function ownText(raw: string): string { - const t = raw.replace(WRAPPED, "").replace(PASTED, PASTE_MARK).trim(); + let t = raw.replace(WRAPPED, "").replace(PASTED, PASTE_MARK).trim(); + if (ARTIFACT_COMMENT.test(t)) t = (ARTIFACT_QUOTE.exec(t)?.[2] ?? "").replace(/^> ?/gm, "").trim(); if (!t || NOT_TYPED.some((re) => re.test(t))) return ""; if (!t.split(PASTE_MARK).join("").trim()) return ""; return t; diff --git a/test/fixtures/skills/trail/SKILL.md b/test/fixtures/skills/trail/SKILL.md index 9950962c4..579ddbfa1 100644 --- a/test/fixtures/skills/trail/SKILL.md +++ b/test/fixtures/skills/trail/SKILL.md @@ -264,8 +264,10 @@ A learning is added to this repo or kept just for the user: with the change it came from, so your teammates' Claude starts from it. Save it before your final commit when you can; if the work is already committed and not pushed, add it to that commit. -- Just for the user, with `--personal`: how this person likes to work, or - their own machine and setup. It stays on their machine, out of the repo. +- Just for the user, with `--personal`: how this person likes to work, + their own machine and setup, and business, growth, sign-up, pricing or + competitor reasoning or internal numbers. It stays on their machine, out of + the repo, even if you also keep your own copy. `trail note` prints the line to close your reply with. For a learning added to this repo it prints two: one for when the learning is committed by the end diff --git a/test/telemetry-contract.test.ts b/test/telemetry-contract.test.ts index 5c6f3fedb..18864b752 100644 --- a/test/telemetry-contract.test.ts +++ b/test/telemetry-contract.test.ts @@ -78,10 +78,10 @@ test('every event carries the common properties, and no identifier beyond them', // Pinned rather than counted, so adding an event is a deliberate edit here and // a matching row in TELEMETRY.md, never something that arrives with a feature. -test('the contract lists exactly the twenty-five documented events', () => { +test('the contract lists exactly the twenty-six documented events', () => { assert.deepEqual(Object.keys(EVENTS).sort(), [ 'brain_signup_opened', 'brain_signup_settled', 'build_completed', 'build_failed', - 'cold_start', 'first_run', 'guardrail', 'hook_ms', 'init_completed', 'install', 'learnings_committed', 'note_saved', 'query', 'repo_seen', 'session_summary', + 'cold_start', 'first_run', 'guardrail', 'hook_ms', 'init_completed', 'install', 'learnings_committed', 'note_asked', 'note_saved', 'query', 'repo_seen', 'session_summary', 'skill_answer', 'skill_changed', 'skill_offered', 'skill_promoted', 'skill_saved', 'skill_stale', 'skill_used', 'skill_value', 'team_pull', 'wall_hit', ]); }); diff --git a/test/trace-typed.test.ts b/test/trace-typed.test.ts index 34e570237..3efbd7268 100644 --- a/test/trace-typed.test.ts +++ b/test/trace-typed.test.ts @@ -105,6 +105,27 @@ test("only origin decides, never promptSource: the desktop and IDE apps mark typ assert.equal(isTypedByPerson(typed("Finished: tests pass.", { entrypoint: "claude-desktop", promptSource: "sdk", origin: { kind: "peer" } })), false); }); +test("Claude Desktop: the status reminder before a message is stripped, and an artifact comment is its quoted text", () => { + const status = "\nLinked sessions now (status, not an instruction): init-offer-wording = local_1 (done, PR #724)\n\n\n\n"; + assert.equal(typedText(typed(`${status}na, these need not go to repo`, { entrypoint: "claude-desktop" })), "na, these need not go to repo"); + const comment = [ + "[Artifact comment sent to Claude]", + "Artifact: https://example.com/artifact/1", + "", + "I sent you this comment from the artifact's comment thread. Treat it as a request about the artifact.", + "", + "=== BEGIN ARTIFACT COMMENT e8e27f ===", + "> dont call it trail cloud", + "> just Trail", + "=== END ARTIFACT COMMENT e8e27f ===", + "", + "Answer this comment in its thread, not here.", + status, + ].join("\n"); + assert.equal(typedText(typed(comment)), "dont call it trail cloud\njust Trail"); + assert.equal(typedText(typed("[Artifact comment sent to Claude]\nArtifact: https://example.com/artifact/1\nno quote here")), "", "no quoted comment, nothing typed"); +}); + test("a prompt typed while the agent was busy (a queued command) is the person's", () => { const q = queuedPrompt("also bump the version"); assert.equal(isTypedByPerson(q), true); diff --git a/test/trail-note-ask.test.ts b/test/trail-note-ask.test.ts index 3e9b62123..175fe10da 100644 --- a/test/trail-note-ask.test.ts +++ b/test/trail-note-ask.test.ts @@ -11,7 +11,7 @@ import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { BRAND_ENV } from '../src/brand.js'; import { main } from '../src/claude/hooks.js'; -import { fileWordRe, noteAsk, noteSignals } from '../src/claude/note-signals.js'; +import { askAnswer, fileWordRe, noteAsk, noteSignals } from '../src/claude/note-signals.js'; import { ensureRepoHome } from '../src/notes/home.js'; let n = 0; @@ -83,6 +83,40 @@ test('the person correcting the agent is a sign, when the agent goes on to chang assert.deepEqual(kinds(jsonl(prompt('x'), say('ok'), prompt('We use pnpm here, not npm'), bash('pnpm install && pnpm lint'))), ['corrected'], 'a rule stated mid-message'); }); +test('Claude Desktop: a correction after the status reminder is seen; compaction summaries and other sessions never are', () => { + const status = '\nLinked sessions now (status, not an instruction): one-line-summary = local_1 (done, PR #729)\n\n\n\n'; + const desktop = (text: string) => [{ type: 'user', uuid: uid(), entrypoint: 'claude-desktop', origin: { kind: 'human' }, message: { role: 'user', content: status + text } }]; + const said = 'na, these learnings need not go to repo , cause these are related to business decisions to get more people to sign up right ?'; + const [s] = noteSignals(jsonl(prompt('x'), say('Two learnings are in the PR.'), desktop(said), bash('git rm -q .trail/learnings/a.md && git commit -qm "Keep it out"')), REPO); + assert.equal(s?.kind, 'corrected'); + assert.equal(s?.what, 'The user corrected you ("na, these learnings need not go to repo , cause these are r…").'); + const comment = '[Artifact comment sent to Claude]\nArtifact: https://example.com/a/1\n\n=== BEGIN ARTIFACT COMMENT 9f ===\n> dont call it trail cloud\n=== END ARTIFACT COMMENT 9f ===\n\nAnswer it in its thread.'; + assert.deepEqual(kinds(jsonl(prompt('x'), say('ok'), desktop(comment), edit('src/nudge.ts'))), ['corrected'], 'an artifact comment'); + + const summary = [{ type: 'user', uuid: uid(), isCompactSummary: true, isVisibleInTranscriptOnly: true, message: { role: 'user', content: 'This session is being continued from a previous conversation. The user said: no, use a channel instead of a mutex.' } }]; + assert.deepEqual(kinds(jsonl(prompt('x'), say('ok'), summary, edit('src/q.go'))), [], 'a compaction summary is not the person'); + assert.deepEqual(kinds(jsonl(prompt('x'), say('ok'), prompt('This session is being continued from a previous conversation. Instead of a mutex, use a channel.'), edit('src/q.go'))), [], 'nor is its text without the flag'); + const peer = [{ type: 'user', uuid: uid(), origin: { kind: 'peer' }, message: { role: 'user', content: 'no, the other session should not use the cache' } }]; + assert.deepEqual(kinds(jsonl(prompt('x'), say('ok'), peer, edit('src/q.go'))), [], "another session's message is not the person"); +}); + +test('keeping something out of the repo is a correction; questions and look-alikes are not', () => { + const after = (text: string) => kinds(jsonl(prompt('x'), say('ok'), prompt(text), edit('src/a.ts'))); + for (const said of ['keep it out of the repo', "this shouldn't go in the repo", 'not in the repo, keep it personal', 'these need not be committed', 'na, leave it']) { + assert.deepEqual(after(said), ['corrected'], said); + } + for (const said of ['is this not in the repo yet?', 'you need not ask before running the tests', 'nan values break the chart', 'can you put this in the repo?', 'naming is fine']) { + assert.deepEqual(after(said), [], said); + } +}); + +test('the corrected and shipped asks say business reasoning goes in --personal, never the repo', () => { + for (const kind of ['corrected', 'shipped'] as const) { + const ask = noteAsk({ kind, key: `${kind}:u`, what: 'x.' }); + assert.match(ask, /Add --personal when it's about how this person likes to work, or about business, growth, sign-up, pricing or competitor reasoning or internal numbers: those never go in the repo, even if you also keep your own copy\./); + } +}); + test('an undone edit is a sign, unless it only undid trail files', () => { assert.deepEqual(kinds(jsonl(prompt('x'), edit('src/a.ts'), bash('git restore src/a.ts'))), ['undone']); assert.match(noteSignals(jsonl(prompt('x'), bash('git revert abc123')), REPO)[0]!.what, /^You undid a change \(`git revert abc123`\)\.$/); @@ -106,6 +140,30 @@ test('committing, pushing or opening a PR for code with no learning is a sign', assert.match(noteSignals(jsonl(prompt('x'), edit('a.go'), bash('gh pr create --fill')), REPO)[0]!.what, /^You opened a PR for this work/); assert.deepEqual(kinds(jsonl(prompt('x'), bash('git commit -m "trail setup"'))), [], 'no code edited this session'); assert.deepEqual(kinds(jsonl(prompt('x'), edit('.trail/learnings/x.md'), bash('git commit -m learning'))), [], 'committing a learning'); + const learned = noteSignals(jsonl(prompt('x'), edit('a.go'), bash('git push')), REPO, () => true); + assert.deepEqual(learned.map((s) => s.kind), [], "the branch's commits already add a learning"); +}); + +test('writes outside the repo are no code edit: Claude memory after a cd, a PR body in a scratch file', () => { + const memory = "cd ~/.claude/projects/-work-app/memory && python3 - <<'PY'\np='MEMORY.md'\ns=open(p).read()\nopen(p,'w').write(s)\nPY"; + assert.deepEqual(kinds(jsonl(prompt('x'), bash(memory), bash('git push -q --force-with-lease'))), []); + const body = 'P=/private/tmp/scratch/pr-body.md && sed -i \'\' \'s/one/two/\' "$P" && gh pr edit 730 --body-file "$P"'; + assert.deepEqual(kinds(jsonl(prompt('x'), bash(body), bash('git push'))), []); + assert.deepEqual(kinds(jsonl(prompt('x'), tool('Write', { file_path: '/home/me/.claude/memory/x.md' }), bash('git push'))), []); + // Still a code edit: a write this can't name, from inside the repo. + assert.deepEqual(kinds(jsonl(prompt('x'), bash("python3 - <<'PY'\nopen(p,'w').write(s)\nPY"), bash('git push'))), ['shipped']); +}); + +test("the stop after an ask reads whether a learning was saved in answer, in the repo or personal; another hook's block isn't trail's", () => { + const ask = noteAsk({ kind: 'corrected', key: 'corrected:u', what: 'The user corrected you ("na").' }); + const asked = [prompt('x'), say('ok'), meta(`Stop hook feedback:\n${ask}`)]; + assert.equal(askAnswer(jsonl(...asked, bash('trail note --personal --title "Growth reasoning stays out of the repo" <<\'EOF\'', '✓ saved just for you'))), 'personal'); + assert.equal(askAnswer(jsonl(...asked, bash('trail note --title "x" <<\'EOF\'', '✓ added to this repo'))), 'repo'); + assert.equal(askAnswer(jsonl(...asked, bash('trail note --title "x"', '✗ not saved yet', true))), 'no', 'the gate turned it away'); + assert.equal(askAnswer(jsonl(...asked, say('Nothing worth keeping.'))), 'no'); + assert.equal(askAnswer(jsonl(bash('trail note --title "x"', '✓'), ...asked)), 'no', 'a learning saved before the ask'); + assert.equal(askAnswer(jsonl(...asked, meta('Stop hook feedback:\nlint failed'))), null, "another hook's block"); + assert.equal(askAnswer(jsonl(prompt('x'), say('ok'))), null, 'no ask'); }); test('a turn that read a lot to find its answer is a sign, with no edit needed', () => { From a81f49c6ab1ccdea24ef3ec496bc58824af741b9 Mon Sep 17 00:00:00 2001 From: anirudhkumar-nanonets Date: Tue, 17 Nov 2026 13:24:33 +0530 Subject: [PATCH 2/2] learning: count a transcript line as typed only when it clearly is --- ...hen-it-s-unclear-who-wrote-anirudhkumar.md | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 .trail/learnings/2026-11-17-when-it-s-unclear-who-wrote-anirudhkumar.md diff --git a/.trail/learnings/2026-11-17-when-it-s-unclear-who-wrote-anirudhkumar.md b/.trail/learnings/2026-11-17-when-it-s-unclear-who-wrote-anirudhkumar.md new file mode 100644 index 000000000..c3c7c9eed --- /dev/null +++ b/.trail/learnings/2026-11-17-when-it-s-unclear-who-wrote-anirudhkumar.md @@ -0,0 +1,28 @@ +--- +title: "When it's unclear who wrote a line in Claude's transcript, trail counts it as not typed by the person" +author: Anirudhkumar-nanonets +date: 2026-11-17 +branch: claude/fix-save-learning-prompts-84888d +cost: { minutes: 1, tokens: 10958 } +id: l-6b0c32a7 +scope: repo +status: active +src: agent +touches: + - "src/trace/claude.ts#candidateText" + - "src/trace/claude.ts#ownText" + - "src/claude/note-signals.ts#typed" +--- + +## Decided +- Claude Code keeps adding lines to the person's slot in its transcript that the person never typed: Desktop's status block before each message, the compaction summary, messages from other sessions, artifact comments, prompts queued while Claude was busy. The format isn't documented as stable, so expect more of these. +- trail counts a line as the person's only when it is clearly theirs, and stays quiet when unsure. A missed save ask costs one learning. A false one ("The user corrected you" quoting a compaction summary) makes people stop trusting every later ask. +- One filter decides this for every learning path. No hook keeps its own copy: the Stop hook's own copy drifted, and that caused the missed Desktop corrections and the false compaction ask. + +## Tried and ruled out +- A per-hook filter tuned to what that hook needs: it fell behind each time Claude Code added a new kind of line, and nobody noticed until a false or missed ask. + +## Watch out +- A missing `origin` still counts as the person, because the CLI and older transcripts don't set one. So a new harness line with no `origin` and none of the skip flags gets through until its text shape is added to the not-typed patterns. When an ask quotes words the person never wrote, add that shape there, and add the real transcript line as a test. + +In the code: src/trace/claude.ts `candidateText` (the skip flags `isMeta`, `isSidechain`, `isCompactSummary`, `isVisibleInTranscriptOnly`, and `origin.kind` must be `human` or absent), `ownText` (`WRAPPED`, `NOT_TYPED`, `ARTIFACT_COMMENT`), `typedText`; src/claude/note-signals.ts `typed` calls it; tests in test/trace-typed.test.ts and test/trail-note-ask.test.ts.