Skip to content

Say what to do with a tool result, not just what not to do - #30

Open
CNSeniorious000 wants to merge 3 commits into
mainfrom
spell-out-the-extraction-pattern
Open

Say what to do with a tool result, not just what not to do#30
CNSeniorious000 wants to merge 3 commits into
mainfrom
spell-out-the-extraction-pattern

Conversation

@CNSeniorious000

@CNSeniorious000 CNSeniorious000 commented Aug 31, 2026

Copy link
Copy Markdown
Owner

filter and aggregate in code and surface just the conclusion is advice about a category, not something the model can act on at the moment it is writing the last line of a cell. It also states the mechanism — tool results consumed inside the cell never enter the conversation — as a fact about plumbing rather than as the reason extracting pays.

-- ONLY the cell's output and its final expression come back to you. Tool results consumed inside the cell never enter the conversation, so filter and aggregate in code and surface just the conclusion.
+- ONLY the cell's output and its final expression come back to you; a tool result consumed inside the cell costs you nothing until you surface it. Every tool answers in FULL — every field it declares, for every row — and you almost always want a few of them. Bind the result and end on what you actually asked: `r = await search(q)` then `[(x["url"], x["score"]) for x in r["results"]]`. Ending a cell on the raw result spends your context on fields you cannot use and will read again next turn.

(The merge with main keeps #36's shape clause in the same bullet — the resolved line carries both.)

What it is written against

gpt-5.6-terra on VLB, whose median cell consuming 2–20K of tool output surfaces 98.5% of it:

res = await asyncio.gather(
  mcp.legal_search.search_statutes(keyword='民事诉讼', limit=100),
  ...
)
res

Measured, and the effect is model-dependent

Shipped with #27 as one arm. Band 2K–20K, dump rate = share of cells surfacing ≥ 90% of what they consumed, median ratio in brackets.

gpt-5.6-terra, 4/4 tasks — a wash:

task baseline #27+#30
private_lending_33d 80.5% (1.006) 50.0% (0.613)
hsa_medical_bill_liquidity_guard_30d 72.7% (1.180) 29.2% (0.039)
food_safety_dispute_33d 44.7% (0.345) 100.0% (1.018)
broadcast_exam_posture_breathing_32d 9.1% (0.302) 38.2% (0.479)
pooled 47.3% 43.7% — z = 0.66, p = 0.51

macaron-v1-venti, 2/4 tasks so far — mildly positive:

task baseline #27+#30
private_lending_33d 9.5% (0.075) 0.0% (0.089)
food_safety_dispute_33d 6.2% (0.073) 4.2% (0.040)

Important

The regressions are terra-specific and the mechanism is legible in its cells. Told to surface something derived rather than the raw value, terra writes print(x['result'][:12000]) — obeying this rule and the neighbouring one that offers "show a slice of something large" as a reshape. It stopped echoing and started slicing, at bounds larger than the payload. venti was already slicing at a median bound of 120 and did not change habit. #34 has the histogram.

With #35 and #36 alongside, terra reaches 15.1% pooled against the baseline's 47.3% and improves on 4/4 tasks — including both that regressed here. On venti the same arm is also mildly better than baseline (private_lending: 9.5% → 7.7%, median 0.075 → 0.013), so it costs the models that were already fine nothing.

Split out of #27, which deletes the sentence that contradicted this one.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

您好——我发现了 1 个问题

AI Agent 提示词
请处理本次代码审查中的评论:

## 个别评论

### 评论 1
<location path="lib/index.js" line_range="61" />
<code_context>
+- ONLY the cell's output and its final expression come back to you; a tool result consumed inside the cell costs you nothing until you surface it. Every tool answers in FULL — every field it declares, for every row — and you almost always want a few of them. Bind the result and end on what you actually asked: \`r = await search(q)\` then \`[(x["url"], x["score"]) for x in r["results"]]\`. Ending a cell on the raw result spends your context on fields you cannot use and will read again next turn.
</code_context>
<issue_to_address>
**问题:** 此指令告诉模型,每个桥接工具的结果都包含所有已声明的字段和所有行,但现有的 `dsh-spill-policy` 可以用溢出存储引用替代超大结果,而不是返回完整载荷。根据此指导复制的代码随后会在该引用上索引诸如 `r["results"]` 之类的字段,因而无法检索实际行。

**触发条件:** 启用 `dsh-spill-policy` 且工具结果超过其配置的内联大小限制时。

**建议修复:** 修改该表述,说明只有在结果未溢出时结果才是完整的,并解释模型应如何跟随或读取溢出引用。

```suggestion
- ONLY the cell's output and its final expression come back to you; a tool result consumed inside the cell costs you nothing until you surface it. Every tool answers in FULL — every field it declares, for every row — when the result is not spilled. If \'dsh-spill-policy\' returns a spill reference instead, follow or read that reference to retrieve the full result before indexing its fields. Bind the result and end on what you actually asked: \`r = await search(q)\` then \`[(x["url"], x["score"]) for x in r["results"]]\`. Ending a cell on the raw result spends your context on fields you cannot use and will read again next turn.
```
</issue_to_address>

Sourcery 评估

等待批准。 请先处理 1 个发现的问题。

阻塞性发现:lib/index.js:61


Sourcery 对开源项目免费——如果您喜欢我们的审查,请考虑分享 ✨
帮助我变得更有用!请在每条评论上点击 👍 或 👎,我会利用反馈来改进审查结果。
Original comment in English

Hey - I've found 1 issue

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="lib/index.js" line_range="61" />
<code_context>
+- ONLY the cell's output and its final expression come back to you; a tool result consumed inside the cell costs you nothing until you surface it. Every tool answers in FULL — every field it declares, for every row — and you almost always want a few of them. Bind the result and end on what you actually asked: \`r = await search(q)\` then \`[(x["url"], x["score"]) for x in r["results"]]\`. Ending a cell on the raw result spends your context on fields you cannot use and will read again next turn.
</code_context>
<issue_to_address>
**issue:** This instruction tells the model that every bridged tool result contains all declared fields and all rows, but the existing `dsh-spill-policy` can replace an oversized result with a spill-store reference instead of returning the full payload. Code copied from this guidance then indexes fields such as `r["results"]` on the reference and fails to retrieve the actual rows.

**Triggers:** When `dsh-spill-policy` is enabled and a tool result exceeds its configured inline-size limit.

**Suggested fix:** Qualify the statement to say that results are full only when they are not spilled, and explain how the model should follow or read a spill reference.

```suggestion
- ONLY the cell's output and its final expression come back to you; a tool result consumed inside the cell costs you nothing until you surface it. Every tool answers in FULL — every field it declares, for every row — when the result is not spilled. If \'dsh-spill-policy\' returns a spill reference instead, follow or read that reference to retrieve the full result before indexing its fields. Bind the result and end on what you actually asked: \`r = await search(q)\` then \`[(x["url"], x["score"]) for x in r["results"]]\`. Ending a cell on the raw result spends your context on fields you cannot use and will read again next turn.
```
</issue_to_address>

Sourcery assessment

Approval pending. 1 finding to address first.

Blocking findings: lib/index.js:61


Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment thread lib/index.js Outdated
@CNSeniorious000

Copy link
Copy Markdown
Owner Author

Merged main in — the conflict was both of the bullets this PR touches, and it was not resolvable by taking either side.

The gather bullet is main's: #35 landed a cost clause there that this branch predates.

The surfacing bullet is main's wording with this PR's two contributions folded back in — the Every tool answers in FULL clause and the named example. Taking this branch's side wholesale would have restored Ending a cell on the raw result spends your context on fields you cannot use, which #27 deleted on main; the merge is clean precisely because that sentence did not come back. post-merge-check confirms all seven landed markers, that one included, and the suite is green.

Repository owner deleted a comment from chatgpt-codex-connector Bot Sep 1, 2026
Repository owner deleted a comment from coderabbitai Bot Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant