docs: name the compile step the quickstart depends on (COR-13656) - #30
Open
Bradenream wants to merge 3 commits into
Open
docs: name the compile step the quickstart depends on (COR-13656)#30Bradenream wants to merge 3 commits into
Bradenream wants to merge 3 commits into
Conversation
A cold agent re-ran the quickstart end to end against the current CLI. It got a working, correctly-configured agent in 5.8 minutes and 50 commands, with one failed invocation — against ~45 minutes on the same task before this round of work. Everything it lost time to was in this file. 1. `vf environment compile` was missing from the workflow the quickstart names. `agent update` writes the definition; `compile` builds the form a conversation actually runs. Skip it and every signal says the change landed — the update returns `Agent updated.`, `vf agent get` reads your text back — while the runtime keeps answering as the template. Nothing errors. The causality was verified rather than assumed, by injecting a marker string: stored but absent from replies before compile, present after. This is the worst failure mode in the product because it returns a confident, plausible, wrong answer, and the paragraph listing the steps to escape the placeholder template omitted the one command that does it. The quickstart is now the three commands in order, with the trap named directly underneath. 2. "Either works immediately" was true and misleading. A new project does have a release, so `published` answers — but that release is a snapshot of the template, so it keeps saying Acme Corp until you publish your own. Test on draft, ship published, and you ship an Acme Corp agent. 3. The response envelope was described but never shown. "Replies arrive as text traces" is accurate and not enough: a typical turn returns five debug traces for one text trace, and a debug payload is a bare string, so the obvious recursive jq errors on it. Measured on a real response, then shown as a worked filter. Verified by following the edited quickstart verbatim on a fresh project: create, update, compile, publish, then converse. Both draft and published answer as the configured agent, with no placeholder text. The project was deleted afterwards. All three edits sit outside the Speakeasy-managed blocks, so regeneration will not undo them.
There was a problem hiding this comment.
🟡 Changes recommended
The updated README contains a couple of factual/inconsistency issues that could mislead users when parsing traces or interpreting agent-mode detection.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR updates the README quickstart guidance for the vf CLI to prevent a common “template agent” trap and to make response parsing more reliable when scripting.
Changes:
- Adds
vf environment compileto the quickstart workflow and explains why it’s required for runtime behavior to reflect updates. - Clarifies the difference between
draftandpublishedversions for new projects to avoid unintentionally shipping the template agent. - Provides a concrete
jqfilter example that extracts onlytexttraces from conversation responses.
File summaries
| File | Description |
|---|---|
| README.md | Updates the quickstart narrative and examples to include compile/publish steps and safer trace parsing guidance. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+101
to
+103
| The agent's replies arrive as `text` traces in the response. Most traces are not replies — | ||
| a typical turn returns five `debug` traces for one `text` — and a `debug` payload is a bare | ||
| string, so a recursive filter errors on it. Select the type first: |
Comment on lines
+118
to
+122
| **`vf environment compile` is required, and it is easy to miss.** `agent update` writes the | ||
| definition; `compile` builds the form a conversation actually runs. Without it every signal | ||
| says the change landed — the update returns `Agent updated.`, and `vf agent get` reads your | ||
| text back — while the runtime keeps answering as the template. Nothing errors. If your agent | ||
| still introduces itself as Acme Corp, this is why. |
| - **`--version-param draft` runs what you are editing; `published` runs the live version.** A new project is created with a first release already in place, so either works immediately — you do not have to publish anything first. | ||
| - **`--version-param draft` runs what you are editing; `published` runs the live version.** A new project is created with a first release already in place, so both work immediately — but that release is a snapshot of the **template**, so `published` keeps answering as Acme Corp until you publish your own. Test on `draft`, and publish before you rely on `published`. | ||
| - **`--action '{"type":"launch"}'` is optional.** Sending a `text` action straight away works; the runtime starts the conversation itself. The launch step is shown because it makes the first turn explicit. | ||
| - **Pass `--output-format json` explicitly when piping.** Inside AI coding agents (`CLAUDECODE`, `CURSOR_AGENT`, …) the default output is TOON, not JSON. |
…656)
A subagent reviewed this PR by running its claims against the live API rather
than reading them. It found three errors, all mine, all verified independently
before this fix.
1. The Acme diagnostic was false, and it was the worst kind of false.
"If your agent still introduces itself as Acme Corp, this is why" pointed at
the missing compile. But "Acme Corp" lives in `agent.prompt`, and this PR's
own example only set `--instructions` — which never touches it. Confirmed by
field scan on a fresh project: `agent.prompt` is the sole match. So a reader
could follow the example, compile correctly, still see Acme Corp, and
conclude that compile was broken. A diagnostic that fires after its own fix
is worse than no diagnostic.
The quickstart never mentioned `--prompt` at all. It now names both fields
and what each one holds, and the example sets both. Verified end to end on a
fresh project: zero occurrences of "Acme" in the configured agent afterwards.
2. `environment publish` compiles too, so the compile line was redundant exactly
where this PR put it.
Proven by injecting a marker, confirming six stale draft probes, then
publishing with no explicit compile — the draft immediately served the marker.
The snippet teaching the trap was therefore the one snippet in which the step
does nothing. Compile now sits in the draft-testing loop, which is the loop
the quickstart actually uses and the only place the step is yours to
remember, and the text says publish compiles as well.
3. The trace hazard was described with the wrong mechanism.
"A debug payload is a bare string, so a recursive filter errors on it" is
wrong twice: every payload is an object, and the naive filter does not error
— it silently returns the runtime's log lines interleaved with the reply
("Main Agent" starting execution, ai result, resources consumption). A quiet
wrong answer, not a loud one, which is the more dangerous failure and the
better reason to select the type. Measured: 5 debug, 2 block, 1 text on a
typical turn; "five" is now "five to seven".
Also from the same review: `vf test run create` requires a `--test-id` the
quickstart never creates, so it is now `vf test create`; the template string is
"Acme Corp", not "Acme Corp support"; and the jq example is a full runnable
command rather than a fragment starting with an ellipsis.
Verified by following the corrected quickstart verbatim on a fresh project:
update both fields, compile, converse. The reply is the configured agent, the
documented jq block runs as printed, and no "Acme" remains. Project deleted.
Second adversarial review, again by running the claims rather than reading them. Two real errors, both verified independently before this fix. 1. The piped example I added made a loud failure silent — in exactly the environment this README claims to serve. Run it with something else holding stdin (a CI step, a coding agent), and the CLI correctly refuses to wait forever — but `jq` succeeds on empty input, so the pipeline exits 0. Measured: the stdin error on stderr, no reply, and PIPELINE EXIT: 0. A scripted caller sees success and no answer. Unpiped commands in the same quickstart exit 1 here. So the one example this PR introduced was the one that swallowed its own failure. It now carries `</dev/null`, and the reason is stated rather than left as a magic incantation. 2. "Five to seven debug traces for one text" is wrong 19% of the time, and it is wrong because of the step this same section recommends. Measured over 52 turns by the reviewer, then reproduced here: a bare template turn gives 6. Add a knowledge base — `vf document create-url`, which this section names as the next thing to do — and the same turns give 10, 14, 10, with one returning two `text` traces rather than one. Every tool round-trip adds roughly four debug traces. The sentence now describes the mechanism instead of quoting a range that the next paragraph invalidates, and tells the reader not to depend on the shape. `end` is added to the trace list; it was missing and is the one a scripted loop most needs. Also from the same review: `--prompt` replaces the entire field, and the template ships Role, Goal, Tone and Guardrails sections — including four explicit safety rules — so the one-line example silently discards all of them. Calling it "the persona" invited exactly that misreading. Now stated, with a pointer to read the current value first. And the bullet that says to publish before relying on `published` now shows the command, with the `--name` its naive form omits. Verified: the corrected piped example run verbatim under an open stdin returns the reply; the publish command runs as printed. Test projects deleted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The quickstart had a 25-minute cliff in it. A cold agent re-running it end to end lost all of its time to this file, so this fixes what it hit.
1.
vf environment compilewas missing from the workflow the quickstart namesagent updatewrites the definition;compilebuilds the form a conversation actually runs. Skip it and every signal says the change landed — the update returnsAgent updated.,vf agent getreads your text back — while the runtime keeps answering as the template. Nothing errors.Verified by marker injection, and separately confirmed there is no lazy compile: an uncompiled update was still serving the template 5m44s later.
vf environment publishcompiles too, so a publish-then-test loop never hits this. It bites ondraft— which is the loop the quickstart itself uses.2. The quickstart never mentioned
--prompt, which is where the brand livesAcme Corpis inagent.prompt.--instructionsnever touches it, so following the old advice and compiling correctly still left an agent introducing itself as Acme Corp. The quickstart now names both fields and sets both.Also notes that
--promptreplaces the whole field — the template ships Role, Goal, Tone and Guardrails sections, including four explicit safety rules, and a one-line prompt discards all four.3.
--version-param publishedserves the templateA new project has a release, but it is a snapshot of the template — so
publishedanswers as Acme Corp until you publish your own. Test ondraft, shippublished, and you ship an Acme Corp agent. The bullet now shows the publish command, with the--nameits naive form omits.4. The response envelope
texttraces arrive mixed withdebug,block,choiceandend. Debug payloads carry their own.message, so an unselectivejqdoes not fail — it quietly returns the runtime's log lines interleaved with the reply. Filter on type.The example carries
</dev/null, and says why: when something else owns stdin,jqsucceeds on empty input and the whole pipeline exits 0 — success, and no reply.Trace counts are described by mechanism rather than by a range: a bare turn gives about six debug traces, each tool call adds roughly four, and a turn can return more than one
text.Two adversarial reviews, five real errors
Both reviews ran the claims against the live API rather than reading them. Between them they found five errors in earlier versions of this PR, all mine, all fixed here:
--instructionscompileplaced beforepublishdocument create-url, recommended 20 lines laterThe last two were introduced by the fixes for the first three, which is the argument for having run the second review at all.
Verification
The corrected quickstart was followed verbatim on fresh projects each round: create → set both fields → compile → converse → publish. The reply is the configured agent, zero
Acmeremains, the piped example returns the reply under an open stdin, and the publish command runs as printed. All test projects deleted.All edits sit outside the Speakeasy-managed blocks, so regeneration will not undo them.
Not fixed here
agent updateshould compile the draft itself, or say that a compile is required. That is platform-side — tracked as COR-13749. This PR is the documentation half: it turns a silent cliff into a step you can follow.