You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 0c2811b
Browse filesBrowse the repository at this point in the historyBrowse files
feat(cli): --dry-run, --base-url, --timeout, and the load report in the artifact (#94)
A dry run loads and checks the dataset, builds the adapter, and applies
the cost guard through the same plan the run starts from, then prints the
case count, endpoint, timeout, semantics, and estimate, and sends nothing.
--base-url and --timeout reach an adapter that takes them, and the timeout
is recorded beside the endpoint. --semantics now overrides any adapter
that takes it, and the report says the operator set it. An option an
adapter does not take is refused by name, read from its signature.
An artifact stores a LoadSummary, so plumbline report rebuilds the Dataset
section it used to drop.
Fixes#58.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: README.md
+10-3Lines changed: 10 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -237,6 +237,13 @@ Without a pricing table the run still works; cost reports as unpriced, and
237
237
`--max-cost-usd` refuses rather than bounding a run it cannot cost. See
238
238
[Limitations](#limitations).
239
239
240
+
Add `--dry-run` to either command first to see what the run would do without
241
+
spending anything. It loads and checks the dataset, builds the adapter, applies
242
+
the guard, and prints the case count, the endpoint, and the estimated cost, then
243
+
exits without sending a request or writing a file. Pass `--base-url` to point
244
+
`typesafe_wire` at a self-hosted endpoint, and `--timeout` to bound one request;
245
+
both are recorded in the artifact.
246
+
240
247
Your own data goes in `datasets/private/`, which is gitignored, and that is the
241
248
only path on which the recalibration numbers mean anything.
242
249
[Your own data](docs/datasets.md) gives the row format, what is refused and why,
@@ -290,7 +297,7 @@ and self-hosted servers speak it, which is why one adapter covers all of them.
290
297
291
298
| Adapter | Transport | Semantics | Run for real | Adding one |
292
299
|---|---|---|---|---|
293
-
|`typesafe_wire`| Jev wire format over HTTP |`calibrated_claim`| Yes, 40 rows against a hosted vendor |A `base_url`. Anything serving the same wire format is a config entry, including self-hosted endpoints and open models behind a compatible server. |
300
+
|`typesafe_wire`| Jev wire format over HTTP |`calibrated_claim`| Yes, 40 rows against a hosted vendor |`--base-url`. Anything serving the same wire format is one flag, recorded in the artifact, including self-hosted endpoints and open models behind a compatible server. |
294
301
|`local_logits`| Option-token logits from a local checkpoint |`restricted_softmax`|**No, tests only**| A HuggingFace model id and a pinned revision. Needs the optional `local` extra. |
295
302
|`generative`| Chat completion, parsed |`none`|**No, tests only**| A model string. |
296
303
|`mock`| None, seeded | configurable | Yes, it is the example report | Built in. A deterministic stand-in, not a system under test. |
@@ -369,8 +376,8 @@ Specific, and none of them are going to surprise you later.
0 commit comments