Skip to content

Commit 0c2811b

Browse files
TMHSDigitalclaude
andauthored
feat(cli): --dry-run, --base-url, --timeout, and the load report in the artifact (#94)
A dry run loads and checks the dataset, builds the adapter, and applies the cost guard through the same plan the run starts from, then prints the case count, endpoint, timeout, semantics, and estimate, and sends nothing. --base-url and --timeout reach an adapter that takes them, and the timeout is recorded beside the endpoint. --semantics now overrides any adapter that takes it, and the report says the operator set it. An option an adapter does not take is refused by name, read from its signature. An artifact stores a LoadSummary, so plumbline report rebuilds the Dataset section it used to drop. Fixes #58. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1 parent b2e062b commit 0c2811b

10 files changed

Lines changed: 528 additions & 31 deletions

File tree

‎.env.example‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,8 @@ TYPESAFE_API_KEY=
1818
# endpoint instead of the hosted service. The endpoint is part of the cache key,
1919
# so a self-hosted run is never served a hosted run's answers, and it is
2020
# recorded in the artifact and named in the report. ANTHROPIC_BASE_URL does the
21-
# same for the generative arm.
21+
# same for the generative arm. `plumbline run --base-url` sets it for one run
22+
# and takes precedence over either variable.
2223
TYPESAFE_BASE_URL=
2324

2425
# Generative control arm, which speaks Anthropic's Messages API. Only needed if

‎CHANGELOG.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,23 @@ different event from one that moved because it was wrong.
1313

1414
### Added
1515

16+
- `plumbline run --dry-run` loads and checks the dataset, builds the adapter,
17+
applies the cost guard, and prints the case count, the endpoint, the timeout,
18+
the semantics, and the estimated cost with the pricing entry behind it, then
19+
exits without sending a request or writing a file (#58). It refuses exactly
20+
what the run would refuse, because the run starts from the same plan.
21+
- `--base-url` and `--timeout` on `plumbline run`, for an adapter that takes
22+
them (#58). The endpoint could be changed only through an environment
23+
variable before; it is recorded in the artifact and the cache key either
24+
way, and the timeout is now recorded too. `--semantics` now overrides the
25+
declared semantics for any adapter that takes it, not only the mock, and the
26+
report says when the operator set it. An option an adapter does not take is
27+
refused by name before anything is built.
28+
- An artifact now stores what loading the dataset found, so a report rebuilt
29+
with `plumbline report` prints the same Dataset section, with the read,
30+
loaded, and refused counts, as the one written at run time (#58). Artifacts
31+
written before this still report, without the section.
32+
1633
- A browser calculator for the ECE floor (`site/`), a JavaScript port of
1734
`synthetic_floor` that reproduces numpy's seeded random stream draw for draw.
1835
`scripts/floor_golden.py` exports golden values from the Python and

‎CONTRIBUTING.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,8 @@ one of them. Adding a vendor should not add a file.
99

1010
- **Anything serving the Jev wire format is a `base_url` for `typesafe_wire`.**
1111
A self-hosted endpoint, a compatible server in front of an open model, a
12-
provider that implemented the same shape. It is a config entry. That includes
12+
provider that implemented the same shape. It is `plumbline run --base-url`,
13+
recorded in the artifact and part of the cache key. That includes
1314
the open decision models that serve this wire format: the unchanged adapter
1415
runs them.
1516
- **Any open-weights checkpoint is a config entry for `local_logits`**, as a

‎README.md‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -237,6 +237,13 @@ Without a pricing table the run still works; cost reports as unpriced, and
237237
`--max-cost-usd` refuses rather than bounding a run it cannot cost. See
238238
[Limitations](#limitations).
239239

240+
Add `--dry-run` to either command first to see what the run would do without
241+
spending anything. It loads and checks the dataset, builds the adapter, applies
242+
the guard, and prints the case count, the endpoint, and the estimated cost, then
243+
exits without sending a request or writing a file. Pass `--base-url` to point
244+
`typesafe_wire` at a self-hosted endpoint, and `--timeout` to bound one request;
245+
both are recorded in the artifact.
246+
240247
Your own data goes in `datasets/private/`, which is gitignored, and that is the
241248
only path on which the recalibration numbers mean anything.
242249
[Your own data](docs/datasets.md) gives the row format, what is refused and why,
@@ -290,7 +297,7 @@ and self-hosted servers speak it, which is why one adapter covers all of them.
290297

291298
| Adapter | Transport | Semantics | Run for real | Adding one |
292299
|---|---|---|---|---|
293-
| `typesafe_wire` | Jev wire format over HTTP | `calibrated_claim` | Yes, 40 rows against a hosted vendor | A `base_url`. Anything serving the same wire format is a config entry, including self-hosted endpoints and open models behind a compatible server. |
300+
| `typesafe_wire` | Jev wire format over HTTP | `calibrated_claim` | Yes, 40 rows against a hosted vendor | `--base-url`. Anything serving the same wire format is one flag, recorded in the artifact, including self-hosted endpoints and open models behind a compatible server. |
294301
| `local_logits` | Option-token logits from a local checkpoint | `restricted_softmax` | **No, tests only** | A HuggingFace model id and a pinned revision. Needs the optional `local` extra. |
295302
| `generative` | Chat completion, parsed | `none` | **No, tests only** | A model string. |
296303
| `mock` | None, seeded | configurable | Yes, it is the example report | Built in. A deterministic stand-in, not a system under test. |
@@ -369,8 +376,8 @@ Specific, and none of them are going to surprise you later.
369376
- **[Mapika/decider](https://github.com/Mapika/decider)**. One-pass typed
370377
decisions with calibrated probabilities, fine-tuned from Qwen3.5-2B, in several
371378
sizes. Its own guidance is to check calibration on your own labels before
372-
routing on confidence. It serves the same typed question shape, so it is a
373-
`base_url` config entry here rather than new code.
379+
routing on confidence. It serves the same typed question shape, so it is
380+
`--base-url` on `typesafe_wire` here rather than new code.
374381
- **[Bespoke Nimble](https://github.com/bespokelabsai/nimble)**. An open recipe
375382
for typed decision models, a LoRA fine-tune on Qwen3.5-9B, trained with
376383
contrastive data curation. It serves the Jev wire format, so the unchanged

‎src/plumbline/adapters/registry.py‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,22 @@ def _missing_arguments(factory: AdapterFactory, config: dict[str, Any]) -> list[
110110
]
111111

112112

113+
def accepts(name: str, setting: str) -> bool:
114+
"""Whether the adapter registered as ``name`` takes ``setting`` when it is built.
115+
116+
Imports a built-in to read its signature, as ``create`` would, so a caller can
117+
name the one setting an adapter refuses before trying to build it.
118+
"""
119+
try:
120+
parameters = inspect.signature(_factory(name)).parameters.values()
121+
except (TypeError, ValueError): # no signature to read, so let create decide
122+
return True
123+
return any(
124+
parameter.name == setting or parameter.kind is inspect.Parameter.VAR_KEYWORD
125+
for parameter in parameters
126+
)
127+
128+
113129
def available() -> tuple[str, ...]:
114130
"""Registered adapter names, sorted, without importing any of them."""
115131
return tuple(sorted({*_REGISTRY, *_BUILTINS}))

‎src/plumbline/cli.py‎

Lines changed: 127 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -76,13 +76,34 @@ def run(
7676
"truncation: use --limit to run fewer.",
7777
),
7878
] = None,
79+
base_url: Annotated[
80+
str | None,
81+
typer.Option(
82+
"--base-url",
83+
help="Send requests here instead of the vendor's endpoint, for an adapter that "
84+
"takes one. Recorded in the artifact and part of the cache key.",
85+
),
86+
] = None,
87+
timeout: Annotated[
88+
float | None,
89+
typer.Option(
90+
"--timeout", help="Seconds one request may take, for an adapter that takes it."
91+
),
92+
] = None,
7993
semantics: Annotated[
8094
str | None,
8195
typer.Option(
82-
help="Override probability_semantics, mock only: one of "
83-
f"{', '.join(PROBABILITY_SEMANTICS)}."
96+
help="Override the probability_semantics the adapter declares, for one that "
97+
f"takes it: one of {', '.join(PROBABILITY_SEMANTICS)}. The report says so."
8498
),
8599
] = None,
100+
dry_run: Annotated[
101+
bool,
102+
typer.Option(
103+
"--dry-run",
104+
help="Load, check, and price the run, print what it would do, and send nothing.",
105+
),
106+
] = False,
86107
seed: Annotated[int, typer.Option(help="Mock seed.")] = 7,
87108
accuracy: Annotated[float, typer.Option(help="Mock target accuracy.")] = 0.8,
88109
escalation_cost: Annotated[
@@ -102,12 +123,15 @@ def run(
102123
103124
Status lines and errors go to stderr, so stdout carries only the report when
104125
no --report is given. The exit code is 1 when no case produced a prediction,
105-
after the artifact and the report are written.
126+
after the artifact and the report are written. With --dry-run, stdout carries
127+
the plan instead, and nothing is sent or written.
106128
"""
107129
# Everything that can be checked before a call goes out is checked here,
108130
# so a mistake costs nothing.
109131
if report is not None and report.is_dir():
110132
_fail(f"--report {report} is a directory; give it a file path, such as {report}/report.md")
133+
if timeout is not None and not timeout > 0:
134+
_fail(f"--timeout must be a number of seconds above 0, got {timeout}")
111135
load = dataclasses.replace(_load(dataset, data_format), source=_shown(dataset))
112136
_status(load.statement())
113137
for refusal in load.refusals:
@@ -124,19 +148,32 @@ def run(
124148
cases,
125149
model=model,
126150
revision=revision,
151+
base_url=base_url,
152+
timeout=timeout,
127153
semantics=semantics,
128154
seed=seed,
129155
accuracy=accuracy,
130156
)
157+
guard = execute.CostGuard(max_cost_usd=max_cost_usd, max_cases=max_cases)
158+
table = _pricing(pricing)
159+
if dry_run:
160+
planned = _guard(lambda: execute.plan(built, cases, guard=guard, pricing_table=table))
161+
typer.echo(_plan_text(built, planned, guard, semantics_set=semantics is not None))
162+
return
163+
164+
extra: dict[str, object] = {"dataset": _shown(dataset), "format": data_format}
165+
if semantics is not None:
166+
extra["semantics_set_by"] = "operator"
131167
result = _guard(
132168
lambda: execute.run(
133169
built,
134170
cases,
135171
cache=Cache(cache_dir) if cache_dir else None,
136-
guard=execute.CostGuard(max_cost_usd=max_cost_usd, max_cases=max_cases),
137-
pricing_table=_pricing(pricing),
172+
guard=guard,
173+
pricing_table=table,
138174
workers=workers,
139-
extra_config={"dataset": _shown(dataset), "format": data_format},
175+
extra_config=extra,
176+
load=load.summary(),
140177
)
141178
)
142179

@@ -192,6 +229,7 @@ def report(
192229
document = _guard(
193230
lambda: markdown.render(
194231
results,
232+
load=_stored_load(results),
195233
options=markdown.ReportOptions(
196234
n_boot=n_boot,
197235
cost_escalation_usd=escalation_cost,
@@ -208,6 +246,20 @@ def report(
208246
typer.echo(document)
209247

210248

249+
def _stored_load(results: list[execute.RunResult]) -> loader.LoadSummary | None:
250+
"""The Dataset section a rebuilt report can print, from what the runs stored.
251+
252+
Only when every run is over one dataset and they all say the same thing
253+
about loading it. Otherwise there is no single section that is true of the
254+
document, and printing one run's counts over another's figures would be
255+
worse than printing none.
256+
"""
257+
if len({result.dataset_hash for result in results}) != 1:
258+
return None
259+
first = results[0].load
260+
return first if all(result.load == first for result in results) else None
261+
262+
211263
@app.command()
212264
def adapters() -> None:
213265
"""List the transports this install can run."""
@@ -253,6 +305,8 @@ def _build(
253305
*,
254306
model: str | None,
255307
revision: str | None,
308+
base_url: str | None = None,
309+
timeout: float | None = None,
256310
semantics: str | None,
257311
seed: int,
258312
accuracy: float,
@@ -263,33 +317,93 @@ def _build(
263317
config["model_requested"] = model
264318
if revision is not None:
265319
config["revision"] = revision
320+
if base_url is not None:
321+
config["base_url"] = base_url
322+
if timeout is not None:
323+
config["timeout"] = timeout
324+
if semantics is not None:
325+
config["probability_semantics"] = _semantics(semantics)
266326

267327
if adapter == "mock":
268328
# The mock is told the answer key up front, because classify() is never
269329
# given the gold label. It is a demo arm, not a system under test.
270330
config["gold_by_text"] = {case.text: case.gold_label for case in cases}
271331
config["seed"] = seed
272332
config["accuracy"] = accuracy
273-
if semantics is not None:
274-
config["probability_semantics"] = _semantics(semantics)
275-
elif semantics is not None:
276-
_fail("--semantics applies to the mock only; a real adapter declares its own.")
333+
334+
# Name the exact option an adapter will not take, before building it: the
335+
# generative arm reports no probability, so it has no semantics to override.
336+
refused_options = [
337+
flag
338+
for setting, flag in _OPTIONS.items()
339+
if setting in config and not _guard(partial(registry.accepts, adapter, setting))
340+
]
341+
if refused_options:
342+
_fail(f"the {adapter} adapter does not take {', '.join(refused_options)}.")
277343

278344
try:
279345
return registry.create(adapter, **config)
280346
except (PlumblineError, ValueError) as refused:
281347
_fail(str(refused))
282348
except TypeError:
283-
given = [f"--{name.replace('_requested', '')}" for name in config if name in _OPTIONS]
349+
given = [flag for setting, flag in _OPTIONS.items() if setting in config]
284350
_fail(f"the {adapter} adapter does not take {', '.join(given) or 'these settings'}.")
285351
except Exception as unavailable: # an SDK that cannot start, such as a missing key
286352
variable = _KEY_VARIABLES.get(adapter)
287353
hint = f" Set {variable} in the environment." if variable else ""
288354
_fail(f"could not set up the {adapter} adapter: {unavailable}.{hint}")
289355

290356

291-
#: Settings the CLI passes to an adapter from its own options.
292-
_OPTIONS = ("model_requested", "revision")
357+
#: Settings the CLI passes to an adapter from its own options, and the option
358+
#: each comes from.
359+
_OPTIONS = {
360+
"model_requested": "--model",
361+
"revision": "--revision",
362+
"base_url": "--base-url",
363+
"timeout": "--timeout",
364+
"probability_semantics": "--semantics",
365+
}
366+
367+
368+
def _plan_text(
369+
adapter: Adapter, planned: execute.Plan, guard: execute.CostGuard, *, semantics_set: bool
370+
) -> str:
371+
"""What a dry run prints: what would be sent, to whom, and what it would cost."""
372+
endpoint = getattr(adapter, "base_url", None)
373+
timeout = getattr(adapter, "timeout", None)
374+
semantics = adapter.probability_semantics + (", set by --semantics" if semantics_set else "")
375+
if planned.estimated_cost_usd is not None and planned.pricing is not None:
376+
cost = (
377+
f"about {planned.estimated_cost_usd:.4f} USD, priced by `{planned.pricing_key}` "
378+
f"as read on {planned.pricing.as_of.isoformat()}. The estimate is rough: it counts "
379+
"the case text, the option names, and a fixed overhead."
380+
)
381+
else:
382+
cost = (
383+
f"not estimated: `{adapter.model_requested}` is not in the pricing table. Pass "
384+
"--pricing with an entry for it to cost the run in advance."
385+
)
386+
cost_limit = (
387+
f"max cost {guard.max_cost_usd} USD" if guard.max_cost_usd is not None else "no cost limit"
388+
)
389+
case_limit = f"max cases {guard.max_cases}" if guard.max_cases is not None else "no case limit"
390+
revision = f", revision `{adapter.revision}`" if adapter.revision else ""
391+
return "\n".join(
392+
[
393+
"dry run: nothing was sent and nothing was written.",
394+
"",
395+
f"- {planned.cases} cases for the {adapter.name} adapter, model "
396+
f"`{adapter.model_requested}`{revision}.",
397+
f"- Endpoint: `{endpoint}`." if endpoint else "- Endpoint: the adapter's default.",
398+
f"- Timeout: {timeout:g} s per request."
399+
if timeout
400+
else "- Timeout: the adapter's default.",
401+
f"- Probability semantics: {semantics}.",
402+
f"- Cost: {cost}",
403+
f"- Guard: {cost_limit}, {case_limit}; the run would start.",
404+
]
405+
)
406+
293407

294408
#: Where each hosted adapter reads its key, for the message when it is missing.
295409
_KEY_VARIABLES = {"typesafe_wire": "TYPESAFE_API_KEY", "generative": "ANTHROPIC_API_KEY"}

0 commit comments

Comments
 (0)