bt scorers create/update

[Cedric / ViaDézo1er (viadezo1er)]

messages.json:

Details

[
  { "role": "system", "content": "You are a strict grader. Reply with a single label." },
  { "role": "user", "content": "Question: {{input}}\nAnswer: {{output}}\nIs the answer correct?" }
]

metadata.yaml

Details

owner: test-team
purpose: demo

scorers_demo.sh

Details

#!/usr/bin/env bash
# Showcase for `bt scorers`. Run `bt auth` and select a project first.
# Reads messages.json and metadata.yaml from this directory (@ = read from file).

# 1. Minimal score scorer: --choice-scores maps each label to a score in [0,1].
bt scorers create "Correctness" --slug correctness \
  --model gpt-5.4-nano \
  --messages @messages.json \
  --choice-scores '{"correct":1,"incorrect":0}' \
  --if-exists replace

# 2. Fully-configured LLM judge (every knob mirrors the web UI).
bt scorers create "Quality judge" --slug quality-judge \
  --model gpt-5.4-nano \
  --messages @messages.json \
  --choice-scores '{"pass":1,"fail":0}' \
  --max-tokens 512 --top-p 0.9 \
  --template-format mustache \
  --pass-threshold 0.7 \
  --metadata @metadata.yaml \
  --if-exists replace

# 3. Classifier: label output instead of a score. --allow-no-match lets it abstain.
bt scorers create "Safety label" --slug safety-label \
  --model gpt-5.4-nano \
  --messages @messages.json \
  --classifications '["safe","unsafe"]' \
  --allow-no-match \
  --if-exists replace

# 4. Update in place; only the fields you pass change. -y skips the prompt.
bt scorers update quality-judge --pass-threshold 0.8 -y
bt scorers update quality-judge --patch '{"prompt_data":{"options":{"params":{"top_p":0.95}}}}' -y

# 5. Inspect.
bt scorers list
bt scorers view quality-judge

# Cleanup (optional).
# bt scorers delete correctness --force
# bt scorers delete quality-judge --force
# bt scorers delete safety-label --force

After using the script:

When trying to update a parameter that isn't supported:

❯ bt scorers update --temperature 1
✔ Select scorer · Correctness [slug: correctness]
error: --temperature is not supported by model 'gpt-5.4-nano' unless reasoning effort is 'none'; pass `--reasoning-effort none` or omit `--temperature`
If this seems like a bug, file an issue at https://github.com/braintrustdata/bt/issues/new and include `bt --version`, `bt status --json`, and the command you ran.

Note: the check is done on bt's side, soon even if it passes it will be done in the backend too. The check is imperfect.

LLM overview:

• New scorer creation

  • Adds bt scorers create for prompt-based LLM scorers and classifiers.
  • Supports messages, model selection, score mappings/classifications, chain-of-thought, pass thresholds, metadata, and --if-exists.

• New update commands

  • Adds bt scorers update, bt prompts update, bt functions update, and bt tools update.
  • Supports targeted configuration flags plus arbitrary deep-merged JSON via --patch.
  • Includes interactive selection/confirmation and JSON output.

• LLM configuration and validation

  • Adds flags for temperature, max tokens, top-p, penalties, stop sequences, tool choice, reasoning effort, verbosity, and template
    format.
  • Validates settings against Braintrust’s model catalog and org/project custom-model metadata.

• Input source helpers

  • Values can be inline, loaded from PATH, or read from stdin with -.
  • Adds YAML metadata parsing and recursive JSON-object merging.

• Scorer/classifier integration

  • Classifiers now appear in scorer listings and use the scorer web page.
  • Scorer-only flags are rejected when updating incompatible function types.

• API additions

  • Adds PATCH helpers for functions and prompts.
  • Adds timestamp bounds to function/dataset BTQL queries via a shared epoch constant.

• Authentication fix

  • Ensures explicitly configured API/app URLs take precedence and remain synchronized with SDK login state.

• Other behavior

  • Allows --force with interactive function deletion.
  • Expands README documentation and adds extensive unit/CLI tests.
  • Updates the skill smoke-test Codex invocation.

[github-actions]

Latest downloadable build artifacts for this PR commit 0e375860ea67:

• Workflow run: https://github.com/braintrustdata/bt/actions/runs/31760466469
• Download all artifacts (GitHub CLI): gh run download 31760466469 --repo braintrustdata/bt
• Installers are published from main automatically. To publish one for a PR branch, run release-canary manually via workflow_dispatch.

Available artifact names

• artifacts-build-global
• artifacts-build-local-aarch64-pc-windows-msvc
• artifacts-build-local-x86_64-pc-windows-msvc
• artifacts-build-local-x86_64-apple-darwin
• artifacts-build-local-x86_64-unknown-linux-musl
• artifacts-build-local-x86_64-unknown-linux-gnu
• artifacts-build-local-aarch64-apple-darwin
• artifacts-build-local-aarch64-unknown-linux-gnu
• artifacts-plan-dist-manifest
• cargo-dist-cache

[Cedric / ViaDézo1er (viadezo1er)]

Split into #306, #307, #308, #309, #310, #312, #313