Skip to content

feat(scorers): create LLM scorers and classifiers - #307

Open
Cedric / ViaDézo1er (viadezo1er) wants to merge 6 commits into
mainfrom
cedric/scorers-create
Open

feat(scorers): create LLM scorers and classifiers#307
Cedric / ViaDézo1er (viadezo1er) wants to merge 6 commits into
mainfrom
cedric/scorers-create

Conversation

@viadezo1er

@viadezo1er Cedric / ViaDézo1er (viadezo1er) commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Add bt scorers create for prompt-based LLM scorers and classifiers.

Parameter validation is a follow-up partly done in #308, and partly done in https://github.com/braintrustdata/braintrust/pull/18155 (and bt will need tweaking once the braintrust pr is merged).
At the moment it is possible to pass incorrect parameters to the scorers, and sometimes the web ui will even show them:

bt scorers create "Demo answer correctness" \
  --slug demo-answer-correctness \
  --description "Synthetic scorer used to demonstrate bt scorers create" \
  --model gpt-5.4-nano \
  --messages @/tmp/bt-scorer-messages.json \
  --choice-scores '{"PASS":1,"FAIL":0}' \
  --use-cot=false \
  --pass-threshold 0.5 \
  --metadata 'purpose: synthetic-demo' \
  --if-exists replace --temperature 1000 --reasoning-effort none
Screenshot 2026-08-14 at 16 16 22

bt-scorer-messages.json:

[
  {
    "role": "system",
    "content": "You are a strict evaluator. Compare the candidate answer with the expected answer. Return exactly PASS or FAIL."
  },
  {
    "role": "user",
    "content": "Question: {{input.question}}\nExpected answer: {{expected}}\nCandidate answer: {{output}}"
  }
]
❯ bt scorers create "Demo answer correctness" \
  --slug demo-answer-correctness \
  --description "Synthetic scorer used to demonstrate bt scorers create" \
  --model gpt-5.4-nano \
  --messages @/tmp/bt-scorer-messages.json \
  --choice-scores '{"PASS":1,"FAIL":0}' \
  --use-cot=false \
  --pass-threshold 0.5 \
  --metadata 'purpose: synthetic-demo' \
  --if-exists replace --json|jq
{
  "found_existing": true,
  "id": "6681c294-0ab3-489b-b979-33351add071d",
  "ignored": false,
  "project_id": "b667bd4c-7e55-49e0-a370-f6526e6bbccd",
  "slug": "demo-answer-correctness",
  "version": "1000197688638740786"
}

@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Latest downloadable build artifacts for this PR commit f541a46695f2:

Available artifact names
  • artifacts-build-global
  • artifacts-build-local-x86_64-apple-darwin
  • artifacts-build-local-aarch64-pc-windows-msvc
  • artifacts-build-local-x86_64-pc-windows-msvc
  • artifacts-build-local-x86_64-unknown-linux-musl
  • artifacts-build-local-x86_64-unknown-linux-gnu
  • artifacts-build-local-aarch64-apple-darwin
  • artifacts-build-local-aarch64-unknown-linux-gnu
  • artifacts-plan-dist-manifest
  • cargo-dist-cache

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9963a33d87

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

Comment thread src/functions/create.rs
…_api_key responses are classified as user/provider errors, not Braintrust authentication failures
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants