Skip to content

fix: price unnamed responses by the requested model, key on option order where it matters, send option descriptions - #88

Merged
TMHSDigital merged 1 commit into
mainfrom
fix/pricing-fallback-order-descriptions
Sep 23, 2026
Merged

TMHSDigital merged 1 commit into
mainfrom
fix/pricing-fallback-order-descriptions

Conversation

@TMHSDigital

Copy link
Copy Markdown
Owner

Fixes #36. Fixes #37. Fixes #39.

#36: pricing a response that named no model.

  • Such a row got cost None with basis model_not_priced, because pricing looked up only model_reported, while the pre-run guard priced the same run by the requested model.
  • It is now priced by the requested model, and its pricing key says so.
  • A response that names a model missing from the table stays unpriced. Billing follows what answered, and a newer version never inherits an older rate. A test pins that.

#37: option order in the cache key. cache_key sorted the labels, but generative and local_logits list the options in their prompts in the given order, so two orderings shared a key while sending different prompts.

  • Adapters now declare label_order_matters, and for those two the key keeps the given order.
  • Every other adapter still sorts, so reordering the options still reuses cached answers.

#39: option descriptions were parsed, then dropped.

  • typesafe_wire sent every criterion as None, the runner never passed descriptions to any adapter, and dataset_hash ignored descriptions and question_type.
  • Adapters now declare uses_label_descriptions. typesafe_wire does: it sends them as the choice's criteria and keys its cache on them.
  • dataset_hash covers question_type and descriptions where a row carries them, so a dataset of plain choice rows keeps its hash. A test pins that.
  • The artifact records how many rows carried descriptions and whether they were sent, and the report says so when they weren't.

Visible change. The public fixture has yes/no rows and descriptions, so its dataset hash changes from c18e9496 to 1b96dc91. The regenerated example report gains the descriptions note and the new hash, and no number changes. The README's sample artifact name follows.

Checked

  • New tests failed on main and pass now:
    • an unnamed response is priced by mock-1;
    • an order-sensitive adapter's keys differ by option order;
    • the hash separates choice, noul, and described rows;
    • descriptions reach TypeSafe's criteria;
    • the report says the descriptions weren't sent.
  • ruff, mypy --strict, and the full pytest suite pass.
  • The site build and all four site checks pass.

🤖 Generated with Claude Code

…der where it matters, and send option descriptions

#36: a row whose response named no model got cost None and basis
model_not_priced, because pricing looked up only model_reported, while
the pre-run guard priced the same run by the requested model. Such a row
is now priced by the requested model and records that key. A response
that names a model missing from the table stays unpriced: billing
follows what answered, and a newer version never inherits an older rate.

#37: cache_key sorted the labels, but generative and local_logits list
the options in their prompts in the order given, so two orderings shared
a key while sending different prompts. An adapter now declares
label_order_matters, and for those two the key keeps the given order.
Every other adapter still sorts, so reordering options reuses answers.

#39: label_descriptions were parsed and then dropped. typesafe_wire sent
criteria as None for every option; the runner never passed descriptions
to any adapter; and dataset_hash ignored descriptions and question_type,
so two datasets differing only in those hashed the same. An adapter now
declares uses_label_descriptions; typesafe_wire does, sends them as the
choice's criteria, and keys its cache on them. dataset_hash covers
question_type and descriptions where a row carries them, so a dataset of
plain choice rows keeps its hash. The artifact records how many rows
carried descriptions and whether they were sent, and the report says so
when they were not.

The public fixture has yes/no rows and descriptions, so its dataset hash
changes from c18e9496 to 1b96dc91. The example report is regenerated: it
gains the descriptions note and the new hash, and no number moves. The
README's sample artifact name follows.

Fixes #36. Fixes #37. Fixes #39.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@TMHSDigital
TMHSDigital merged commit 3aef2a2 into main Sep 23, 2026
17 checks passed
@TMHSDigital
TMHSDigital deleted the fix/pricing-fallback-order-descriptions branch September 23, 2026 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant