Repository navigation
fix: price unnamed responses by the requested model, key on option order where it matters, send option descriptions - #88
Merged
Conversation
…der where it matters, and send option descriptions #36: a row whose response named no model got cost None and basis model_not_priced, because pricing looked up only model_reported, while the pre-run guard priced the same run by the requested model. Such a row is now priced by the requested model and records that key. A response that names a model missing from the table stays unpriced: billing follows what answered, and a newer version never inherits an older rate. #37: cache_key sorted the labels, but generative and local_logits list the options in their prompts in the order given, so two orderings shared a key while sending different prompts. An adapter now declares label_order_matters, and for those two the key keeps the given order. Every other adapter still sorts, so reordering options reuses answers. #39: label_descriptions were parsed and then dropped. typesafe_wire sent criteria as None for every option; the runner never passed descriptions to any adapter; and dataset_hash ignored descriptions and question_type, so two datasets differing only in those hashed the same. An adapter now declares uses_label_descriptions; typesafe_wire does, sends them as the choice's criteria, and keys its cache on them. dataset_hash covers question_type and descriptions where a row carries them, so a dataset of plain choice rows keeps its hash. The artifact records how many rows carried descriptions and whether they were sent, and the report says so when they were not. The public fixture has yes/no rows and descriptions, so its dataset hash changes from c18e9496 to 1b96dc91. The example report is regenerated: it gains the descriptions note and the new hash, and no number moves. The README's sample artifact name follows. Fixes #36. Fixes #37. Fixes #39. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #36. Fixes #37. Fixes #39.
#36: pricing a response that named no model.
Nonewith basismodel_not_priced, because pricing looked up onlymodel_reported, while the pre-run guard priced the same run by the requested model.#37: option order in the cache key.
cache_keysorted the labels, butgenerativeandlocal_logitslist the options in their prompts in the given order, so two orderings shared a key while sending different prompts.label_order_matters, and for those two the key keeps the given order.#39: option descriptions were parsed, then dropped.
typesafe_wiresent every criterion asNone, the runner never passed descriptions to any adapter, anddataset_hashignored descriptions andquestion_type.uses_label_descriptions.typesafe_wiredoes: it sends them as the choice's criteria and keys its cache on them.dataset_hashcoversquestion_typeand descriptions where a row carries them, so a dataset of plain choice rows keeps its hash. A test pins that.Visible change. The public fixture has yes/no rows and descriptions, so its dataset hash changes from
c18e9496to1b96dc91. The regenerated example report gains the descriptions note and the new hash, and no number changes. The README's sample artifact name follows.Checked
mainand pass now:mock-1;ruff,mypy --strict, and the fullpytestsuite pass.🤖 Generated with Claude Code