Fix TeleMath answer parsing and numeric tolerance - #2
Merged
Conversation
The answer parser under-scored every model in the pool by 3 to 8 points. The
error is not uniform across models, so it moved per-cluster error rates
relative to each other and changed the Stage 1 routing, not only accuracy
levels.
The scoring change is confined to TeleMath. parse_telemath_answer and
numeric_match are reached only by the telemath, telemath_nothink and
telemath_gemma4 tasks; AIME and TeleQnA match exactly and their verdicts are
untouched. The version stamp and both guards below apply to every task but
change no score.
Three parsing faults:
- A fraction in a final answer was split into separate digit tokens by the
last-number fallback, so 7/6 scored as 6.0 rather than 1.1667. \frac,
\dfrac, \tfrac and bare a/b are now read as one value.
- \boxed{} had to hold a bare number, so a trailing unit or comma thousands
separators fell through to a much noisier full-text scan. A leading value
followed by a \text{...} unit is now accepted, while other trailing content
is still refused as unevaluated maths, so \boxed{2e^{-2}} is not truncated
to 2.
- Only the final \boxed{} was tried, so a model that boxed a clean decimal and
then restated it symbolically had the unparseable second box shadow the
first. Boxes are now scanned last to first.
numeric_match used abs_tol=1e-9, but five TeleMath gold answers are smaller
than that, down to 1e-10, so any answer within 1e-9 including zero scored
correct against them. Set to 0 and rely on the relative tolerance.
Add SCORER_VERSION, stamped into stats, generations and outcomes, so an
artifact records which grader produced it. prep_qe.qe_row refuses a generation
stamped with an older version and warns when it copies a stored verdict.
Labels taken from the old field are what produced the first quality
estimators, which learned to predict parser failures rather than wrong
answers.
Add check_pre_rendered, which refuses un-templated prompts on a task whose
chat template is baked into the dataset. Feeding raw data to such a task
truncated 52% of outputs and gave an accuracy of 0.005.
ymoslem
force-pushed
the
fix/telemath-scorer
branch
from
August 14, 2026 02:38
264fe6e to
21e4fa1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes blind spots in the TeleMath scorer.
Affects TeleMath only. AIME and TeleQnA match answers exactly and are unchanged.
Problem
7/6scored as6.0.\boxed{}had to hold a bare number, so a trailing unit or comma separators were missed.\boxed{}was tried, so a clean decimal followed by a symbolic restatement was lost.abs_tol=1e-9was larger than five gold answers, so answering zero scored correct against them.Together these under-scored every model in the pool by 3 to 8 points. The amount varies by model, so the Stage 1 routing shifts too.
Changes
abs_tolto 0 and rely on the relative tolerance.SCORER_VERSIONto stats, generations and outcomes.prep_qe.qe_rowrefuses a generation graded by an older scorer.check_pre_renderedfor tasks whose chat template is already in the data.Testing
185 pass, with new cases for each fault and both guards.