Repository navigation
Publish the reproducible specialist evaluation suite - #6
Merged
Merged
Conversation
hudsonaikins
marked this pull request as ready for review
November 13, 2026 04:45
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 25f37b657c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Publish the reproducible four-case specialist evaluation package so a clean checkout can prepare the fixtures, run the scripted SmolVM reference suite, and compare qualified live models through LangSmith. The cases cover sales/refunds, missing-cost refusal, irregular energy and service availability; private references remain outside runtime mounts. README and roadmap point to runnable commands and distinguish measured capability from release acceptance.
Review fixes enforce declared metric IDs and cell bindings, reject negated/ambiguous missing-input explanations, and make live CLI success require complete applicable accounting and review eligibility. Numerical references and model inputs are unchanged. SQLite dataset identity is checked within each frozen environment rather than hardcoding a host-specific file encoding. Historical feedback stays intact; any revised grading is recorded separately with its evaluator identity.
Validation: the full offline suite passed 204 tests with 15 optional skips before one additional refusal-control case; the final focused suite passes 47 tests. Ruff, format, Vulture and Gitleaks checks pass. The real scripted SmolVM suite passed 4/4 with three exported bundles and no provider calls. The original two-case SmolVM integration passed 7 tests. Refreshed six-store runtime validation passed all stores, captured commit 3d61a72 and source/runner identities, used zero model calls or data reloads, and restored service state.
No runtime default or specialist prompt changes are included. Prompt guidance is isolated in stacked PR #7. One trial per case is not general reliability or release acceptance.
Tracks INT-451.