Repository navigation
Make friendly the default output level and score subtests - #45
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A test that uses pytest's subtests fixture gets the fraction of its subtests that passed, and its message shows the first 5 failures. After any query times out, later queries on the sandbox raise SandboxStopped, so a late reply can't be read as the answer to the next call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
eliotwrobson
marked this pull request as ready for review
November 17, 2026 17:50
Merged
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tests without an output level now use
"friendly"instead of"message". Every converted PrairieLearn example question setfriendlyitself, mostly on each test. Themessagedefault starts every wrong answer with "Student code grading failed with an exception: AssertionError", which reads as if the student's code crashed.get_output_level_markerfalls back toFriendly.ConfigObject(output_level=...)and@pytest.mark.outputstill override it.quick_start.md,converting.mdand theConfigObjectdocstring sayfriendlyis the default. Examples now show overriding it withtraceback. The level tables also no longer say thatfriendlynever shows a traceback: when a call to a student function raises, its message includes the traceback from the student's code.test_global_output_levelnow setsmessage, so it still checks that theConfigObjectsetting applies.test_fixture_exceptionpassed its level positionally, which the grader ignores, so it now passeslevel="message".This changes what students see in existing questions that don't set a level. For example, a startup error shows "division by zero" instead of "Student code execution failed with an exception: ZeroDivisionError / Exception Message: division by zero".
Partial credit across inputs
Tests can use pytest's built-in
subtestsfixture to split their points across inputs. This replaces the 15–25 linequery_function_rawloops in the example course.set_score, andset_score_finalstill overrides.SandboxStoppedwithout running student code. In a subtests loop, the remaining checks fail and the message says so once. This also stops a late reply from being read as the next call's answer, including across tests that share amodule_sandbox.query_functionnow name the call with its arguments, such asfib(23) timed out after 1.0 seconds.>=9.0,<10, since subtests were added in 9.0 and pytest calls them experimental.tests/test_subtests.pyruns them end to end, so a pytest change to how subtests report fails CI.pytest_runtest_makereportfor each subtest too, so the call result is held untilpytest_runtest_logreportsays whether it was aSubtestReport.🤖 Generated with Claude Code