Skip to content

Make friendly the default output level and score subtests - #45

Merged
eliotwrobson merged 3 commits into
mainfrom
claude/friendly-default
Nov 17, 2026
Merged

eliotwrobson merged 3 commits into
mainfrom
claude/friendly-default

Conversation

@eliotwrobson

@eliotwrobson eliotwrobson commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Tests without an output level now use "friendly" instead of "message". Every converted PrairieLearn example question set friendly itself, mostly on each test. The message default starts every wrong answer with "Student code grading failed with an exception: AssertionError", which reads as if the student's code crashed.

  • Default: get_output_level_marker falls back to Friendly. ConfigObject(output_level=...) and @pytest.mark.output still override it.
  • Docs: README, quick_start.md, converting.md and the ConfigObject docstring say friendly is the default. Examples now show overriding it with traceback. The level tables also no longer say that friendly never shows a traceback: when a call to a student function raises, its message includes the traceback from the student's code.
  • Tests: scenarios that relied on the old default now expect the friendly wording. test_global_output_level now sets message, so it still checks that the ConfigObject setting applies. test_fixture_exception passed its level positionally, which the grader ignores, so it now passes level="message".

This changes what students see in existing questions that don't set a level. For example, a startup error shows "division by zero" instead of "Student code execution failed with an exception: ZeroDivisionError / Exception Message: division by zero".

Partial credit across inputs

Tests can use pytest's built-in subtests fixture to split their points across inputs. This replaces the 15–25 line query_function_raw loops in the example course.

  • Scoring: a test with subtests gets the fraction of them that passed. A skipped subtest counts as failed. A failure outside any subtest gives 0, or the last set_score, and set_score_final still overrides.
  • Messages: "7/10 checks passed.", then the first 5 failures, each after its label if it has one.
  • Timeouts: after any query times out, later queries on the sandbox raise SandboxStopped without running student code. In a subtests loop, the remaining checks fail and the message says so once. This also stops a late reply from being read as the next call's answer, including across tests that share a module_sandbox.
  • Messages from query_function now name the call with its arguments, such as fib(23) timed out after 1.0 seconds.
  • pytest is now >=9.0,<10, since subtests were added in 9.0 and pytest calls them experimental. tests/test_subtests.py runs them end to end, so a pytest change to how subtests report fails CI.
  • Plugin: pytest calls pytest_runtest_makereport for each subtest too, so the call result is held until pytest_runtest_logreport says whether it was a SubtestReport.

🤖 Generated with Claude Code

eliotwrobson and others added 2 commits November 17, 2026 11:25
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A test that uses pytest's subtests fixture gets the fraction of its subtests
that passed, and its message shows the first 5 failures. After any query
times out, later queries on the sandbox raise SandboxStopped, so a late reply
can't be read as the answer to the next call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@eliotwrobson eliotwrobson changed the title Make friendly the default output level Make friendly the default output level and score subtests Nov 17, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@eliotwrobson
eliotwrobson marked this pull request as ready for review November 17, 2026 17:50
@eliotwrobson
eliotwrobson merged commit 1cce490 into main Nov 17, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant