Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 21 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,28 +34,28 @@ Student code runs in a separate subprocess via Unix sockets, providing:

### Student-Friendly Feedback Mode

Control how much information students see when their code fails. Use `@pytest.mark.output(level=...)` per test, or set `output_level` globally in your `ConfigObject`:
Control how much information students see when their code fails. The default is `"friendly"`. Use `@pytest.mark.output(level=...)` per test, or set `output_level` globally in your `ConfigObject`:

```python
from pytest_prairielearn_grader import ConfigObject

autograder_config = ConfigObject(
output_level="friendly", # Global default for all tests
output_level="traceback", # Global default for all tests
)
```

**Output levels:**
| Level | Shows |
|-------|-------|
| `"friendly"` | Only the exception message, with no class name *(default)*. If a call to a student function raised the error, this includes the traceback from the student's code. |
| `"none"` | Exception class name only (e.g., `AssertionError`) |
| `"message"` | Exception name + first line of message *(default)* |
| `"message"` | Exception name + first line of message |
| `"traceback"` | Full exception with traceback |
| `"friendly"` | Only the exception message text — no class name, no traceback |

Per-test markers override the global setting:

```python
@pytest.mark.output(level="friendly")
@pytest.mark.output(level="traceback")
@pytest.mark.grading_data(name="Test add", points=5)
def test_add(sandbox: StudentFixture) -> None:
assert_fn_equal(sandbox, "add", args=(2, 3), expected=5)
Expand All @@ -68,7 +68,6 @@ Student-friendly assertion functions that produce clean, readable failure messag
```python
from pytest_prairielearn_grader.assertions import assert_equal, assert_fn_equal

@pytest.mark.output(level="friendly")
@pytest.mark.grading_data(name="Test calculation", points=5)
def test_calc(sandbox: StudentFixture) -> None:
# Produces: "Checking: add(2, 3)\nExpected output: 5\nYour code output: 4\n..."
Expand All @@ -83,6 +82,22 @@ Available helpers:
- `assert_true(condition)` / `assert_false(condition)` — boolean checks
- `assert_plot_equal(plot, expected)` — compare a matplotlib plot with a reference plot

### Partial Credit Across Inputs

Check each input in a `subtests.test()` block, using pytest's built-in `subtests` fixture. The
test gets the fraction of blocks that passed, and the student sees the first few failures:

```python
@pytest.mark.grading_data(name="Check random values", points=3)
def test_random_values(sandbox: StudentFixture, subtests: pytest.Subtests) -> None:
for n in [2, 3, 5, 8, 13]:
with subtests.test():
assert_fn_equal(sandbox, "fib", args=(n,), expected=fib(n))
```

After a call to student code times out, the sandbox stops answering queries, so the remaining
inputs fail without running. See [Checking Many Inputs](quick_start.md#checking-many-inputs).

### Ungradable Submission Detection

When student code has a `SyntaxError`, the submission is automatically marked as **ungradable** (`"gradable": false` in results). This means the student does not lose a grading attempt and sees a clear error message.
Expand Down
16 changes: 8 additions & 8 deletions converting.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,8 +162,6 @@ import pytest
from pytest_prairielearn_grader.assertions import assert_approx_equal
from pytest_prairielearn_grader.fixture import StudentFixture

pytestmark = pytest.mark.output(level="friendly")


@pytest.mark.grading_data(name="x_sq", points=10)
def test_x_sq(sandbox: StudentFixture, data_json: dict) -> None:
Expand All @@ -189,7 +187,7 @@ regrade of a variant, and `question.html` could show it with `{{params.x}}`.
| `Feedback.set_score(s)` | `feedback.set_score(s)` with the `feedback` fixture. See [Scoring](#scoring). |
| `Feedback.add_feedback("...")` | `feedback.add_message("...")` |
| `accuracy_critical=True` | A failed assertion ends the test. Other tests still run. |
| `total_iters = 3` and `@not_repeated` | `@pytest.mark.parametrize` over the inputs, or a loop with partial credit |
| `total_iters = 3` and `@not_repeated` | `@pytest.mark.parametrize` over the inputs, or a loop of `subtests.test()` blocks for partial credit |
| `Feedback.not_allowed` | `import_blacklist=["numpy.linalg.inv"]` in the `ConfigObject`. See [Update the setup code](#4-update-the-setup-code). |
| `repeated_setup()` | Not needed: setup code runs again for each sandbox. |

Expand Down Expand Up @@ -234,9 +232,8 @@ The `plots` example question is a converted plot question.

### Feedback messages

Use `@pytest.mark.output(level=GradingOutputLevel.Friendly)` on each test, as the example questions
do, or `output_level="friendly"` in the `ConfigObject`. At this level the student sees only the
assertion's message.
The default `friendly` [output level](quick_start.md#output-levels) shows the student only the
assertion's message, so tests don't need `@pytest.mark.output`.

Follow [Hiding Expected Values](quick_start.md#hiding-expected-values): don't use a bare `assert`
or numpy testing functions, and use `assert_true` for hidden test cases. The comparison helpers show
Expand All @@ -249,6 +246,9 @@ legacy feedback where you can, such as `"x is inaccurate"`.
- A test that fails gets the last score it set with `feedback.set_score`, or 0. Call `set_score`
after each check that earns partial credit.
- `feedback.set_score_final(s)` sets the score even if the test then fails.
- A legacy loop that adds up how many inputs were correct becomes a loop of `subtests.test()`
blocks. The test gets the fraction of blocks that passed. See
[Checking Many Inputs](quick_start.md#checking-many-inputs).
- The legacy pattern of `if check: set_score(1) else: set_score(0)` becomes a plain assertion.

### Querying
Expand All @@ -257,8 +257,8 @@ legacy feedback where you can, such as `"x is inaccurate"`.
the test with a message that says so. You don't need to check this yourself.
- Each call to `query_function` has a 1 second timeout by default. Pass `query_timeout=` for slower
functions.
- When a test calls a student function several times, use `query_function_raw` and stop after the
first `"timeout"` status: see the `randomFunctionName` example question.
- After a call times out, later queries on the sandbox fail without running student code, so a
slow function can't use up the grading job's time. You don't need to stop a loop yourself.

## 4. Update the setup code

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description = "A pytest plugin for autograding Python code. Designed for use wit
readme = "README.md"
authors = [{ name = "Eliot Robson", email = "eliot.robson24@gmail.com" }]
requires-python = ">=3.11"
dependencies = ["dill>=0.4.0", "pytest", "prettytable>=3.8.0"]
dependencies = ["dill>=0.4.0", "pytest>=9.0,<10", "prettytable>=3.8.0"]
license = { file = "LICENSE" }
classifiers = [
"Development Status :: 3 - Alpha",
Expand Down
98 changes: 75 additions & 23 deletions quick_start.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,12 +110,9 @@ checks the result:

```python
import pytest
from pytest_prairielearn_grader import ConfigObject
from pytest_prairielearn_grader.assertions import assert_fn_approx_equal
from pytest_prairielearn_grader.fixture import StudentFixture

autograder_config = ConfigObject(output_level="friendly")


@pytest.mark.grading_data(name="Average of one number", points=1)
def test_single(sandbox: StudentFixture) -> None:
Expand All @@ -134,7 +131,8 @@ def test_several(sandbox: StudentFixture, values: list[float]) -> None:
tolerance. On a failure, the student sees which call was checked, the expected value and their
value. See [Checking Answers](#checking-answers) for the other helpers and for checks that hide the
expected value.
- The `friendly` output level shows students only the assertion's message, without a traceback.
- Students see only the assertion's message, because `friendly` is the default
[output level](#output-levels).
- Expected values are computed in the test. Don't put a reference solution in a file that student
code can read.

Expand Down Expand Up @@ -262,6 +260,9 @@ These fixtures are provided by the `pytest-prairielearn-grader` package:
6. **`workspace_sandbox: WorkspaceFixture`**: For multi-file workspace projects. See
[Workspace Grading](#workspace-grading).

pytest's built-in **`subtests: pytest.Subtests`** fixture splits a test's points across many
inputs. See [Checking Many Inputs](#checking-many-inputs).

For implementation details, see the [fixture source code](https://github.com/prairielearn/grader-python-v2/blob/main/src/pytest_prairielearn_grader/fixture.py).

### Querying Student Code
Expand Down Expand Up @@ -312,9 +313,8 @@ def test_function(sandbox: StudentFixture) -> None:

You don't need to catch these errors: they fail the test with a message saying what went wrong.

Each call has a 1 second timeout by default. See [Function Timeout](#4-function-timeout). When a
test calls a student function many times, use `query_function_raw` and stop after the first
`"timeout"` status, so that a slow function doesn't use up the grading job's time.
Each call has a 1 second timeout by default. See [Function Timeout](#4-function-timeout). To give
partial credit across many calls, see [Checking Many Inputs](#checking-many-inputs).

**Testing for specific exceptions with `query_function_raw`:**

Expand All @@ -341,7 +341,7 @@ def test_raises_value_error(sandbox: StudentFixture, feedback: FeedbackFixture)

The `query_function_raw` response contains:

- `status`: One of `"success"`, `"exception"`, `"timeout"`, `"not_found"`, or `"not_serializable"`. `"not_found"` means the function doesn't exist, and only the lookup can cause it: an error raised while the function runs, even a `KeyError` or `ImportError`, is `"exception"`. `"not_serializable"` means the function finished, but its return value can't be sent back to the grader. `exception_message` explains this, and later queries on the sandbox still work.
- `status`: One of `"success"`, `"exception"`, `"timeout"`, `"not_found"`, or `"not_serializable"`. After a `"timeout"`, later queries on the sandbox raise an error: see [Function Timeout](#4-function-timeout). `"not_found"` means the function doesn't exist, and only the lookup can cause it: an error raised while the function runs, even a `KeyError` or `ImportError`, is `"exception"`. `"not_serializable"` means the function finished, but its return value can't be sent back to the grader. `exception_message` explains this, and later queries on the sandbox still work.
- `value`: The return value (when status is `"success"`)
- `stdout` / `stderr`: Captured output from the function call
- `exception_name`: The exception class name (e.g., `"ValueError"`)
Expand Down Expand Up @@ -375,6 +375,12 @@ Control execution time for individual function calls:
result = sandbox.query_function("slow_computation", data, query_timeout=2.0)
```

After any query times out, every later query on that sandbox raises `SandboxStopped`. The student
code may still send the late answer, which would otherwise be read as the answer to the next call.
So a timeout fails the rest of the test, and with a [module-scoped sandbox](#module-scoped-sandbox),
the later tests that use it too. In a [subtests loop](#checking-many-inputs), the inputs before the
timeout keep their credit.

#### 5. Query Setup Variables

```python
Expand Down Expand Up @@ -474,17 +480,18 @@ shows both frames.

The output level sets how much of a failure students see:

| Level | Shows to student |
| ------------- | ------------------------------------------------------------- |
| `"none"` | Exception class name only (e.g., `AssertionError`) |
| `"message"` | Exception name + first line of message _(default when unset)_ |
| `"traceback"` | Exception name + message + full traceback |
| `"friendly"` | Only the exception message text — no class name, no traceback |
| Level | Shows to student |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `"friendly"` | The exception message only, with no class name _(default)_. If a call to a student function raised the error, the message includes the traceback from the student's code. |
| `"none"` | Exception class name only (e.g., `AssertionError`) |
| `"message"` | Exception name + first line of message |
| `"traceback"` | Exception name + message + full traceback |

Use `"friendly"` with the assertion helpers. Set it on one test:
The default, `"friendly"`, works well with the assertion helpers. To show more, such as the full
traceback for advanced students, set a different level on one test:

```python
@pytest.mark.output(level="friendly")
@pytest.mark.output(level="traceback")
@pytest.mark.grading_data(name="Test addition", points=5)
def test_add(sandbox: StudentFixture) -> None:
assert_fn_equal(sandbox, "add", args=(2, 3), expected=5)
Expand All @@ -493,18 +500,17 @@ def test_add(sandbox: StudentFixture) -> None:
On every test in a file, with a module-level `pytestmark`:

```python
pytestmark = pytest.mark.output(level="friendly")
pytestmark = pytest.mark.output(level="traceback")
```

Or in the `ConfigObject`:

```python
autograder_config = ConfigObject(output_level="friendly")
autograder_config = ConfigObject(output_level="traceback")
```

With the `ConfigObject` approach, individual tests can still override it with
`@pytest.mark.output(level="traceback")`. The `"message"` and `"traceback"` levels can help
advanced students, who can learn from reading tracebacks.
`@pytest.mark.output(level=...)`.

### Partial Credit and Feedback

Expand Down Expand Up @@ -540,6 +546,52 @@ def test_with_partial_credit(sandbox: StudentFixture, feedback: FeedbackFixture)
- `feedback.add_message(msg)`: Add custom feedback message
- `feedback.add_image(image, label=None)`: Show an image with the test's results. See [Showing Plots to Students](#showing-plots-to-students)

### Checking Many Inputs

To split a test's points across many inputs, check each input in a `subtests.test()` block, using
pytest's built-in `subtests` fixture:

```python
@pytest.mark.grading_data(name="Check random values", points=3)
def test_random_values(sandbox: StudentFixture, subtests: pytest.Subtests) -> None:
for n in [2, 3, 5, 8, 13, 17, 19, 23, 27, 29]:
with subtests.test():
assert_fn_equal(sandbox, "fib", args=(n,), expected=fib(n))
```

A failed assertion or exception inside a block fails only that block, and the loop goes on. The
test's score is the fraction of blocks that passed:

- A block that calls `pytest.skip` counts as failed.
- If the test fails outside any block, its score is 0, or the last score it set with
`feedback.set_score`.
- `feedback.set_score_final` replaces the fraction.

The test's message starts with how many blocks passed, such as `7/10 checks passed.`, followed by
the messages of the first 5 failed blocks. Passed blocks aren't listed. Pass a label to show it
before a block's message, such as `subtests.test(f"last_element({lst})")`. The `assert_fn_*`
helpers already name the call, so they don't need one.

A block can check several things about one input:

```python
@pytest.mark.grading_data(name="last_element", points=10)
def test_last_element(sandbox: StudentFixture, subtests: pytest.Subtests) -> None:
for _ in range(10):
lst = random.sample(range(1000), k=random.randint(1, 20))
with subtests.test(f"last_element({lst})"):
value, lst_after = sandbox.query_function("last_element_and_list", lst)
assert_equal(value, lst[-1])
assert_equal(lst_after, lst, msg="Your function should not modify the input list.")
```

When a call times out, the remaining blocks fail without running student code, and the message
says so once: `The remaining 3 checks didn't run because fib(23) timed out after 1.0 seconds.`
See [Function Timeout](#4-function-timeout).

Use `@pytest.mark.parametrize` instead when each input should be its own test, with its own name
and points.

### Capturing and Testing Output

Control whether student code output appears in feedback:
Expand Down Expand Up @@ -901,7 +953,7 @@ All parameters are optional:
| Parameter | Default | Description |
| -------------------------- | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `sandbox_timeout` | `1.0` | Seconds allowed for starting the sandbox, which runs the setup code and student code. See [Timeout Configuration](#timeout-configuration). |
| `output_level` | `None` | Default [output level](#output-levels) for every test. |
| `output_level` | `None` | Default [output level](#output-levels) for every test. `None` means `"friendly"`. |
| `names_for_user` | `None` | Variables to give student code. See [Giving Values to Student Code](#giving-values-to-student-code). |
| `starting_vars` | `{}` | Values for names in `names_for_user`. |
| `student_code_pattern` | `"student_code*.py"` | Glob pattern for the student's file. |
Expand Down Expand Up @@ -1955,8 +2007,8 @@ To see more examples of what is possible in these test files, look at the test c

1. **Keep code in the tests**: See [Where Code Belongs](#where-code-belongs).
2. **Use descriptive test names**: The `name` in `@pytest.mark.grading_data` is shown to students.
3. **Use friendly messages that don't give away answers**: Use the assertion helpers with the
`friendly` output level, and `assert_true` for hidden test cases.
3. **Use friendly messages that don't give away answers**: Use the assertion helpers, and
`assert_true` for hidden test cases.
4. **Test incrementally**: Break complex problems into smaller tests with partial credit.
5. **Prefer `parametrize`**: Use `@pytest.mark.parametrize` for several inputs instead of a loop,
unless the inputs share partial credit.
Expand Down
Loading
Loading