This guide shows how to write a PrairieLearn question that's graded with the pytest-based Python
autograder, and then documents the grader's full API. The grader runs in the
prairielearn/grader-python-pytest Docker images and is built on the pytest-prairielearn-grader
pytest plugin.
Tests are ordinary pytest test functions. Each test gets a fresh sandbox that runs the student's code, and the test asks the sandbox for the values and functions it wants to check. Student code never runs in the same process as the tests.
If you're converting a question that uses the legacy prairielearn/grader-python image, follow
converting.md instead.
Install the required packages in your Python environment for IDE support (e.g., VS Code with Pylance):
pip install pytest pytest-prairielearn-graderThis enables IDE features like autocomplete, type checking, and inline documentation when writing test cases.
This section builds a question that asks students to write a function average(values). A
question that's graded by this grader has these files:
average/
├── info.json
├── question.html
└── tests/
├── initial_code.py (starter code shown in the file editor)
└── test_student.py (the tests)
Set gradingMethod to "External" and pick a grader image:
{
"uuid": "...",
"title": "Average of a list",
"topic": "Functions",
"type": "v3",
"gradingMethod": "External",
"externalGradingOptions": {
"enabled": true,
"image": "prairielearn/grader-python-pytest:lite",
"timeout": 30
}
}Two image variants are published. Pick the smallest one that includes what your question needs — a smaller image means faster cold-start pull times and a reduced security surface.
| Tag | Size (approx.) | Includes | Best for |
|---|---|---|---|
:latest |
~400–500 MB | Full scientific stack: numpy, pandas, scipy, matplotlib + matplotcheck2, seaborn, scikit-learn, sympy, networkx, nltk, nbformat, and more |
Any course that needs scientific computing, data science, plotting, or Jupyter notebook grading |
:lite |
~60–80 MB | Grader core only: pytest-prairielearn-grader, Pygments, beautifulsoup4, defusedxml, requests |
Introductory CS courses where student code uses only the Python standard library |
Note: If a student's code imports a package that isn't present in the image (e.g.,
numpyin:lite), it will raise anImportErrorand the test will fail with an exception. Make sure the image you choose contains everything both your test code and the expected student solution require.
Pinned version tags (e.g., :v1.2.3 and :v1.2.3-lite) are also published alongside each release if you need a reproducible build.
Students write their code in a file editor. The grader looks for a file named student_code*.py,
so name the file student_code.py, or set student_code_pattern:
<pl-question-panel>
<p>
Write a function <code>average(values)</code> that returns the mean of a
non-empty list of numbers.
</p>
<pl-file-editor
file-name="student_code.py"
ace-mode="ace/mode/python"
source-file-name="tests/initial_code.py"
></pl-file-editor>
</pl-question-panel>
<pl-submission-panel>
<pl-external-grader-results></pl-external-grader-results>
<pl-file-preview></pl-file-preview>
</pl-submission-panel>The starter code. The grader doesn't read this file, so it's only used by the file editor. Make sure it parses: a submission with a syntax error is ungradable, so an unchanged starter file should run and get 0 points with feedback.
def average(values):
return 0Each test is a function that takes the sandbox fixture, calls the student's code through it, and
checks the result:
import pytest
from pytest_prairielearn_grader.assertions import assert_fn_approx_equal
from pytest_prairielearn_grader.fixture import StudentFixture
@pytest.mark.grading_data(name="Average of one number", points=1)
def test_single(sandbox: StudentFixture) -> None:
assert_fn_approx_equal(sandbox, "average", args=([5],), expected=5)
@pytest.mark.parametrize("values", [[1, 2, 3], [2.5, -1, 0, 7], list(range(100))])
@pytest.mark.grading_data(name="Average of several numbers", points=2)
def test_several(sandbox: StudentFixture, values: list[float]) -> None:
assert_fn_approx_equal(sandbox, "average", args=(values,), expected=sum(values) / len(values))grading_datasets the name that students see and the test's points. A parametrized test gets those points for each parameter.assert_fn_approx_equalcallsaveragein the sandbox and compares the result with a tolerance. On a failure, the student sees which call was checked, the expected value and their value. See Checking Answers for the other helpers and for checks that hide the expected value.- Students see only the assertion's message, because
friendlyis the default output level. - Expected values are computed in the test. Don't put a reference solution in a file that student code can read.
The grader image's entrypoint expects PrairieLearn's /grade layout. Build it from your question
and a solution, run the image, and read the results:
mkdir -p grade/tests grade/student grade/data
cp -R average/tests/. grade/tests/
cp solution.py grade/student/student_code.py
echo '{"params": {}}' > grade/data/data.json
docker run --rm -v "$PWD/grade:/grade" prairielearn/grader-python-pytest:lite
cat grade/results/results.jsonThe image moves files around, so make a fresh copy of the layout for each run. Put in
data.json whatever params your server.py and elements would set. Try at least a correct
solution, the unchanged starter code and a wrong answer for each test, and read each test's
message in results.json.
To run pytest directly while you write tests, install the packages from Editor Setup and put the files that the grader reads in a folder named after the test file, next to it:
tests/
├── test_student.py
└── test_student/
├── student_code_correct.py (each student_code*.py file is graded separately)
├── student_code_wrong.py
├── setup_code.py (if the question has one)
└── data.json (if the tests need params)
Then run pytest tests/test_student.py. To run the tests of several questions at once, pass their
folder, such as pytest questions/. The grader uses pytest's importlib import mode, so each
question's test_student.py is imported separately, unless you pass a different --import-mode.
Besides test_student.py, the tests folder can hold these files:
| File | What the grader does with it |
|---|---|
setup_code.py |
Runs in the sandbox before the student's code. See Giving Values to Student Code. |
leading_code.py |
Prepended to the student's code, in the same namespace. |
trailing_code.py |
Appended to the student's code, in the same namespace. |
| Helper modules | Modules that student code can import, when listed in helper_modules. |
| Data files | Files that student code can read, when listed in data_files. |
Keep as much as possible in test_student.py, where each test runs on its own and controls its
feedback. Most questions need only initial_code.py and test_student.py. Put each piece of code
in the first place in this table that works:
| Code | Where to put it |
|---|---|
| Expected values and reference solutions | The test |
| Calls to the student's functions | sandbox.query_function or an assertion helper in the test |
| Inputs to the student's functions, including random ones | The test, passed as arguments. Use @pytest.mark.parametrize for several inputs. |
Variables that student code reads and that server.py randomizes |
data["params"] in server.py, listed in names_for_user. Tests read them from the data_json fixture. This works for plain JSON values, not ones stored with pl.to_json. |
| Fixed variables that student code reads | starting_vars in the ConfigObject, listed in names_for_user |
| Variables that student code reads and that need Python to build | setup_code.py, such as numpy arrays or values decoded with from_server_json |
| Blocking a library function | import_blacklist, such as ["numpy.linalg.inv"] |
| Imports or definitions that the student's code relies on | The starter code, so that the student sees them. Use helper_modules for a module the student imports. Use leading_code.py only when neither works. |
Why:
- Trailing code runs when the sandbox starts. If it calls a student function that raises, every
test fails with the same error, including tests of other functions. When a test calls the function
with
query_function, only that test fails, and the message names the function. - Setup code runs again for each test, in the sandbox, with the same restrictions as student code. Random values in it change between tests, and errors in it are reported as if the student's code failed.
- Leading code is hidden. The student's file only works with code they can't see, so it behaves differently when they run it in a workspace or on their own computer.
- Values in
data["params"]are the same in every test and match whatquestion.htmlshows, and tests can read them without starting a sandbox.
test_student.py can't import the other files in tests/. Put helper code for the tests in
test_student.py, and read data files in tests with the tests_dir
fixture.
Tests are pytest test functions in tests/test_student.py. They run in the order they appear in
the file. The examples below leave out their imports, which usually are:
import pytest
from pytest_prairielearn_grader import ConfigObject
from pytest_prairielearn_grader.assertions import assert_equal, assert_fn_equal, assert_true
from pytest_prairielearn_grader.fixture import FeedbackFixture, StudentFixtureThe @pytest.mark.grading_data decorator sets what PrairieLearn shows for the test:
name: Test name displayed to studentspoints: Maximum points for the testinclude_stdout_feedback: (Optional, default=True) Whether to include student code's stdout in feedback
@pytest.mark.output(level=...) sets how much of a failure students see. See
Output Levels.
These fixtures are provided by the pytest-prairielearn-grader package:
-
sandbox: StudentFixture: Runs the setup code and student code in a new sandbox for each test. Use it to query variables and call functions from the student's submission. -
module_sandbox: StudentFixture: Likesandbox, but shares one sandbox across all tests in the file. See Module-Scoped Sandbox. -
feedback: FeedbackFixture: Sets partial credit and adds feedback messages. See Partial Credit and Feedback. -
data_json: dict | None: The contents of PrairieLearn'sdata.json, such asdata_json["params"]. -
tests_dir: Path: The question's tests folder. Use it to read instructor files in a test, for examplenp.loadtxt(tests_dir / "pagerank_large.txt"), instead of building the path from__file__. -
workspace_sandbox: WorkspaceFixture: For multi-file workspace projects. See Workspace Grading.
pytest's built-in subtests: pytest.Subtests fixture splits a test's points across many
inputs. See Checking Many Inputs.
For implementation details, see the fixture source code.
The sandbox fixture provides methods to interact with student code:
value = sandbox.query("variable_name")Retrieves the value of a variable defined in the student code or setup_code.py.
Raises a NameError if the variable doesn't exist, and a RuntimeError with a clear message if its value can't be sent back to the grader. Later queries on the sandbox still work after either error.
Example:
@pytest.mark.grading_data(name="Check Variable", points=2)
def test_variable(sandbox: StudentFixture) -> None:
coefficient = sandbox.query("coefficient")
assert_true(coefficient > 0, msg="coefficient must be positive")result = sandbox.query_function("function_name", arg1, arg2, kwarg1=value1)Calls a function defined in the student code with the given arguments and returns the value directly.
- On success: Returns the function's return value
- On error: Raises a
RuntimeErrorwith details about the exception - If the function doesn't exist: Raises a
NameErrorsaying so
Example:
@pytest.mark.grading_data(name="Test Function", points=5)
def test_function(sandbox: StudentFixture) -> None:
result = sandbox.query_function("calculate", 10, 20)
assert_equal(result, 30, description="calculate(10, 20)")
result = sandbox.query_function("process", x=5, y=10)
assert_equal(result, 15, description="process(x=5, y=10)")You don't need to catch these errors: they fail the test with a message saying what went wrong.
Each call has a 1 second timeout by default. See Function Timeout. To give partial credit across many calls, see Checking Many Inputs.
Testing for specific exceptions with query_function_raw:
When you need to verify that student code raises a specific exception (e.g., testing input validation),
use query_function_raw to inspect the response directly:
@pytest.mark.grading_data(name="Test Exception Handling", points=3)
def test_raises_value_error(sandbox: StudentFixture, feedback: FeedbackFixture) -> None:
response = sandbox.query_function_raw("validate_input", -1)
assert_true(response["status"] == "exception", msg="validate_input(-1) should raise an exception")
feedback.set_score(0.5)
assert_equal(response["exception_name"], "ValueError", description="the exception raised by validate_input(-1)")
feedback.set_score(0.75)
assert_true(
"negative" in response["exception_message"].lower(),
msg="The error message should mention negative values",
)
feedback.set_score(1.0)The query_function_raw response contains:
status: One of"success","exception","timeout","not_found", or"not_serializable". After a"timeout", later queries on the sandbox raise an error: see Function Timeout."not_found"means the function doesn't exist, and only the lookup can cause it: an error raised while the function runs, even aKeyErrororImportError, is"exception"."not_serializable"means the function finished, but its return value can't be sent back to the grader.exception_messageexplains this, and later queries on the sandbox still work.value: The return value (when status is"success")stdout/stderr: Captured output from the function callexception_name: The exception class name (e.g.,"ValueError")exception_message: The exception message stringtraceback: Full traceback string for debugging
output = sandbox.get_accumulated_stdout()Retrieves stdout captured from student code execution across all function calls.
Example:
@pytest.mark.grading_data(name="Test Output", points=2)
def test_output(sandbox: StudentFixture) -> None:
sandbox.query_function("print_greeting", "Alice")
output = sandbox.get_accumulated_stdout()
assert_true("Hello, Alice!" in output, msg="Greeting not found in output")Control execution time for individual function calls:
# Set a 2-second timeout for this specific function call
result = sandbox.query_function("slow_computation", data, query_timeout=2.0)After any query times out, every later query on that sandbox raises SandboxStopped. The student
code may still send the late answer, which would otherwise be read as the answer to the next call.
So a timeout fails the rest of the test, and with a module-scoped sandbox,
the later tests that use it too. In a subtests loop, the inputs before the
timeout keep their credit.
x = sandbox.query_setup("x")Retrieves a variable defined by setup_code.py, whether or not it's in names_for_user. The value
comes from the same sandbox as the student's code, so a random value matches the one the student's
code received.
Note: Values sent back from the sandbox must be serializable. Supported types include: int, float, str, list, dict, bool, None, numpy arrays, pandas DataFrames,
and matplotlib plots (see Testing Matplotlib Plots).
The pytest_prairielearn_grader.assertions module has helpers that raise an AssertionError with a
structured message:
Checking: add(2, 3)
Expected output: 5
Your code output: -1
The expected and actual output do not match.
assert_equal(actual, expected, *, msg=None, description=None)
Compare two values for equality:
assert_equal(sandbox.query("x"), 42, description="variable 'x'")assert_approx_equal(actual, expected, *, rtol=1e-5, atol=1e-8, msg=None, description=None)
Compare numbers with a tolerance. It's for scalars: see below for arrays.
result = sandbox.query_function("compute_pi")
assert_approx_equal(result, 3.14159, rtol=1e-3, description="compute_pi()")assert_fn_equal(sandbox, func_name, *, args=(), kwargs=None, expected, msg=None, description=None, query_timeout=1.0)
Call a student function and compare its return value. The description defaults to the call, such as
add(2, 3):
assert_fn_equal(sandbox, "add", args=(2, 3), expected=5)assert_fn_approx_equal(sandbox, func_name, *, args=(), kwargs=None, expected, rtol=1e-5, atol=1e-8, msg=None, query_timeout=1.0)
Same as assert_fn_equal, with a tolerance:
assert_fn_approx_equal(sandbox, "compute_area", args=(3.0,), expected=28.274, rtol=1e-2)assert_plot_equal(plot, expected, *, name="your plot", rtol=1e-5, atol=1e-8, ..., msg=None)
Compares a student's matplotlib plot with a reference plot. See Comparing with a Reference Plot.
assert_true(condition, *, msg=None) / assert_false(condition, *, msg=None)
Check a condition, and show only msg when it fails:
assert_true(sandbox.query_function("is_valid", data), msg="is_valid() should accept valid input.")The comparison helpers show the expected value, which is fine when the student can see the test's
inputs. For hidden test cases, use assert_true with a message that doesn't include the answer.
Don't use a bare assert or numpy and pandas testing functions for checks that students see.
pytest rewrites a bare assert, so its message includes both values, such as assert 6 == 12345.
np.testing.assert_allclose and similar include both arrays. Use
assert_true(np.allclose(...), msg=...) instead.
For a numpy array, check the type, dtype, shape and values in that order, so that each failure has
a clear message and a wrong shape doesn't reach np.allclose:
value = sandbox.query("x")
assert_true(isinstance(value, np.ndarray), msg="x should be a numpy array")
assert_true(np.issubdtype(value.dtype, np.number), msg="x should contain numbers")
assert_true(value.shape == expected.shape, msg=f"x should have shape {expected.shape}")
assert_true(np.allclose(value, expected), msg="x is inaccurate")For a pandas DataFrame, pandas.testing.assert_frame_equal(expected, df) works, but its message
shows both frames.
The output level sets how much of a failure students see:
| Level | Shows to student |
|---|---|
"friendly" |
The exception message only, with no class name (default). If a call to a student function raised the error, the message includes the traceback from the student's code. |
"none" |
Exception class name only (e.g., AssertionError) |
"message" |
Exception name + first line of message |
"traceback" |
Exception name + message + full traceback |
The default, "friendly", works well with the assertion helpers. To show more, such as the full
traceback for advanced students, set a different level on one test:
@pytest.mark.output(level="traceback")
@pytest.mark.grading_data(name="Test addition", points=5)
def test_add(sandbox: StudentFixture) -> None:
assert_fn_equal(sandbox, "add", args=(2, 3), expected=5)On every test in a file, with a module-level pytestmark:
pytestmark = pytest.mark.output(level="traceback")Or in the ConfigObject:
autograder_config = ConfigObject(output_level="traceback")With the ConfigObject approach, individual tests can still override it with
@pytest.mark.output(level=...).
Tests flow linearly, allowing partial credit after certain assertions pass. When an assertion fails, the student receives the last partial credit value set before the failure.
@pytest.mark.grading_data(name="Multi-step Test", points=10)
def test_with_partial_credit(sandbox: StudentFixture, feedback: FeedbackFixture) -> None:
# Check basic requirements (worth 30%)
result = sandbox.query("data_loaded")
assert_true(result is not None, msg="Data must be loaded")
feedback.set_score(0.3)
# Check intermediate computation (worth 60%)
intermediate = sandbox.query("processed_data")
assert_true(len(intermediate) > 0, msg="Data processing failed")
feedback.set_score(0.6)
# Check final result (worth 100%)
final = sandbox.query("final_result")
assert_true(final == expected_value, msg="Final result incorrect")
feedback.set_score(1.0)
# Add custom feedback
feedback.add_message("Excellent work! All steps completed correctly.")Key Methods:
feedback.set_score(fraction): Set partial credit (0.0 to 1.0)feedback.set_score_final(fraction): Set final score (prevents further updates)feedback.add_message(msg): Add custom feedback messagefeedback.add_image(image, label=None): Show an image with the test's results. See Showing Plots to Students
To split a test's points across many inputs, check each input in a subtests.test() block, using
pytest's built-in subtests fixture:
@pytest.mark.grading_data(name="Check random values", points=3)
def test_random_values(sandbox: StudentFixture, subtests: pytest.Subtests) -> None:
for n in [2, 3, 5, 8, 13, 17, 19, 23, 27, 29]:
with subtests.test():
assert_fn_equal(sandbox, "fib", args=(n,), expected=fib(n))A failed assertion or exception inside a block fails only that block, and the loop goes on. The test's score is the fraction of blocks that passed:
- A block that calls
pytest.skipcounts as failed. - If the test fails outside any block, its score is 0, or the last score it set with
feedback.set_score. feedback.set_score_finalreplaces the fraction.
To make some blocks count for more, pass a weight, which is 1 by default:
for n, weight in [(2, 1), (10, 1), (29, 2)]:
with subtests.test(weight=weight):
assert_fn_equal(sandbox, "fib", args=(n,), expected=fib(n))The score is then the weight of the blocks that passed divided by the total weight: here, passing
only fib(29) gives half the points. Weights are non-negative ints or floats. A block with weight 0
is still checked and shown, but doesn't change the score.
The test's message starts with how many blocks passed, such as 7/10 checks passed., followed by
the messages of the first 5 failed blocks. Passed blocks aren't listed. Pass a label to show it
before a block's message, such as subtests.test(f"last_element({lst})"). The assert_fn_*
helpers already name the call, so they don't need one.
A block can check several things about one input:
@pytest.mark.grading_data(name="last_element", points=10)
def test_last_element(sandbox: StudentFixture, subtests: pytest.Subtests) -> None:
for _ in range(10):
lst = random.sample(range(1000), k=random.randint(1, 20))
with subtests.test(f"last_element({lst})"):
value, lst_after = sandbox.query_function("last_element_and_list", lst)
assert_equal(value, lst[-1])
assert_equal(lst_after, lst, msg="Your function should not modify the input list.")When a call times out, the remaining blocks fail without running student code, and the message
says so once: The remaining 3 checks didn't run because fib(23) timed out after 1.0 seconds.
See Function Timeout.
Use @pytest.mark.parametrize instead when each input should be its own test, with its own name
and points.
Control whether student code output appears in feedback:
@pytest.mark.grading_data(name="With Output", points=2, include_stdout_feedback=True)
def test_with_output(sandbox: StudentFixture) -> None:
"""Student's print statements will appear in feedback."""
result = sandbox.query_function("process_data")
# Any print() calls in student code are captured and shown
assert_equal(result, expected)
@pytest.mark.grading_data(name="Without Output", points=2, include_stdout_feedback=False)
def test_without_output(sandbox: StudentFixture) -> None:
"""Student's print statements will NOT appear in feedback."""
result = sandbox.query_function("process_data")
assert_equal(result, expected)
@pytest.mark.grading_data(name="Manual Output Check", points=2)
def test_output_manually(sandbox: StudentFixture) -> None:
"""Manually inspect and test stdout."""
sandbox.query_function("print_greeting", "Alice")
output = sandbox.get_accumulated_stdout()
assert_true("Hello, Alice!" in output, msg="Greeting message not found in output")Use module_sandbox instead of sandbox when you want student code state to persist
across multiple tests:
@pytest.mark.grading_data(name="Initialize", points=1)
def test_initialization(module_sandbox: StudentFixture) -> None:
"""First test initializes state."""
result = module_sandbox.query_function("initialize_counter")
assert_equal(result, 0)
@pytest.mark.grading_data(name="Increment 1", points=1)
def test_increment_1(module_sandbox: StudentFixture) -> None:
"""State persists - counter should be 1."""
result = module_sandbox.query_function("increment_counter")
assert_equal(result, 1)
@pytest.mark.grading_data(name="Increment 2", points=1)
def test_increment_2(module_sandbox: StudentFixture) -> None:
"""State still persists - counter should be 2."""
result = module_sandbox.query_function("increment_counter")
assert_equal(result, 2)Use Cases for module_sandbox:
- Testing stateful classes or modules
- Expensive initialization that should only run once
- Testing persistent data structures (databases, file systems, etc.)
- Simulating multi-step workflows
Important: With module_sandbox, student code is loaded once and shared across all tests
in the module. Use regular sandbox for independent test execution.
Student code can use matplotlib as usual and provide the plot's Axes or Figure. The test receives a matplotlib Figure.
# Student code
import matplotlib.pyplot as plt
def create_plot(data):
plt.plot(data)
plt.xlabel("Time")
plt.ylabel("Value")
plt.title("Data Visualization")
return plt.gca()An Axes value brings its whole figure, so subplots are never lost: use figure.axes in your test to get each one, in the order the student created them.
What's sent to the test:
- Lines, with their data, label, color, width, style, marker and draw style. This includes
plot,semilogy,loglogandstep. - Horizontal and vertical lines from
axhlineandaxvline, with their position, extent, label, color, width and style. - Text from
text, in data or axes coordinates, with its position, alignment and rotation. - Annotations from
annotate, with the annotated point in data or axes coordinates, the text position in data or axes coordinates or offset points, and whether it has an arrow. - Scatter points, with their positions, sizes and colors.
- Bars and histograms, with their positions, sizes, colors and orientation.
- Axes details: scales, labels, titles, limits, position and legend labels. Twin axes come back as separate axes in
figure.axes. - Figure details: size and overall title.
Other plot content makes the query fail with a message saying what isn't supported, rather than losing data. This includes errorbar, stem, fill_between, pie, imshow, contour, vlines and hlines, axline, axvspan, text in figure coordinates, colormapped scatter plots, polar axes, inset axes and subfigures.
Some details aren't sent, so don't grade them: grid lines, tick positions and labels, line and marker transparency set separately from color, scatter marker shapes, the base of a log scale, symlog settings, text fonts and colors, and arrow styles, since every annotation arrow comes back as a plain arrow. Values are sent as decimal numbers, and dates come back as matplotlib date numbers (use matplotlib.dates.num2date to convert them). Categories on a line come back as strings.
Importing matplotlib can take most of the default 1-second sandbox timeout, so set a longer one in plot questions, such as sandbox_timeout = 15 at the top of the test file.
matplotlib is included in the :latest image but not in :lite.
assert_plot_equal checks a student's plot against a reference plot that the test draws itself:
import numpy as np
import pytest
from matplotlib.figure import Figure
from pytest_prairielearn_grader.assertions import assert_plot_equal
from pytest_prairielearn_grader.fixture import StudentFixture
from pytest_prairielearn_grader.utils import GradingOutputLevel
@pytest.mark.output(level=GradingOutputLevel.Friendly)
@pytest.mark.grading_data(name="plot", points=1)
def test_plot(sandbox: StudentFixture) -> None:
reference = Figure()
axes = reference.subplots()
x = np.arange(0, 11)
axes.semilogy(x, np.exp(x))
assert_plot_equal(sandbox.query("plot"), reference)Draw the reference on a Figure made directly, as here, rather than with plt, so that it doesn't
share pyplot's current figure with anything else in the test. Both plots can be a Figure or an
Axes, and subplots are compared in order.
By default, it checks:
- Subplots: the number of axes.
- Scales: the x and y axis scales, such as linear or log.
- Data: the lines,
axhlineandaxvlinelines, scatter plots and bar plots, which must have the same number of each kind. Data is compared withnumpy.allclose, usingrtolandatol.
The series can be in any order, and a line also matches when its points are in reverse order, since it looks the same. The points in a scatter plot and the bars in a bar plot can be in any order.
Keyword arguments change what's checked:
| Argument | Default | Checks |
|---|---|---|
check_scales |
True |
The x and y axis scales. |
check_data |
True |
The data in lines, axhline and axvline lines, scatter plots and bar plots. |
ordered |
False |
Requires lines, scatter plots, bar plots and legend labels to be in the reference's order. |
allow_extra |
False |
Allows more lines, scatter plots, bar plots, text or annotations than the reference, such as a student's extra reference line. |
check_labels |
False |
The x and y axis labels, exactly. |
check_titles |
False |
The title of each subplot and the figure, exactly. |
check_legend |
False |
The legend's labels and title, or that neither plot has a legend. |
check_limits |
False |
The x and y axis limits, within rtol and atol. |
check_styles |
False |
Line colors, styles and markers, scatter point colors, and bar colors. |
check_texts |
False |
The strings of text and annotate labels, and the point each annotation points at. The position of plain text isn't compared. |
A failure describes the first difference found, such as Expected a log scale on the y-axis of your plot, but found a linear scale. or No line in your plot matches the line labeled 'exp'. Lines
are named by their labels when the reference has them. Use name to call the plot something else
in messages, such as name="plot" for a variable, or msg to show your own message instead.
For checks that assert_plot_equal doesn't cover, use the Figure directly or
matplotcheck, which is included in the :latest
image. Its PlotTester can check that a label or title contains a word, histogram bins, the slope
of a fitted line, and more. It can be used in the same test as assert_plot_equal:
# Test code
import matplotlib
matplotlib.use("Agg") # Non-GUI backend for server environments
import pytest
from matplotlib.figure import Figure
from matplotcheck.base import PlotTester
from pytest_prairielearn_grader.assertions import assert_equal, assert_true
from pytest_prairielearn_grader.fixture import StudentFixture
@pytest.mark.grading_data(name="Plot Test", points=5)
def test_student_plot(sandbox: StudentFixture) -> None:
"""Test that student creates a correct plot."""
# Student function returns an Axes or Figure, which arrives here as a Figure
plot = sandbox.query_function("create_plot", data)
assert_true(isinstance(plot, Figure))
assert_equal(len(plot.axes), 1)
# Use matplotcheck for detailed plot testing
ax = plot.axes[0]
pt = PlotTester(ax)
# Check plot properties
pt.assert_plot_type("line")
pt.assert_axis_label_contains(axis="x", strings_expected=["Time"])
pt.assert_axis_label_contains(axis="y", strings_expected=["Value"])
pt.assert_title_contains(["Data Visualization"])To show the student's plot with a test's results, pass it to feedback.add_image:
@pytest.mark.grading_data(name="Plot Test", points=5)
def test_student_plot(sandbox: StudentFixture, feedback: FeedbackFixture) -> None:
plot = sandbox.query_function("create_plot", data)
feedback.add_image(plot, label="Your plot")
...- Values:
add_imagetakes a matplotlibFigureorAxes, or the bytes of a PNG file, and adds it to the test'simagesinresults.json.<pl-external-grader-results>shows it under the test, whether the test passes or fails. - Call it first: a failed assertion ends the test, so add the image before checking the plot.
- A copy of the plot: the image is drawn from the
Figurethat the test received, which the grader rebuilt from the student's plot. It shows the data, scales, labels and other parts the grader copies, but other styling can look different from what the student saw. - Size: PrairieLearn limits the whole
results.jsonto 1 MB. A typical plot is tens of kilobytes, but don't add an image in every case of a test with many parameters.
The test can also draw a plot of its own and show it, such as the expected plot, or a plot of
values that student code returned for a question that doesn't ask for a plot. Draw it on a new
Figure instead of with pyplot, so that one test's plot doesn't end up in another test's figure:
from matplotlib.figure import Figure
@pytest.mark.grading_data(name="Simulation", points=5)
def test_simulation(sandbox: StudentFixture, feedback: FeedbackFixture) -> None:
times = np.linspace(0, 10, 50)
positions = sandbox.query_function("simulate", times)
expected = expected_positions(times)
figure = Figure()
axes = figure.subplots()
axes.plot(times, positions, label="Your values")
axes.plot(times, expected, "--", label="Expected values")
axes.set_xlabel("Time")
axes.legend()
feedback.add_image(figure, label="Your values and the expected values")
assert_true(np.allclose(positions, expected), msg="The positions aren't correct")To show an image made some other way, pass the bytes of a PNG file to add_image.
The grader automatically serializes/deserializes:
- Matplotlib
AxesandFigurevalues (received as aFigure) - NumPy arrays
- Pandas DataFrames
- Standard Python types (int, float, str, list, dict, bool, None)
Some questions give students variables to use, such as a matrix to factor. Student code only gets
the variables named in names_for_user. Each name's value comes from the first of these that
defines it:
- A variable that
setup_code.pydefines. starting_varsin theConfigObject.data["params"]fromdata.json.
names_for_user comes from the ConfigObject if it's set there, as a list of strings. Otherwise
it comes from data["params"]["names_for_user"], a list of {"name": ..., "description": ..., "type": ...} objects, which <pl-external-grader-variables params-name="names_for_user"> in
question.html usually sets. Without names_for_user, student code gets no variables, and using
one fails with a NameError. names_from_user only displays a table in the question, and the
grader ignores it.
When server.py randomizes plain values, list them in names_for_user and read them in tests with
the data_json fixture:
# server.py
import random
def generate(data):
data["params"]["n"] = random.randint(5, 10)<!-- question.html -->
<pl-external-grader-variables params-name="names_for_user">
<pl-variable name="n" type="integer">The number of terms</pl-variable>
</pl-external-grader-variables># tests/test_student.py
@pytest.mark.grading_data(name="total", points=1)
def test_total(sandbox: StudentFixture, data_json: dict) -> None:
n = data_json["params"]["n"]
assert_equal(sandbox.query("total"), n * (n + 1) // 2, description="total")For a value that's the same in every variant, use starting_vars:
autograder_config = ConfigObject(starting_vars={"threshold": 5.0}, names_for_user=["threshold"])Use setup code when a value needs Python to build, such as a numpy array. It runs in the sandbox before the student's code:
import numpy as np
import numpy.linalg as la
n = np.random.randint(4, 16)
# A random full-rank matrix
X = la.qr(np.random.random_sample((n, n)))[0]
D = np.diag(np.random.random_sample(n) * 10 + 1)
A = X.T @ D @ X
b = np.random.random(n)With names_for_user set to A, b and n, student code gets those three variables. To stop
students from calling a library function such as la.inv, use
import_blacklist rather than setup code.
-
Parameters. Import
data["params"]frompytest_prairielearn_grader.server_data. Values thatserver.pystored withpl.to_jsonare still encoded: decode them withfrom_server_json. Theprairielearnpackage isn't available.from pytest_prairielearn_grader.server_data import from_server_json, params np.random.seed(params["seed"]) A = from_server_json(params["A"])
Only setup code can import this module, whatever the import rules. Older setup code can still use
__data_paramsand__from_server_json, which hold the same values. -
Restrictions. Setup code has the same builtins and import rules as student code. For example, it can't call
openor importos. To read a data file, list it indata_files. -
Fresh values. With the
sandboxfixture, setup code runs again for each test, so random values differ between tests. Read a setup value in the test that uses it, withsandbox.query_setup.
For complex test scenarios, you can use the ConfigObject class to configure all autograder settings
in a single, type-safe, immutable object. This is especially useful when you need to override multiple
settings or want better IDE support with autocomplete and type checking.
Key Features:
- Type-safe: All parameters are type-checked with validation
- Immutable: Configuration cannot be modified after creation (frozen dataclass)
- Keyword-only: Must use explicit parameter names for clarity
- Comprehensive: Supports all security and execution settings
Import and create a ConfigObject at the module level in your test_student.py:
from pytest_prairielearn_grader import ConfigObject
from pytest_prairielearn_grader.assertions import assert_equal, assert_true
from pytest_prairielearn_grader.fixture import StudentFixture
import pytest
# Module-level configuration - detected automatically by the plugin
autograder_config = ConfigObject(
sandbox_timeout=2.0,
import_whitelist=["numpy", "pandas"],
builtin_whitelist=["getattr", "hasattr"],
starting_vars={"coefficient": 10, "threshold": 5.0},
names_for_user=["coefficient", "threshold"],
)
@pytest.mark.grading_data(name="Test with Config", points=5)
def test_with_config(sandbox: StudentFixture) -> None:
# Configuration is automatically applied
result = sandbox.query_function("process", data)
assert_equal(result, expected_value)All parameters are optional:
| Parameter | Default | Description |
|---|---|---|
sandbox_timeout |
1.0 |
Seconds allowed for starting the sandbox, which runs the setup code and student code. See Timeout Configuration. |
output_level |
None |
Default output level for every test. None means "friendly". |
names_for_user |
None |
Variables to give student code. See Giving Values to Student Code. |
starting_vars |
{} |
Values for names in names_for_user. |
student_code_pattern |
"student_code*.py" |
Glob pattern for the student's file. |
import_whitelist |
None |
Modules that code in the sandbox may import. None allows all. See Import Control. |
import_blacklist |
["os", "sys", "subprocess", "pathlib", "shutil"] |
Modules and functions that code in the sandbox may not use. See Blocking Functions. |
builtin_whitelist |
None |
Builtins to allow beyond the safe defaults. See Builtin Function Control. |
helper_modules |
None |
Modules from the tests folder that can be imported. See Helper Modules. |
data_files |
None |
Files from the tests folder that code in the sandbox can read by relative path. See Data Files. |
syntax_errors_ungradable |
True |
Whether a syntax error makes the submission ungradable. See Ungradable Submission Detection. |
notebook_cell_tag |
None |
Grade only notebook cells that start with this tag. See Jupyter Notebook Grading. |
workspace_mode |
False |
Grade a multi-file project. See Workspace Grading. |
workspace_student_dir |
None |
The student's project folder in workspace mode. |
workspace_exec_entry |
None |
A file to run at startup in workspace mode. |
The tests folder isn't on the import path in the sandbox, and import_whitelist only permits a
name. To let setup code and student code import a file from that folder, name it
in helper_modules:
# tests/test_student.py, with the helper in tests/helper_function.py
autograder_config = ConfigObject(helper_modules=["helper_function"])# the student's code, or setup code
import helper_function as hf- Names: each entry is a plain module name, without dots, paths or
.py. Packages aren't supported. - Everything else stays out: other files in the folder, such as
test_student.py, still can't be imported, even if they're inimport_whitelist. - Import restrictions: a listed module doesn't need to be in
import_whitelist. Listing one thatimport_blacklistalso blocks is a configuration error. - Inside a helper module: it runs with the same builtins and import rules as student code. For
example, it can't use
getattror importnumpyunless those are allowed. - Configuration errors: if a listed file is missing or has a syntax error, or the module is blacklisted, the submission is ungradable. The students don't lose an attempt, and each test's message says what's wrong.
- Fixtures: it works with
sandboxandmodule_sandbox, for.pyand.ipynbsubmissions, and withworkspace_sandbox. - Workspace mode: students can have several files that import each other, so a listed name is
reserved. If a student submits a
helper_function.pyor ahelper_function/folder, the submission fails with a message asking them to rename it and counts as an attempt. Don't give students a copy of a helper module in their workspace. A listed module keeps the restricted builtins that the student's own modules don't have. Dotted queries such asquery_function("helper_function.triple")only search the student's project, so they won't call a helper module.
Warning: Students can read everything in a helper module, so it must not contain answers. This setting also doesn't make the rest of the tests folder private.
importlibisn't blacklisted by default, so student code can load files in that folder by path.
To share a module between questions, put it in the course's serverFilesCourse folder and list it
in the question's info.json:
"externalGradingOptions": {
"image": "prairielearn/grader-python-pytest:latest",
"serverFilesCourse": ["course_lib.py"]
}PrairieLearn copies it to /grade/serverFilesCourse, which the grader images put on the import
path, as the legacy grader did. Tests, setup code and student code can then import course_lib.
- Import restrictions: if you set
import_whitelist, list the module in it so that setup code and student code can import it. Tests aren't restricted. - Ordinary modules: unlike helper modules, these load like any installed library, without the sandbox's restricted builtins.
- Testing locally: add the folder to
PYTHONPATHwhen you run pytest.
Warning: Student code can import these modules too, so they must not contain answers.
In a workspace, code such as np.loadtxt("pagerank_large.txt") reads a file next to the notebook. To
make the same code work during grading, put the file in the tests folder and name it in data_files:
# tests/test_student.py, with the data in tests/pagerank_large.txt
autograder_config = ConfigObject(
data_files=["pagerank_large.txt"],
student_code_pattern="Markov-Chains-3.ipynb",
notebook_cell_tag="#grade",
)
def test_pagerank(sandbox: StudentFixture, tests_dir: Path) -> None:
edges = np.loadtxt(tests_dir / "pagerank_large.txt") # the instructor's original
...# the student's code, or setup code
edges = np.loadtxt("pagerank_large.txt")- Paths: each entry is relative to the tests folder, such as
"pagerank_large.txt"or"data/points.csv", and keeps that relative path in the sandbox. Absolute paths and..aren't accepted. - Starting folder: the sandbox starts in a new folder that holds only copies of the listed files.
Without
data_files, the starting folder isn't changed. - Read-only copies: the copies and their folder are read-only for student code, and a fresh copy is
made for each sandbox. Tests should read the originals through the
tests_dirfixture. - Reading, not importing: the folder isn't on the import path, so a listed
.pyfile can't be imported. Usehelper_modulesfor that. - Configuration errors: if a listed file is missing, the submission is ungradable. The students don't lose an attempt, and each test's message says what's wrong.
- Fixtures: it works with
sandboxandmodule_sandbox, for.pyand.ipynbsubmissions, and withworkspace_sandbox. In workspace mode, relative paths resolve against the data files' folder, not the student's workspace folder.
Warning: Students can read the listed files, so they must not contain answers.
When ConfigObject is defined, it overrides all other configuration sources:
- Highest priority:
ConfigObject(whenautograder_configvariable exists) - Medium priority: Module-level variables (e.g.,
sandbox_timeout = 2.0) - Lowest priority:
data.jsonparams from PrairieLearn
# This ConfigObject overrides everything
autograder_config = ConfigObject(
sandbox_timeout=3.0, # Overrides module-level timeout
import_whitelist=["numpy"], # Overrides data.json import_whitelist
starting_vars={"x": 10}, # Overrides data.json params["x"]
names_for_user=["x"],
)
# This module-level timeout is ignored when ConfigObject is present
sandbox_timeout = 1.0from pytest_prairielearn_grader import ConfigObject
from pytest_prairielearn_grader.assertions import assert_equal, assert_true
from pytest_prairielearn_grader.fixture import StudentFixture, FeedbackFixture
import pytest
# Comprehensive configuration
autograder_config = ConfigObject(
# Execution settings
sandbox_timeout=2.5,
# Security: only allow scientific computing libraries
import_whitelist=["numpy", "scipy", "matplotlib"],
# Allow builtins that the defaults leave out
builtin_whitelist=["getattr", "hasattr"],
# Provide test data
starting_vars={
"input_data": [1, 2, 3, 4, 5],
"multiplier": 2.5,
"threshold": 10,
},
# Explicitly list what gets injected
names_for_user=["input_data", "multiplier", "threshold"],
)
@pytest.mark.grading_data(name="Test Processing", points=10)
def test_data_processing(sandbox: StudentFixture, feedback: FeedbackFixture) -> None:
# All configuration from ConfigObject is automatically applied
result = sandbox.query_function("process_data")
assert_true(len(result) > 0, msg="Result should not be empty")
feedback.set_score(0.5)
assert_true(result == expected_output, msg="Incorrect processing result")
feedback.set_score(1.0)ConfigObject validates all parameters at creation time:
# ✓ Valid configuration
config = ConfigObject(
sandbox_timeout=2.0,
import_whitelist=["numpy"],
)
# ✓ Valid: can combine whitelist and blacklist
config = ConfigObject(
import_whitelist=["numpy", "matplotlib"],
import_blacklist=["matplotlib.pyplot"] # Block pyplot specifically
)
# ✗ Error: timeout must be positive
config = ConfigObject(sandbox_timeout=-1.0) # ValueError
# ✗ Error: must use keyword arguments
config = ConfigObject(2.0) # TypeError
# ✗ Error: names_for_user must contain strings
config = ConfigObject(names_for_user=[123, 456]) # ValueErrorControl execution time limits for sandbox initialization and function calls to prevent infinite loops or slow student code.
Important: Timeouts apply to:
- Sandbox initialization (loading and executing student code at startup)
- Function calls via
sandbox.query_function()withquery_timeoutparameter
Variable queries via sandbox.query() do not have timeouts since they're simple lookups.
Set a default timeout for sandbox initialization in all tests in a file. This variable and the
sandbox_timeout marker below are ignored when the file has an autograder_config, so use
ConfigObject(sandbox_timeout=...) there instead.
import pytest
from pytest_prairielearn_grader.assertions import assert_equal, assert_true
from pytest_prairielearn_grader.fixture import StudentFixture
sandbox_timeout = 2.0 # 2 second timeout for initialization
@pytest.mark.grading_data(name="Test 1", points=1)
def test_with_default_timeout(sandbox: StudentFixture) -> None:
# The sandbox was initialized with 2 second timeout
# Now we can safely query variables and call functions
result = sandbox.query_function("compute_result")
assert_equal(result, 5)Override the default initialization timeout with the @pytest.mark.sandbox_timeout marker:
@pytest.mark.grading_data(name="Fast Test", points=1)
@pytest.mark.sandbox_timeout(0.5) # 0.5 second initialization timeout
def test_with_custom_timeout(sandbox: StudentFixture) -> None:
# This sandbox was initialized with 0.5 second timeout
result = sandbox.query_function("quick_computation")
assert_equal(result, 5)Set timeout for individual function calls using the query_timeout parameter:
@pytest.mark.grading_data(name="Function Test", points=2)
def test_function_timeout(sandbox: StudentFixture) -> None:
# This specific function call has a 1 second timeout
result = sandbox.query_function("compute", data, query_timeout=1.0)
assert_equal(result, expected_value)
# This function call uses the default timeout (no per-function limit)
result2 = sandbox.query_function("another_compute", data)
assert_equal(result2, expected_value2)Using ConfigObject for timeouts:
from pytest_prairielearn_grader import ConfigObject
autograder_config = ConfigObject(
sandbox_timeout=3.0, # Sets initialization timeout
)
@pytest.mark.grading_data(name="Test", points=1)
def test_with_config_timeout(sandbox: StudentFixture) -> None:
# Initialized with 3 second timeout from ConfigObject
result = sandbox.query_function("process", query_timeout=1.0)
assert_equal(result, expected)Control which Python modules and builtin functions students can use in their code. This is crucial for:
- Security: Preventing access to file system, network, or system operations
- Pedagogical constraints: Requiring students to implement functionality from scratch
- Resource management: Blocking operations that consume excessive resources
Restrict which Python modules students can import using whitelists or blacklists.
Import Whitelist: Only allow specific modules:
# In server.py (PrairieLearn question)
def generate(data):
data["params"]["import_whitelist"] = ["numpy", "math", "statistics"]
# Only numpy, math, and statistics can be imported
# Attempting to import any other module raises ImportErrorImport Blacklist: Block specific modules while allowing all others:
# In server.py
def generate(data):
data["params"]["import_blacklist"] = ["os", "subprocess", "sys"]
# os, subprocess, and sys cannot be imported
# All other modules are allowedCombining Whitelist and Blacklist: Use both for fine-grained control (blacklist checked first):
# In server.py
def generate(data):
# Allow scientific libraries but block specific dangerous submodules
data["params"]["import_whitelist"] = ["numpy", "scipy", "matplotlib"]
data["params"]["import_blacklist"] = ["matplotlib.pyplot"] # Block pyplot even though matplotlib is allowed
# Result: numpy, scipy, and matplotlib are allowed, but matplotlib.pyplot is blockedExample student code with import whitelist:
# data.json has "import_whitelist": ["numpy", "math"]
import numpy as np # ✓ Allowed
import math # ✓ Allowed
from numpy import array # ✓ Allowed
import os # ✗ ImportError: Module 'os' is not allowed to be imported
import pandas # ✗ ImportError: Module 'pandas' is not allowed to be importedExample with import inside function:
def my_function():
import os # ✗ This will raise ImportError when the function is called
return os.getcwd()Note: Blacklist is checked first, then whitelist. If a whitelist is specified, only those modules (minus any in blacklist) can be imported. If only a blacklist is specified, those modules are blocked but all others are allowed.
A blacklisted module also blocks its submodules, however they're imported. With "os" in the blacklist, import os.path is blocked as well as import os. With "matplotlib.pyplot", both import matplotlib.pyplot and from matplotlib import pyplot are blocked, but import matplotlib still works.
The blacklist only checks import statements, so it isn't a security boundary. For example, modules that student code is allowed to import can expose os as an attribute.
To stop students from using one library function, such as numpy.linalg.inv in a question about
solving linear systems, add the function to import_blacklist:
autograder_config = ConfigObject(
import_blacklist=["os", "sys", "subprocess", "pathlib", "shutil", "numpy.linalg.inv", "numpy.linalg.pinv"],
sandbox_timeout=15,
)Setting import_blacklist replaces the default list, so keep its entries.
Before setup code runs, the sandbox replaces each listed function in its module with one that
raises an error. So np.linalg.inv(A) and la.inv(A) raise
RuntimeError: 'numpy.linalg.inv' is blacklisted and cannot be used., and
from numpy.linalg import inv raises an ImportError. This is the preferred way to block a
function: don't override it in setup code.
- Which entries: an entry is treated as a function when the part before its last dot is a module
that can be imported and the last part is a callable that isn't a module. Other entries, such as
"os"or"matplotlib.pyplot", block imports as described above. If the module isn't installed in the image, the entry does nothing. - Setup code and helper modules can't call the function either, because they share the sandbox's modules. Compute values that need it in the test instead.
- Startup time: the function's module is imported when the sandbox starts, even if the student's code doesn't use it.
- Not a security boundary: the function can still be reached under another name, such as
numpy's internal
numpy.linalg._linalg.inv.
Setup code, student code and helper modules get only these builtins by default:
- Types:
bool,bytearray,bytes,complex,dict,float,frozenset,int,list,object,set,slice,str,tuple,type - Type checking:
callable,isinstance,issubclass - Iteration:
all,any,enumerate,filter,iter,map,next,range,reversed,sorted,zip - Math:
abs,divmod,max,min,pow,round,sum - Conversion:
ascii,bin,chr,format,hash,hex,id,oct,ord,repr - Classes: class definitions,
classmethod,property,staticmethod,super - Other:
len,print,None,True,False,Ellipsis,NotImplemented - The standard exception classes, such as
ValueError
Builtins that read or change attributes and namespaces by name, such as getattr, hasattr,
setattr, vars, dir and globals, aren't in this list. Student code that uses one fails with a
NameError, so add the ones your question's solutions need to builtin_whitelist. A builtin in
the default list can't be removed with builtin_whitelist.
Dangerous builtins automatically blocked include: open, eval, exec, compile, __import__, input, exit, quit, and others that could compromise security.
Builtin Whitelist: Grant access to additional builtin functions:
# In server.py
def generate(data):
data["params"]["builtin_whitelist"] = ["getattr", "hasattr"]
# Students can now use getattr() and hasattr() in addition to the defaultsExample with builtin whitelist:
# data.json has "builtin_whitelist": ["getattr"]
# Default builtins work normally:
my_list = [1, 2, 3] # ✓ list is a default
length = len(my_list) # ✓ len is a default
print(length) # ✓ print is a default
# Whitelisted builtins work:
count = getattr(my_list, "count")(1) # ✓ getattr is in the whitelist
# Other builtins are blocked:
names = dir(my_list) # ✗ NameError: name 'dir' is not defined
# Dangerous builtins are blocked:
f = open("file.txt") # ✗ NameError: 'open' is not defined
result = eval("1+1") # ✗ NameError: 'eval' is not definedYou can combine import and builtin restrictions for comprehensive control:
# In server.py
def generate(data):
# Allow only numpy and math imports
data["params"]["import_whitelist"] = ["numpy", "math"]
# Allow getattr for student convenience
data["params"]["builtin_whitelist"] = ["getattr"]
# Now students can:
# - Import numpy and math
# - Use getattr() plus the default builtins
# - Cannot import other modules
# - Cannot use dangerous builtins like open(), eval()In your PrairieLearn question's server.py, add these parameters during the generate() step:
import prairielearn as pl
def generate(data):
# Set up question parameters
data["params"]["n"] = 5
data["params"]["A"] = pl.to_json(np.random.rand(5, 5))
# Configure security restrictions
data["params"]["import_whitelist"] = ["numpy", "numpy.linalg"]
data["params"]["builtin_whitelist"] = ["getattr"]
# Define variables accessible to student code
data["params"]["names_for_user"] = [
{"name": "n", "description": "Matrix dimension", "type": "integer"},
{"name": "A", "description": "Input matrix", "type": "numpy array"}
]These parameters are automatically passed to the autograder via /grade/data/data.json and enforced during student code execution.
In the grader images, run.sh passes this path to pytest with --data-json-path and makes the file readable only by root, so student code can't open it. The file can contain correct_answers. Without that option, for example when running locally, the fixture reads data.json next to your test file. The sandbox still receives params for setup code, so don't rely on values in params staying hidden from student code.
To test your restrictions in the test scenarios:
# tests/test_student.py
import pytest
from pytest_prairielearn_grader.assertions import assert_equal, assert_true
from pytest_prairielearn_grader.fixture import StudentFixture
@pytest.mark.grading_data(name="Test with restrictions", points=5)
def test_security(sandbox: StudentFixture) -> None:
"""Test that imports are properly restricted."""
# If student code tries to import blocked modules,
# the sandbox will raise an ImportError during initialization
result = sandbox.query_function("safe_function", data)
assert_equal(result, expected)# data.json
{
"params": {
"import_whitelist": ["numpy", "math"],
"builtin_whitelist": ["getattr"],
"names_for_user": [...]
}
}1. Force students to implement algorithms from scratch:
# Require students to implement algorithms without libraries
data["params"]["import_whitelist"] = [] # No imports allowed
# Default builtins such as sorted() and sum() are still available.2. Allow numerical computing but block system access:
# Scientific computing course
data["params"]["import_whitelist"] = ["numpy", "scipy", "matplotlib", "pandas"]
# Safe builtins are automatically enforced (no open, eval, etc.)3. Block dangerous operations explicitly:
# Block system and file operations
data["params"]["import_blacklist"] = ["os", "sys", "subprocess", "pathlib", "shutil"]
# All other imports allowed, but dangerous ones explicitly blockedWhen student code has a SyntaxError (e.g., missing colons, unmatched brackets, bad indentation), the grader automatically marks the submission as ungradable. This means:
- The student does not lose a grading attempt
- They see a clear message: "Your code could not be parsed. Please fix the syntax errors and resubmit."
- The
format_errorsfield in results contains the specific error
This is especially useful for introductory courses where students may accidentally submit code with trivial syntax mistakes.
The grader detects SyntaxError during sandbox initialization (when compile() is called on
student code). Instead of failing all tests with 0%, it produces:
{
"gradable": false,
"format_errors": ["SyntaxError: invalid syntax (student_code.py, line 3)"],
"message": "Your code could not be parsed. Please fix the syntax errors and resubmit.",
"tests": [...]
}PrairieLearn's external grading framework handles the "gradable": false field — the student's
submission is marked as "invalid, not gradable", no points are awarded or lost, and the student
is not penalized an attempt. See the
PrairieLearn external grading docs
for full details on the results.json format.
If you prefer syntax errors to score 0% (counting as a normal attempt), opt out in your config:
from pytest_prairielearn_grader import ConfigObject
autograder_config = ConfigObject(
syntax_errors_ungradable=False, # SyntaxError → 0% instead of ungradable
)The grader also detects when test files themselves fail to collect (e.g., import errors in your test code, missing dependencies). These are marked as ungradable with a message directing course staff to check the logs — students are never penalized for grader bugs.
PrairieLearn supports JupyterLab as a workspace environment
(prairielearn/workspace-jupyterlab-python), and students commonly submit .ipynb files from
other IDE-based workspaces too. The autograder handles notebooks natively: it extracts the Python
source from the relevant code cells, concatenates them into a single script, and executes that
script in the sandbox exactly like a normal .py submission. Every fixture, query method, and
partial credit feature works identically — no special test-writing API is needed.
Optional dependency: Notebook support requires
nbformat. Install it withpip install 'pytest-prairielearn-grader[notebook]'. The core package deliberately does not depend onnbformatso that grader images without Jupyter installed remain lightweight.
The only required change is updating student_code_pattern in your ConfigObject to match
.ipynb files:
from pytest_prairielearn_grader import ConfigObject
autograder_config = ConfigObject(
student_code_pattern="student_code*.ipynb",
sandbox_timeout=5.0,
)All code cells are extracted in notebook order and concatenated, so functions and variables defined in earlier cells are available in later ones — exactly as they are when the notebook is executed top-to-bottom in JupyterLab.
Real student notebooks often contain scratch work, exploratory plots, and print statements that
you don't want to run during grading. Use notebook_cell_tag to restrict extraction to only the
cells the student intended to submit:
autograder_config = ConfigObject(
student_code_pattern="student_code*.ipynb",
notebook_cell_tag="#grade", # only cells whose first line starts with "#grade"
sandbox_timeout=5.0,
)A code cell is included if and only if its first non-empty line starts with the tag string. All other cells — including cells that would raise errors or produce unwanted side effects — are silently skipped.
Student notebook example:
# Cell 1 — scratch work (NOT included, no tag)
import matplotlib.pyplot as plt
plt.plot([1, 2, 3])
plt.show()
# Cell 2 — graded solution (included, starts with #grade)
#grade
def solve(A, b):
import numpy as np
return np.linalg.solve(A, b)
# Cell 3 — another graded cell (included, starts with #grade)
#grade
x = solve([[1, 0], [0, 2]], [3, 8])Only cells 2 and 3 are extracted and run. Cell 1 is completely ignored, so the plt.show() call
never reaches the sandbox.
notebook_cell_tag=None(the default) includes all code cells. Use this when you control the notebook template and don't need students to tag their answers.
{
"uuid": "...",
"title": "Linear Algebra — Notebook Submission",
"topic": "...",
"tags": ["..."],
"type": "v3",
"singleVariant": true,
"gradingMethod": "External",
"workspaceOptions": {
"image": "prairielearn/workspace-jupyterlab-python",
"port": 8080,
"home": "/home/user",
"gradedFiles": ["notebook.ipynb"]
},
"externalGradingOptions": {
"enabled": true,
"image": "prairielearn/grader-python-pytest:latest",
"timeout": 60
}
}Set
singleVariant: trueso the workspace — and any work saved in it — persists across multiple submission attempts.
Tests are written the same way as for any other submission type. There is nothing
notebook-specific in the test file beyond student_code_pattern (and optionally
notebook_cell_tag) in the ConfigObject:
import pytest
import numpy as np
from pytest_prairielearn_grader import ConfigObject
from pytest_prairielearn_grader.assertions import assert_equal, assert_true
from pytest_prairielearn_grader.fixture import FeedbackFixture, StudentFixture
autograder_config = ConfigObject(
student_code_pattern="notebook*.ipynb",
notebook_cell_tag="#grade",
sandbox_timeout=10.0,
)
@pytest.mark.grading_data(name="solve() is correct", points=5)
def test_solve(sandbox: StudentFixture, feedback: FeedbackFixture) -> None:
A = [[2, 1], [1, 3]]
b = [5, 10]
feedback.set_score(0.0)
result = sandbox.query_function("solve", A, b)
feedback.set_score(0.5)
expected = np.linalg.solve(A, b)
assert_true(np.allclose(result, expected), msg="solve(A, b) is inaccurate")
feedback.set_score(1.0)
@pytest.mark.grading_data(name="result variable is set", points=2)
def test_result_variable(sandbox: StudentFixture, feedback: FeedbackFixture) -> None:
feedback.set_score(0.0)
x = sandbox.query("x")
assert_true(x is not None, msg="Variable 'x' was not defined")
feedback.set_score(1.0)| Parameter | Type | Default | Description |
|---|---|---|---|
student_code_pattern |
str |
"student_code*.py" |
Glob pattern — change to "*.ipynb" or "notebook*.ipynb" to match notebook files. |
notebook_cell_tag |
str | None |
None |
First-line tag for cell filtering. None = all cells. Cannot be used with workspace_mode=True. |
sandbox_timeout |
float |
1.0 |
Increase to 5–15 s for notebooks with heavy imports or large cell computations. |
In production PrairieLearn, the submitted gradedFiles end up in /grade/student/. The grader
searches for matching files there automatically. For local development, place the notebook in the
tests folder next to your test file (the folder named after it):
questions/my_notebook_question/
└── tests/
├── test_student.py
└── test_student/
└── notebook.ipynb # local stand-in for the student's submission
PrairieLearn workspace questions give students a full IDE (VS Code, JupyterLab, etc.) running in a
browser. When a student clicks Submit, PrairieLearn copies the files listed in gradedFiles
out of their workspace container and into /grade/student/, preserving all directory structure.
The workspace_sandbox fixture is designed specifically for this use case. Unlike the regular
sandbox fixture (which execs a single file), the workspace fixture adds the student's project
directory to sys.path and lets tests interact with the project using Python's normal import
machinery — exactly the same way you would test a local Python package.
sandbox |
workspace_sandbox |
|
|---|---|---|
| Student code | Single file | Multi-file project |
| Startup | exec the file |
Set sys.path, import on demand |
| Querying | Flat variable/function names | Dotted module paths ("models.predict") |
| Use case | pl-file-editor, pl-file-upload |
Workspace questions |
For a PrairieLearn workspace question the grader sees this layout at runtime:
/grade/
├── data/
│ └── data.json # question parameters
├── student/ # gradedFiles copied from student's workspace
│ ├── calculator.py
│ ├── utils/
│ │ ├── __init__.py
│ │ └── helpers.py
│ └── main.py
└── tests/ # copies of your question's tests/ directory
├── setup_code.py # (optional)
└── test_student.py
For local development, mirror this by creating a student/ sub-directory next to your test
file. The fixture finds it automatically without any configuration:
questions/my_workspace_question/
├── info.json
├── question.html
└── tests/
├── test_student.py # your test file
├── setup_code.py # (optional)
└── test_student/ # ← mirrors the tests folder layout
└── student/ # ← local stand-in for /grade/student
├── calculator.py
└── utils/
├── __init__.py
└── helpers.py
{
"uuid": "...",
"title": "Calculator Project",
"topic": "...",
"tags": ["..."],
"type": "v3",
"singleVariant": true,
"gradingMethod": "External",
"workspaceOptions": {
"image": "prairielearn/workspace-vscode-python",
"port": 8080,
"home": "/home/user",
"gradedFiles": ["calculator.py", "utils/*.py", "utils/__init__.py"]
},
"externalGradingOptions": {
"enabled": true,
"image": "prairielearn/grader-python-pytest:latest",
"timeout": 30
}
}Key points:
gradedFilescontrols which files are copied from the workspace into/grade/student/. Use glob patterns (utils/*.py) to capture whole directories.singleVariant: trueprevents the workspace from resetting between attempts.
Use workspace_sandbox and query student code with dotted module paths:
from pytest_prairielearn_grader import ConfigObject
from pytest_prairielearn_grader.assertions import assert_equal, assert_true
from pytest_prairielearn_grader.fixture import WorkspaceFixture
import pytest
autograder_config = ConfigObject(
workspace_mode=True,
sandbox_timeout=10.0,
)
@pytest.mark.grading_data(name="Add two numbers", points=2)
def test_add(workspace_sandbox: WorkspaceFixture) -> None:
result = workspace_sandbox.query_function("calculator.add", 3, 4)
assert_equal(result, 7)
@pytest.mark.grading_data(name="Clamp from sub-package", points=2)
def test_clamp(workspace_sandbox: WorkspaceFixture) -> None:
result = workspace_sandbox.query_function("utils.helpers.clamp", 15, 0, 10)
assert_equal(result, 10)
@pytest.mark.grading_data(name="Module-level constant", points=1)
def test_constant(workspace_sandbox: WorkspaceFixture) -> None:
pi = workspace_sandbox.query("calculator.PI_APPROX")
assert_true(abs(pi - 3.14159) < 1e-5)The dotted path is split on the last dot: "calculator.add" imports the module calculator
and calls getattr(calculator, "add"). This means:
| Query string | Module imported | Attribute retrieved |
|---|---|---|
"calculator.add" |
calculator |
add |
"utils.helpers.clamp" |
utils.helpers |
clamp |
"models.nn.Model.predict" |
models.nn.Model |
predict |
Both query and query_function understand dotted paths:
# Query a module-level variable
eps = workspace_sandbox.query("utils.helpers.EPSILON")
# Query a class defined in a module
result = workspace_sandbox.query_function("models.Classifier.predict", X_test)
# query_function_raw — inspects the full response without raising
response = workspace_sandbox.query_function_raw("calculator.divide", 1, 0)
assert_equal(response["status"], "exception")
assert_equal(response["exception_name"], "ValueError")Some workspace questions have a main.py that sets up global state before tests run. Use
workspace_exec_entry to exec it at sandbox startup:
autograder_config = ConfigObject(
workspace_mode=True,
workspace_exec_entry="main.py", # exec'd from workspace root at startup
sandbox_timeout=15.0,
)
# Now the sandbox namespace includes everything main.py defined at module level
def test_global_state(workspace_sandbox: WorkspaceFixture) -> None:
db = workspace_sandbox.query("database") # flat name - set by main.py via exec
assert_true(db is not None)Without workspace_exec_entry (the default), no file is executed at startup and all access
goes through the import machinery.
The sandbox loads some modules for itself before any student or setup code runs, such as
random, json, types, typing, logging and math. Python reuses a module that's already
loaded, so a student file named random.py (or a folder named json/) at the top of the project
would be silently ignored in favor of the standard library module.
Instead, startup fails and every test is an error worth 0 points, with a message listing the files and asking the student to rename them:
Student code execution failed with an exception: ImportError
Exception Message: These names are already used by modules that the grader has loaded: json/, random.py. Rename yours and update any imports that use them.
Only top-level .py files and folders are checked, and only against modules the sandbox has
actually loaded. A file such as calendar.py or utils.py that doesn't match a loaded module is
imported normally. The list of loaded modules can change between Python versions.
| Parameter | Type | Default | Description |
|---|---|---|---|
workspace_mode |
bool |
False |
Enable workspace grading mode |
workspace_student_dir |
str | None |
None |
Path to student project root. Defaults to student/ next to tests. Set to "/grade/student" for production. |
workspace_exec_entry |
str | None |
None |
Relative path (from workspace root) to an entry-point file to exec at startup. |
sandbox_timeout |
float |
1.0 |
Timeout for sandbox startup (increase for larger projects). |
import_whitelist |
list[str] | None |
None |
Allowed imports. None = allow all. |
import_blacklist |
list[str] | None |
default list | Blocked imports and functions. |
helper_modules |
list[str] | None |
None |
Modules from the tests folder to make importable. See Helper Modules. |
The workspace_student_dir path differs between environments:
# Production (running on PrairieLearn): set an absolute path
autograder_config = ConfigObject(
workspace_mode=True,
workspace_student_dir="/grade/student",
)
# Local dev: omit the field — the fixture finds student/ automatically
autograder_config = ConfigObject(
workspace_mode=True,
# workspace_student_dir defaults to student/ next to your test file
)A common pattern is to use an environment variable so one config works in both environments:
import os
autograder_config = ConfigObject(
workspace_mode=True,
workspace_student_dir=os.environ.get("STUDENT_DIR"), # None → auto-detect locally
sandbox_timeout=10.0,
)- The default
import_blacklist(["os", "sys", "subprocess", "pathlib", "shutil"]) is applied in workspace mode too. If the student's project legitimately usespathlibfor internal file operations, setimport_blacklistto a list without it. Adding it toimport_whitelistdoesn't help, because the blacklist is checked first. import_whitelistandimport_blacklistapply to imports in the student's project, setup code and the entry point. They don't apply to imports made inside library code, so an allowed library such asnumpyorpandascan still loadosand its other dependencies.sys.pathinside the subprocess is modified to includeworkspace_student_dir, so all relative imports within the project work naturally.- Each call to
workspace_sandbox.query_functionruns in the same subprocess, so module-level state (caches, open files) persists across calls within a test — exactly like normal Python.
tests/test_student.py:
import pytest
from pytest_prairielearn_grader import ConfigObject
from pytest_prairielearn_grader.assertions import assert_equal, assert_true
from pytest_prairielearn_grader.fixture import FeedbackFixture, WorkspaceFixture
autograder_config = ConfigObject(
workspace_mode=True,
sandbox_timeout=10.0,
import_whitelist=["math", "statistics"],
)
@pytest.mark.grading_data(name="add() returns correct value", points=3)
def test_add(workspace_sandbox: WorkspaceFixture, feedback: FeedbackFixture) -> None:
feedback.set_score(0.0)
response = workspace_sandbox.query_function_raw("calculator.add", 2, 3)
assert_true(response["status"] != "not_found", msg="calculator.add is not defined")
feedback.set_score(0.5)
assert_true(
response["status"] == "success",
msg=f"calculator.add raised {response['exception_name']}: {response['exception_message']}",
)
feedback.set_score(0.8)
assert_equal(response["value"], 5, description="calculator.add(2, 3)")
feedback.set_score(1.0)
@pytest.mark.grading_data(name="divide() raises ValueError on zero", points=2)
def test_divide_guard(workspace_sandbox: WorkspaceFixture) -> None:
response = workspace_sandbox.query_function_raw("calculator.divide", 10, 0)
assert_true(response["status"] == "exception", msg="divide() should raise on zero denominator")
assert_equal(response["exception_name"], "ValueError")
@pytest.mark.grading_data(name="PI_APPROX constant exists", points=1)
def test_constant(workspace_sandbox: WorkspaceFixture) -> None:
pi = workspace_sandbox.query("calculator.PI_APPROX")
assert_true(isinstance(pi, float), msg="PI_APPROX should be a float")
assert_true(abs(pi - 3.14) < 0.01)Student workspace (student/calculator.py):
PI_APPROX = 3.14159
def add(a, b):
return a + b
def divide(a, b):
if b == 0:
raise ValueError("Cannot divide by zero")
return a / bTo see more examples of what is possible in these test files, look at the test cases in
this folder. Each test file is called scenario.py.
- Keep code in the tests: See Where Code Belongs.
- Use descriptive test names: The
namein@pytest.mark.grading_datais shown to students. - Use friendly messages that don't give away answers: Use the assertion helpers, and
assert_truefor hidden test cases. - Test incrementally: Break complex problems into smaller tests with partial credit.
- Prefer
parametrize: Use@pytest.mark.parametrizefor several inputs instead of a loop, unless the inputs share partial credit. - Control output visibility: Use
include_stdout_feedback=Falsefor tests where student output would be confusing or reveal answers. - Set appropriate timeouts: Student code that imports numpy or pandas usually needs a
sandbox_timeoutof several seconds. - Use
module_sandboxsparingly: Only when you need persistent state across tests. - Watch for
__main__blocks: Student code runs with__name__ == "__main__", so code underif __name__ == "__main__":runs during grading. - Configure security restrictions: Use
import_whitelist,import_blacklistandbuiltin_whitelistto control what modules and functions students can access.