This guide explains how to convert a PrairieLearn question from the legacy Python autograder
(prairielearn/grader-python, with tests/test.py and PLTestCase) to this pytest grader
(prairielearn/grader-python-pytest). It's written so that you can give it to a person or an AI
coding agent and have them do the conversion from start to finish.
quick_start.md shows how to write a question for this grader and documents its full API. This guide covers only what's specific to converting: what changes, in what order to do it, and the differences from the legacy grader that are easy to miss.
Converted questions in the PrairieLearn example course are good references:
Question (under exampleCourse/questions/demo/) |
Shows |
|---|---|
autograder/python/square |
A scalar with a tolerance. It still keeps x in setup_code.py, unlike the version below |
autograder/python/pandas |
Checking a DataFrame |
autograder/python/randomFunctionName |
server.py parameters, data_json, partial credit, timeouts |
autograder/codeEditor |
A function question |
annotated/MarkovChainGroupActivity/* |
Notebook workspaces, helper_modules, data_files |
Legacy questions often run grading code next to the student's code, in leading_code.py,
trailing_code.py and setup_code.py. Don't carry that over. Follow
Where Code Belongs, which says where each kind of code goes in
this grader and why. A converted question should usually have only initial_code.py and
test_student.py, plus setup_code.py only when student code needs something from it:
ans.py: compute the expected values in the tests.trailing_code.py: call the student's functions from the tests withquery_function.leading_code.py: move imports and definitions to the starter code, or tohelper_modules.setup_code.py: move random plain values toserver.py, fixed values tostarting_varsand values that only tests use to the tests. Keep what's left, such as numpy arrays and blocked functions.
Don't change what the student has to write to make this work. If the student's task is to set
variables at the top level, keep it that way, and read the variables with sandbox.query. If
students have already submitted to the question, check that removing leading_code.py doesn't break
those submissions when they're regraded.
Read every file in the question, including info.json, question.html, server.py and everything in
tests/. Stop and report back instead of converting if the question needs one of these:
- A custom grader image or an
entrypointinexternalGradingOptions. - Values the sandbox can't send back. Tests can only receive values that can be serialized:
numbers, strings,
None, lists, dicts, numpy arrays and scalars, pandas DataFrames and matplotlib plots. A test that needs a student's class instance, generator or open file has to be rewritten to call student functions that return plain values. Do that only if the student's task stays the same.
| Legacy file | What to do |
|---|---|
info.json |
Change externalGradingOptions.image. Keep timeout and everything else. See Choosing an image. |
question.html |
Change the file editor's file-name to student_code.py, or keep it and set student_code_pattern. |
tests/test.py |
Replace with tests/test_student.py. See Rewrite the tests. |
tests/ans.py |
Delete it. Compute expected values in test_student.py. Don't leave a reference solution in tests/. |
tests/setup_code.py |
Move what you can out of it, then delete it or update it. See Update the setup code. |
tests/leading_code.py |
Move its code to the starter code or helper_modules, and delete it. See Where code belongs. |
tests/trailing_code.py |
Move its code into the tests and delete it. See Where code belongs. |
tests/initial_code.py |
Keep it. The grader doesn't read it. Fix starter code that doesn't parse (see below). |
server.py |
Usually unchanged. |
Other changes:
- Starter code that doesn't parse. A submission with a
SyntaxErroris ungradable, so it doesn't cost the student an attempt, but the student sees no test results. Ifinitial_code.pyhas placeholders likex = # fill this in, replace them with something that parses, such asx = 0 # fill this in, so an unchanged submission gets 0 points with feedback. <pl-external-grader-variables>. Keep these elements. Thenames_for_userlist still controls which setup variables student code gets. The grader ignoresnames_from_user, which only displays the table.- The file name. The grader looks for
student_code*.pyby default, while the legacy grader looked foruser_code.py. Changefile-name="user_code.py"tofile-name="student_code.py"inquestion.html, or addstudent_code_pattern="user_code.py"to theConfigObject. For workspace and notebook questions, setstudent_code_patternto the graded file's name, such as"Markov-Chains-1.ipynb".
prairielearn/grader-python-pytest:litewhen the student code, setup code and tests use only the standard library.prairielearn/grader-python-pytest:latestfor anything else, including numpy, pandas, matplotlib, sympy and notebooks.
See Choosing an Image for what each includes.
Each test_* method on the legacy Test class becomes a test_* function in
tests/test_student.py. Keep each test's name, points and order, and keep checking the same things
with the same tolerances. Don't make the question easier or harder.
Here's the square question, which asks students to set x_sq to the square of a random x.
The legacy version has four files in tests/:
# tests/setup_code.py
import random
x = random.random()# tests/ans.py
x_sq = x**2# tests/initial_code.py
import numpy as np
x_sq = # fill in this code here...# tests/test.py
class Test(PLTestCaseWithPlot):
@points(10)
@name("x_sq")
def test_0(self):
if Feedback.check_scalar("x_sq", self.ref.x_sq, self.st.x_sq, accuracy_critical=False):
Feedback.set_score(1)
else:
Feedback.set_score(0)Converted, following Where code belongs:
xis a plain random value that student code reads, soserver.pygenerates it instead ofsetup_code.py.question.htmlalready listsxin<pl-external-grader-variables params-name="names_for_user">, so student code still gets it.- The expected value comes from the test, so
ans.pyis deleted. - The starter code parses, so an unchanged submission gets 0 points with feedback.
question.htmlchangesfile-name="user_code.py"tofile-name="student_code.py", andinfo.jsonchanges the image toprairielearn/grader-python-pytest:lite.
# server.py
import random
def generate(data):
data["params"]["x"] = random.random()# tests/initial_code.py
x_sq = 0 # fill in this code here...# tests/test_student.py
import pytest
from pytest_prairielearn_grader.assertions import assert_approx_equal
from pytest_prairielearn_grader.fixture import StudentFixture
@pytest.mark.grading_data(name="x_sq", points=10)
def test_x_sq(sandbox: StudentFixture, data_json: dict) -> None:
x = data_json["params"]["x"]
assert_approx_equal(sandbox.query("x_sq"), x**2, description="x_sq")The converted question has no setup_code.py. x is now the same in every test and every
regrade of a variant, and question.html could show it with {{params.x}}.
| Legacy | pytest grader |
|---|---|
class Test(PLTestCase) with test_* methods |
Module-level test_* functions that take the sandbox fixture |
@points(10) and @name("...") |
@pytest.mark.grading_data(name="...", points=10) |
| Tests run in sorted name order | Tests run in file order |
self.st.x |
sandbox.query("x") |
Feedback.call_user(self.st.f, a, b) |
sandbox.query_function("f", a, b), or query_function_raw to inspect exceptions and timeouts |
self.ref.x from ans.py |
Compute the expected value in the test |
A setup variable, such as x from setup_code.py |
sandbox.query_setup("x") |
self.data["params"]["k"] |
The data_json fixture: data_json["params"]["k"] |
Feedback.set_score(s) |
feedback.set_score(s) with the feedback fixture. See Scoring. |
Feedback.add_feedback("...") |
feedback.add_message("...") |
accuracy_critical=True |
A failed assertion ends the test. Other tests still run. |
total_iters = 3 and @not_repeated |
@pytest.mark.parametrize over the inputs, or a loop of subtests.test() blocks for partial credit |
Feedback.not_allowed |
import_blacklist=["numpy.linalg.inv"] in the ConfigObject. See Update the setup code. |
repeated_setup() |
Not needed: setup code runs again for each sandbox. |
Replace the Feedback.check_* functions with the helpers in pytest_prairielearn_grader.assertions,
or with numpy and pandas functions wrapped in assert_true:
| Legacy | pytest grader |
|---|---|
check_scalar(name, ref, st) |
assert_approx_equal(value, expected, description=name), keeping rtol=1e-5, atol=1e-8 unless the test set others |
| Exact comparisons | assert_equal(value, expected, description=name) |
call_user plus a comparison |
assert_fn_equal(sandbox, "f", args=(...), expected=...) or assert_fn_approx_equal |
check_numpy_array_allclose(name, ref, st) |
The four array checks on type, dtype, shape and values |
check_numpy_array_features(name, ref, st) |
The first three of those array checks |
check_list(name, ref, st, entry_type=T) |
assert_true(isinstance(value, list), ...), a length check and an isinstance check on each entry |
check_tuple, check_dict |
assert_equal |
check_dataframe(name, ref, st) |
pandas.testing.assert_frame_equal(expected, df, check_like=True), as in the pandas example question |
check_plot(name, ref, st, check_axes_scale=...) |
assert_plot_equal(plot, reference) with a reference plot drawn in the test. It checks the scales by default. See Plots. |
Keep check_scalar's default tolerances of rtol=1e-5, atol=1e-8 unless the legacy test set others.
PLTestCaseWithPlot gave tests the student's pyplot module as self.plt, showed its current
figure with display_plot, and added an optional test that the plot has labels and a title. The
pytest grader has no current figure: student code saves its plot in a variable or returns it from
a function, and the test gets a matplotlib Figure from sandbox.query or query_function.
- Replace
Feedback.check_plotwithassert_plot_equal, drawing the reference plot on a newFigurein the test. See Comparing with a Reference Plot. - To keep showing the student's plot, call
feedback.add_image(plot, label="Your plot")before the checks. See Showing Plots to Students. - If the legacy test read
self.pltand the student's task doesn't save the plot anywhere, the task has to change to save it, such asplot = plt.gca(). Report that back. - To keep the optional label test, check the labels with
matplotcheckor on theFigurein a test with its own points. - Set
sandbox_timeout = 15, because importing matplotlib can take most of the 1 second default. - Use the
:latestimage, which includes matplotlib.
The plots example question is a converted plot question.
The default friendly output level shows the student only the
assertion's message, so tests don't need @pytest.mark.output.
Follow Hiding Expected Values: don't use a bare assert
or numpy testing functions, and use assert_true for hidden test cases. The comparison helpers show
the expected value, which is fine when the legacy feedback showed it too. Keep the wording of the
legacy feedback where you can, such as "x is inaccurate".
- A test that finishes without failing gets full points, even if it never calls
feedback.set_score. - A test that fails gets the last score it set with
feedback.set_score, or 0. Callset_scoreafter each check that earns partial credit. feedback.set_score_final(s)sets the score even if the test then fails.- A legacy loop that adds up how many inputs were correct becomes a loop of
subtests.test()blocks. The test gets the fraction of blocks that passed. See Checking Many Inputs. - The legacy pattern of
if check: set_score(1) else: set_score(0)becomes a plain assertion.
queryandquery_functionraiseNameErrorif the student didn't define the name, which fails the test with a message that says so. You don't need to check this yourself.- Each call to
query_functionhas a 1 second timeout by default. Passquery_timeout=for slower functions. - After a call times out, later queries on the sandbox fail without running student code, so a slow function can't use up the grading job's time. You don't need to stop a loop yourself.
First, move what you can out of the setup code, as described in Where code belongs:
- Values that only tests use go in
test_student.py. - Random variables that student code reads go in
server.py, if they're plain JSON values. - Fixed variables that student code reads go in
starting_vars. - Functions that the setup code blocks, such as
la.inv = not_allowed, go inimport_blacklist.
If nothing is left, delete setup_code.py. Otherwise, update what's left. Setup code runs inside the sandbox, so some things that worked in the legacy grader don't:
-
Parameters.
dataisn't defined. Importparams, which isdata["params"], withfrom pytest_prairielearn_grader.server_data import from_server_json, params. Values thatserver.pystored withpl.to_jsonarrive still encoded: decode them withfrom_server_json(params["A"]). Theprairielearnpackage isn't available. -
Restrictions. Setup code runs with the same builtins and import rules as student code. For example, it can't call
openor importos. To read a data file, list it indata_filesand use a library function such asnp.loadtxt. -
Variables for the student. Student code only gets the setup variables named in
names_for_user. This list usually comes from<pl-external-grader-variables params-name="names_for_user">inquestion.html. If the question defines these names somewhere else, or not at all, setnames_for_user=[...]in theConfigObject. -
Fresh values per test. The
sandboxfixture runs setup code and student code again for each test, so random values in setup code change between tests. Read them withsandbox.query_setupin the same test that uses them. Don't compute expected values from random setup values at module level. -
Disallowed functions.
Feedback.not_allowedisn't available. Instead of overriding the function in setup code, list it inimport_blacklist, which blocks it however student code reaches it. The default blacklist is replaced, so keep its entries:autograder_config = ConfigObject( import_blacklist=["os", "sys", "subprocess", "pathlib", "shutil", "numpy.linalg.inv"], sandbox_timeout=15, )
See Blocking Functions. If the setup code computed a value with the blocked function, compute it in the test instead.
Put a ConfigObject named autograder_config at the top of test_student.py when the defaults
don't fit:
from pytest_prairielearn_grader import ConfigObject
autograder_config = ConfigObject(
student_code_pattern="Markov-Chains-1.ipynb",
notebook_cell_tag="#grade",
# Importing numpy in student code can take longer than the default timeout.
sandbox_timeout=15,
)When a ConfigObject is present, the module-level sandbox_timeout variable and the
@pytest.mark.sandbox_timeout marker are ignored.
| Situation | Setting |
|---|---|
| Student or setup code imports numpy, pandas or other large libraries | sandbox_timeout=15. The default of 1 second covers starting the student's code. |
The graded file isn't student_code*.py |
student_code_pattern="..." |
| A notebook where only some cells should run | notebook_cell_tag="#grade" |
Student code or leading_code.py imports a module from tests/ |
helper_modules=["module_name"]. See Helper Modules. |
Student code or leading_code.py reads a file from tests/ |
data_files=["file.txt"], and read it in tests with the tests_dir fixture. See Data Files. |
Student code imports os, sys, subprocess, pathlib or shutil |
import_blacklist=[...] without that module. These are blocked by default. |
Solutions use builtins that aren't defaults, such as getattr |
builtin_whitelist=[...]. See Builtin Function Control. |
| The legacy setup code blocked imports or builtins | import_whitelist, import_blacklist or builtin_whitelist |
A multi-file workspace project with .py files |
workspace_mode=True. See Workspace Grading. |
test_student.py itself can't import other files from tests/. Put helper code for the tests in
test_student.py, and read data files through tests_dir.
For questions where students submit a Jupyter notebook from a workspace:
- Set
student_code_patternto the notebook's name fromworkspaceOptions.gradedFiles. - Legacy notebook questions grade only the cells that start with
#grade. Keep that withnotebook_cell_tag="#grade". Without it, every code cell runs, including plots and scratch work. - Set
sandbox_timeout=15if the notebook imports numpy or similar libraries. - Notebook questions use the normal
sandboxfixture, notworkspace_mode. - The grader runs the graded cells as a script, so anything a graded cell needs must be defined in
a graded cell. Tag the cells with the notebook's imports
#grade. Useleading_code.pyonly when students already have copies of the notebook that you can't change. - If the notebook reads a data file, list it in
data_files. If the notebook imports a.pyfile that's in the workspace, copy that file intotests/and list it inhelper_modules.
These differences from the legacy grader are easy to miss:
__name__. Student code runs with__name__ == "__main__", so code underif __name__ == "__main__":runs during grading. The legacy grader skipped it. If the starter code has such a block, check that running it during grading is harmless. Aleading_code.pythat only sets__name__ = "__main__"is no longer needed.- No reference run.
ans.pyisn't run. Expected values come from the test code. - Setup code runs for each test, and so does the student's code. Output that student code prints
at the top level appears in each test's feedback, unless the test sets
include_stdout_feedback=Falseingrading_data. - Restricted builtins. Student code and setup code get only a default set of builtins. It
includes common ones such as
sum,anyandsuper, but notgetattr,hasattr,setattr,vars,dir,globals,open,evalorexec. Check the solution and the starter code, and add what they use tobuiltin_whitelist. A missing builtin shows up as aNameErrorwhen you test the conversion. - Default import blacklist.
os,sys,subprocess,pathlibandshutilare blocked unless you setimport_blacklist. - Syntax errors make the submission ungradable instead of scoring 0.
- Course libraries from
externalGradingOptions.serverFilesCourseare still on the import path. If you setimport_whitelist, list them in it. See Course Modules.
Test the converted question with the grader image before calling the conversion done, as described in Testing Locally. Run each case in a fresh copy of the layout, because the image moves files around:
mkdir -p grade/tests grade/student grade/data
cp -R path/to/question/tests/. grade/tests/
cp solution.py grade/student/student_code.py
echo '{"params": {}}' > grade/data/data.json
docker run --rm -v "$PWD/grade:/grade" prairielearn/grader-python-pytest:latest
cat grade/results/results.jsondata.json. Put inparamswhatever the question'sserver.pyand elements would set. In particular,<pl-external-grader-variables params-name="names_for_user">setsparams["names_for_user"]to a list of{"name": ..., "description": ..., "type": ...}objects. Without it, student code doesn't get the setup variables.- Cases to run. At least a correct solution, which should get full points, the unchanged starter code, which should get 0 without crashing the grader, and one wrong answer for each test. Compare the scores with what the legacy grader would give.
- Messages. Read
messagefor each test inresults.json, and check that hidden test cases don't reveal their expected values. - Notebooks. Use the solution notebook as the submission, with the file name from
gradedFiles.
Finally, test the question in PrairieLearn itself, with a local PrairieLearn server and the example or course repository. A person usually does this step. If you couldn't, say so when you report back.
When you're done, report:
- Which files you changed, added and deleted.
- Any check whose behavior changed, such as a different tolerance or message, and why.
- The cases you ran and their scores.
- Anything you couldn't convert or test.