Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 18 additions & 14 deletions quick_start.md
Original file line number Diff line number Diff line change
Expand Up @@ -627,14 +627,17 @@ An `Axes` value brings its whole figure, so subplots are never lost: use `figure
What's sent to the test:

- **Lines**, with their data, label, color, width, style, marker and draw style. This includes `plot`, `semilogy`, `loglog` and `step`.
- **Horizontal and vertical lines** from `axhline` and `axvline`, with their position, extent, label, color, width and style.
- **Text** from `text`, in data or axes coordinates, with its position, alignment and rotation.
- **Annotations** from `annotate`, with the annotated point in data or axes coordinates, the text position in data or axes coordinates or offset points, and whether it has an arrow.
- **Scatter points**, with their positions, sizes and colors.
- **Bars and histograms**, with their positions, sizes, colors and orientation.
- **Axes details:** scales, labels, titles, limits, position and legend labels. Twin axes come back as separate axes in `figure.axes`.
- **Figure details:** size and overall title.

Other plot content makes the query fail with a message saying what isn't supported, rather than losing data. This includes `errorbar`, `stem`, `fill_between`, `pie`, `imshow`, `contour`, `vlines` and `hlines`, `axhline` and `axvline`, `axvspan`, text and annotations, colormapped scatter plots, polar axes, inset axes and subfigures.
Other plot content makes the query fail with a message saying what isn't supported, rather than losing data. This includes `errorbar`, `stem`, `fill_between`, `pie`, `imshow`, `contour`, `vlines` and `hlines`, `axline`, `axvspan`, text in figure coordinates, colormapped scatter plots, polar axes, inset axes and subfigures.

Some details aren't sent, so don't grade them: grid lines, tick positions and labels, line and marker transparency set separately from color, scatter marker shapes, the base of a log scale, and `symlog` settings. Values are sent as decimal numbers, and dates come back as matplotlib date numbers (use `matplotlib.dates.num2date` to convert them). Categories on a line come back as strings.
Some details aren't sent, so don't grade them: grid lines, tick positions and labels, line and marker transparency set separately from color, scatter marker shapes, the base of a log scale, `symlog` settings, text fonts and colors, and arrow styles, since every annotation arrow comes back as a plain arrow. Values are sent as decimal numbers, and dates come back as matplotlib date numbers (use `matplotlib.dates.num2date` to convert them). Categories on a line come back as strings.

Importing matplotlib can take most of the default 1-second sandbox timeout, so set a longer one in plot questions, such as `sandbox_timeout = 15` at the top of the test file.

Expand Down Expand Up @@ -672,24 +675,25 @@ By default, it checks:

- **Subplots:** the number of axes.
- **Scales:** the x and y axis scales, such as linear or log.
- **Data:** the lines, scatter plots and bar plots, which must have the same number of each kind. Data is compared with `numpy.allclose`, using `rtol` and `atol`.
- **Data:** the lines, `axhline` and `axvline` lines, scatter plots and bar plots, which must have the same number of each kind. Data is compared with `numpy.allclose`, using `rtol` and `atol`.

The series can be in any order, and a line also matches when its points are in reverse order, since
it looks the same. The points in a scatter plot and the bars in a bar plot can be in any order.

Keyword arguments change what's checked:

| Argument | Default | Checks |
| -------------- | ------- | ----------------------------------------------------------------------------------------------------------- |
| `check_scales` | `True` | The x and y axis scales. |
| `check_data` | `True` | The data in lines, scatter plots and bar plots. |
| `ordered` | `False` | Requires lines, scatter plots, bar plots and legend labels to be in the reference's order. |
| `allow_extra` | `False` | Allows more lines, scatter plots or bar plots than the reference, such as a student's extra reference line. |
| `check_labels` | `False` | The x and y axis labels, exactly. |
| `check_titles` | `False` | The title of each subplot and the figure, exactly. |
| `check_legend` | `False` | The legend's labels and title, or that neither plot has a legend. |
| `check_limits` | `False` | The x and y axis limits, within `rtol` and `atol`. |
| `check_styles` | `False` | Line colors, styles and markers, scatter point colors, and bar colors. |
| Argument | Default | Checks |
| -------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `check_scales` | `True` | The x and y axis scales. |
| `check_data` | `True` | The data in lines, `axhline` and `axvline` lines, scatter plots and bar plots. |
| `ordered` | `False` | Requires lines, scatter plots, bar plots and legend labels to be in the reference's order. |
| `allow_extra` | `False` | Allows more lines, scatter plots, bar plots, text or annotations than the reference, such as a student's extra reference line. |
| `check_labels` | `False` | The x and y axis labels, exactly. |
| `check_titles` | `False` | The title of each subplot and the figure, exactly. |
| `check_legend` | `False` | The legend's labels and title, or that neither plot has a legend. |
| `check_limits` | `False` | The x and y axis limits, within `rtol` and `atol`. |
| `check_styles` | `False` | Line colors, styles and markers, scatter point colors, and bar colors. |
| `check_texts` | `False` | The strings of `text` and `annotate` labels, and the point each annotation points at. The position of plain text isn't compared. |

A failure describes the first difference found, such as `Expected a log scale on the y-axis of your
plot, but found a linear scale.` or `No line in your plot matches the line labeled 'exp'.` Lines
Expand Down
14 changes: 10 additions & 4 deletions src/pytest_prairielearn_grader/assertions.py
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,7 @@ def assert_plot_equal(
check_legend: bool = False,
check_limits: bool = False,
check_styles: bool = False,
check_texts: bool = False,
msg: str | None = None,
) -> None:
"""Assert that a student's matplotlib plot matches a reference plot.
Expand All @@ -213,8 +214,8 @@ def assert_plot_equal(
test itself. Subplots are compared in order.

By default this checks the number of subplots, the axis scales, and the
data in each line, scatter plot and bar plot, which must have the same
number of each kind. Series can be in any order, and a line also matches
data in each line, ``axhline`` and ``axvline`` line, scatter plot and bar
plot, which must have the same number of each kind. Series can be in any order, and a line also matches
when its points are in reverse order. Scatter points and bars can be in
any order. Numbers are compared with ``numpy.allclose``.

Expand All @@ -237,8 +238,8 @@ def assert_plot_equal(
Require lines, scatter plots, bar plots and legend labels to be in
the same order as in the reference.
allow_extra:
Allow the student's plot to have more lines, scatter plots or bar
plots than the reference.
Allow the student's plot to have more lines, scatter plots, bar
plots, text or annotations than the reference.
check_labels:
Compare the x and y axis labels exactly.
check_titles:
Expand All @@ -251,6 +252,10 @@ def assert_plot_equal(
check_styles:
Compare line colors, line styles and markers, scatter point colors,
and bar colors and orientation.
check_texts:
Compare the strings of ``text`` and ``annotate`` labels, and the point
each annotation points at. The position of plain text isn't compared.
Extra text is allowed with *allow_extra*.
msg:
Optional message to show instead of the description of the
difference.
Expand Down Expand Up @@ -280,6 +285,7 @@ def assert_plot_equal(
check_legend=check_legend,
check_limits=check_limits,
check_styles=check_styles,
check_texts=check_texts,
)
if mismatch is not None:
raise AssertionError(msg or mismatch)
Expand Down
11 changes: 8 additions & 3 deletions src/pytest_prairielearn_grader/fixture.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,9 @@

SCRIPT_PATH = str(files("pytest_prairielearn_grader").joinpath("_student_code_runner.py"))
DEFAULT_TIMEOUT = 1.0
# Extra time to wait for the start response, so that the sandbox's own
# initialization timeout is reported instead of racing against the socket timeout.
START_RESPONSE_GRACE = 0.5

# Modes for the data files folder (see _make_data_files_dir). Everyone can read the files and list the
# folders, and nobody can change them; only the owner (root, in the Docker image) can make them writable again.
Expand Down Expand Up @@ -305,8 +308,10 @@ def _connect_to_server(self, timeout: float) -> None:
self.student_socket.settimeout(timeout)
self.student_socket.connect((host, int(port)))

def _read_start_response(self) -> ProcessStartResponse:
def _read_start_response(self, initialization_timeout: float) -> ProcessStartResponse:
"""Read and return the ProcessStartResponse sent after the start message."""
assert self.student_socket is not None
self.student_socket.settimeout(initialization_timeout + START_RESPONSE_GRACE)
try:
data = self._read_from_socket().decode()
res: ProcessStartResponse = json.loads(data)
Expand Down Expand Up @@ -488,7 +493,7 @@ def start_student_code_server(self, *, initialization_timeout: float = DEFAULT_T
helper_modules=self.helper_modules,
)
self._send_json_object(json_message)
return self._read_start_response()
return self._read_start_response(initialization_timeout)

def query_setup_raw(self, var_to_query: str) -> SetupQueryResponse:
self._assert_process_running()
Expand Down Expand Up @@ -581,7 +586,7 @@ def start_workspace_server(self, *, initialization_timeout: float = DEFAULT_TIME
helper_modules=self.helper_modules,
)
self._send_json_object(json_message)
return self._read_start_response()
return self._read_start_response(initialization_timeout)

def __repr__(self) -> str:
return f"WorkspaceFixture(workspace_dir={self.workspace_dir})"
Loading
Loading