python-tools.md

Python practice helpers — agent reference

These optional helpers implement a particular Python practice workflow. Adapt or replace them when the user's interview calls for another language, editor, test framework, ML environment, or discussion format. They do not define the repo's scope. Python 3.10+ is required; the CLI runner has no external dependencies.

VS Code practice handoff

For an assigned exercise, the agent prepares and opens a separate practice window:

python3 practice_vscode.py handoff exercise_name --level 1

The specification opens on the left and solution.py on the right. Click Run Public Tests in the status bar or the Python editor's ▶ button; Cmd+Enter (Ctrl+Enter elsewhere) does the same. Changes are saved before the run. Results, tracebacks and timings appear in Public Tests, without requiring a terminal. The button detects the level automatically on each run, then runs public tests cumulatively through that level; it excludes candidate and hidden tests. Hidden submissions remain available to the agent through the existing CLI when requested.

Setup is automatic: the launcher locates the installed VS Code, selects the current Python interpreter, and installs only Pyright (ordinary Python completion) and Ruff (syntax/basic lint) plus the small local test-button extension. Setup needs internet on first installation; tests and editing then work locally. Dependencies and settings live under ignored .practice-vscode/, separately from normal VS Code user data and extensions. On macOS, each handoff also creates Open Practice.command in the exercise folder for double-click reopening, with the same timeout. No new handoff is needed as you implement later levels. For a fixed public-test level, disable interviewPractice.autoLevel and set interviewPractice.level in workspace settings.

AI/Copilot and inline suggestions are disabled. The practice window hides sidebars, Git and debugging controls, disables unrelated built-in extensions and common shortcuts for terminal/debug/refactor/formatting, and turns off snippets, auto-import completion, formatting and quick fixes from the practice tools. Basic editing, syntax highlighting, indentation, completion, parameter hints and lint remain. This is a minimal local practice environment: built-in menus/commands can still expose other editor features; hidden files are excluded from browsing/search, but remain readable on disk.

When preparing a task, the agent owns the setup and verification. The candidate does not need to select an interpreter, install extensions or change settings. Opening the raw exercise folder in normal VS Code bypasses this environment; use the prepared handoff. No exercise is created by the launcher.

Automatic timing and history

The practice window records solution.py and Python files directly under tests/candidate/, including unsaved edits, undo/redo, saves and window focus. The status-bar clock starts on the first recorded content edit. Reading, inactivity, test waits, time away and reopening all count toward elapsed time. Opening/saving an unchanged starter does not start it. Earlier edits made before recording was enabled are not recoverable.

Click the clock or run Practice: Finish or Resume Attempt to save and stop the timer. Click again, run Practice: Resume Attempt, or make a content edit to resume the same attempt from its original first edit. The intervening time counts; an accidental finish is archived for review. Opening/saving alone does not resume it. Public passes and intermediate submissions do not stop it automatically: all four levels belong to one attempt. Practice: Start New Attempt explicitly starts a new history with the clock waiting for its first edit; it does not erase your code or old history. Resuming preserves all previously recorded edits.

History stays local under ignored .practice-history/<exercise>/<attempt-id>/. Each editor session has an ordered journal with exact text snapshots, timestamps, hashes and edit details. Public runs and CLI submissions record frozen code versions, levels, results and aggregate group counts, including public level-detection probes. While recording or detecting a level, tests run in a temporary copy of the exercise so concurrent edits cannot change the tested version. That copy's files form the worker's working directory; edits remain in the original exercise. Test-only filesystem changes in the copy are disposable. Hidden test names, inputs and tracebacks are not placed in the history.

For the reviewing agent:

python3 practice_history.py summary artifact_catalog
python3 practice_history.py export artifact_catalog
python3 practice_history.py replay artifact_catalog --segment UUID --seq 12
python3 practice_history.py finish artifact_catalog
python3 practice_history.py resume artifact_catalog

export writes review.md and review.json inside the history attempt, separate from the exercise's solution-feedback review.md. It provides a timeline and saved checkpoint diffs; raw journals allow reconstruction of unsaved intermediate states. replay prints a recorded version without modifying the solution. Use --attempt UUID to review older attempts. A 120-second idle threshold is used only for a labelled inter-edit activity estimate; --idle-seconds changes that analysis, never the elapsed timer. An edit gap does not prove inactivity or reveal the candidate's thinking. Clock changes, interrupted sessions and unobserved intervals are reported as limitations.

The recorder is local observation, not proof of assistance or paste provenance. Writes are ordered and flushed at checkpoints; a crash may still lose queued records. The exporter reports a torn final record without changing the original journal and rejects corrupt internal records. No AI runs during practice.

python3 practice_vscode.py prepare                    # Prepare tools without a task
python3 practice_vscode.py handoff exercise_name --level 2 --no-open
python3 tests/check_vscode.py                         # Optional real-editor check

The integration check briefly opens a disposable VS Code test window, checks layout, saved edits, public-only execution, non-AI completion, lint and disabled formatting, then closes it; it creates no practice questions. Implementation uses VS Code's isolated CLI directories, AI disable setting, and command contributions.

Commands

Run these from this workspace (or use the absolute path to run_tests.py elsewhere):

python3 run_tests.py list
python3 run_tests.py new exercise_name
python3 run_tests.py run exercise_name
python3 run_tests.py submit exercise_name
python3 run_tests.py run exercise_name --level 2
python3 run_tests.py run exercise_name --level 2 --public-only
python3 run_tests.py run exercise_name --only-level --level 3
python3 run_tests.py run exercise_name -k boundary --timeout 5
python3 run_tests.py submit exercise_name --level 2
python3 run_tests.py submit exercise_name --level 2 --reveal-hidden

The runner lives at the repository root, not inside an exercise. From any directory:

python3 /absolute/path/to/this/repo/run_tests.py run artifact_catalog

Without --level, public tests probe one level at a time, independently of debug filters. The first level raising NotImplementedError marks unfinished work: evaluate all earlier levels (at least Level 1, so a blank starter fails). For example, an unfinished Level 3 selects cumulative Levels 1–2. The output states the selection. An assertion failure, other exception or import error keeps that level in evaluation; it cannot be used to skip a regression. Probes stop at the first unfinished/failing level; they never import candidate or hidden suites. If there is no Level 1 public coverage, detection falls back to Level 4 without inferring implementation progress. Missing public coverage at a later level ends detection at the preceding covered level.

Use --level N to bypass detection and evaluate partial work at Level N. Automatic selection is a public-test heuristic, not proof that every requirement is implemented. Detection and evaluation use one frozen candidate version; each probe has a disposable copy so its filesystem writes cannot alter subsequent probes or the evaluation. Public tests run again for the cumulative result; the shared timeout includes both.

new creates only a blank specification, solution.py, and three empty test groups. It refuses to overwrite an existing directory. The exercise folder is the working directory during a run, and tests can import the implementation with from solution import ....

Exercises use standard unittest.TestCase tests in these files:

exercises/<name>/
  README.md                 # Specification, extended across levels
  solution.py               # Candidate implementation shared by all levels
  tests/
    __init__.py
    public/
      __init__.py
      test_level_1.py       # Visible examples and basic behavior
      test_level_2.py
      test_level_3.py
      test_level_4.py
    candidate/
      __init__.py
      test_level_1.py       # Your own debugging/edge-case tests (optional)
    hidden/
      __init__.py
      test_level_1.py       # Broader specification coverage, withheld in a mock
      test_level_2.py
      test_level_3.py
      test_level_4.py

Additional files such as test_level_2_boundaries.py work too. Keep test files directly in their group; other filenames and nested directories are not selected. Legacy test files directly in tests/ are treated as public. Every package needs an __init__.py; the scaffold creates them. Tests import the shared solution.py and should create fresh state per test. Each group runs in a fresh Python process.

• run executes public and candidate tests, showing names, tracebacks and output. It never imports hidden tests. -k, --only-level, --failfast and --buffer are available for debugging; a visible pass does not imply a submission pass.
• run --public-only executes public tests only, for the editor button or CLI.
• submit executes all three groups, even if a visible group fails. Public and candidate feedback remains detailed. Hidden feedback shows only overall status and aggregate test/failure/error/skip counts; test names, inputs, assertion messages, tracebacks and output are withheld, including import output.
• submit --reveal-hidden reruns the current implementation with full hidden feedback for post-mock review. Leave the implementation unchanged if you want to review the same attempt. When history is enabled, frozen submission versions remain in the local attempt history; revealing hidden feedback runs the current code.

Both commands run levels 1 through the automatically selected or explicit --level to check backward compatibility. submit does not accept filters, single-level shortcuts, or failfast. A submission needs nonempty public and hidden suites for the selected levels to record a pass; candidate tests are optional. A pass covers only the tests present, so exercise authors must supply coverage for each level. Hidden tests should cover the stated contract's boundaries, sequences and regressions, never unstated requirements.

Hidden files stay local and readable: the candidate is trusted not to inspect them or request --reveal-hidden during a mock. Level selection controls execution only; it does not hide later requirements or test files. This is a local runner, not a sandbox.

--timeout is the total budget across all selected groups, including imports, and stops a worker if it hangs. The default is 10 seconds. Public and candidate test execution times are printed; hidden timing details appear with --reveal-hidden.

Exit codes: 0 passed (or a successful list/new command), 1 test failure/error, 2 invalid command/setup, 3 no matching visible tests or incomplete submission, 124 timeout, 130 interrupted. An empty suite never counts as a pass. No CodeSignal score is estimated.

Development verification

Run from the repository root:

python3 -m unittest discover -s tests -v
node --test tests/test_recorder.cjs

These use disposable synthetic fixtures, not the unfinished bundled exercises. Node is needed for recorder development checks; the editor uses its bundled runtime. For a real editor check, first run python3 practice_vscode.py prepare, then python3 tests/check_vscode.py. It briefly opens and closes a disposable window, checking layout, completion, lint, public-only execution and recording. macOS was the original tested platform; Linux/Windows paths exist but have not been verified. VS Code 1.100+ is required by the extension manifest.

Adapting the existing environment

vscode-practice/settings.json and keybindings.json define the restrictive unassisted profile; practice_vscode.py selects its extensions. Inspect those files against the actual allowed tools. For an AI-permitted or debugger-permitted interview, prepare a separate appropriate environment. The bundled profile intentionally disables AI, debugging, formatting and many shortcuts. Do not apply it unchanged to every stage. No editor profile enforces assessment security or certifies an unassisted attempt.