Skip to content

Add test catalogue generator (scripts/test_catalogue.py + docs/TEST_CASES.md) #169

Open

Description

@amar-python

Context

Automated test catalogue, follow-up from closed PR #41.

PR #41 proposed scripts/test_catalogue.py — a script that walks the test tree and generates docs/TEST_CASES.md, listing every test case with ID, file and objective, grouped by category. CI then runs scripts/test_catalogue.py --check to keep the doc current (fails CI if the generated doc is out of date relative to the actual tests).

The current state: docs/TEST_CASES.md doesn't exist. Tests are discoverable only by pytest --collect-only, which doesn't give you a per-test objective or a stable ID.

Scope

  • scripts/test_catalogue.py reading pytest.ini markers and test files, generating docs/TEST_CASES.md
  • The script extracts test IDs from docstrings or from a @pytest.mark.id("...") decorator (decide which during implementation)
  • CI check mode (--check) that fails if the committed docs/TEST_CASES.md differs from a fresh generation
  • Add the check to the free-tier Quality Gate job

Out of scope

  • Backfilling IDs or docstrings on existing tests (that's a separate, larger housekeeping PR)
  • Any cross-referencing with VCRM or BR requirements (#A, #B, #C aside, that's its own problem)

Done when

  • python scripts/test_catalogue.py --check runs in CI and passes
  • docs/TEST_CASES.md exists on main with current test inventory
  • A new test added without updating the doc fails CI with a clear "run scripts/test_catalogue.py and commit the result" message

Related

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions