From 6439ef8b8f2c3b3fd01f2a147ba68f4bb48d7e33 Mon Sep 17 00:00:00 2001 From: Marius Storhaug Date: Mon, 2 Nov 2026 19:30:24 +0100 Subject: [PATCH 1/2] Testing guidance prioritizes supported behavior and real APIs --- src/docs/Coding-Standards/Testing.md | 50 +++++++++++++------ src/docs/Coding-Standards/index.md | 2 +- .../Principles/Engineering-Practices.md | 4 +- 3 files changed, 38 insertions(+), 18 deletions(-) diff --git a/src/docs/Coding-Standards/Testing.md b/src/docs/Coding-Standards/Testing.md index a66942b4..fd153fee 100644 --- a/src/docs/Coding-Standards/Testing.md +++ b/src/docs/Coding-Standards/Testing.md @@ -1,11 +1,19 @@ --- title: Testing -description: The executable specification — test-first, locally runnable, deterministic. +description: Prove supported behavior through real entry points and APIs, with justified test doubles and honest coverage. --- # Testing -Tests are the executable specification. They define the intended behavior, prove it was built, and catch the day it breaks. They are written as part of the work — not bolted on afterward. +Tests are the executable specification. They define the intended behavior, prove it was built, and catch the day it breaks. This page owns the testing policy; language standards and agent skills build on it or link to it rather than define another one. + +## Test supported behavior + +Tests MUST assert observable functionality and public contracts through supported entry points: functions, commands, and APIs. Exercise the real implementation with controlled inputs and independently established expected results. A behavior-preserving refactor MUST NOT break tests merely because implementation text or internal structure changed. + +Do not add assertions whose only purpose is detecting source, documentation, or workflow edits: required phrases, headings, source-string searches, structural snapshots, or drift checks that freeze implementation shape. Linting, formatting, static analysis, link validation, and builds remain separate safeguards; this rule does not remove them. + +An independent expected result for a public functional contract is valid. So is testing a product whose supported behavior is change detection. Invoke a link checker or index generator on temporary fixture inputs and assert its output, exit code, or resulting files; do not search its source or policy prose merely to enforce an edit. ## Test-first @@ -17,31 +25,41 @@ See [Test-Driven Development](../Ways-of-Working/Principles/Engineering-Practice ## Testable locally -A developer or an agent must be able to run the full suite **on their own machine** — no cloud resources, no special access, no secrets that cannot be mocked. If you cannot test it locally, you cannot reason about it in your editor, and the cost of every change goes up. +A developer or an agent MUST have a fast, credential-free local inner loop. Exercise pure logic directly and use real implementations with isolated local dependencies where feasible. Document the local command and setup so someone can reason about behavior in their editor without cloud access. -This is a design constraint, not a nicety. When building anything new, ask early: *can someone run this locally?* If the answer is no, the design is wrong. +The local loop is not a claim that every external integration was tested. Identify [protected live-API checks](#protected-live-api-checks) separately, with their prerequisites and execution evidence. Do not require slow remote calls for logic that can be exercised directly. ## The test pyramid -Most tests should be fast and narrow; few should be slow and broad. +Choose the narrowest test that proves the behavior using the real implementation. Keep feedback fast without substituting a mock for a contract that needs an actual API. -- **Unit** — the foundation. Fast, isolated, no I/O. The bulk of the suite. -- **Integration** — components together, real boundaries. Fewer, slower, higher-value. -- **End-to-end** — the whole system. Fewest of all; reserved for critical paths. +- **Unit** — exercise logic directly through its supported interface, with controlled inputs. +- **Integration** — exercise real components together through commands and actual APIs in safe, isolated local, test, or sandbox environments. +- **End-to-end** — exercise critical workflows through the system's supported entry points, with the wider resource and setup cost made explicit. -Inverting the pyramid — leaning on slow, brittle end-to-end tests — produces a suite that is painful to run and quick to be ignored. +These levels describe scope, not a required ratio of mocked unit tests. Prefer real implementations and actual APIs by default; keep broad checks focused on behavior the narrower checks cannot establish. ## Don't mock what you don't own -Mock the boundary you control, not someone else's surface. Wrap a third-party API or SDK behind your own thin interface and fake *that*. Mocking the vendor's client directly only freezes your assumptions about how it behaves — the mock keeps passing after the real dependency changes, and the break reaches production unseen. +Mocks and fakes are justified exceptions, not the default. Record why a safe real case is not feasible, which boundary is simulated, and what the test does and does not prove. Controlled simulation can cover dangerous, unavailable, or rate-limited failure modes without performing unsafe calls. + +When a test double is necessary, prefer the owned boundary rather than inventing the vendor's behavior. A fake proves how the caller handles the simulated response; it does not prove that the external API, authentication, or deployed integration works. Where that external contract matters, require a separately identified real-API check rather than treating the fake as integration evidence. -Where the real contract matters, back the mocked unit tests with a small integration suite that exercises the live dependency on a schedule, so drift surfaces early. +## Protected live-API checks + +Real network checks are allowed. Use designated test or sandbox endpoints, least-privilege test identities, controlled inputs, and stable observable contracts. Identify the protected checks separately from the credential-free local loop. + +- **Prerequisites and authorization.** Document the environment, credential source, opt-in command, required approvals, and safe resource scope. Never embed secrets in code, fixtures, logs, or evidence. A test grants no permission: production writes and unsafe, destructive, or paid calls require explicit scoped authorization and are never defaults. +- **Isolation and cleanup.** Create or reserve resources isolated to the run. Clean them up on success and failure; report cleanup failures and remaining resources rather than hiding them. +- **Execution and evidence.** Record the command, tested environment, examined cases, result, and safe evidence in the Task or PR. Distinguish a functional failure from an unavailable prerequisite, authentication failure, rate limit, or network outage. + +Missing credentials, approvals, resources, or a live run mean **unexecuted coverage**, not a passing integration check. Report skipped or blocked checks and their reason explicitly. Local or simulated success cannot close that evidence gap. ## The oracle must be independent A test proves nothing when it computes the expected value the same way the code computes the actual one. If a function derives anchors from headings, and the test reimplements that same derivation to check it, both are free to be wrong together and the suite stays green forever. The expected value has to come from somewhere the implementation cannot influence: the library the real consumer runs, a fixture recorded from the real system, the published specification, or a second implementation written independently. -This is [don't mock what you don't own](#dont-mock-what-you-dont-own) one level up. There the mock freezes your assumption about a dependency; here the assertion freezes your assumption about correctness. +The same limit applies to [test doubles](#dont-mock-what-you-dont-own): neither a simulated dependency nor an assertion derived from the implementation independently establishes correctness. When the oracle is an external tool or service, confirm it is answering the question you think you asked. A wrong mode flag, an expired token, or a renamed field can turn a rich response into an empty one — and every assertion built on it into a vacuous truth. So assert on the oracle too: an empty response is a failure of the check, not a pass (see [nothing checked is not a pass](#nothing-checked-is-not-a-pass)). Where two independent oracles are available cheaply, agreeing them once is worth more than trusting either. @@ -53,6 +71,8 @@ So make the count part of the result: **a check fails when it checked nothing.** This costs three lines next to the exit, and it covers a whole class at once — the wrong root directory, a filter that matches more than intended, a glob that missed, an empty checkout, a query that returned no rows, an oracle that answered with an empty response. None of those need to be anticipated individually. Each one drives the count to zero, and zero is a failure. +Count meaningful examined units, not an adjacent proxy. An empty result can be a supported functional outcome, but test it with a controlled case and an independent expected result. Any scope-specific zero-count exception MUST document that real behavior; it cannot turn missing inputs or an unexecuted check into success. + ## Prove the test can fail A check that has never failed has not been checked. Before trusting a new one, make it red on purpose: break the behavior it guards, confirm it fails, read the message it produces, then restore. If breaking the behavior leaves the check green, the check is decoration. @@ -61,7 +81,7 @@ Counting is not a substitute for this. A healthy count only says the check exami ## Properties of a good test -- **Deterministic.** Same input, same result, every time. A test that passes intermittently is worse than no test — it trains people to ignore failures. No reliance on wall-clock time, network, ordering, or shared mutable state. +- **Deterministic.** Use controlled inputs and stable observable contracts. Control time, ordering, and mutable state. Real network or API checks have explicit prerequisites and report external failures; do not hide them behind a passing fallback. - **Isolated.** Each test sets up and tears down its own world. Tests do not depend on each other or on run order. - **Fast.** The inner loop is where engineering time is spent; slow tests get skipped. - **Readable.** A test is also documentation. Arrange–act–assert, one behavior per test, a name that states the expectation. @@ -79,4 +99,6 @@ A bug that reached production is a missing test. The fix is incomplete until a t ## Tests run in CI -Every pull request runs the suite before a human review begins. Validation that depends on a reviewer remembering to check is validation that eventually fails. Automate it once; it protects every PR after. See [Shift Left](../Ways-of-Working/Principles/Engineering-Practices.md#shift-left). +Every pull request runs the required local tests and validation gates before a human review begins. Run protected live-API checks in authorized environments on documented triggers, such as relevant changes or a risk-based schedule. A required protected check that could not execute remains an open gate, not a pass. + +Review tests against the supported behavior and the evidence they establish, including justified test doubles and unexecuted coverage. Validation that depends on a reviewer remembering to run it eventually fails; automate the appropriate checks. See [Shift Left](../Ways-of-Working/Principles/Engineering-Practices.md#shift-left). diff --git a/src/docs/Coding-Standards/index.md b/src/docs/Coding-Standards/index.md index 68e7a07a..cb35df45 100644 --- a/src/docs/Coding-Standards/index.md +++ b/src/docs/Coding-Standards/index.md @@ -39,7 +39,7 @@ The baseline pages apply to all code and come first; the per-language standards | [Error Handling](Error-Handling.md) | Fail fast, never swallow, and write messages that help the next person. | | [Documentation](Documentation.md) | Help that lives next to the code and explains the why. | | [Natural Language](Natural-Language.md) | Which language each artifact is written in, and the plain-language writing rules for docs, issues, pull requests, comments, prompts, and agent-facing text — with American English as the project dialect. | -| [Testing](Testing.md) | The executable specification — test-first, locally runnable, deterministic. | +| [Testing](Testing.md) | Prove supported behavior through real entry points and APIs, with justified test doubles and honest coverage. | | [Performance](Performance.md) | Scale with the input, measure before optimizing, clarity first. | | [Security](Security.md) | Least privilege, secret hygiene, and the OWASP baseline. | | [Dependencies](Dependencies.md) | How dependencies are pinned and kept current — the locking spectrum from identity pins to floating latest, and the balance between update velocity and supply-chain exposure. | diff --git a/src/docs/Ways-of-Working/Principles/Engineering-Practices.md b/src/docs/Ways-of-Working/Principles/Engineering-Practices.md index 6c4b0e9e..32969b70 100644 --- a/src/docs/Ways-of-Working/Principles/Engineering-Practices.md +++ b/src/docs/Ways-of-Working/Principles/Engineering-Practices.md @@ -63,9 +63,7 @@ Move quality gates as early as possible in the development cycle. The later a pr ### Testable locally -A developer (or agent) must be able to run the full test suite locally, without requiring cloud resources, special access, or environment secrets that can't be mocked. If you can't test it on your machine, you can't reason about it in your editor. - -This is a design constraint, not an optional addition. A solution that is untestable locally has a hidden cost that compounds with every change. When building new workflows, modules, or automation, ask early: *can someone run this locally?* +Keep a fast local inner loop so a developer or agent can reason about behavior in the editor. The [Testing standard](../../Coding-Standards/Testing.md#testable-locally) owns the local-loop requirements, real-API checks, test-double exceptions, and execution evidence. ### Pre-commit hooks From 607cca0010fa08b9fee43c6f5b07c17c5a051253 Mon Sep 17 00:00:00 2001 From: Marius Storhaug Date: Mon, 2 Nov 2026 19:31:51 +0100 Subject: [PATCH 2/2] Add a thin testing skill to the shared MSX plugin --- .github/plugin/README.md | 2 +- .github/plugin/marketplace.json | 2 +- .github/plugin/msx/plugin.json | 2 +- .github/plugin/msx/skills/msx-coding-testing/SKILL.md | 8 ++++++++ 4 files changed, 11 insertions(+), 3 deletions(-) create mode 100644 .github/plugin/msx/skills/msx-coding-testing/SKILL.md diff --git a/.github/plugin/README.md b/.github/plugin/README.md index bb7499d5..8782c33a 100644 --- a/.github/plugin/README.md +++ b/.github/plugin/README.md @@ -26,7 +26,7 @@ corresponding directory. Every plugin directory must contain `plugin.json`; its ## Skill granularity Each skill is a single pointer to one canonical document. The shared `msx` -plugin uses `msx-coding-*` skills for each coding language or tool, +plugin uses `msx-coding-*` skills for testing and each coding language or tool, `msx-documentation-*` skills for each documentation artifact type, and `msx-ways-of-working-*` skills for each child listed by the Ways of Working index. Skill bodies do not copy the documentation; they identify the one page diff --git a/.github/plugin/marketplace.json b/.github/plugin/marketplace.json index 2ff23ce8..1db8bfbd 100644 --- a/.github/plugin/marketplace.json +++ b/.github/plugin/marketplace.json @@ -11,7 +11,7 @@ { "name": "msx", "description": "Routes coding, documentation, and work lifecycle tasks to current MSX standards.", - "version": "0.1.1", + "version": "0.1.2", "source": ".github/plugin/msx" } ] diff --git a/.github/plugin/msx/plugin.json b/.github/plugin/msx/plugin.json index fb1d7e0f..f72f71ca 100644 --- a/.github/plugin/msx/plugin.json +++ b/.github/plugin/msx/plugin.json @@ -1,7 +1,7 @@ { "$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json", "name": "msx", - "version": "0.1.1", + "version": "0.1.2", "description": "Routes coding, documentation, and work lifecycle tasks to current MSX standards.", "author": { "name": "MSXOrg", diff --git a/.github/plugin/msx/skills/msx-coding-testing/SKILL.md b/.github/plugin/msx/skills/msx-coding-testing/SKILL.md new file mode 100644 index 00000000..48ffc1ec --- /dev/null +++ b/.github/plugin/msx/skills/msx-coding-testing/SKILL.md @@ -0,0 +1,8 @@ +--- +name: msx-coding-testing +description: Apply the MSX testing standard when designing, writing, or reviewing tests. +--- + +# Apply the MSX testing standard + +Read and follow the [testing standard](https://github.com/MSXOrg/docs/blob/main/src/docs/Coding-Standards/Testing.md).