Repository navigation
[functional-test-alignment] Prove framework behavior without freezing implementation text #550
Description
Context and request
Align Process-PSModule tests with the MSX Testing standard. Maintainers must be able to change implementation text without breaking tests when supported behavior stays the same. Real regressions in functionality, failure handling, or shipped validation must still fail.
The audit starts from default-branch source 64ffb1d87f86bd5d487891fdb140b78385a00531. The scoped implementation is in draft #551 at 7a00cbb9cd72baddf14d990019cf5da3c9c3600e. No open PR branch is a source.
Acceptance criteria
- Every action helper, recovery/WhatIf suite, shipped Module/SourceCode validator, fixture module test, setup assertion, and workflow-embedded assertion is classified below. The record identifies the disposition, real entry point, observable expectation, and policy reason.
- No remaining assertion fails only because source, documentation wording, a private parameter inventory, or workflow shape changed. Standard lint/static analysis, supported public contracts, and the shipped consumer validation product remain intact.
- Replacement checks call existing real functions or scripts with controlled fixtures and independent expected results. Deliberate behavior faults fail them; behavior-preserving wording changes do not.
- The existing combined action tests and applicable fixture/validation commands pass, or a concrete missing prerequisite or pre-existing failure is recorded without calling it a pass. No coverage target, required check, lint rule, or security control is weakened.
- A separate draft PR targets
main, closes only this Task, and contains exact-head validation and review evidence. No merge, auto-merge, or publication is authorized.
Technical decisions
Use existing Pester 6.1 tests and current-main action entry points. Keep settings routing, version calculation, SemVer outputs, publication recovery, and WhatIf coverage. Replace the log-phrase check with observed release-notes content and a real isolated Git commit. Verify that generated documentation preserves its authored input, not a stored set of phrases. This is a content-transfer contract against input bytes, not a snapshot of repository prose or generator implementation.
Remove the helper parameter-inventory guard; all six version-output/invalid-input checks remain. Keep the Pester configuration instance that is actually passed to Invoke-Pester, so shuffle, its seed, and start markers are exercised rather than echoed and discarded. Keep file execution explicitly serial because these suites share process-wide environment variables. This preserves the baseline execution mode.
This Task does not change resolver, publisher, recovery, settings, or safety-owner contracts. It does not import or modify #548, #549, #531, #524, #525, #535, #537, or #541. It is independent of PSModule/.github#89. Preserve existing publication/evidence holds and original-tested-artifact decisions. Necessary runtime repairs are separate work.
Complete test inventory and classification
Counts describe this audited source; no repository inventory/count assertion is added. Action test paths are under .github/actions/<action>/tests. The shipped suites are under Test-PSModule/src/tests.
| Surface | Disposition | Real entry point | Observable expectation | Policy reason |
|---|---|---|---|---|
| Get-PSModuleSettings event cases (7) | Retained | Resolve-WorkflowEventRouting | Push/dispatch/PR inputs select release, test, and cleanup decisions, including closed-PR rejection | Controlled function results, not workflow text |
| Get-PSModuleSettings push association (2) | Retained | Select-PullRequestForPush | Only a merged PR with the matching commit is selected | Supported input/output filtering |
| Get-PSModuleSettings tree/compare cases (4) | Retained | Get-FilesFromGitTree; Get-FilesFromGitHubComparison | Return controlled filenames; reject truncated trees and the 300-file API limit | Completeness/failure handling; filenames and limit come from controlled input/API contract, not repository inventory |
| Resolve-PSModuleVersion release JSON (7) | Retained | ConvertFrom-GitHubReleaseJson; LogGroup; Get-LatestGitHubVersion | Empty/null input, flat arrays, single items, and composition preserve values | Counts are controlled response cardinality; actual JSON parsing and composed functions run |
| Resolve-PSModuleVersion Gallery-only resolution (2) | Retained | Get-ResolvedModuleVersion | Resume the supported current-main version case and preserve the ordinary baseline otherwise | Existing behavior, not a publication receipt or safety claim; held proposals are not imported |
| Resolve-PSModuleVersion GitHub version (8) | Retained | Get-LatestGitHubVersion | Controlled absent/latest/prerelease metadata resolves the expected version | Independent version results, not source strings |
| Resolve-PSModuleVersion published version (6) | Retained | Get-LatestPublishedVersion | Handle missing sources and choose the higher controlled version | Real version comparison |
| Resolve-PSModuleVersion prerelease number (4) | Retained | Get-NextPrereleaseNumber | Controlled GitHub releases produce 001/006 | Real function; existing anonymous Gallery reads are not independent proof of Gallery availability or semantics |
| Resolve-PSModuleVersion next version (8) and composition (3) | Retained | Get-NextModuleVersion and composed existing helpers | First patch/minor/major, null baselines, prerelease suffix, and existing patch increment | Independent SemVer expectations |
| Resolve-PSModuleVersion normalized context (2) and release decisions (5) | Retained | Get-GitHubPullRequest; Resolve-ReleaseDecision | Use normalized push context, direct-release defaults, and explicit label conflicts | Real input/output and failure contracts |
| Publish-PSModule version output (6) | Retained | Get-ModuleVersionString | Stable/prerelease SemVer, whitespace handling, and invalid empty version | Output text is the product contract |
| Publish-PSModule VersionPrefix parameter absence (1) | Removed | Get-Command parameter inventory | No independent functional expectation beyond the retained SemVer cases | Private helper signature freeze; output behavior remains covered |
| Publish-PSModule recovery (3) | Retained | Real publish.ps1 | Skip an existing package, publish on simulated absence, fail on another simulated lookup error | Unsafe vendor writes are intercepted; proves caller branching, not Gallery API behavior or safe publication ownership |
| Release-PSModule existing cases (3) | Retained; fail-closed WhatIf boundary and per-test cleanup added | Real release.ps1 | Dry-run action outputs, resumed upload/comment inputs, and stable-push target | Observable script outputs/command inputs; no source searches |
| Release-PSModule manual-dispatch log phrase (1) | Rewritten | Real release.ps1 with a real isolated Git repository | Selected commit content reaches the notes file supplied at the intercepted GitHub write boundary; target is that commit | Log prose is not the oracle; actual Git and file handling run |
| update-index catalog (1) | Retained | Update-ModuleList; Update-MDSection; New-ModuleCatalogPage | Write the repository page and catalog links for a controlled repository | Generated files/links are the supported output; owned remote metadata helpers are simulated |
| Shipped SourceCode suite (10 definitions, data-driven links) | Retained | Actual Pester suite/container through Test-PSModule | Diagnostics for NumberOfProcessors, Verbose, OutNull, LowercaseKeywords, FunctionCount, FunctionName, CmdletBinding, ParamBlock, PublicHelpLink, FunctionTest | This is the shipped static-analysis/validation product, not an edit-detection test of its own implementation |
| Shipped Module suite (7 definitions, data-driven types) | Retained | Actual Pester suite/container; Import-Module; Test-ModuleManifest; Remove-Module | Valid/importable installed module, matching version folder, public type accelerators and cleanup | Runtime quality product; source metadata is used for discovery, not a new structural snapshot |
| Two shipped containers and two configurations | Retained | Test-PSModule and the existing Invoke-Pester workflow route | Supply real module/source paths, reports, and coverage options | Execution plumbing, not assertion inventories |
| srcTestRepo and srcWithManifestTestRepo module suites (5 each) | Retained | Four real exported greeting functions and the exported SolarSystems variable | Return Hello, World! and expose Solar System | Controlled fixture behavior and exported data, not frozen implementation |
| Both fixture Environment suites (2 each) | Retained | Actual process environment after TestData exposure | Known non-secret fixture values arrive intact through secret/variable maps | Exact controlled data and lengths are transport expectations, not source snapshots |
| Both TestData.Assertions scripts; BeforeAll/AfterAll phase scripts | Retained | Real setup/teardown phase invocation | Both known inputs are present and equal in each phase | Supported environment plumbing; logs are not asserted |
| Workflow-Test-Default and Workflow-Test-WithManifest root index checks | Rewritten | Existing Build-Docs/Build-PSModuleDocumentation; then exact workflow assertion step | Generated root file preserves a nonempty authored input; different headings/prose are accepted, lost content is rejected | Independent input/output file contract replaces six frozen phrase checks |
| Test-Actions configuration echo loop | Removed and replaced by actual execution | Invoke-Pester with the configured instance | Configured shuffle, seed, and start markers reach the real serial run; no-test-folder failure and Run.Throw remain | Tautological property echoes do not prove execution |
| Test-Module, Test-SourceCode, Test-ModuleLocal, setup/teardown, report/coverage workflows | Retained | Existing reusable workflows, Test-PSModule, Invoke-ScriptAnalyzer, Get-PesterTestResults, Get-PesterCodeCoverage | Run configured suites; keep failure, missing/unexecuted-result, and coverage gates | Required runtime quality gates are not source-shape assertions |
| Build-Docs/Lint-Repository/Lint-SourceCode/Linter and PSScriptAnalyzer.Tests.psd1 files | Retained | Existing linter/build/schema tools | Validate source, Markdown, settings schemas, and built artifacts | Standard static analysis/build safeguards are explicitly separate from prohibited snapshots |
| Guidance benchmarks, sample class invariants, and site JavaScript | Not test entry points | Inspected direct scripts and discovery routes | No additional assertion runner was found | Examples/benchmarks/runtime guards are not mislabeled as test coverage |
Evidence and limits
The first release selection passed 4/4. Deliberately wrong notes failed the replacement assertion; a diagnostic-only edit passed 4/4. All experimental runtime edits were restored. The complete original action selection passed 73/73; after removal of only the signature-inventory test, the actual updated workflow step passes with 72 action cases, shuffled order, and explicit serial execution.
The real documentation generator and the exact two workflow assertion scripts passed 12 controlled checks: original and changed authored wording accepted for both layouts, lost content rejected, and empty input rejected. The actual shipped SourceCode suite accepted a valid controlled fixture (10/10) and rejected the injected OutNull defect (9 passed, 1 intended failure).
GitHub release calls and Gallery writes in recovery tests are simulations that prevent unsafe external effects. They do not prove live service behavior, credentials, publication provenance, safety ownership, or a protected integration. No real release, Gallery upload, deletion, or label/permission change was performed. Existing fixture skip settings remain unchanged and must be reported as unexecuted coverage, not new passes.
Exact-head CI and Copilot review are complete: 52 checks succeeded, 7 existing conditional jobs were skipped, and Copilot approved the final immutable head with no active review issues. This is automated review, not a claimed human approval. PSScriptAnalyzer reports no findings on touched tests. Actionlint 1.7.12 passes; it was downloaded to isolated scratch storage only after the installed command was absent, and its official SHA-256 digest was verified. The exact pinned CI YAML settings pass exact-head LF blobs with warnings. Zizmor reports the same four pre-existing broad-permission findings on base and head; this Task does not change permissions.
The repository's Release route calls Release-GHRepository v2.0.3 at 750abed0ecab02394554b610dc3aa191e75e3248. Its released interface uses legacy labels and has no verified adapter to the canonical decision contract. Publication/source baselines are not established by this PR. No release decision, release label, consumer-release claim, or readiness claim is fabricated; the draft remains held.
Exact-head review and retained-validator evidence
The complete first review at dbd162f0001b73cfa03207dd18f0ce607adc6114 identified two in-scope gaps. Both are fixed and the threads are resolved: the Git fixture now makes the selected commit differ from HEAD, and file execution stays serial because environment variables are shared. A controlled Pester 6.1 parallel probe produced one expected failure and one pass in the same process. Replacing CommitSha with HEAD in the release script failed the improved notes test with the newer commit's text; the experimental production edit was restored. A fresh review was explicitly requested on 7a00cbb9cd72baddf14d990019cf5da3c9c3600e.
The retained Module validator was also exercised against the real PSModuleTest2 artifact from workflow run 41824863677, source dbd162f0001b73cfa03207dd18f0ce607adc6114; the underlying compiler and shipped validator are unchanged in the final test-only head. It passes 26/26 cases. Removing public type registration in an isolated artifact copy causes 21 expected failures. A pre-existing discovery fault is now tracked separately in #553: renaming an exporter comment hides the type cases, and missing registration can then return 4 passed/1 skipped. The shipped suite was not removed, weakened, or rewritten in this PR.
The first local compiled-artifact run correctly failed because ThreadJob 1.x was missing. Only that declared fixture dependency was saved to isolated scratch storage; the subsequent real run passes. No global installation was changed. A preliminary Windows scratch build hit mixed line endings in the formatter; no compiler repair or bypass is included. The successful Module evidence uses the real, source-bound Ubuntu CI artifact instead.
The four pre-existing Zizmor permission findings are tracked separately in #552. They are identical on base and head. No permission change or lint suppression is part of this PR.
Final verification and disposition
Final scoped head: 7a00cbb9cd72baddf14d990019cf5da3c9c3600e; base/main remains 64ffb1d87f86bd5d487891fdb140b78385a00531. Seven ordinary commits change only the five test/CI files listed in the PR. The branch is pushed and the worktree is clean.
| Evidence | Final result |
|---|---|
| Exact-head action CI | Pester 6.2.0, within the unchanged allowed range: 72 passed, 0 failed, 0 skipped |
| Local runner | Actual workflow step passes with Pester 6.1.0; an empty selection throws instead of returning a pass |
| Default fixture CI | Build, source/module checks, local fixtures, generated-index verification, reports/coverage, and site build pass |
| Manifest fixture CI | Configured build/module/local fixture/index/report paths pass; existing disabled source test/lint paths remain skipped |
| Review 5768041720 | Copilot APPROVED on the exact final head; full summary read, no active issues, both earlier threads resolved |
| Script/workflow checks | Clean-profile Pester-aware PSScriptAnalyzer: 0 findings; actionlint 1.7.12: pass; pinned CI YAML rules: 0 errors, 23 warnings |
| Manual baseline Zizmor | Same four permission findings on base and head, tracked separately in #552 |
| Retained validator limit | Proven comment-driven false green is tracked separately in #553; no shipped validation code is changed here |
An inherited-shell analyzer run emitted an intermittent Context/Get-Context resolution warning. Rechecking the same baseline and candidate paths returned 0/0, and a clean-profile Pester-aware run returned 0. No suppression or linter configuration change was made.
The seven CI skips are the four fixture publication jobs, the default fixture's repository lint, and the manifest fixture's source test/lint jobs. They are not reported as passing coverage. Live Gallery/GitHub publication and safety-owner/provenance checks remain unexecuted. The complete review and validation record is in #551.
The final alignment pass verified the Testing, PowerShell, YAML/Actions, contribution/commit/PR rules, and the existing framework contracts. Relevant canonical policy files were unchanged through MSXOrg/docs commit 1b0d4893c8fefabe604d04393068a8ff6f848082. Existing docs/native help remain correct for this scoped behavior and are unchanged.
The convergence search covered functional-test alignment, snapshot/change-detection terms, touched components, and the two new follow-ups. Only this Task is closed by the draft PR. #184 and unrelated/held proposals are not changed or closed.
All scoped audit/delivery steps are complete. The PR remains a draft, with auto-merge off, because the legacy producer route and publication/source evidence have not been reconciled. This Task does not claim release readiness or bypass the existing holds. No merge, publication, label removal, permission change, or held-branch integration was performed.
Implementation plan
- Read README/AGENTS, canonical Testing policy, applicable language/process standards, and all current-main assertion/test entry points; search for a matching ready Task.
- Establish the first passing baseline, define an observable replacement, and prove that it fails on incorrect behavior but not a wording-only change.
- Commit the first focused useful change, push it, and open the requested draft PR before the full audit is complete.
- Read every assertion and real invoked behavior, including workflow checks and shipped validators; record the complete classification above.
- Remove or rewrite the prohibited phrase/signature guards and preserve all supported behavioral and quality coverage.
- Finish exact-head combined validation, controlled negative cases, and directly coupled linter/CI evidence. Record any missing prerequisite or existing failure plainly.
- Finish exact-head review, read complete feedback, record standards/framework alignment and the issue-convergence result, and report real blockers. Keep the requested PR in draft; do not merge or enable auto-merge.
Activity
MariusStorhaug commentedon Nov 4, 2026
The first focused check uses the real release script and intercepts the unsafe GitHub boundary. The original four release tests pass. A deliberate notes-content fault fails the replacement test with the expected content mismatch; a harmless diagnostic wording change leaves all four tests passing. Both experimental source edits were restored; no runtime source change is retained.
The full assertion audit is still in progress. The early draft will keep existing producer/publication holds explicit; no release labels or merge action are part of this Task.
MariusStorhaug commentedon Nov 4, 2026
Recorded the complete source-bound assertion inventory and the focused cleanup results. The draft stays held while exact-head CI/review and producer evidence are reconciled. Restored structured Markdown after the earlier progress update flattened line breaks.
MariusStorhaug commentedon Nov 4, 2026
Addressed both exact-head review findings with a two-commit selection fixture and explicit serial execution. Recorded the deliberate wrong-commit and parallel-environment counterexamples, the retained Module/SourceCode validator evidence, and separate follow-up records for the verified pre-existing gaps. Final review and CI are refreshed on the new head.
MariusStorhaug commentedon Nov 4, 2026
Completed the scoped audit and delivery plan on 7a00cbb. Exact-head CI has 52 successful checks and 7 documented conditional skips; all 72 action cases pass. Copilot approved the exact head after both findings were fixed and resolved. The PR remains draft with the existing producer/publication holds intact; the two verified out-of-scope gaps remain separate open records.