From 2b6acb02f6afeaf6dea95dd2e3bb41a63e1dbc59 Mon Sep 17 00:00:00 2001 From: Adrian Webb Date: Tue, 27 Oct 2026 08:14:11 -0400 Subject: [PATCH 1/9] docs: require permanent test tiers (#325) --- .agents/skills/release-workflow/SKILL.md | 34 ++++++++++++++++++++++++ AGENTS.md | 32 ++++++++++++++++++++++ CHANGELOG.md | 4 +++ tests/README.md | 34 ++++++++++++++++++++++++ 4 files changed, 104 insertions(+) create mode 100644 tests/README.md diff --git a/.agents/skills/release-workflow/SKILL.md b/.agents/skills/release-workflow/SKILL.md index 564328e..24a3b22 100644 --- a/.agents/skills/release-workflow/SKILL.md +++ b/.agents/skills/release-workflow/SKILL.md @@ -141,6 +141,38 @@ Open PR to `cordis`, verify required CI/acceptance/review, then merge if authori **Why this order:** documentation written after code describes what *was* built, not what *should* be — it captures implementation detail instead of intent and is perpetually behind. Tests written after code verify the implementation, not the specification. The docs ARE the spec, the tests validate the spec, the code satisfies the tests. +## ⚠️ MANDATORY: Permanent Test Organization + +All permanent executable test specifications MUST live in the owning repository under exactly +these tier directories, named for the capability or behavior being verified: + +- `tests/acceptance/`: user-observable flows, end-to-end/browser coverage and smoke tests of built programs. +- `tests/regression/`: cases that pin an identified prior defect and prevent recurrence. +- `tests/integration/`: real collaborator, provider, package, filesystem and service boundaries. +- `tests/unit/`: isolated contracts, computations and transformations. + +Write tests outside-in in that order, after committing documentation and before implementation. +Split mixed-tier suites. A negative case is classified by its purpose and boundary; it is not +automatically a regression. Permanent test files MUST NOT be flat under `tests/`, colocated with +source, or named after a migration/task/phase number. Do not disguise test suites as helpers. +Shared fixture classes, fixture programs and data may use `tests/support/`, with no test +registrations there. Historical migration evidence, exact composition pins and immutable archives +retain provenance outside executable test suites; durable coverage moves into the proper tier. + +Every reorganization preserves assertions, inputs, timeouts, negative cases and discovery/count +parity. Update imports, fixture/build entrypoints, public test-helper exports, package scripts, +partition inventories, CI and current docs together. Verify all four tiers are discovered; an +empty tier must be recorded honestly rather than filled with redundant tests or silently skipped. +Include an executable layout/discovery check in the normal owner verification and CI gates; +exercise rejection of misplaced tests, phase-based names and hidden helper registrations. +For a setup-only repository, install that check with its first implementation, without adding +application scaffolding or dependencies during initialization. + +Every injectable seam still needs default production construction coverage, and every user +program needs a built-program acceptance test. Completion requires applicable tier suites, +checks, build, CI and assembled-artifact smoke evidence. Report missing prerequisites or gates +explicitly. A green unit suite or policy prose alone is never sufficient. + ## ⚠️ MANDATORY: Testing Tiers and the Composition Root Outside-in testing only works if the **outer** rung exists. Unit tests of injected, isolated @@ -536,6 +568,8 @@ Run this every time, immediately before opening the PR (this is the artifact the - [ ] On a `feature/*` or `fix/*` branch off verified `origin/cordis`, never on `cordis`/`staging`/`main` directly - [ ] Issue exists and is linked; branch name carries its number - [ ] Documentation updated **and committed** before code (diff shows the doc files changed) +- [ ] Permanent tests live in the four required tier directories with capability-based names; layout/discovery checks reject misplaced or hidden tests. +- [ ] Reorganizations preserve assertion bodies, inputs, timeouts and discovered case counts; imports, fixtures, exports, scripts, partitions and CI agree. - [ ] Tests written outside-in and committed per layer; `` run now → 0 failures - [ ] `` and `` run now → clean / exit 0 - [ ] Seams and user-runnable surfaces have ≥1 test on the **default construction path**, nothing injected (see Testing Tiers and the Composition Root) diff --git a/AGENTS.md b/AGENTS.md index 60f5dfd..411c147 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -180,6 +180,38 @@ Why: docs written after code describe what was built, not what should be built. after code verify the implementation, not the specification. This cycle forces spec-driven development. See the `release-workflow` skill for full details. +## Permanent Test Organization (MANDATORY) + +All permanent executable test specifications MUST live in the owning repository under exactly +these tier directories, named for the capability or behavior being verified: + +- `tests/acceptance/`: user-observable flows, end-to-end/browser coverage and smoke tests of built programs. +- `tests/regression/`: cases that pin an identified prior defect and prevent recurrence. +- `tests/integration/`: real collaborator, provider, package, filesystem and service boundaries. +- `tests/unit/`: isolated contracts, computations and transformations. + +Write tests outside-in in that order, after committing documentation and before implementation. +Split mixed-tier suites. A negative case is classified by its purpose and boundary; it is not +automatically a regression. Permanent test files MUST NOT be flat under `tests/`, colocated with +source, or named after a migration/task/phase number. Do not disguise test suites as helpers. +Shared fixture classes, fixture programs and data may use `tests/support/`, with no test +registrations there. Historical migration evidence, exact composition pins and immutable archives +retain provenance outside executable test suites; durable coverage moves into the proper tier. + +Every reorganization preserves assertions, inputs, timeouts, negative cases and discovery/count +parity. Update imports, fixture/build entrypoints, public test-helper exports, package scripts, +partition inventories, CI and current docs together. Verify all four tiers are discovered; an +empty tier must be recorded honestly rather than filled with redundant tests or silently skipped. +Include an executable layout/discovery check in the normal owner verification and CI gates; +exercise rejection of misplaced tests, phase-based names and hidden helper registrations. +For a setup-only repository, install that check with its first implementation, without adding +application scaffolding or dependencies during initialization. + +Every injectable seam still needs default production construction coverage, and every user +program needs a built-program acceptance test. Completion requires applicable tier suites, +checks, build, CI and assembled-artifact smoke evidence. Report missing prerequisites or gates +explicitly. A green unit suite or policy prose alone is never sufficient. + ## Testing Tiers and the Composition Root (MANDATORY) For **every injectable seam**: diff --git a/CHANGELOG.md b/CHANGELOG.md index 4cc988c..4e7905a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## [Unreleased] +### Changed + +- Organize permanent tests by tier and enforce the shared development/testing strategy (#325). + ### Changed - Qualify built plugin commands against exact packaged Git, approved Git build and native live probe installation artifacts (#317). diff --git a/tests/README.md b/tests/README.md new file mode 100644 index 0000000..37d2d5e --- /dev/null +++ b/tests/README.md @@ -0,0 +1,34 @@ +# Owner testing strategy + + +All permanent executable test specifications MUST live in the owning repository under exactly +these tier directories, named for the capability or behavior being verified: + +- `tests/acceptance/`: user-observable flows, end-to-end/browser coverage and smoke tests of built programs. +- `tests/regression/`: cases that pin an identified prior defect and prevent recurrence. +- `tests/integration/`: real collaborator, provider, package, filesystem and service boundaries. +- `tests/unit/`: isolated contracts, computations and transformations. + +Write tests outside-in in that order, after committing documentation and before implementation. +Split mixed-tier suites. A negative case is classified by its purpose and boundary; it is not +automatically a regression. Permanent test files MUST NOT be flat under `tests/`, colocated with +source, or named after a migration/task/phase number. Do not disguise test suites as helpers. +Shared fixture classes, fixture programs and data may use `tests/support/`, with no test +registrations there. Historical migration evidence, exact composition pins and immutable archives +retain provenance outside executable test suites; durable coverage moves into the proper tier. + +Every reorganization preserves assertions, inputs, timeouts, negative cases and discovery/count +parity. Update imports, fixture/build entrypoints, public test-helper exports, package scripts, +partition inventories, CI and current docs together. Verify all four tiers are discovered; an +empty tier must be recorded honestly rather than filled with redundant tests or silently skipped. +Include an executable layout/discovery check in the normal owner verification and CI gates; +exercise rejection of misplaced tests, phase-based names and hidden helper registrations. +For a setup-only repository, install that check with its first implementation, without adding +application scaffolding or dependencies during initialization. + +Every injectable seam still needs default production construction coverage, and every user +program needs a built-program acceptance test. Completion requires applicable tier suites, +checks, build, CI and assembled-artifact smoke evidence. Report missing prerequisites or gates +explicitly. A green unit suite or policy prose alone is never sufficient. + + From 88faeaa1d4e79d1da44dab31a511f26232b1ffbf Mon Sep 17 00:00:00 2001 From: Adrian Webb Date: Tue, 27 Oct 2026 08:46:42 -0400 Subject: [PATCH 2/9] test: organize permanent coverage and enforce discovery (#325) --- .agents/skills/release-workflow/SKILL.md | 11 +- .../release-workflow/scripts/test-layout.ts | 49 ++++ .github/workflows/verify.yml | 14 +- CHANGELOG.md | 2 - package.json | 13 +- tests/{ => acceptance}/cli-program.test.ts | 237 ++++-------------- tests/integration/cli-program.test.ts | 24 ++ .../command-classification.test.ts | 25 ++ tests/layout-migration.json | 34 +++ tests/regression/test-layout.test.ts | 19 ++ tests/support/cli-program-test-support.ts | 160 ++++++++++++ tests/unit/README.md | 6 + 12 files changed, 393 insertions(+), 201 deletions(-) create mode 100644 .agents/skills/release-workflow/scripts/test-layout.ts rename tests/{ => acceptance}/cli-program.test.ts (66%) create mode 100644 tests/integration/cli-program.test.ts create mode 100644 tests/integration/command-classification.test.ts create mode 100644 tests/layout-migration.json create mode 100644 tests/regression/test-layout.test.ts create mode 100644 tests/support/cli-program-test-support.ts create mode 100644 tests/unit/README.md diff --git a/.agents/skills/release-workflow/SKILL.md b/.agents/skills/release-workflow/SKILL.md index 24a3b22..5d9768b 100644 --- a/.agents/skills/release-workflow/SKILL.md +++ b/.agents/skills/release-workflow/SKILL.md @@ -173,6 +173,14 @@ program needs a built-program acceptance test. Completion requires applicable ti checks, build, CI and assembled-artifact smoke evidence. Report missing prerequisites or gates explicitly. A green unit suite or policy prose alone is never sufficient. +The copied development-only checker is `scripts/test-layout.ts` inside this skill. After +installing the owner's declared development dependencies, wire `bun .agents/skills/release-workflow/scripts/test-layout.ts` +into the normal check command and CI. It rejects misplaced or undiscoverable registrations and +phase-based specification paths, and verifies complete tier commands. Platform checks each +assembled specification is explicitly assigned to a script or CI closure because historical +exact artifact selections run in separate jobs. Reusable test contracts with assertions belong +in their actual tier; support holds fixture setup and executable fixture programs. + ## ⚠️ MANDATORY: Testing Tiers and the Composition Root Outside-in testing only works if the **outer** rung exists. Unit tests of injected, isolated @@ -441,7 +449,8 @@ gh pr create --base cordis \ - [ ] CI green ( + + ) - [ ] Manual verification: - [ ] New tests added for new behavior -- [ ] Existing tests pass unmodified (for refactors) +- [ ] Acceptance, regression, integration and unit tiers are verified; any empty tier or missing prerequisite is explicitly recorded +- [ ] Test layout/discovery check passes; reorganization preserves cases, assertions, arguments and timeouts ## Related Part of # diff --git a/.agents/skills/release-workflow/scripts/test-layout.ts b/.agents/skills/release-workflow/scripts/test-layout.ts new file mode 100644 index 0000000..6f90250 --- /dev/null +++ b/.agents/skills/release-workflow/scripts/test-layout.ts @@ -0,0 +1,49 @@ +import ts from 'typescript'; +import {readdirSync,readFileSync,existsSync} from 'node:fs'; +import {join,relative} from 'node:path'; +/** Shared development-only policy. No runtime or target project depends on it. */ +export class TestLayout { + private static readonly ignored=new Set(['.git','.agents','.codegraph','node_modules','dist','vendor']); + private static readonly tier=/^tests\/(acceptance|regression|integration|unit)\//; + inspect(root=process.cwd()):string[]{const errors:string[]=[],specifications:string[]=[];this.directory(root,root,errors,specifications);this.discovery(root,specifications,errors);return errors;} + check(root=process.cwd()):void{const errors=this.inspect(root);if(errors.length)throw new Error(errors.join('\n'));} + private discovery(root:string,specifications:string[],errors:string[]):void{ + const path=join(root,'package.json');if(!existsSync(path))return; + const metadata=JSON.parse(readFileSync(path,'utf8')) as {name?:string;scripts?:Record},scripts=metadata.scripts??{}; + if(metadata.name==='@treeseed-cordis/platform-verification'){ + let commands=Object.values(scripts).join('\n');const workflows=join(root,'.github/workflows');if(existsSync(workflows))for(const name of readdirSync(workflows))if(/\.ya?ml$/.test(name))commands+='\n'+readFileSync(join(workflows,name),'utf8'); + for(const file of specifications)if(!commands.includes(file))errors.push(file+': exact assembled test must be assigned to a script or CI closure');return; + } + if(!Object.keys(scripts).some(this.tierScript))return; + for(const tier of ['acceptance','regression','integration','unit']){const command=scripts['test:'+tier];if(!command||!command.includes('tests/'+tier)||command.includes('--test-name-pattern'))errors.push('test:'+tier+': command must discover the complete tier directory');} + } + private tierScript(name:string):boolean{return ['test:acceptance','test:regression','test:integration','test:unit'].includes(name);} + private directory(root:string,directory:string,errors:string[],specifications:string[]):void{ + for(const entry of readdirSync(directory,{withFileTypes:true})){ + if(TestLayout.ignored.has(entry.name))continue; + const path=join(directory,entry.name),file=relative(root,path).replaceAll('\\','/'); + if(entry.isSymbolicLink()){if(file.startsWith('tests/'))errors.push(file+': test symlink forbidden');continue;} + if(entry.isDirectory()){if(existsSync(join(path,'.git')))continue;this.directory(root,path,errors,specifications);continue;} + if(!/\.[cm]?[jt]sx?$/.test(file))continue; + const spec=/\.(test|spec)\.[cm]?[jt]sx?$/.test(file),tier=TestLayout.tier.test(file); + if(spec)specifications.push(file); + if(spec&&!tier)errors.push(file+': specifications require tests/{acceptance,regression,integration,unit}'); + if(spec&&/(?:^|[\/-])(?:task|phase)[-_]?\d/i.test(file))errors.push(file+': permanent specifications require capability names'); + const source=ts.createSourceFile(file,readFileSync(path,'utf8'),ts.ScriptTarget.Latest,true),visitor=new TestRegistrationVisitor();visitor.inspect(source); + if(visitor.registrations&&(!tier||!spec))errors.push(file+': test registrations require a discoverable tier specification'); + if(spec&&!visitor.registrations)errors.push(file+': specification has no discoverable test registrations'); + } + } +} +class TestRegistrationVisitor { + registrations=0; + private readonly bindings=new Set(); + private readonly namespaces=new Set(); + inspect(source:ts.SourceFile):void{ + for(const node of source.statements){if(!ts.isImportDeclaration(node)||!ts.isStringLiteral(node.moduleSpecifier)||!['bun:test','node:test'].includes(node.moduleSpecifier.text))continue;const clause=node.importClause;if(!clause)continue;if(clause.name)this.bindings.add(clause.name.text);const names=clause.namedBindings;if(names&&ts.isNamespaceImport(names))this.namespaces.add(names.name.text);if(names&&ts.isNamedImports(names))for(const item of names.elements)if(['test','it','describe'].includes((item.propertyName??item.name).text))this.bindings.add(item.name.text);} + this.visit(source); + } + private visit(node:ts.Node):void{if(ts.isCallExpression(node)&&this.test(node.expression))this.registrations++;ts.forEachChild(node,this.visit.bind(this));} + private test(expression:ts.Expression):boolean{if(ts.isIdentifier(expression))return this.bindings.has(expression.text);if(ts.isPropertyAccessExpression(expression)){if(ts.isIdentifier(expression.expression)&&this.namespaces.has(expression.expression.text))return ['test','it','describe'].includes(expression.name.text);return this.test(expression.expression);}if(ts.isCallExpression(expression))return this.test(expression.expression);return false;} +} +if(import.meta.main)new TestLayout().check(); diff --git a/.github/workflows/verify.yml b/.github/workflows/verify.yml index af1dbf1..67e0897 100644 --- a/.github/workflows/verify.yml +++ b/.github/workflows/verify.yml @@ -13,19 +13,19 @@ jobs: matrix: include: - group: foundation - pattern: "^(unit:|integration:|smoke: (?!built CLI (installs|selects registered)))" + pattern: "^(regression:|unit:|integration:|acceptance: (?!built CLI (installs|selects registered)))" - group: npm - pattern: "^smoke: built CLI installs two typed" + pattern: "^acceptance: built CLI installs two typed" - group: git - pattern: "^smoke: built CLI installs packaged" + pattern: "^acceptance: built CLI installs packaged" - group: git-build - pattern: "^smoke: built CLI installs approved" + pattern: "^acceptance: built CLI installs approved" - group: git-registered - pattern: "^smoke: built CLI selects registered packaged" + pattern: "^acceptance: built CLI selects registered packaged" - group: build-registered - pattern: "^smoke: built CLI selects registered built" + pattern: "^acceptance: built CLI selects registered built" - group: live - pattern: "^smoke: built CLI installs and selects" + pattern: "^acceptance: built CLI installs and selects" runs-on: ubuntu-26.04 timeout-minutes: 10 env: diff --git a/CHANGELOG.md b/CHANGELOG.md index 4e7905a..7396583 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,8 +5,6 @@ ### Changed - Organize permanent tests by tier and enforce the shared development/testing strategy (#325). - -### Changed - Qualify built plugin commands against exact packaged Git, approved Git build and native live probe installation artifacts (#317). ### Added diff --git a/package.json b/package.json index 97804c6..b5d7791 100644 --- a/package.json +++ b/package.json @@ -6,12 +6,15 @@ "license": "Apache-2.0", "packageManager": "bun@1.4.4", "scripts": { - "check": "bun scripts/build-cli.ts check", + "check": "bun run check:test-layout && bun scripts/build-cli.ts check", "build": "bun scripts/build-cli.ts build", - "test:unit": "bun test ./tests --test-name-pattern '^unit:'", - "test:integration": "bun test ./tests --timeout 30000 --test-name-pattern '^integration:'", - "test:smoke": "bun test ./tests --timeout 60000 --test-name-pattern '^smoke:'", - "verify": "bun run check && bun run build && bun run test:unit && bun run test:integration && bun run test:smoke" + "test:unit": "bun test ./tests/unit --pass-with-no-tests", + "test:integration": "bun test ./tests/integration --timeout 30000", + "test:smoke": "bun run test:acceptance", + "verify": "bun run check && bun run build && bun run test:acceptance && bun run test:regression && bun run test:integration && bun run test:unit", + "check:test-layout": "bun .agents/skills/release-workflow/scripts/test-layout.ts", + "test:regression": "bun test ./tests/regression", + "test:acceptance": "bun test ./tests/acceptance --timeout 60000" }, "dependencies": { "@treeseed-cordis/sdk": "file:vendor/sdk.tgz", diff --git a/tests/cli-program.test.ts b/tests/acceptance/cli-program.test.ts similarity index 66% rename from tests/cli-program.test.ts rename to tests/acceptance/cli-program.test.ts index 0d58ccf..7fba455 100644 --- a/tests/cli-program.test.ts +++ b/tests/acceptance/cli-program.test.ts @@ -6,27 +6,14 @@ import { spawnSync, type SpawnSyncReturns } from 'node:child_process'; import { createServer, type Server } from 'node:net'; import { createHash } from 'node:crypto'; import { PreparedReleaseFixture } from '@treeseed-cordis/deployment/testing'; -import { CliApplication } from '../src/main.js'; +import { CliApplication } from '../../src/main.js'; import { ServiceRuntimeFixture } from '@treeseed-cordis/deployment/testing/runtime'; import { ServiceWorkerBuild, ServicePlanStore } from '@treeseed-cordis/deployment'; import { PluginInstallationFixture } from '@treeseed-cordis/deployment/testing/plugins'; import { CanonicalJson, ServiceSchemas, PlatformSchemas, PluginManagementSchemas, type PluginInstallResult, type ServiceResourceReceipt, type CliEnvelope } from '@treeseed-cordis/sdk'; +import {CliProgramFixture,CliTests,ProviderFixture,ExtensiblePlanProgramTests,InstalledProgramTests,InstalledPluginCliTests,LivePluginCliTests} from '../support/cli-program-test-support.js'; +class CliTestsAcceptance extends CliTests { -class CliProgramFixture { - readonly root=mkdtempSync(join(tmpdir(),'cordis cli ')); - readonly cwd=join(this.root,'foreign cwd'); - readonly binary=join(this.root,'path with spaces','trsd'); - readonly symlink=join(this.root,'trsd-link'); - constructor() {mkdirSync(this.cwd);mkdirSync(join(this.root,'path with spaces'));copyFileSync(resolve('dist/trsd-linux-x64'),this.binary);symlinkSync(this.binary,this.symlink);} - run(args:string[],environment:NodeJS.ProcessEnv={},binary=this.binary):SpawnSyncReturns {return spawnSync(binary,args,{cwd:this.cwd,env:{...process.env,...environment},encoding:'utf8',stdio:['ignore','pipe','pipe'],timeout:15000});} - close():void {rmSync(this.root,{recursive:true,force:true});} -} -class CliTests { - server:Server|null=null; - resolvePort:((port:number)=>void)|null=null; - listen(resolvePort:(port:number)=>void):void {this.resolvePort=resolvePort;this.server=createServer();this.server.listen(0,'127.0.0.1',this.listening.bind(this));} - listening():void {const address=this.server!.address();if(address&&typeof address==='object')this.resolvePort!(address.port);} - async port(occupied=false):Promise {const port=await new Promise(this.listen.bind(this));if(!occupied)this.server!.close();return port;} pluginArguments():void { const fixture=new CliProgramFixture(); try { @@ -45,10 +32,12 @@ class CliTests { const empty=fixture.run(['plugin','list','--installation',join(fixture.root,'missing'),'--json'],{PATH:''});expect(empty.status).toBe(3);expect(JSON.parse(empty.stdout).result).toBeNull();expect(existsSync(join(fixture.root,'missing'))).toBe(false); } finally {fixture.close();} } + installedArguments():void { const fixture=new CliProgramFixture(); try{for(const command of ["apply","status","application"]){const result=fixture.run(["platform",command,"--json"],{PATH:""});expect(result.status).toBe(2);expect(result.stderr).toBe("");expect(JSON.parse(result.stdout).problem.code).toBe("INVALID_ARGUMENTS");}}finally{fixture.close();} } + identity():void { const fixture=new CliProgramFixture(); try { @@ -59,6 +48,7 @@ class CliTests { expect(readdirSync(fixture.cwd)).toEqual([]); }finally{fixture.close();} } + async diagnosis():Promise { const fixture=new CliProgramFixture(); try { @@ -75,6 +65,7 @@ class CliTests { if(envelope.result&&'host' in envelope.result&&envelope.result.host.distribution==='ubuntu'&&envelope.result.host.release==='26.04') {expect(result.status).toBe(0);expect(envelope.success).toBe(true);} else {expect(result.status).toBe(3);expect(envelope.success).toBe(false);} }finally{fixture.close();} } + async blockers():Promise { const fixture=new CliProgramFixture(); try { @@ -87,6 +78,7 @@ class CliTests { expect(existsSync(join(fixture.root,'new'))).toBe(false); }finally{fixture.close();} } + invalidProgram():void { const fixture=new CliProgramFixture(); try { @@ -96,14 +88,7 @@ class CliTests { const unsupported=fixture.run(['platform','install']);expect(unsupported.status).toBe(3);expect(unsupported.stderr).toContain('UNSUPPORTED_COMMAND'); }finally{fixture.close();} } - async defaultComposition():Promise { - const root=mkdtempSync(join(tmpdir(),'cordis-default-')); - try {const result=await new CliApplication().execute(['doctor','--installation',join(root,'new'),'--port',String(await this.port())]);expect(result.envelope.result).not.toBeNull();expect(result.envelope.command).toBe('doctor');expect(result.envelope.result&&'host' in result.envelope.result).toBe(true);expect(readdirSync(root)).toEqual([]);} - finally {rmSync(root,{recursive:true,force:true});} - } - snapshot():string { - const rows:string[]=[];for(const args of [['ps','-aq'],['image','ls','--no-trunc','-q'],['volume','ls','-q'],['network','ls','-q']]){const result=spawnSync('docker',['--context','default',...args],{encoding:'utf8'});expect(result.status).toBe(0);rows.push(result.stdout.split('\n').sort().join('\n'));}return JSON.stringify(rows); - } + async plan_changes_no_runtime():Promise { const program=new CliProgramFixture(),fixture=await new PreparedReleaseFixture().prepare(await this.port()); try { @@ -124,11 +109,13 @@ class CliTests { writeFileSync(join(fixture.bundle,'archives/plugin.tgz'),'tampered');const blocked=program.run(['platform','plan','--file',configuration,'--json']);expect(blocked.status).toBe(3);const rejected=JSON.parse(blocked.stdout) as CliEnvelope;expect(rejected.result).toBeNull();expect(rejected.references).toEqual([]);expect(readdirSync(join(fixture.installation,'.cordis/plans'))).toEqual([plan.id+'.json']); }finally{fixture.close();program.close();} } + async installationPaths():Promise { const program=new CliProgramFixture(),fixture=await new PreparedReleaseFixture().prepare(await this.port()); const installation=join(fixture.root,'installation with spaces Ω'); try {const file=join(program.cwd,'desired path.json');writeFileSync(file,JSON.stringify({...fixture.configuration,installationDirectory:installation}));const result=program.run(['platform','plan','--file',file,'--json']);const qualified=/^VERSION_ID="?26\.04/m.test(readFileSync('/etc/os-release','utf8'))&&process.arch==='x64';if(!qualified&&process.env.CORDIS_REQUIRE_PLAN!=='1'){expect(result.status).toBe(3);expect(existsSync(installation)).toBe(false);return;}expect(result.status).toBe(0);const plan=PlatformSchemas.plan.parse(JSON.parse(result.stdout).result);expect(plan.configuration.installationDirectory).toBe(installation);expect(readdirSync(installation)).toEqual(['.cordis']);expect(existsSync(join(installation,'.cordis/plans',plan.id+'.json'))).toBe(true);}finally{fixture.close();program.close();} } + async invalidPlans():Promise { const fixture=new CliProgramFixture(); try { @@ -139,102 +126,20 @@ class CliTests { for(const args of [['platform','plan','--json'],['platform','plan','--file',file,'--file',file,'--json'],['platform','plan','--file',file,'--apply','--json']]){const result=fixture.run(args);expect(result.status).toBe(2);expect(JSON.parse(result.stdout).command).toBe('platform.plan');} }finally{fixture.close();} } - async parser():Promise { - const app=new CliApplication(); - expect((await app.execute(['--help'])).code).toBe(0);expect((await app.execute(['--version'])).code).toBe(0); - expect((await app.execute(['doctor','--json','--port','bad'])).code).toBe(2); - expect((await app.execute(['apply'])).code).toBe(3); - } } -const suite=new CliTests(); -test('smoke: standalone identity from spaces, symlink, foreign cwd and closed stdin',suite.identity.bind(suite)); -test('smoke: doctor JSON and no filesystem/container mutation or cwd autoload',suite.diagnosis.bind(suite)); -test('smoke: wrong engine, missing Docker, foreign installation and occupied port',suite.blockers.bind(suite)); -test('smoke: invalid arguments and unsupported commands are actionable',suite.invalidProgram.bind(suite)); -test('integration: default CLI construction uses real packaged diagnostics',suite.defaultComposition.bind(suite)); -test('unit: parser help, version and error classifications',suite.parser.bind(suite)); - -test('smoke: plan_changes_no_runtime using real prepared artifacts',suite.plan_changes_no_runtime.bind(suite)); - -test('smoke: invalid and duplicate planning inputs never echo secrets or claim saved state',suite.invalidPlans.bind(suite)); - -test('smoke: plan new installation directory with spaces and Unicode',suite.installationPaths.bind(suite)); +class CompositionProgramTestsAcceptance { -class ProviderFixture { - readonly root=mkdtempSync(join(tmpdir(),'cordis installed provider Ω ')); - tree=join(this.root,'prepared package'); - readonly marker=join(this.root,'live-effect'); - readonly importMarker=join(this.root,'module-imported'); - get file():string{return join(this.tree,'composition.json');} - readonly declaration={id:'fixture-one',pluginId:'fixture',capability:'deployment',contractVersion:'0.3.0-cordis.1',target:'cli',provide:['deployment'],inject:[]}; - digest=''; - constructor(readonly label='one') {} - static sha(value:Uint8Array|string):string {return createHash('sha256').update(value).digest('hex');} - copy(source:string,destination:string):void { - mkdirSync(destination,{recursive:true}); - for(const entry of readdirSync(source,{withFileTypes:true})) { - const from=join(source,entry.name),to=join(destination,entry.name); - if(entry.isDirectory())this.copy(from,to);else if(entry.isFile())copyFileSync(from,to);else throw new Error('Fixture dependency is not a regular package tree'); - } - } - async prepare():Promise { - mkdirSync(this.tree);this.declaration.id='fixture-'+this.label; - // These are real pinned installed third-party packages, copied as fixture bytes, not mocked modules. - for(const name of ['cordis','cosmokit','@standard-schema/spec'])this.copy(resolve('node_modules',name),join(this.tree,'node_modules',name)); - const helper=join(this.tree,'node_modules/@fixture/helper');mkdirSync(helper,{recursive:true}); - writeFileSync(join(helper,'package.json'),JSON.stringify({name:'@fixture/helper',version:'1.0.0',type:'module',exports:'./index.js'})); - const helperSource=join(this.root,'helper.ts');writeFileSync(helperSource,'export class Helper {static label():string{return "ordinary-dependency";}}'); - let build=spawnSync('bun',['build',helperSource,'--target=node','--outfile',join(helper,'index.js')],{encoding:'utf8'});expect(build.status).toBe(0); - const source=join(this.root,'provider.ts'); - writeFileSync(source,`import {Context} from 'cordis';import {Helper} from '@fixture/helper';import {writeFileSync,unlinkSync} from 'node:fs'; -class ImportCanary {static run(){writeFileSync(${JSON.stringify(this.importMarker)},'imported');}}ImportCanary.run(); -class Configuration {static validate(value:unknown){return value&&typeof value==='object'&&Object.keys(value).length===0?{value}:{issues:[{message:'Invalid configuration'}]};}} -export class Provider { -static readonly declaration=${JSON.stringify(this.declaration)}; -static readonly provide=Provider.declaration.provide;static readonly inject=Provider.declaration.inject; -static readonly Config={'~standard':{version:1,vendor:'fixture',validate:Configuration.validate.bind(Configuration)}}; -constructor(readonly ctx:Context){if(!Context.is(ctx)||ctx.registry.get(Provider)!==ctx[Context.root].registry.get(Provider))throw new Error('not native shared graph');writeFileSync(${JSON.stringify(this.marker)},'active');ctx.provide('deployment',this);ctx.effect(this.effect.bind(this));if(${JSON.stringify(this.label)}==='constructor-fail')throw new Error('secret-constructor-canary');} -effect(){return this.close.bind(this);}close(){unlinkSync(${JSON.stringify(this.marker)});if(${JSON.stringify(this.label)}==='cleanup-fail')throw new Error('secret-cleanup-canary');} -async diagnose(){return {ready:true,host:{os:'fixture',architecture:'x64',distribution:'fixture',release:'1'},engine:{context:${JSON.stringify(this.label)},endpoint:null,serverVersion:null,serverArchitecture:null,composeVersion:null,cpus:null,memoryBytes:null},installation:{path:'/fixture',exists:false,freeBytes:null},port:8443,checks:[{id:'native-provider',status:'pass',code:'FIXTURE',message:Helper.label()}]};} -async plan(){throw new Error('fixture is diagnosis-only');}} -`); - build=spawnSync('bun',['build',source,'--target=node','--external','cordis','--external','@fixture/helper','--outfile',join(this.tree,'index.js')],{encoding:'utf8'});expect(build.status).toBe(0); - writeFileSync(join(this.tree,'package.json'),JSON.stringify({name:'@fixture/provider',version:'1.0.0',type:'module',exports:'./index.js',dependencies:{cordis:'4.0.0-rc.12','@fixture/helper':'1.0.0'},cordis:this.declaration})); - this.refresh();return this; - } - inventory(directory=this.tree,prefix=''):Array<{path:string;sha256:string}> { - const files:Array<{path:string;sha256:string}>=[]; - for(const entry of readdirSync(directory,{withFileTypes:true}).sort(ProviderFixture.order)){ - if(directory===this.tree&&entry.name==='composition.json')continue; - const path=join(directory,entry.name),relative=prefix+entry.name; - if(entry.isDirectory())files.push(...this.inventory(path,relative+'/'));else files.push({path:relative,sha256:ProviderFixture.sha(readFileSync(path))}); - } - return files; - } - static order(a:{name:string},b:{name:string}):number {return a.name.localeCompare(b.name);} - permissions(directory:string,locked:boolean):void {chmodSync(directory,0o700);for(const entry of readdirSync(directory,{withFileTypes:true})){const path=join(directory,entry.name);if(entry.isSymbolicLink())continue;if(entry.isDirectory())this.permissions(path,locked);else chmodSync(path,locked?0o400:0o600);}chmodSync(directory,locked?0o500:0o700);} - refresh():void { - this.permissions(this.tree,false); - const manifest={formatVersion:1,contractVersion:'0.3.0-cordis.1',target:'cli',scopeId:'local-cli',providerId:this.declaration.id,package:{name:'@fixture/provider',version:'1.0.0'},entry:'index.js',configuration:{},files:this.inventory()}; - writeFileSync(this.file,JSON.stringify(manifest));this.address();this.permissions(this.tree,true); - } - address():void {this.digest=ProviderFixture.sha(readFileSync(this.file));const destination=join(this.root,this.digest);if(destination!==this.tree){renameSync(this.tree,destination);this.tree=destination;}} - run(extra:string[]=[]):SpawnSyncReturns { - const binary=join(this.root,'trsd link');if(!existsSync(binary))symlinkSync(resolve('dist/trsd-linux-x64'),binary); - return spawnSync(binary,['doctor','--json','--composition',this.file,'--composition-digest',this.digest,...extra],{encoding:'utf8',cwd:this.root,env:{...process.env,PATH:''},timeout:15000}); - } - close():void {this.permissions(this.tree,false);rmSync(this.root,{recursive:true,force:true});} -} -class CompositionProgramTests { async selected():Promise { for(const label of ['one','two']){ const fixture=await new ProviderFixture(label).prepare(); try {const result=fixture.run();expect(result.status).toBe(0);expect(result.stderr).toBe('');const envelope=JSON.parse(String(result.stdout));expect(envelope.result.engine.context).toBe(label);expect(envelope.result.checks).toContainEqual({id:'native-provider',status:'pass',code:'FIXTURE',message:'ordinary-dependency'});expect(existsSync(fixture.marker)).toBe(false);const repeat=fixture.run();expect(repeat.status).toBe(0);expect(repeat.stdout).toBe(result.stdout);expect(existsSync(fixture.marker)).toBe(false);}finally{fixture.close();} } } + async lifecycleFailures():Promise { for(const label of ['constructor-fail','cleanup-fail']){const fixture=await new ProviderFixture(label).prepare();try{const result=fixture.run();expect(result.status).toBe(1);expect(JSON.parse(result.stdout).success).toBe(false);expect(JSON.parse(result.stdout).result).toBeNull();expect(result.stdout+result.stderr).not.toContain('canary');expect(existsSync(fixture.marker)).toBe(false);}finally{fixture.close();}} } + async metadataFailures():Promise { for(const kind of ['hash','duplicate','schema','symlink','extra','configuration','address','closure','scope','file-mode','directory-mode','capability']){ const fixture=await new ProviderFixture().prepare(); @@ -254,6 +159,7 @@ class CompositionProgramTests { }finally{fixture.close();} } } + async rejected():Promise { const fixture=await new ProviderFixture().prepare(); try { @@ -264,29 +170,16 @@ class CompositionProgramTests { try {missing.permissions(missing.tree,false);rmSync(join(missing.tree,'node_modules/@fixture/helper'),{recursive:true});const result=missing.run();expect(result.status).toBe(3);expect(existsSync(missing.marker)).toBe(false);}finally{missing.close();} } } -const compositionSuite=new CompositionProgramTests(); -test('smoke: compiled_cli_uses_cordis_and_selected_deployment_provider without Node Bun or cwd modules',compositionSuite.selected.bind(compositionSuite)); -test('smoke: installed provider byte and dependency failures block before import',compositionSuite.rejected.bind(compositionSuite)); - -test('smoke: provider constructors and failed disposal cannot report successful CLI completion',compositionSuite.lifecycleFailures.bind(compositionSuite)); -test('smoke: prepared-provider metadata scope ownership and closure are checked before activation',compositionSuite.metadataFailures.bind(compositionSuite)); - -test('smoke: installed commands validate arguments before contacting worker',suite.installedArguments.bind(suite)); +class UiOutputProgramTestsAcceptance { -class UiOutputProgramTests { human():void{const fixture=new CliProgramFixture();try{const result=fixture.run(['doctor','--installation',join(fixture.root,'new')],{PATH:''});expect(result.status).toBe(3);expect(result.stdout).toContain('Diagnostic checks');expect(result.stdout).toContain('Check | Result | Code | Detail');expect(result.stderr).toContain('PREREQUISITES_UNMET');expect(result.stdout+result.stderr).not.toContain('\u001b');}finally{fixture.close();}} } -const uiOutput=new UiOutputProgramTests();test('smoke: built CLI renders structured diagnostic models through reusable terminal providers',uiOutput.human.bind(uiOutput)); +class ExtensiblePlanProgramTestsAcceptance extends ExtensiblePlanProgramTests { -class ExtensiblePlanProgramTests extends CliTests { - private listeners():string{const rows:string[]=[];for(const path of ['/proc/net/tcp','/proc/net/tcp6'])for(const line of readFileSync(path,'utf8').trim().split('\n').slice(1)){const fields=line.trim().split(/\s+/);if(fields[3]==='0A')rows.push(fields[1]!);}return rows.sort().join('\n');} async plan():Promise{const {ServiceReleaseFixture}=await import('@treeseed-cordis/deployment/testing'),{ServiceSchemas}=await import('@treeseed-cordis/sdk'),fixture=await new ServiceReleaseFixture().services(),program=new CliProgramFixture();try{const port=await this.port();fixture.configuration.origin='https://localhost:'+port;fixture.release.composition.services[0]!.definition.ports[0]!.publication!.port=port;fixture.recipe.services['instance-0']!.ports![0]!.published=port;fixture.refreshService();const path=join(program.cwd,'service-config.json');writeFileSync(path,JSON.stringify(fixture.configuration));const before=this.snapshot(),listeners=this.listeners(),result=program.run(['platform','plan','--file',path,'--json']);expect(result.status).toBe(0);expect(result.stderr).toBe('');const plan=ServiceSchemas.plan.parse(JSON.parse(result.stdout).result);expect(plan.formatVersion).toBe(3);expect(plan.composition.services).toHaveLength(5);expect(plan.changes).toHaveLength(5);expect(plan.blockers).toEqual([]);const restart=program.run(['platform','apply','--installation',fixture.installation,'--plan',plan.id,'--digest',plan.digest,'--idempotency-key','not-an-additive-restart','--restart','instance-0','--json']);expect(restart.status).toBe(3);expect(JSON.parse(restart.stdout).blockers[0].code).toBe('RESTART_SCOPE_NOT_QUALIFIED');expect(existsSync(join(fixture.installation,'.cordis/launch'))).toBe(false);const saved=join(fixture.installation,'.cordis/plans',plan.id+'.json');expect(ServiceSchemas.plan.parse(JSON.parse(readFileSync(saved,'utf8')))).toEqual(plan);expect(statSync(saved).mode&0o777).toBe(0o600);const again=program.run(['platform','plan','--file',path,'--json']);expect(again.status).toBe(0);expect(again.stdout).toBe(result.stdout);const human=program.run(['platform','plan','--file',path]);expect(human.status).toBe(0);expect(human.stdout).toContain('Services\nService | Image');expect(human.stdout).toContain('instance-4');expect(human.stdout).toContain('Future effects only');const tools=join(program.root,'docker-only');mkdirSync(tools);let docker:string|null=null;for(const directory of(process.env.PATH??'').split(':'))if(existsSync(join(directory,'docker'))){docker=join(directory,'docker');break;}expect(docker).not.toBeNull();symlinkSync(docker!,join(tools,'docker'));const standalone=program.run(['platform','plan','--file',path,'--json'],{PATH:tools});expect(standalone.status).toBe(0);expect(standalone.stdout).toBe(result.stdout);expect(this.snapshot()).toBe(before);expect(this.listeners()).toBe(listeners);expect(existsSync(join(fixture.installation,'.cordis/credentials'))).toBe(false);expect(existsSync(join(fixture.root,'SCRIPT_CANARY'))).toBe(false);writeFileSync(join(fixture.bundle,'archives/plugin.tgz'),'tampered');const blocked=program.run(['platform','plan','--file',path,'--json']);expect(blocked.status).toBe(3);const envelope=JSON.parse(blocked.stdout);expect(envelope.result).toBeNull();expect(envelope.references).toEqual([]);expect(readdirSync(join(fixture.installation,'.cordis/plans'))).toEqual([plan.id+'.json']);}finally{fixture.close();program.close();}} } -const extensiblePlan=new ExtensiblePlanProgramTests();test('smoke: standalone CLI plans genuine five-instance format3 bundle deterministically without runtime effects',extensiblePlan.plan.bind(extensiblePlan),60000); +class InstalledProgramTestsAcceptance extends InstalledProgramTests { - -class InstalledProgramTests { - private worker: { pid: number; dev: number; ino: number } | null = null; async installed(): Promise { const fixture = new ServiceRuntimeFixture(), program = new CliProgramFixture(); try { @@ -316,55 +209,13 @@ class InstalledProgramTests { this.signal('SIGKILL'); fixture.close(); program.close(); } } - private json(program: CliProgramFixture, args: string[], environment: NodeJS.ProcessEnv = {}): ReturnType { const result = program.run(args, environment, program.symlink); if (result.status !== 0) throw new Error('Compiled installed command failed: ' + result.stdout + result.stderr); expect(result.stderr).toBe(''); return ServiceSchemas.envelope.parse(JSON.parse(result.stdout)); } - private signal(signal: NodeJS.Signals): void { if (!this.worker) return; try { const current = statSync('/proc/' + this.worker.pid + '/exe'); if (current.dev === this.worker.dev && current.ino === this.worker.ino) process.kill(this.worker.pid, signal); } catch { } } - private static delay(done: (value: void) => void): void { setTimeout(done, 100); } } -const installedPrograms = new InstalledProgramTests(); -test('smoke: standalone CLI applies and observes five format3 services through its default durable host', installedPrograms.installed.bind(installedPrograms), 240000); - -test('smoke: built plugin commands validate arguments with safe SDK JSON envelopes and do not create missing installations', suite.pluginArguments.bind(suite),30000); +class InstalledPluginCliTestsAcceptance extends InstalledPluginCliTests { -class InstalledPluginCliTests { - private worker: { pid: number; dev: number; ino: number } | null = null; - private static delay(done: (value: void) => void): void { setTimeout(done, 50); } - private static hash(bytes: Uint8Array): string { return createHash('sha256').update(bytes).digest('hex'); } - private request(program: CliProgramFixture, args: string[], expected?: number): ReturnType { - const result=program.run(['plugin',...args,'--json']); - if(result.status===null)throw new Error('Built CLI terminated without an exit status'); - if(expected!==undefined&&result.status!==expected){const envelope=PluginManagementSchemas.envelope.parse(JSON.parse(result.stdout));const codes:string[]=[];for(const blocker of envelope.blockers)codes.push(blocker.code);throw new Error('CLI_PLUGIN_READBACK:'+args[0]+':'+result.status+':'+envelope.problem?.code+':'+codes.join(','));} - if(expected===undefined)expect([0,3]).toContain(result.status);else expect(result.status).toBe(expected); - expect(result.stderr).toBe('');expect(result.stdout).not.toContain('NEVER_INHERIT');return PluginManagementSchemas.envelope.parse(JSON.parse(result.stdout)); - } - private async settled(program: CliProgramFixture, installation: string, input: PluginInstallResult): Promise { - let result=input;const deadline=Date.now()+150000; - while(result.state==='pending'&&Date.now() { - const result=new Map();const all=store.resources(); - for(const receipt of all)for(const binding of store.current()!.resources)if(receipt.resourceId===binding.resourceId){const prior=result.get(receipt.resourceId);if(!prior||prior.references.length, files: Map): void { - const current=this.receipts(store); - for(const [id,receipt]of before){const next=current.get(id);expect(next).toBeDefined();expect(next!.actualId).toBe(receipt.actualId);expect(next!.creatingGeneration).toBe(receipt.creatingGeneration);expect(next!.definitionDigest).toBe(receipt.definitionDigest);expect(next!.contentDigest).toBe(receipt.contentDigest);expect(next!.path).toBe(receipt.path);} - for(const [path,digest]of files)expect(InstalledPluginCliTests.hash(readFileSync(join(store.path,path)))).toBe(digest); - } - private async direct(store: ServicePlanStore, id: string): Promise { - for(const receipt of this.receipts(store).values())if(receipt.kind==='container'&&receipt.serviceId===id&&receipt.actualId){ - const inspected=spawnSync('docker',['container','inspect','--format','{{json .NetworkSettings.Networks}}',receipt.actualId],{encoding:'utf8'});expect(inspected.status).toBe(0);const networks=JSON.parse(inspected.stdout) as Record;let address:string|undefined;for(const network of Object.values(networks))address=network.IPAddress;expect(address).toBeDefined(); - const response=await fetch('http://'+address+':8080/'),body=await response.json() as {serviceId:string;passwordPresent:boolean;volumeMarker:string};expect(response.ok).toBe(true);expect(body.serviceId).toBe(id);expect(body.passwordPresent).toBe(true);expect(body.volumeMarker).toBe(id);return; - } - throw new Error('Installed CLI fixture endpoint missing'); - } - private identify(installation: string): void { - const child=JSON.parse(readFileSync(join(installation,'.cordis/launch/child.json'),'utf8')),file=statSync(join(installation,'.cordis/worker')),running=statSync('/proc/'+child.pid+'/exe'); - expect(file.dev).toBe(running.dev);expect(file.ino).toBe(running.ino);expect(file.uid).toBe(process.getuid!());this.worker={pid:child.pid,dev:running.dev,ino:running.ino}; - } - private signal(signal: NodeJS.Signals): void {if(!this.worker)return;try{const current=statSync('/proc/'+this.worker.pid+'/exe');if(current.dev===this.worker.dev&¤t.ino===this.worker.ino)process.kill(this.worker.pid,signal);}catch{}} async git():Promise{await this.program('git');} + async gitBuild():Promise{await this.program('build');} + async program(source:'npm'|'git'|'build'='npm',registeredOnly=false): Promise { const fixture=new PluginInstallationFixture(),program=new CliProgramFixture(); try { @@ -402,22 +253,36 @@ class InstalledPluginCliTests { } finally {if(!this.worker){try{this.identify(fixture.installation);}catch{}}this.signal('SIGTERM');if(this.worker){const deadline=Date.now()+15000;while(existsSync(join(fixture.installation,'.cordis/writer.lock'))&&Date.now()void):void{setTimeout(done,50);} - private run(program:CliProgramFixture,args:string[],code?:number):ReturnType{const output=program.run(['plugin',...args,'--json']);expect(output.stderr).toBe('');if(code!==undefined)expect(output.status).toBe(code);else expect([0,3]).toContain(output.status!);expect(output.stdout).not.toContain('NEVER_INHERIT');return PluginManagementSchemas.envelope.parse(JSON.parse(output.stdout));} - private async settled(program:CliProgramFixture,installation:string,input:PluginInstallResult):Promise{let result=input;const until=Date.now()+180000;while(result.state==='pending'&&Date.now(){const{LiveProbeFixture}=await import('@treeseed-cordis/deployment/testing/live'),fixture=await new LiveProbeFixture().prepareLive(),program=new CliProgramFixture(),path=process.env.PATH,token=process.env.NPM_TOKEN;try{await fixture.startRegistry();const configuration=join(program.cwd,'live.json');writeFileSync(configuration,JSON.stringify(fixture.configuration));const planned=program.run(['platform','plan','--file',configuration,'--json']);expect(planned.status).toBe(0);const plan=ServiceSchemas.plan.parse(JSON.parse(planned.stdout).result),tools=join(program.root,'docker-only');mkdirSync(tools);let docker:string|undefined;for(const dir of(path??'').split(':'))if(existsSync(join(dir,'docker'))){docker=join(dir,'docker');break;}expect(docker).toBeDefined();symlinkSync(docker!,join(tools,'docker'));process.env.PATH=tools;process.env.NPM_TOKEN='NEVER_INHERIT';const{ComposeDeploymentHost,ServiceWorkerOwnerClient}=await import('@treeseed-cordis/deployment'),host=new ComposeDeploymentHost(),owner=new ServiceWorkerOwnerClient({installationDirectory:fixture.installation}),accepted=await host.apply(plan.id,plan.digest,{installationDirectory:fixture.installation,idempotencyKey:'cli-live-base'});let application=await owner.application(accepted.id);const until=Date.now()+180000;while(!['completed','failed'].includes(application.phase)&&Date.now()(),files=new Map();for(const binding of before.resources){const receipt=store.resources().find(LivePluginCliTests.digest.bind(null,binding.receiptDigest))!;physical.set(receipt.resourceId,LivePluginCliTests.physical(receipt));if(receipt.path&&existsSync(join(fixture.installation,receipt.path)))files.set(receipt.path,createHash('sha256').update(readFileSync(join(fixture.installation,receipt.path))).digest('hex'));}const child=JSON.parse(readFileSync(join(fixture.installation,'.cordis/launch/child.json'),'utf8')),provider=plan.composition.services[2]!.readiness,config=join(program.cwd,'probe.json');writeFileSync(config,JSON.stringify(provider.configuration));const args=['install',fixture.source(),'--installation',fixture.installation,'--target','deployment','--service','instance-2','--provider','fixture-live.next','--configuration',config,'--idempotency-key','cli-live-install'],live=await this.settled(program,fixture.installation,PluginManagementSchemas.result.parse(this.run(program,args).result));expect(live.state).toBe('active');expect(live.applicationId).not.toBeNull();expect(live.acknowledgements.length).toBeGreaterThan(0);for(const binding of store.current()!.resources){const receipt=store.resources().find(LivePluginCliTests.digest.bind(null,binding.receiptDigest))!;expect(LivePluginCliTests.physical(receipt)).toBe(physical.get(receipt.resourceId)!);}for(const[file,hash]of files)expect(createHash('sha256').update(readFileSync(join(fixture.installation,file))).digest('hex')).toBe(hash);expect((await owner.status()).services.find(LivePluginCliTests.next)).toBeDefined();renameSync(fixture.bundle,join(fixture.root,'source-bundle-offline'));await fixture.stopRegistry();expect(PluginManagementSchemas.result.parse(this.run(program,args,0).result)).toEqual(live);const select=['select','probe.service','fixture-live.next','--installation',fixture.installation,'--target','deployment','--service','instance-2','--configuration',config,'--idempotency-key','cli-live-current'],current=await this.settled(program,fixture.installation,PluginManagementSchemas.result.parse(this.run(program,select).result));expect(current.state).toBe('already-active');const count=store.records().length;select[2]='fixture-live.old';select[select.length-1]='cli-unsafe-reverse';const unsafe=await this.settled(program,fixture.installation,PluginManagementSchemas.result.parse(this.run(program,select).result));expect(unsafe.state).toBe('prepared');expect(unsafe.nextActions).toContain('review-restart');expect(unsafe.applicationId).toBeNull();expect(store.records().length).toBe(count);expect((await owner.status()).services.find(LivePluginCliTests.next)).toBeDefined();expect(JSON.parse(readFileSync(join(fixture.installation,'.cordis/launch/child.json'),'utf8')).pid).toBe(child.pid);expect(fixture.authorization).toBe(false);}finally{process.env.PATH=path;if(token===undefined)delete process.env.NPM_TOKEN;else process.env.NPM_TOKEN=token;await fixture.dispose();program.close();}} } -const livePlugins=new LivePluginCliTests(); -test('smoke: built CLI installs and selects a qualified live probe with offline exact replay and unsafe reverse refusal',livePlugins.program.bind(livePlugins),420000); +const suite=new CliTestsAcceptance(); +const compositionSuite=new CompositionProgramTestsAcceptance(); +const uiOutput=new UiOutputProgramTestsAcceptance(); +const extensiblePlan=new ExtensiblePlanProgramTestsAcceptance(); +const installedPrograms=new InstalledProgramTestsAcceptance(); +const installedPlugins=new InstalledPluginCliTestsAcceptance(); +const livePlugins=new LivePluginCliTestsAcceptance(); +test("acceptance: standalone identity from spaces, symlink, foreign cwd and closed stdin",suite.identity.bind(suite)); +test("acceptance: doctor JSON and no filesystem/container mutation or cwd autoload",suite.diagnosis.bind(suite)); +test("acceptance: wrong engine, missing Docker, foreign installation and occupied port",suite.blockers.bind(suite)); +test("acceptance: invalid arguments and unsupported commands are actionable",suite.invalidProgram.bind(suite)); +test("acceptance: plan_changes_no_runtime using real prepared artifacts",suite.plan_changes_no_runtime.bind(suite)); +test("acceptance: invalid and duplicate planning inputs never echo secrets or claim saved state",suite.invalidPlans.bind(suite)); +test("acceptance: plan new installation directory with spaces and Unicode",suite.installationPaths.bind(suite)); +test("acceptance: compiled_cli_uses_cordis_and_selected_deployment_provider without Node Bun or cwd modules",compositionSuite.selected.bind(compositionSuite)); +test("acceptance: installed provider byte and dependency failures block before import",compositionSuite.rejected.bind(compositionSuite)); +test("acceptance: provider constructors and failed disposal cannot report successful CLI completion",compositionSuite.lifecycleFailures.bind(compositionSuite)); +test("acceptance: prepared-provider metadata scope ownership and closure are checked before activation",compositionSuite.metadataFailures.bind(compositionSuite)); +test("acceptance: installed commands validate arguments before contacting worker",suite.installedArguments.bind(suite)); +test("acceptance: built CLI renders structured diagnostic models through reusable terminal providers",uiOutput.human.bind(uiOutput)); +test("acceptance: standalone CLI plans genuine five-instance format3 bundle deterministically without runtime effects",extensiblePlan.plan.bind(extensiblePlan),60000); +test("acceptance: standalone CLI applies and observes five format3 services through its default durable host", installedPrograms.installed.bind(installedPrograms), 240000); +test("acceptance: built plugin commands validate arguments with safe SDK JSON envelopes and do not create missing installations", suite.pluginArguments.bind(suite),30000); +test("acceptance: built CLI installs two typed plugin instances with retained credentials and safe UI readback using exact public fixtures",installedPlugins.program.bind(installedPlugins,'npm',false),420000); +test("acceptance: built CLI installs packaged HTTPS Git with managed helpers",installedPlugins.git.bind(installedPlugins),420000); +test("acceptance: built CLI installs approved source-only HTTPS Git with managed helpers",installedPlugins.gitBuild.bind(installedPlugins),420000); +test("acceptance: built CLI selects registered packaged Git instances with sources stopped",installedPlugins.program.bind(installedPlugins,'git',true),420000); +test("acceptance: built CLI selects registered built Git instances with sources stopped",installedPlugins.program.bind(installedPlugins,'build',true),420000); +test("acceptance: built CLI installs and selects a qualified live probe with offline exact replay and unsafe reverse refusal",livePlugins.program.bind(livePlugins),420000); diff --git a/tests/integration/cli-program.test.ts b/tests/integration/cli-program.test.ts new file mode 100644 index 0000000..bf2a970 --- /dev/null +++ b/tests/integration/cli-program.test.ts @@ -0,0 +1,24 @@ +import { test, expect } from 'bun:test'; +import { mkdtempSync, rmSync, mkdirSync, copyFileSync, symlinkSync, writeFileSync, readdirSync, existsSync, readFileSync, statSync, chmodSync, renameSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join, resolve, dirname } from 'node:path'; +import { spawnSync, type SpawnSyncReturns } from 'node:child_process'; +import { createServer, type Server } from 'node:net'; +import { createHash } from 'node:crypto'; +import { PreparedReleaseFixture } from '@treeseed-cordis/deployment/testing'; +import { CliApplication } from '../../src/main.js'; +import { ServiceRuntimeFixture } from '@treeseed-cordis/deployment/testing/runtime'; +import { ServiceWorkerBuild, ServicePlanStore } from '@treeseed-cordis/deployment'; +import { PluginInstallationFixture } from '@treeseed-cordis/deployment/testing/plugins'; +import { CanonicalJson, ServiceSchemas, PlatformSchemas, PluginManagementSchemas, type PluginInstallResult, type ServiceResourceReceipt, type CliEnvelope } from '@treeseed-cordis/sdk'; +import {CliProgramFixture,CliTests,ProviderFixture,ExtensiblePlanProgramTests,InstalledProgramTests,InstalledPluginCliTests,LivePluginCliTests} from '../support/cli-program-test-support.js'; +class CliTestsIntegration extends CliTests { + + async defaultComposition():Promise { + const root=mkdtempSync(join(tmpdir(),'cordis-default-')); + try {const result=await new CliApplication().execute(['doctor','--installation',join(root,'new'),'--port',String(await this.port())]);expect(result.envelope.result).not.toBeNull();expect(result.envelope.command).toBe('doctor');expect(result.envelope.result&&'host' in result.envelope.result).toBe(true);expect(readdirSync(root)).toEqual([]);} + finally {rmSync(root,{recursive:true,force:true});} + } +} +const suite=new CliTestsIntegration(); +test('integration: default CLI construction uses real packaged diagnostics',suite.defaultComposition.bind(suite)); diff --git a/tests/integration/command-classification.test.ts b/tests/integration/command-classification.test.ts new file mode 100644 index 0000000..abd887b --- /dev/null +++ b/tests/integration/command-classification.test.ts @@ -0,0 +1,25 @@ +import { test, expect } from 'bun:test'; +import { mkdtempSync, rmSync, mkdirSync, copyFileSync, symlinkSync, writeFileSync, readdirSync, existsSync, readFileSync, statSync, chmodSync, renameSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join, resolve, dirname } from 'node:path'; +import { spawnSync, type SpawnSyncReturns } from 'node:child_process'; +import { createServer, type Server } from 'node:net'; +import { createHash } from 'node:crypto'; +import { PreparedReleaseFixture } from '@treeseed-cordis/deployment/testing'; +import { CliApplication } from '../../src/main.js'; +import { ServiceRuntimeFixture } from '@treeseed-cordis/deployment/testing/runtime'; +import { ServiceWorkerBuild, ServicePlanStore } from '@treeseed-cordis/deployment'; +import { PluginInstallationFixture } from '@treeseed-cordis/deployment/testing/plugins'; +import { CanonicalJson, ServiceSchemas, PlatformSchemas, PluginManagementSchemas, type PluginInstallResult, type ServiceResourceReceipt, type CliEnvelope } from '@treeseed-cordis/sdk'; +import {CliProgramFixture,CliTests,ProviderFixture,ExtensiblePlanProgramTests,InstalledProgramTests,InstalledPluginCliTests,LivePluginCliTests} from '../support/cli-program-test-support.js'; +class CliTestsIntegration extends CliTests { + + async parser():Promise { + const app=new CliApplication(); + expect((await app.execute(['--help'])).code).toBe(0);expect((await app.execute(['--version'])).code).toBe(0); + expect((await app.execute(['doctor','--json','--port','bad'])).code).toBe(2); + expect((await app.execute(['apply'])).code).toBe(3); + } +} +const suite=new CliTestsIntegration(); +test("integration: parser help, version and error classifications",suite.parser.bind(suite)); diff --git a/tests/layout-migration.json b/tests/layout-migration.json new file mode 100644 index 0000000..097f673 --- /dev/null +++ b/tests/layout-migration.json @@ -0,0 +1,34 @@ +{ + "repository": "cli", + "baseline": "c3287671197c589f83983ad89bbd4b4abc7847ed", + "registrationCount": 24, + "entries": [ + {"from":"tests/cli-program.test.ts","to":"tests/acceptance/cli-program.test.ts","tier":"acceptance","name":"smoke: standalone identity from spaces, symlink, foreign cwd and closed stdin","currentName":"acceptance: standalone identity from spaces, symlink, foreign cwd and closed stdin"}, + {"from":"tests/cli-program.test.ts","to":"tests/acceptance/cli-program.test.ts","tier":"acceptance","name":"smoke: doctor JSON and no filesystem/container mutation or cwd autoload","currentName":"acceptance: doctor JSON and no filesystem/container mutation or cwd autoload"}, + {"from":"tests/cli-program.test.ts","to":"tests/acceptance/cli-program.test.ts","tier":"acceptance","name":"smoke: wrong engine, missing Docker, foreign installation and occupied port","currentName":"acceptance: wrong engine, missing Docker, foreign installation and occupied port"}, + {"from":"tests/cli-program.test.ts","to":"tests/acceptance/cli-program.test.ts","tier":"acceptance","name":"smoke: invalid arguments and unsupported commands are actionable","currentName":"acceptance: invalid arguments and unsupported commands are actionable"}, + {"from":"tests/cli-program.test.ts","to":"tests/acceptance/cli-program.test.ts","tier":"acceptance","name":"smoke: plan_changes_no_runtime using real prepared artifacts","currentName":"acceptance: plan_changes_no_runtime using real prepared artifacts"}, + {"from":"tests/cli-program.test.ts","to":"tests/acceptance/cli-program.test.ts","tier":"acceptance","name":"smoke: invalid and duplicate planning inputs never echo secrets or claim saved state","currentName":"acceptance: invalid and duplicate planning inputs never echo secrets or claim saved state"}, + {"from":"tests/cli-program.test.ts","to":"tests/acceptance/cli-program.test.ts","tier":"acceptance","name":"smoke: plan new installation directory with spaces and Unicode","currentName":"acceptance: plan new installation directory with spaces and Unicode"}, + {"from":"tests/cli-program.test.ts","to":"tests/acceptance/cli-program.test.ts","tier":"acceptance","name":"smoke: compiled_cli_uses_cordis_and_selected_deployment_provider without Node Bun or cwd modules","currentName":"acceptance: compiled_cli_uses_cordis_and_selected_deployment_provider without Node Bun or cwd modules"}, + {"from":"tests/cli-program.test.ts","to":"tests/acceptance/cli-program.test.ts","tier":"acceptance","name":"smoke: installed provider byte and dependency failures block before import","currentName":"acceptance: installed provider byte and dependency failures block before import"}, + {"from":"tests/cli-program.test.ts","to":"tests/acceptance/cli-program.test.ts","tier":"acceptance","name":"smoke: provider constructors and failed disposal cannot report successful CLI completion","currentName":"acceptance: provider constructors and failed disposal cannot report successful CLI completion"}, + {"from":"tests/cli-program.test.ts","to":"tests/acceptance/cli-program.test.ts","tier":"acceptance","name":"smoke: prepared-provider metadata scope ownership and closure are checked before activation","currentName":"acceptance: prepared-provider metadata scope ownership and closure are checked before activation"}, + {"from":"tests/cli-program.test.ts","to":"tests/acceptance/cli-program.test.ts","tier":"acceptance","name":"smoke: installed commands validate arguments before contacting worker","currentName":"acceptance: installed commands validate arguments before contacting worker"}, + {"from":"tests/cli-program.test.ts","to":"tests/acceptance/cli-program.test.ts","tier":"acceptance","name":"smoke: built CLI renders structured diagnostic models through reusable terminal providers","currentName":"acceptance: built CLI renders structured diagnostic models through reusable terminal providers"}, + {"from":"tests/cli-program.test.ts","to":"tests/acceptance/cli-program.test.ts","tier":"acceptance","name":"smoke: standalone CLI plans genuine five-instance format3 bundle deterministically without runtime effects","currentName":"acceptance: standalone CLI plans genuine five-instance format3 bundle deterministically without runtime effects"}, + {"from":"tests/cli-program.test.ts","to":"tests/acceptance/cli-program.test.ts","tier":"acceptance","name":"smoke: standalone CLI applies and observes five format3 services through its default durable host","currentName":"acceptance: standalone CLI applies and observes five format3 services through its default durable host"}, + {"from":"tests/cli-program.test.ts","to":"tests/acceptance/cli-program.test.ts","tier":"acceptance","name":"smoke: built plugin commands validate arguments with safe SDK JSON envelopes and do not create missing installations","currentName":"acceptance: built plugin commands validate arguments with safe SDK JSON envelopes and do not create missing installations"}, + {"from":"tests/cli-program.test.ts","to":"tests/acceptance/cli-program.test.ts","tier":"acceptance","name":"smoke: built CLI installs two typed plugin instances with retained credentials and safe UI readback using exact public fixtures","currentName":"acceptance: built CLI installs two typed plugin instances with retained credentials and safe UI readback using exact public fixtures"}, + {"from":"tests/cli-program.test.ts","to":"tests/acceptance/cli-program.test.ts","tier":"acceptance","name":"smoke: built CLI installs packaged HTTPS Git with managed helpers","currentName":"acceptance: built CLI installs packaged HTTPS Git with managed helpers"}, + {"from":"tests/cli-program.test.ts","to":"tests/acceptance/cli-program.test.ts","tier":"acceptance","name":"smoke: built CLI installs approved source-only HTTPS Git with managed helpers","currentName":"acceptance: built CLI installs approved source-only HTTPS Git with managed helpers"}, + {"from":"tests/cli-program.test.ts","to":"tests/acceptance/cli-program.test.ts","tier":"acceptance","name":"smoke: built CLI selects registered packaged Git instances with sources stopped","currentName":"acceptance: built CLI selects registered packaged Git instances with sources stopped"}, + {"from":"tests/cli-program.test.ts","to":"tests/acceptance/cli-program.test.ts","tier":"acceptance","name":"smoke: built CLI selects registered built Git instances with sources stopped","currentName":"acceptance: built CLI selects registered built Git instances with sources stopped"}, + {"from":"tests/cli-program.test.ts","to":"tests/acceptance/cli-program.test.ts","tier":"acceptance","name":"smoke: built CLI installs and selects a qualified live probe with offline exact replay and unsafe reverse refusal","currentName":"acceptance: built CLI installs and selects a qualified live probe with offline exact replay and unsafe reverse refusal"}, + {"from":"tests/cli-program.test.ts","to":"tests/integration/cli-program.test.ts","tier":"integration","name":"integration: default CLI construction uses real packaged diagnostics","currentName":"integration: default CLI construction uses real packaged diagnostics"}, + {"from":"tests/cli-program.test.ts","to":"tests/integration/command-classification.test.ts","tier":"integration","name":"unit: parser help, version and error classifications","currentName":"integration: parser help, version and error classifications"} + ], + "mapping": { + "tests/cli-program.test.ts": ["tests/acceptance/cli-program.test.ts", "tests/integration/cli-program.test.ts", "tests/unit/cli-program.test.ts"] + } +} diff --git a/tests/regression/test-layout.test.ts b/tests/regression/test-layout.test.ts new file mode 100644 index 0000000..7b6cde1 --- /dev/null +++ b/tests/regression/test-layout.test.ts @@ -0,0 +1,19 @@ +import {test,expect} from 'bun:test'; +import {mkdtempSync,mkdirSync,writeFileSync,rmSync} from 'node:fs'; +import {tmpdir} from 'node:os'; +import {join,dirname} from 'node:path'; +import {TestLayout} from '../../.agents/skills/release-workflow/scripts/test-layout.js'; +class TestLayoutRegression { + private check(path:string,source:string):string[]{const root=mkdtempSync(join(tmpdir(),'test-layout-'));try{mkdirSync(dirname(join(root,path)),{recursive:true});writeFileSync(join(root,path),source);return new TestLayout().inspect(root);}finally{rmSync(root,{recursive:true,force:true});}} + valid():void{for(const tier of ['acceptance','regression','integration','unit'])expect(this.check('tests/'+tier+'/provider-selection.test.ts',"import {test} from 'bun:test';class Cases {run():void{}}const suite=new Cases();test('case',suite.run.bind(suite));")).toEqual([]);} + misplaced():void{for(const path of ['tests/flat.test.ts','src/provider.test.ts','tests/support/hidden.ts','tests/unit/hidden.ts'])expect(this.check(path,"import {test as scenario} from 'bun:test';class Cases {run():void{}}const suite=new Cases();scenario('case',suite.run.bind(suite));").length).toBeGreaterThan(0);} + phases():void{for(const name of ['task3d-agent','phase-4-provider','task-3c'])expect(this.check('tests/unit/'+name+'.test.ts',"import {test} from 'bun:test';class Cases {run():void{}}const suite=new Cases();test('case',suite.run.bind(suite));").length).toBeGreaterThan(0);} + discovery():void{const root=mkdtempSync(join(tmpdir(),'test-discovery-'));try{mkdirSync(join(root,'tests/unit'),{recursive:true});writeFileSync(join(root,'tests/unit/value.test.ts'),"import {test} from 'bun:test';class Cases {run():void{}}const suite=new Cases();test('case',suite.run.bind(suite));");const scripts:Record={};for(const tier of ['acceptance','regression','integration','unit'])scripts['test:'+tier]='bun test ./tests/'+tier;writeFileSync(join(root,'package.json'),JSON.stringify({scripts}));expect(new TestLayout().inspect(root)).toEqual([]);scripts['test:unit']='bun test ./tests --test-name-pattern ^unit:';writeFileSync(join(root,'package.json'),JSON.stringify({scripts}));expect(new TestLayout().inspect(root).length).toBeGreaterThan(0);}finally{rmSync(root,{recursive:true,force:true});}} + support():void{expect(this.check('tests/support/provider.ts','export class Fixture {run():void{}}')).toEqual([]);expect(this.check('tests/support/hidden.ts',"import * as cases from 'node:test';class Cases {run():void{}}const suite=new Cases();cases.test('case',suite.run.bind(suite));").length).toBeGreaterThan(0);} +} +const suite=new TestLayoutRegression(); +test('regression: every required test tier accepts capability names',suite.valid.bind(suite)); +test('regression: misplaced and aliased hidden test registrations fail layout verification',suite.misplaced.bind(suite)); +test('regression: task and phase filenames cannot become permanent suites',suite.phases.bind(suite)); +test('regression: shared fixture classes are permitted without hidden namespace tests',suite.support.bind(suite)); +test('regression: tier commands cannot silently omit files through name filters',suite.discovery.bind(suite)); diff --git a/tests/support/cli-program-test-support.ts b/tests/support/cli-program-test-support.ts new file mode 100644 index 0000000..05ca1f8 --- /dev/null +++ b/tests/support/cli-program-test-support.ts @@ -0,0 +1,160 @@ +import { test, expect } from 'bun:test'; +import { mkdtempSync, rmSync, mkdirSync, copyFileSync, symlinkSync, writeFileSync, readdirSync, existsSync, readFileSync, statSync, chmodSync, renameSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join, resolve, dirname } from 'node:path'; +import { spawnSync, type SpawnSyncReturns } from 'node:child_process'; +import { createServer, type Server } from 'node:net'; +import { createHash } from 'node:crypto'; +import { PreparedReleaseFixture } from '@treeseed-cordis/deployment/testing'; +import { CliApplication } from '../../src/main.js'; +import { ServiceRuntimeFixture } from '@treeseed-cordis/deployment/testing/runtime'; +import { ServiceWorkerBuild, ServicePlanStore } from '@treeseed-cordis/deployment'; +import { PluginInstallationFixture } from '@treeseed-cordis/deployment/testing/plugins'; +import { CanonicalJson, ServiceSchemas, PlatformSchemas, PluginManagementSchemas, type PluginInstallResult, type ServiceResourceReceipt, type CliEnvelope } from '@treeseed-cordis/sdk'; + +export class CliProgramFixture { + readonly root=mkdtempSync(join(tmpdir(),'cordis cli ')); + readonly cwd=join(this.root,'foreign cwd'); + readonly binary=join(this.root,'path with spaces','trsd'); + readonly symlink=join(this.root,'trsd-link'); + constructor() {mkdirSync(this.cwd);mkdirSync(join(this.root,'path with spaces'));copyFileSync(resolve('dist/trsd-linux-x64'),this.binary);symlinkSync(this.binary,this.symlink);} + run(args:string[],environment:NodeJS.ProcessEnv={},binary=this.binary):SpawnSyncReturns {return spawnSync(binary,args,{cwd:this.cwd,env:{...process.env,...environment},encoding:'utf8',stdio:['ignore','pipe','pipe'],timeout:15000});} + close():void {rmSync(this.root,{recursive:true,force:true});} +} +export class CliTests { + server:Server|null=null; + resolvePort:((port:number)=>void)|null=null; + listen(resolvePort:(port:number)=>void):void {this.resolvePort=resolvePort;this.server=createServer();this.server.listen(0,'127.0.0.1',this.listening.bind(this));} + listening():void {const address=this.server!.address();if(address&&typeof address==='object')this.resolvePort!(address.port);} + async port(occupied=false):Promise {const port=await new Promise(this.listen.bind(this));if(!occupied)this.server!.close();return port;} + snapshot():string { + const rows:string[]=[];for(const args of [['ps','-aq'],['image','ls','--no-trunc','-q'],['volume','ls','-q'],['network','ls','-q']]){const result=spawnSync('docker',['--context','default',...args],{encoding:'utf8'});expect(result.status).toBe(0);rows.push(result.stdout.split('\n').sort().join('\n'));}return JSON.stringify(rows); + } +} + +export class ProviderFixture { + readonly root=mkdtempSync(join(tmpdir(),'cordis installed provider Ω ')); + tree=join(this.root,'prepared package'); + readonly marker=join(this.root,'live-effect'); + readonly importMarker=join(this.root,'module-imported'); + get file():string{return join(this.tree,'composition.json');} + readonly declaration={id:'fixture-one',pluginId:'fixture',capability:'deployment',contractVersion:'0.3.0-cordis.1',target:'cli',provide:['deployment'],inject:[]}; + digest=''; + constructor(readonly label='one') {} + static sha(value:Uint8Array|string):string {return createHash('sha256').update(value).digest('hex');} + copy(source:string,destination:string):void { + mkdirSync(destination,{recursive:true}); + for(const entry of readdirSync(source,{withFileTypes:true})) { + const from=join(source,entry.name),to=join(destination,entry.name); + if(entry.isDirectory())this.copy(from,to);else if(entry.isFile())copyFileSync(from,to);else throw new Error('Fixture dependency is not a regular package tree'); + } + } + async prepare():Promise { + mkdirSync(this.tree);this.declaration.id='fixture-'+this.label; + // These are real pinned installed third-party packages, copied as fixture bytes, not mocked modules. + for(const name of ['cordis','cosmokit','@standard-schema/spec'])this.copy(resolve('node_modules',name),join(this.tree,'node_modules',name)); + const helper=join(this.tree,'node_modules/@fixture/helper');mkdirSync(helper,{recursive:true}); + writeFileSync(join(helper,'package.json'),JSON.stringify({name:'@fixture/helper',version:'1.0.0',type:'module',exports:'./index.js'})); + const helperSource=join(this.root,'helper.ts');writeFileSync(helperSource,'export class Helper {static label():string{return "ordinary-dependency";}}'); + let build=spawnSync('bun',['build',helperSource,'--target=node','--outfile',join(helper,'index.js')],{encoding:'utf8'});expect(build.status).toBe(0); + const source=join(this.root,'provider.ts'); + writeFileSync(source,`import {Context} from 'cordis';import {Helper} from '@fixture/helper';import {writeFileSync,unlinkSync} from 'node:fs'; +class ImportCanary {static run(){writeFileSync(${JSON.stringify(this.importMarker)},'imported');}}ImportCanary.run(); +class Configuration {static validate(value:unknown){return value&&typeof value==='object'&&Object.keys(value).length===0?{value}:{issues:[{message:'Invalid configuration'}]};}} +export class Provider { +static readonly declaration=${JSON.stringify(this.declaration)}; +static readonly provide=Provider.declaration.provide;static readonly inject=Provider.declaration.inject; +static readonly Config={'~standard':{version:1,vendor:'fixture',validate:Configuration.validate.bind(Configuration)}}; +constructor(readonly ctx:Context){if(!Context.is(ctx)||ctx.registry.get(Provider)!==ctx[Context.root].registry.get(Provider))throw new Error('not native shared graph');writeFileSync(${JSON.stringify(this.marker)},'active');ctx.provide('deployment',this);ctx.effect(this.effect.bind(this));if(${JSON.stringify(this.label)}==='constructor-fail')throw new Error('secret-constructor-canary');} +effect(){return this.close.bind(this);}close(){unlinkSync(${JSON.stringify(this.marker)});if(${JSON.stringify(this.label)}==='cleanup-fail')throw new Error('secret-cleanup-canary');} +async diagnose(){return {ready:true,host:{os:'fixture',architecture:'x64',distribution:'fixture',release:'1'},engine:{context:${JSON.stringify(this.label)},endpoint:null,serverVersion:null,serverArchitecture:null,composeVersion:null,cpus:null,memoryBytes:null},installation:{path:'/fixture',exists:false,freeBytes:null},port:8443,checks:[{id:'native-provider',status:'pass',code:'FIXTURE',message:Helper.label()}]};} +async plan(){throw new Error('fixture is diagnosis-only');}} +`); + build=spawnSync('bun',['build',source,'--target=node','--external','cordis','--external','@fixture/helper','--outfile',join(this.tree,'index.js')],{encoding:'utf8'});expect(build.status).toBe(0); + writeFileSync(join(this.tree,'package.json'),JSON.stringify({name:'@fixture/provider',version:'1.0.0',type:'module',exports:'./index.js',dependencies:{cordis:'4.0.0-rc.12','@fixture/helper':'1.0.0'},cordis:this.declaration})); + this.refresh();return this; + } + inventory(directory=this.tree,prefix=''):Array<{path:string;sha256:string}> { + const files:Array<{path:string;sha256:string}>=[]; + for(const entry of readdirSync(directory,{withFileTypes:true}).sort(ProviderFixture.order)){ + if(directory===this.tree&&entry.name==='composition.json')continue; + const path=join(directory,entry.name),relative=prefix+entry.name; + if(entry.isDirectory())files.push(...this.inventory(path,relative+'/'));else files.push({path:relative,sha256:ProviderFixture.sha(readFileSync(path))}); + } + return files; + } + static order(a:{name:string},b:{name:string}):number {return a.name.localeCompare(b.name);} + permissions(directory:string,locked:boolean):void {chmodSync(directory,0o700);for(const entry of readdirSync(directory,{withFileTypes:true})){const path=join(directory,entry.name);if(entry.isSymbolicLink())continue;if(entry.isDirectory())this.permissions(path,locked);else chmodSync(path,locked?0o400:0o600);}chmodSync(directory,locked?0o500:0o700);} + refresh():void { + this.permissions(this.tree,false); + const manifest={formatVersion:1,contractVersion:'0.3.0-cordis.1',target:'cli',scopeId:'local-cli',providerId:this.declaration.id,package:{name:'@fixture/provider',version:'1.0.0'},entry:'index.js',configuration:{},files:this.inventory()}; + writeFileSync(this.file,JSON.stringify(manifest));this.address();this.permissions(this.tree,true); + } + address():void {this.digest=ProviderFixture.sha(readFileSync(this.file));const destination=join(this.root,this.digest);if(destination!==this.tree){renameSync(this.tree,destination);this.tree=destination;}} + run(extra:string[]=[]):SpawnSyncReturns { + const binary=join(this.root,'trsd link');if(!existsSync(binary))symlinkSync(resolve('dist/trsd-linux-x64'),binary); + return spawnSync(binary,['doctor','--json','--composition',this.file,'--composition-digest',this.digest,...extra],{encoding:'utf8',cwd:this.root,env:{...process.env,PATH:''},timeout:15000}); + } + close():void {this.permissions(this.tree,false);rmSync(this.root,{recursive:true,force:true});} +} + +export class ExtensiblePlanProgramTests extends CliTests { + protected listeners():string{const rows:string[]=[];for(const path of ['/proc/net/tcp','/proc/net/tcp6'])for(const line of readFileSync(path,'utf8').trim().split('\n').slice(1)){const fields=line.trim().split(/\s+/);if(fields[3]==='0A')rows.push(fields[1]!);}return rows.sort().join('\n');} +} + + +export class InstalledProgramTests { + protected worker: { pid: number; dev: number; ino: number } | null = null; + protected json(program: CliProgramFixture, args: string[], environment: NodeJS.ProcessEnv = {}): ReturnType { const result = program.run(args, environment, program.symlink); if (result.status !== 0) throw new Error('Compiled installed command failed: ' + result.stdout + result.stderr); expect(result.stderr).toBe(''); return ServiceSchemas.envelope.parse(JSON.parse(result.stdout)); } + protected signal(signal: NodeJS.Signals): void { if (!this.worker) return; try { const current = statSync('/proc/' + this.worker.pid + '/exe'); if (current.dev === this.worker.dev && current.ino === this.worker.ino) process.kill(this.worker.pid, signal); } catch { } } + protected static delay(done: (value: void) => void): void { setTimeout(done, 100); } +} + +export class InstalledPluginCliTests { + protected worker: { pid: number; dev: number; ino: number } | null = null; + protected static delay(done: (value: void) => void): void { setTimeout(done, 50); } + protected static hash(bytes: Uint8Array): string { return createHash('sha256').update(bytes).digest('hex'); } + protected request(program: CliProgramFixture, args: string[], expected?: number): ReturnType { + const result=program.run(['plugin',...args,'--json']); + if(result.status===null)throw new Error('Built CLI terminated without an exit status'); + if(expected!==undefined&&result.status!==expected){const envelope=PluginManagementSchemas.envelope.parse(JSON.parse(result.stdout));const codes:string[]=[];for(const blocker of envelope.blockers)codes.push(blocker.code);throw new Error('CLI_PLUGIN_READBACK:'+args[0]+':'+result.status+':'+envelope.problem?.code+':'+codes.join(','));} + if(expected===undefined)expect([0,3]).toContain(result.status);else expect(result.status).toBe(expected); + expect(result.stderr).toBe('');expect(result.stdout).not.toContain('NEVER_INHERIT');return PluginManagementSchemas.envelope.parse(JSON.parse(result.stdout)); + } + protected async settled(program: CliProgramFixture, installation: string, input: PluginInstallResult): Promise { + let result=input;const deadline=Date.now()+150000; + while(result.state==='pending'&&Date.now() { + const result=new Map();const all=store.resources(); + for(const receipt of all)for(const binding of store.current()!.resources)if(receipt.resourceId===binding.resourceId){const prior=result.get(receipt.resourceId);if(!prior||prior.references.length, files: Map): void { + const current=this.receipts(store); + for(const [id,receipt]of before){const next=current.get(id);expect(next).toBeDefined();expect(next!.actualId).toBe(receipt.actualId);expect(next!.creatingGeneration).toBe(receipt.creatingGeneration);expect(next!.definitionDigest).toBe(receipt.definitionDigest);expect(next!.contentDigest).toBe(receipt.contentDigest);expect(next!.path).toBe(receipt.path);} + for(const [path,digest]of files)expect(InstalledPluginCliTests.hash(readFileSync(join(store.path,path)))).toBe(digest); + } + protected async direct(store: ServicePlanStore, id: string): Promise { + for(const receipt of this.receipts(store).values())if(receipt.kind==='container'&&receipt.serviceId===id&&receipt.actualId){ + const inspected=spawnSync('docker',['container','inspect','--format','{{json .NetworkSettings.Networks}}',receipt.actualId],{encoding:'utf8'});expect(inspected.status).toBe(0);const networks=JSON.parse(inspected.stdout) as Record;let address:string|undefined;for(const network of Object.values(networks))address=network.IPAddress;expect(address).toBeDefined(); + const response=await fetch('http://'+address+':8080/'),body=await response.json() as {serviceId:string;passwordPresent:boolean;volumeMarker:string};expect(response.ok).toBe(true);expect(body.serviceId).toBe(id);expect(body.passwordPresent).toBe(true);expect(body.volumeMarker).toBe(id);return; + } + throw new Error('Installed CLI fixture endpoint missing'); + } + protected identify(installation: string): void { + const child=JSON.parse(readFileSync(join(installation,'.cordis/launch/child.json'),'utf8')),file=statSync(join(installation,'.cordis/worker')),running=statSync('/proc/'+child.pid+'/exe'); + expect(file.dev).toBe(running.dev);expect(file.ino).toBe(running.ino);expect(file.uid).toBe(process.getuid!());this.worker={pid:child.pid,dev:running.dev,ino:running.ino}; + } + protected signal(signal: NodeJS.Signals): void {if(!this.worker)return;try{const current=statSync('/proc/'+this.worker.pid+'/exe');if(current.dev===this.worker.dev&¤t.ino===this.worker.ino)process.kill(this.worker.pid,signal);}catch{}} +} + +export class LivePluginCliTests { + protected static pause(done:(value:void)=>void):void{setTimeout(done,50);} + protected run(program:CliProgramFixture,args:string[],code?:number):ReturnType{const output=program.run(['plugin',...args,'--json']);expect(output.stderr).toBe('');if(code!==undefined)expect(output.status).toBe(code);else expect([0,3]).toContain(output.status!);expect(output.stdout).not.toContain('NEVER_INHERIT');return PluginManagementSchemas.envelope.parse(JSON.parse(output.stdout));} + protected async settled(program:CliProgramFixture,installation:string,input:PluginInstallResult):Promise{let result=input;const until=Date.now()+180000;while(result.state==='pending'&&Date.now() Date: Tue, 27 Oct 2026 09:13:43 -0400 Subject: [PATCH 3/9] docs: keep shared behavioral contracts in their test tier (#325) --- tests/README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tests/README.md b/tests/README.md index 37d2d5e..2ec8442 100644 --- a/tests/README.md +++ b/tests/README.md @@ -32,3 +32,7 @@ checks, build, CI and assembled-artifact smoke evidence. Report missing prerequi explicitly. A green unit suite or policy prose alone is never sufficient. + +Assertion-bearing reusable acceptance contracts live in `tests/acceptance/`; shared support +contains fixture setup and program inputs. Moving a case must also move its shared behavioral +assertions, rather than hiding the scenario in a support base class. From a5e500d32f5fd3afcb392c4730c851baed499196 Mon Sep 17 00:00:00 2001 From: Adrian Webb Date: Tue, 27 Oct 2026 09:16:12 -0400 Subject: [PATCH 4/9] refactor: keep shared acceptance assertions in their tier (#325) --- tests/acceptance/cli-program-contract.ts | 141 ++++++++++++++++ tests/acceptance/cli-program.test.ts | 3 +- tests/integration/cli-program.test.ts | 2 +- .../command-classification.test.ts | 2 +- tests/layout-migration.json | 2 +- tests/support/cli-program-test-support.ts | 152 +----------------- 6 files changed, 153 insertions(+), 149 deletions(-) create mode 100644 tests/acceptance/cli-program-contract.ts diff --git a/tests/acceptance/cli-program-contract.ts b/tests/acceptance/cli-program-contract.ts new file mode 100644 index 0000000..ad91000 --- /dev/null +++ b/tests/acceptance/cli-program-contract.ts @@ -0,0 +1,141 @@ +import { expect } from 'bun:test'; +import { mkdtempSync, rmSync, mkdirSync, copyFileSync, symlinkSync, writeFileSync, readdirSync, existsSync, readFileSync, statSync, chmodSync, renameSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join, resolve, dirname } from 'node:path'; +import { spawnSync, type SpawnSyncReturns } from 'node:child_process'; +import { createServer, type Server } from 'node:net'; +import { createHash } from 'node:crypto'; +import { PreparedReleaseFixture } from '@treeseed-cordis/deployment/testing'; +import { CliApplication } from '../../src/main.js'; +import { ServiceRuntimeFixture } from '@treeseed-cordis/deployment/testing/runtime'; +import { ServiceWorkerBuild, ServicePlanStore } from '@treeseed-cordis/deployment'; +import { PluginInstallationFixture } from '@treeseed-cordis/deployment/testing/plugins'; +import { CanonicalJson, ServiceSchemas, PlatformSchemas, PluginManagementSchemas, type PluginInstallResult, type ServiceResourceReceipt, type CliEnvelope } from '@treeseed-cordis/sdk'; + +import {CliProgramFixture,CliNetworkFixture} from '../support/cli-program-test-support.js'; +export class CliTests extends CliNetworkFixture { +snapshot():string { + const rows:string[]=[];for(const args of [['ps','-aq'],['image','ls','--no-trunc','-q'],['volume','ls','-q'],['network','ls','-q']]){const result=spawnSync('docker',['--context','default',...args],{encoding:'utf8'});expect(result.status).toBe(0);rows.push(result.stdout.split('\n').sort().join('\n'));}return JSON.stringify(rows); + } +} +export class ProviderFixture { + readonly root=mkdtempSync(join(tmpdir(),'cordis installed provider Ω ')); + tree=join(this.root,'prepared package'); + readonly marker=join(this.root,'live-effect'); + readonly importMarker=join(this.root,'module-imported'); + get file():string{return join(this.tree,'composition.json');} + readonly declaration={id:'fixture-one',pluginId:'fixture',capability:'deployment',contractVersion:'0.3.0-cordis.1',target:'cli',provide:['deployment'],inject:[]}; + digest=''; + constructor(readonly label='one') {} + static sha(value:Uint8Array|string):string {return createHash('sha256').update(value).digest('hex');} + copy(source:string,destination:string):void { + mkdirSync(destination,{recursive:true}); + for(const entry of readdirSync(source,{withFileTypes:true})) { + const from=join(source,entry.name),to=join(destination,entry.name); + if(entry.isDirectory())this.copy(from,to);else if(entry.isFile())copyFileSync(from,to);else throw new Error('Fixture dependency is not a regular package tree'); + } + } + async prepare():Promise { + mkdirSync(this.tree);this.declaration.id='fixture-'+this.label; + // These are real pinned installed third-party packages, copied as fixture bytes, not mocked modules. + for(const name of ['cordis','cosmokit','@standard-schema/spec'])this.copy(resolve('node_modules',name),join(this.tree,'node_modules',name)); + const helper=join(this.tree,'node_modules/@fixture/helper');mkdirSync(helper,{recursive:true}); + writeFileSync(join(helper,'package.json'),JSON.stringify({name:'@fixture/helper',version:'1.0.0',type:'module',exports:'./index.js'})); + const helperSource=join(this.root,'helper.ts');writeFileSync(helperSource,'export class Helper {static label():string{return "ordinary-dependency";}}'); + let build=spawnSync('bun',['build',helperSource,'--target=node','--outfile',join(helper,'index.js')],{encoding:'utf8'});expect(build.status).toBe(0); + const source=join(this.root,'provider.ts'); + writeFileSync(source,`import {Context} from 'cordis';import {Helper} from '@fixture/helper';import {writeFileSync,unlinkSync} from 'node:fs'; +class ImportCanary {static run(){writeFileSync(${JSON.stringify(this.importMarker)},'imported');}}ImportCanary.run(); +class Configuration {static validate(value:unknown){return value&&typeof value==='object'&&Object.keys(value).length===0?{value}:{issues:[{message:'Invalid configuration'}]};}} +export class Provider { +static readonly declaration=${JSON.stringify(this.declaration)}; +static readonly provide=Provider.declaration.provide;static readonly inject=Provider.declaration.inject; +static readonly Config={'~standard':{version:1,vendor:'fixture',validate:Configuration.validate.bind(Configuration)}}; +constructor(readonly ctx:Context){if(!Context.is(ctx)||ctx.registry.get(Provider)!==ctx[Context.root].registry.get(Provider))throw new Error('not native shared graph');writeFileSync(${JSON.stringify(this.marker)},'active');ctx.provide('deployment',this);ctx.effect(this.effect.bind(this));if(${JSON.stringify(this.label)}==='constructor-fail')throw new Error('secret-constructor-canary');} +effect(){return this.close.bind(this);}close(){unlinkSync(${JSON.stringify(this.marker)});if(${JSON.stringify(this.label)}==='cleanup-fail')throw new Error('secret-cleanup-canary');} +async diagnose(){return {ready:true,host:{os:'fixture',architecture:'x64',distribution:'fixture',release:'1'},engine:{context:${JSON.stringify(this.label)},endpoint:null,serverVersion:null,serverArchitecture:null,composeVersion:null,cpus:null,memoryBytes:null},installation:{path:'/fixture',exists:false,freeBytes:null},port:8443,checks:[{id:'native-provider',status:'pass',code:'FIXTURE',message:Helper.label()}]};} +async plan(){throw new Error('fixture is diagnosis-only');}} +`); + build=spawnSync('bun',['build',source,'--target=node','--external','cordis','--external','@fixture/helper','--outfile',join(this.tree,'index.js')],{encoding:'utf8'});expect(build.status).toBe(0); + writeFileSync(join(this.tree,'package.json'),JSON.stringify({name:'@fixture/provider',version:'1.0.0',type:'module',exports:'./index.js',dependencies:{cordis:'4.0.0-rc.12','@fixture/helper':'1.0.0'},cordis:this.declaration})); + this.refresh();return this; + } + inventory(directory=this.tree,prefix=''):Array<{path:string;sha256:string}> { + const files:Array<{path:string;sha256:string}>=[]; + for(const entry of readdirSync(directory,{withFileTypes:true}).sort(ProviderFixture.order)){ + if(directory===this.tree&&entry.name==='composition.json')continue; + const path=join(directory,entry.name),relative=prefix+entry.name; + if(entry.isDirectory())files.push(...this.inventory(path,relative+'/'));else files.push({path:relative,sha256:ProviderFixture.sha(readFileSync(path))}); + } + return files; + } + static order(a:{name:string},b:{name:string}):number {return a.name.localeCompare(b.name);} + permissions(directory:string,locked:boolean):void {chmodSync(directory,0o700);for(const entry of readdirSync(directory,{withFileTypes:true})){const path=join(directory,entry.name);if(entry.isSymbolicLink())continue;if(entry.isDirectory())this.permissions(path,locked);else chmodSync(path,locked?0o400:0o600);}chmodSync(directory,locked?0o500:0o700);} + refresh():void { + this.permissions(this.tree,false); + const manifest={formatVersion:1,contractVersion:'0.3.0-cordis.1',target:'cli',scopeId:'local-cli',providerId:this.declaration.id,package:{name:'@fixture/provider',version:'1.0.0'},entry:'index.js',configuration:{},files:this.inventory()}; + writeFileSync(this.file,JSON.stringify(manifest));this.address();this.permissions(this.tree,true); + } + address():void {this.digest=ProviderFixture.sha(readFileSync(this.file));const destination=join(this.root,this.digest);if(destination!==this.tree){renameSync(this.tree,destination);this.tree=destination;}} + run(extra:string[]=[]):SpawnSyncReturns { + const binary=join(this.root,'trsd link');if(!existsSync(binary))symlinkSync(resolve('dist/trsd-linux-x64'),binary); + return spawnSync(binary,['doctor','--json','--composition',this.file,'--composition-digest',this.digest,...extra],{encoding:'utf8',cwd:this.root,env:{...process.env,PATH:''},timeout:15000}); + } + close():void {this.permissions(this.tree,false);rmSync(this.root,{recursive:true,force:true});} +} +export class ExtensiblePlanProgramTests extends CliTests { + protected listeners():string{const rows:string[]=[];for(const path of ['/proc/net/tcp','/proc/net/tcp6'])for(const line of readFileSync(path,'utf8').trim().split('\n').slice(1)){const fields=line.trim().split(/\s+/);if(fields[3]==='0A')rows.push(fields[1]!);}return rows.sort().join('\n');} +} +export class InstalledProgramTests { + protected worker: { pid: number; dev: number; ino: number } | null = null; + protected json(program: CliProgramFixture, args: string[], environment: NodeJS.ProcessEnv = {}): ReturnType { const result = program.run(args, environment, program.symlink); if (result.status !== 0) throw new Error('Compiled installed command failed: ' + result.stdout + result.stderr); expect(result.stderr).toBe(''); return ServiceSchemas.envelope.parse(JSON.parse(result.stdout)); } + protected signal(signal: NodeJS.Signals): void { if (!this.worker) return; try { const current = statSync('/proc/' + this.worker.pid + '/exe'); if (current.dev === this.worker.dev && current.ino === this.worker.ino) process.kill(this.worker.pid, signal); } catch { } } + protected static delay(done: (value: void) => void): void { setTimeout(done, 100); } +} +export class InstalledPluginCliTests { + protected worker: { pid: number; dev: number; ino: number } | null = null; + protected static delay(done: (value: void) => void): void { setTimeout(done, 50); } + protected static hash(bytes: Uint8Array): string { return createHash('sha256').update(bytes).digest('hex'); } + protected request(program: CliProgramFixture, args: string[], expected?: number): ReturnType { + const result=program.run(['plugin',...args,'--json']); + if(result.status===null)throw new Error('Built CLI terminated without an exit status'); + if(expected!==undefined&&result.status!==expected){const envelope=PluginManagementSchemas.envelope.parse(JSON.parse(result.stdout));const codes:string[]=[];for(const blocker of envelope.blockers)codes.push(blocker.code);throw new Error('CLI_PLUGIN_READBACK:'+args[0]+':'+result.status+':'+envelope.problem?.code+':'+codes.join(','));} + if(expected===undefined)expect([0,3]).toContain(result.status);else expect(result.status).toBe(expected); + expect(result.stderr).toBe('');expect(result.stdout).not.toContain('NEVER_INHERIT');return PluginManagementSchemas.envelope.parse(JSON.parse(result.stdout)); + } + protected async settled(program: CliProgramFixture, installation: string, input: PluginInstallResult): Promise { + let result=input;const deadline=Date.now()+150000; + while(result.state==='pending'&&Date.now() { + const result=new Map();const all=store.resources(); + for(const receipt of all)for(const binding of store.current()!.resources)if(receipt.resourceId===binding.resourceId){const prior=result.get(receipt.resourceId);if(!prior||prior.references.length, files: Map): void { + const current=this.receipts(store); + for(const [id,receipt]of before){const next=current.get(id);expect(next).toBeDefined();expect(next!.actualId).toBe(receipt.actualId);expect(next!.creatingGeneration).toBe(receipt.creatingGeneration);expect(next!.definitionDigest).toBe(receipt.definitionDigest);expect(next!.contentDigest).toBe(receipt.contentDigest);expect(next!.path).toBe(receipt.path);} + for(const [path,digest]of files)expect(InstalledPluginCliTests.hash(readFileSync(join(store.path,path)))).toBe(digest); + } + protected async direct(store: ServicePlanStore, id: string): Promise { + for(const receipt of this.receipts(store).values())if(receipt.kind==='container'&&receipt.serviceId===id&&receipt.actualId){ + const inspected=spawnSync('docker',['container','inspect','--format','{{json .NetworkSettings.Networks}}',receipt.actualId],{encoding:'utf8'});expect(inspected.status).toBe(0);const networks=JSON.parse(inspected.stdout) as Record;let address:string|undefined;for(const network of Object.values(networks))address=network.IPAddress;expect(address).toBeDefined(); + const response=await fetch('http://'+address+':8080/'),body=await response.json() as {serviceId:string;passwordPresent:boolean;volumeMarker:string};expect(response.ok).toBe(true);expect(body.serviceId).toBe(id);expect(body.passwordPresent).toBe(true);expect(body.volumeMarker).toBe(id);return; + } + throw new Error('Installed CLI fixture endpoint missing'); + } + protected identify(installation: string): void { + const child=JSON.parse(readFileSync(join(installation,'.cordis/launch/child.json'),'utf8')),file=statSync(join(installation,'.cordis/worker')),running=statSync('/proc/'+child.pid+'/exe'); + expect(file.dev).toBe(running.dev);expect(file.ino).toBe(running.ino);expect(file.uid).toBe(process.getuid!());this.worker={pid:child.pid,dev:running.dev,ino:running.ino}; + } + protected signal(signal: NodeJS.Signals): void {if(!this.worker)return;try{const current=statSync('/proc/'+this.worker.pid+'/exe');if(current.dev===this.worker.dev&¤t.ino===this.worker.ino)process.kill(this.worker.pid,signal);}catch{}} +} +export class LivePluginCliTests { + protected static pause(done:(value:void)=>void):void{setTimeout(done,50);} + protected run(program:CliProgramFixture,args:string[],code?:number):ReturnType{const output=program.run(['plugin',...args,'--json']);expect(output.stderr).toBe('');if(code!==undefined)expect(output.status).toBe(code);else expect([0,3]).toContain(output.status!);expect(output.stdout).not.toContain('NEVER_INHERIT');return PluginManagementSchemas.envelope.parse(JSON.parse(output.stdout));} + protected async settled(program:CliProgramFixture,installation:string,input:PluginInstallResult):Promise{let result=input;const until=Date.now()+180000;while(result.state==='pending'&&Date.now() { diff --git a/tests/integration/command-classification.test.ts b/tests/integration/command-classification.test.ts index abd887b..d995e2a 100644 --- a/tests/integration/command-classification.test.ts +++ b/tests/integration/command-classification.test.ts @@ -11,7 +11,7 @@ import { ServiceRuntimeFixture } from '@treeseed-cordis/deployment/testing/runti import { ServiceWorkerBuild, ServicePlanStore } from '@treeseed-cordis/deployment'; import { PluginInstallationFixture } from '@treeseed-cordis/deployment/testing/plugins'; import { CanonicalJson, ServiceSchemas, PlatformSchemas, PluginManagementSchemas, type PluginInstallResult, type ServiceResourceReceipt, type CliEnvelope } from '@treeseed-cordis/sdk'; -import {CliProgramFixture,CliTests,ProviderFixture,ExtensiblePlanProgramTests,InstalledProgramTests,InstalledPluginCliTests,LivePluginCliTests} from '../support/cli-program-test-support.js'; +import {CliNetworkFixture as CliTests} from '../support/cli-program-test-support.js'; class CliTestsIntegration extends CliTests { async parser():Promise { diff --git a/tests/layout-migration.json b/tests/layout-migration.json index 097f673..fdf452b 100644 --- a/tests/layout-migration.json +++ b/tests/layout-migration.json @@ -29,6 +29,6 @@ {"from":"tests/cli-program.test.ts","to":"tests/integration/command-classification.test.ts","tier":"integration","name":"unit: parser help, version and error classifications","currentName":"integration: parser help, version and error classifications"} ], "mapping": { - "tests/cli-program.test.ts": ["tests/acceptance/cli-program.test.ts", "tests/integration/cli-program.test.ts", "tests/unit/cli-program.test.ts"] + "tests/cli-program.test.ts": ["tests/acceptance/cli-program.test.ts", "tests/integration/cli-program.test.ts", "tests/integration/command-classification.test.ts"] } } diff --git a/tests/support/cli-program-test-support.ts b/tests/support/cli-program-test-support.ts index 05ca1f8..9406219 100644 --- a/tests/support/cli-program-test-support.ts +++ b/tests/support/cli-program-test-support.ts @@ -1,17 +1,8 @@ -import { test, expect } from 'bun:test'; -import { mkdtempSync, rmSync, mkdirSync, copyFileSync, symlinkSync, writeFileSync, readdirSync, existsSync, readFileSync, statSync, chmodSync, renameSync } from 'node:fs'; -import { tmpdir } from 'node:os'; -import { join, resolve, dirname } from 'node:path'; -import { spawnSync, type SpawnSyncReturns } from 'node:child_process'; -import { createServer, type Server } from 'node:net'; -import { createHash } from 'node:crypto'; -import { PreparedReleaseFixture } from '@treeseed-cordis/deployment/testing'; -import { CliApplication } from '../../src/main.js'; -import { ServiceRuntimeFixture } from '@treeseed-cordis/deployment/testing/runtime'; -import { ServiceWorkerBuild, ServicePlanStore } from '@treeseed-cordis/deployment'; -import { PluginInstallationFixture } from '@treeseed-cordis/deployment/testing/plugins'; -import { CanonicalJson, ServiceSchemas, PlatformSchemas, PluginManagementSchemas, type PluginInstallResult, type ServiceResourceReceipt, type CliEnvelope } from '@treeseed-cordis/sdk'; - +import {mkdtempSync,rmSync,mkdirSync,copyFileSync,symlinkSync} from 'node:fs'; +import {tmpdir} from 'node:os'; +import {join,resolve} from 'node:path'; +import {spawnSync,type SpawnSyncReturns} from 'node:child_process'; +import {createServer,type Server} from 'node:net'; export class CliProgramFixture { readonly root=mkdtempSync(join(tmpdir(),'cordis cli ')); readonly cwd=join(this.root,'foreign cwd'); @@ -21,140 +12,11 @@ export class CliProgramFixture { run(args:string[],environment:NodeJS.ProcessEnv={},binary=this.binary):SpawnSyncReturns {return spawnSync(binary,args,{cwd:this.cwd,env:{...process.env,...environment},encoding:'utf8',stdio:['ignore','pipe','pipe'],timeout:15000});} close():void {rmSync(this.root,{recursive:true,force:true});} } -export class CliTests { +export class CliNetworkFixture { server:Server|null=null; resolvePort:((port:number)=>void)|null=null; listen(resolvePort:(port:number)=>void):void {this.resolvePort=resolvePort;this.server=createServer();this.server.listen(0,'127.0.0.1',this.listening.bind(this));} listening():void {const address=this.server!.address();if(address&&typeof address==='object')this.resolvePort!(address.port);} async port(occupied=false):Promise {const port=await new Promise(this.listen.bind(this));if(!occupied)this.server!.close();return port;} - snapshot():string { - const rows:string[]=[];for(const args of [['ps','-aq'],['image','ls','--no-trunc','-q'],['volume','ls','-q'],['network','ls','-q']]){const result=spawnSync('docker',['--context','default',...args],{encoding:'utf8'});expect(result.status).toBe(0);rows.push(result.stdout.split('\n').sort().join('\n'));}return JSON.stringify(rows); - } -} - -export class ProviderFixture { - readonly root=mkdtempSync(join(tmpdir(),'cordis installed provider Ω ')); - tree=join(this.root,'prepared package'); - readonly marker=join(this.root,'live-effect'); - readonly importMarker=join(this.root,'module-imported'); - get file():string{return join(this.tree,'composition.json');} - readonly declaration={id:'fixture-one',pluginId:'fixture',capability:'deployment',contractVersion:'0.3.0-cordis.1',target:'cli',provide:['deployment'],inject:[]}; - digest=''; - constructor(readonly label='one') {} - static sha(value:Uint8Array|string):string {return createHash('sha256').update(value).digest('hex');} - copy(source:string,destination:string):void { - mkdirSync(destination,{recursive:true}); - for(const entry of readdirSync(source,{withFileTypes:true})) { - const from=join(source,entry.name),to=join(destination,entry.name); - if(entry.isDirectory())this.copy(from,to);else if(entry.isFile())copyFileSync(from,to);else throw new Error('Fixture dependency is not a regular package tree'); - } - } - async prepare():Promise { - mkdirSync(this.tree);this.declaration.id='fixture-'+this.label; - // These are real pinned installed third-party packages, copied as fixture bytes, not mocked modules. - for(const name of ['cordis','cosmokit','@standard-schema/spec'])this.copy(resolve('node_modules',name),join(this.tree,'node_modules',name)); - const helper=join(this.tree,'node_modules/@fixture/helper');mkdirSync(helper,{recursive:true}); - writeFileSync(join(helper,'package.json'),JSON.stringify({name:'@fixture/helper',version:'1.0.0',type:'module',exports:'./index.js'})); - const helperSource=join(this.root,'helper.ts');writeFileSync(helperSource,'export class Helper {static label():string{return "ordinary-dependency";}}'); - let build=spawnSync('bun',['build',helperSource,'--target=node','--outfile',join(helper,'index.js')],{encoding:'utf8'});expect(build.status).toBe(0); - const source=join(this.root,'provider.ts'); - writeFileSync(source,`import {Context} from 'cordis';import {Helper} from '@fixture/helper';import {writeFileSync,unlinkSync} from 'node:fs'; -class ImportCanary {static run(){writeFileSync(${JSON.stringify(this.importMarker)},'imported');}}ImportCanary.run(); -class Configuration {static validate(value:unknown){return value&&typeof value==='object'&&Object.keys(value).length===0?{value}:{issues:[{message:'Invalid configuration'}]};}} -export class Provider { -static readonly declaration=${JSON.stringify(this.declaration)}; -static readonly provide=Provider.declaration.provide;static readonly inject=Provider.declaration.inject; -static readonly Config={'~standard':{version:1,vendor:'fixture',validate:Configuration.validate.bind(Configuration)}}; -constructor(readonly ctx:Context){if(!Context.is(ctx)||ctx.registry.get(Provider)!==ctx[Context.root].registry.get(Provider))throw new Error('not native shared graph');writeFileSync(${JSON.stringify(this.marker)},'active');ctx.provide('deployment',this);ctx.effect(this.effect.bind(this));if(${JSON.stringify(this.label)}==='constructor-fail')throw new Error('secret-constructor-canary');} -effect(){return this.close.bind(this);}close(){unlinkSync(${JSON.stringify(this.marker)});if(${JSON.stringify(this.label)}==='cleanup-fail')throw new Error('secret-cleanup-canary');} -async diagnose(){return {ready:true,host:{os:'fixture',architecture:'x64',distribution:'fixture',release:'1'},engine:{context:${JSON.stringify(this.label)},endpoint:null,serverVersion:null,serverArchitecture:null,composeVersion:null,cpus:null,memoryBytes:null},installation:{path:'/fixture',exists:false,freeBytes:null},port:8443,checks:[{id:'native-provider',status:'pass',code:'FIXTURE',message:Helper.label()}]};} -async plan(){throw new Error('fixture is diagnosis-only');}} -`); - build=spawnSync('bun',['build',source,'--target=node','--external','cordis','--external','@fixture/helper','--outfile',join(this.tree,'index.js')],{encoding:'utf8'});expect(build.status).toBe(0); - writeFileSync(join(this.tree,'package.json'),JSON.stringify({name:'@fixture/provider',version:'1.0.0',type:'module',exports:'./index.js',dependencies:{cordis:'4.0.0-rc.12','@fixture/helper':'1.0.0'},cordis:this.declaration})); - this.refresh();return this; - } - inventory(directory=this.tree,prefix=''):Array<{path:string;sha256:string}> { - const files:Array<{path:string;sha256:string}>=[]; - for(const entry of readdirSync(directory,{withFileTypes:true}).sort(ProviderFixture.order)){ - if(directory===this.tree&&entry.name==='composition.json')continue; - const path=join(directory,entry.name),relative=prefix+entry.name; - if(entry.isDirectory())files.push(...this.inventory(path,relative+'/'));else files.push({path:relative,sha256:ProviderFixture.sha(readFileSync(path))}); - } - return files; - } - static order(a:{name:string},b:{name:string}):number {return a.name.localeCompare(b.name);} - permissions(directory:string,locked:boolean):void {chmodSync(directory,0o700);for(const entry of readdirSync(directory,{withFileTypes:true})){const path=join(directory,entry.name);if(entry.isSymbolicLink())continue;if(entry.isDirectory())this.permissions(path,locked);else chmodSync(path,locked?0o400:0o600);}chmodSync(directory,locked?0o500:0o700);} - refresh():void { - this.permissions(this.tree,false); - const manifest={formatVersion:1,contractVersion:'0.3.0-cordis.1',target:'cli',scopeId:'local-cli',providerId:this.declaration.id,package:{name:'@fixture/provider',version:'1.0.0'},entry:'index.js',configuration:{},files:this.inventory()}; - writeFileSync(this.file,JSON.stringify(manifest));this.address();this.permissions(this.tree,true); - } - address():void {this.digest=ProviderFixture.sha(readFileSync(this.file));const destination=join(this.root,this.digest);if(destination!==this.tree){renameSync(this.tree,destination);this.tree=destination;}} - run(extra:string[]=[]):SpawnSyncReturns { - const binary=join(this.root,'trsd link');if(!existsSync(binary))symlinkSync(resolve('dist/trsd-linux-x64'),binary); - return spawnSync(binary,['doctor','--json','--composition',this.file,'--composition-digest',this.digest,...extra],{encoding:'utf8',cwd:this.root,env:{...process.env,PATH:''},timeout:15000}); - } - close():void {this.permissions(this.tree,false);rmSync(this.root,{recursive:true,force:true});} -} - -export class ExtensiblePlanProgramTests extends CliTests { - protected listeners():string{const rows:string[]=[];for(const path of ['/proc/net/tcp','/proc/net/tcp6'])for(const line of readFileSync(path,'utf8').trim().split('\n').slice(1)){const fields=line.trim().split(/\s+/);if(fields[3]==='0A')rows.push(fields[1]!);}return rows.sort().join('\n');} -} - - -export class InstalledProgramTests { - protected worker: { pid: number; dev: number; ino: number } | null = null; - protected json(program: CliProgramFixture, args: string[], environment: NodeJS.ProcessEnv = {}): ReturnType { const result = program.run(args, environment, program.symlink); if (result.status !== 0) throw new Error('Compiled installed command failed: ' + result.stdout + result.stderr); expect(result.stderr).toBe(''); return ServiceSchemas.envelope.parse(JSON.parse(result.stdout)); } - protected signal(signal: NodeJS.Signals): void { if (!this.worker) return; try { const current = statSync('/proc/' + this.worker.pid + '/exe'); if (current.dev === this.worker.dev && current.ino === this.worker.ino) process.kill(this.worker.pid, signal); } catch { } } - protected static delay(done: (value: void) => void): void { setTimeout(done, 100); } -} - -export class InstalledPluginCliTests { - protected worker: { pid: number; dev: number; ino: number } | null = null; - protected static delay(done: (value: void) => void): void { setTimeout(done, 50); } - protected static hash(bytes: Uint8Array): string { return createHash('sha256').update(bytes).digest('hex'); } - protected request(program: CliProgramFixture, args: string[], expected?: number): ReturnType { - const result=program.run(['plugin',...args,'--json']); - if(result.status===null)throw new Error('Built CLI terminated without an exit status'); - if(expected!==undefined&&result.status!==expected){const envelope=PluginManagementSchemas.envelope.parse(JSON.parse(result.stdout));const codes:string[]=[];for(const blocker of envelope.blockers)codes.push(blocker.code);throw new Error('CLI_PLUGIN_READBACK:'+args[0]+':'+result.status+':'+envelope.problem?.code+':'+codes.join(','));} - if(expected===undefined)expect([0,3]).toContain(result.status);else expect(result.status).toBe(expected); - expect(result.stderr).toBe('');expect(result.stdout).not.toContain('NEVER_INHERIT');return PluginManagementSchemas.envelope.parse(JSON.parse(result.stdout)); - } - protected async settled(program: CliProgramFixture, installation: string, input: PluginInstallResult): Promise { - let result=input;const deadline=Date.now()+150000; - while(result.state==='pending'&&Date.now() { - const result=new Map();const all=store.resources(); - for(const receipt of all)for(const binding of store.current()!.resources)if(receipt.resourceId===binding.resourceId){const prior=result.get(receipt.resourceId);if(!prior||prior.references.length, files: Map): void { - const current=this.receipts(store); - for(const [id,receipt]of before){const next=current.get(id);expect(next).toBeDefined();expect(next!.actualId).toBe(receipt.actualId);expect(next!.creatingGeneration).toBe(receipt.creatingGeneration);expect(next!.definitionDigest).toBe(receipt.definitionDigest);expect(next!.contentDigest).toBe(receipt.contentDigest);expect(next!.path).toBe(receipt.path);} - for(const [path,digest]of files)expect(InstalledPluginCliTests.hash(readFileSync(join(store.path,path)))).toBe(digest); - } - protected async direct(store: ServicePlanStore, id: string): Promise { - for(const receipt of this.receipts(store).values())if(receipt.kind==='container'&&receipt.serviceId===id&&receipt.actualId){ - const inspected=spawnSync('docker',['container','inspect','--format','{{json .NetworkSettings.Networks}}',receipt.actualId],{encoding:'utf8'});expect(inspected.status).toBe(0);const networks=JSON.parse(inspected.stdout) as Record;let address:string|undefined;for(const network of Object.values(networks))address=network.IPAddress;expect(address).toBeDefined(); - const response=await fetch('http://'+address+':8080/'),body=await response.json() as {serviceId:string;passwordPresent:boolean;volumeMarker:string};expect(response.ok).toBe(true);expect(body.serviceId).toBe(id);expect(body.passwordPresent).toBe(true);expect(body.volumeMarker).toBe(id);return; - } - throw new Error('Installed CLI fixture endpoint missing'); - } - protected identify(installation: string): void { - const child=JSON.parse(readFileSync(join(installation,'.cordis/launch/child.json'),'utf8')),file=statSync(join(installation,'.cordis/worker')),running=statSync('/proc/'+child.pid+'/exe'); - expect(file.dev).toBe(running.dev);expect(file.ino).toBe(running.ino);expect(file.uid).toBe(process.getuid!());this.worker={pid:child.pid,dev:running.dev,ino:running.ino}; - } - protected signal(signal: NodeJS.Signals): void {if(!this.worker)return;try{const current=statSync('/proc/'+this.worker.pid+'/exe');if(current.dev===this.worker.dev&¤t.ino===this.worker.ino)process.kill(this.worker.pid,signal);}catch{}} -} - -export class LivePluginCliTests { - protected static pause(done:(value:void)=>void):void{setTimeout(done,50);} - protected run(program:CliProgramFixture,args:string[],code?:number):ReturnType{const output=program.run(['plugin',...args,'--json']);expect(output.stderr).toBe('');if(code!==undefined)expect(output.status).toBe(code);else expect([0,3]).toContain(output.status!);expect(output.stdout).not.toContain('NEVER_INHERIT');return PluginManagementSchemas.envelope.parse(JSON.parse(output.stdout));} - protected async settled(program:CliProgramFixture,installation:string,input:PluginInstallResult):Promise{let result=input;const until=Date.now()+180000;while(result.state==='pending'&&Date.now() Date: Tue, 27 Oct 2026 09:16:49 -0400 Subject: [PATCH 5/9] style: normalize fixture whitespace (#325) --- tests/support/cli-program-test-support.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/support/cli-program-test-support.ts b/tests/support/cli-program-test-support.ts index 9406219..502fde6 100644 --- a/tests/support/cli-program-test-support.ts +++ b/tests/support/cli-program-test-support.ts @@ -18,5 +18,5 @@ export class CliNetworkFixture { listen(resolvePort:(port:number)=>void):void {this.resolvePort=resolvePort;this.server=createServer();this.server.listen(0,'127.0.0.1',this.listening.bind(this));} listening():void {const address=this.server!.address();if(address&&typeof address==='object')this.resolvePort!(address.port);} async port(occupied=false):Promise {const port=await new Promise(this.listen.bind(this));if(!occupied)this.server!.close();return port;} - + } From 437c701d605e49db6f1d0fad20af88eb4ffcf2d0 Mon Sep 17 00:00:00 2001 From: Adrian Webb Date: Tue, 27 Oct 2026 10:00:46 -0400 Subject: [PATCH 6/9] docs: require complete discovery evidence (#325) --- .agents/skills/release-workflow/SKILL.md | 2 +- tests/README.md | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/.agents/skills/release-workflow/SKILL.md b/.agents/skills/release-workflow/SKILL.md index 5d9768b..bd75544 100644 --- a/.agents/skills/release-workflow/SKILL.md +++ b/.agents/skills/release-workflow/SKILL.md @@ -179,7 +179,7 @@ into the normal check command and CI. It rejects misplaced or undiscoverable reg phase-based specification paths, and verifies complete tier commands. Platform checks each assembled specification is explicitly assigned to a script or CI closure because historical exact artifact selections run in separate jobs. Reusable test contracts with assertions belong -in their actual tier; support holds fixture setup and executable fixture programs. +in their actual tier; support holds fixture setup and executable fixture programs. Tier scripts run the complete tier directory directly; an individual-file selector does not qualify. When CI partitions cases with name filters, every static registered case must match exactly one nonempty partition. Missing, overlapping or malformed partitions fail verification. ## ⚠️ MANDATORY: Testing Tiers and the Composition Root diff --git a/tests/README.md b/tests/README.md index 2ec8442..46a242e 100644 --- a/tests/README.md +++ b/tests/README.md @@ -36,3 +36,8 @@ explicitly. A green unit suite or policy prose alone is never sufficient. Assertion-bearing reusable acceptance contracts live in `tests/acceptance/`; shared support contains fixture setup and program inputs. Moving a case must also move its shared behavioral assertions, rather than hiding the scenario in a support base class. + +Complete tier commands use `bun test ./tests/` with only non-filtering timeout or +empty-tier reporting options. Individual-file selectors and shell wrappers do not prove +complete discovery. Name-filtered CI matrices must cover each registered case exactly once +and every partition must be nonempty; the shared layout guard verifies that inventory. From fce86982c023ab1f48055ae56cf8399a5878b968 Mon Sep 17 00:00:00 2001 From: Adrian Webb Date: Tue, 27 Oct 2026 10:02:00 -0400 Subject: [PATCH 7/9] test: reject incomplete commands and CI partitions (#325) --- tests/regression/test-layout.test.ts | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/tests/regression/test-layout.test.ts b/tests/regression/test-layout.test.ts index 7b6cde1..a414a7f 100644 --- a/tests/regression/test-layout.test.ts +++ b/tests/regression/test-layout.test.ts @@ -8,7 +8,19 @@ class TestLayoutRegression { valid():void{for(const tier of ['acceptance','regression','integration','unit'])expect(this.check('tests/'+tier+'/provider-selection.test.ts',"import {test} from 'bun:test';class Cases {run():void{}}const suite=new Cases();test('case',suite.run.bind(suite));")).toEqual([]);} misplaced():void{for(const path of ['tests/flat.test.ts','src/provider.test.ts','tests/support/hidden.ts','tests/unit/hidden.ts'])expect(this.check(path,"import {test as scenario} from 'bun:test';class Cases {run():void{}}const suite=new Cases();scenario('case',suite.run.bind(suite));").length).toBeGreaterThan(0);} phases():void{for(const name of ['task3d-agent','phase-4-provider','task-3c'])expect(this.check('tests/unit/'+name+'.test.ts',"import {test} from 'bun:test';class Cases {run():void{}}const suite=new Cases();test('case',suite.run.bind(suite));").length).toBeGreaterThan(0);} - discovery():void{const root=mkdtempSync(join(tmpdir(),'test-discovery-'));try{mkdirSync(join(root,'tests/unit'),{recursive:true});writeFileSync(join(root,'tests/unit/value.test.ts'),"import {test} from 'bun:test';class Cases {run():void{}}const suite=new Cases();test('case',suite.run.bind(suite));");const scripts:Record={};for(const tier of ['acceptance','regression','integration','unit'])scripts['test:'+tier]='bun test ./tests/'+tier;writeFileSync(join(root,'package.json'),JSON.stringify({scripts}));expect(new TestLayout().inspect(root)).toEqual([]);scripts['test:unit']='bun test ./tests --test-name-pattern ^unit:';writeFileSync(join(root,'package.json'),JSON.stringify({scripts}));expect(new TestLayout().inspect(root).length).toBeGreaterThan(0);}finally{rmSync(root,{recursive:true,force:true});}} + discovery():void{const root=mkdtempSync(join(tmpdir(),'test-discovery-'));try{ + mkdirSync(join(root,'tests/unit'),{recursive:true});writeFileSync(join(root,'tests/unit/value.test.ts'),"import {test} from 'bun:test';class Cases {run():void{}}const suite=new Cases();test('case',suite.run.bind(suite));"); + const scripts:Record={};for(const tier of ['acceptance','regression','integration','unit'])scripts['test:'+tier]='bun test ./tests/'+tier; + for(const command of ['bun test ./tests/unit','bun test tests/unit/ --timeout 30000','bun test ./tests/unit --pass-with-no-tests']){scripts['test:unit']=command;writeFileSync(join(root,'package.json'),JSON.stringify({scripts}));expect(new TestLayout().inspect(root)).toEqual([]);} + for(const command of ['bun test ./tests --test-name-pattern ^unit:','bun test ./tests/unit/only-one.test.ts','echo tests/unit && bun test ./tests/unit/only-one.test.ts','bun test ./tests/unit --test-name-pattern selected','bun test ./tests/unit ./tests/acceptance/one.test.ts']){scripts['test:unit']=command;writeFileSync(join(root,'package.json'),JSON.stringify({scripts}));expect(new TestLayout().inspect(root).length).toBeGreaterThan(0);} + }finally{rmSync(root,{recursive:true,force:true});}} + private matrix(root:string,patterns:string[]):void{let source='jobs:\n programs:\n strategy:\n matrix:\n include:\n';for(const pattern of patterns)source+=' - pattern: '+JSON.stringify(pattern)+'\n';source+=' steps:\n - run: bun test ./tests --test-name-pattern "${{ matrix.pattern }}"\n';writeFileSync(join(root,'.github/workflows/verify.yml'),source);} + partitions():void{const root=mkdtempSync(join(tmpdir(),'test-partitions-'));try{ + mkdirSync(join(root,'tests/unit'),{recursive:true});mkdirSync(join(root,'.github/workflows'),{recursive:true});writeFileSync(join(root,'tests/unit/value.test.ts'),"import {test} from 'bun:test';class Cases {run():void{}}const suite=new Cases();test('case',suite.run.bind(suite));"); + const scripts:Record={};for(const tier of ['acceptance','regression','integration','unit'])scripts['test:'+tier]='bun test ./tests/'+tier;writeFileSync(join(root,'package.json'),JSON.stringify({scripts})); + this.matrix(root,['^case$']);expect(new TestLayout().inspect(root)).toEqual([]); + for(const patterns of [['^different$'],['^case$','^case$'],['^case$','^unused$'],['['],[]]){this.matrix(root,patterns);expect(new TestLayout().inspect(root).length).toBeGreaterThan(0);} + }finally{rmSync(root,{recursive:true,force:true});}} support():void{expect(this.check('tests/support/provider.ts','export class Fixture {run():void{}}')).toEqual([]);expect(this.check('tests/support/hidden.ts',"import * as cases from 'node:test';class Cases {run():void{}}const suite=new Cases();cases.test('case',suite.run.bind(suite));").length).toBeGreaterThan(0);} } const suite=new TestLayoutRegression(); @@ -17,3 +29,4 @@ test('regression: misplaced and aliased hidden test registrations fail layout ve test('regression: task and phase filenames cannot become permanent suites',suite.phases.bind(suite)); test('regression: shared fixture classes are permitted without hidden namespace tests',suite.support.bind(suite)); test('regression: tier commands cannot silently omit files through name filters',suite.discovery.bind(suite)); +test('regression: CI name partitions are exhaustive disjoint nonempty and valid',suite.partitions.bind(suite)); From d7a777b87b6d91e6f2d13b6fec061bf32c929769 Mon Sep 17 00:00:00 2001 From: Adrian Webb Date: Tue, 27 Oct 2026 10:03:40 -0400 Subject: [PATCH 8/9] fix: enforce complete tier and partition discovery (#325) --- .../release-workflow/scripts/test-layout.ts | 29 +++++++++++++++---- 1 file changed, 23 insertions(+), 6 deletions(-) diff --git a/.agents/skills/release-workflow/scripts/test-layout.ts b/.agents/skills/release-workflow/scripts/test-layout.ts index 6f90250..44d992b 100644 --- a/.agents/skills/release-workflow/scripts/test-layout.ts +++ b/.agents/skills/release-workflow/scripts/test-layout.ts @@ -5,9 +5,9 @@ import {join,relative} from 'node:path'; export class TestLayout { private static readonly ignored=new Set(['.git','.agents','.codegraph','node_modules','dist','vendor']); private static readonly tier=/^tests\/(acceptance|regression|integration|unit)\//; - inspect(root=process.cwd()):string[]{const errors:string[]=[],specifications:string[]=[];this.directory(root,root,errors,specifications);this.discovery(root,specifications,errors);return errors;} + inspect(root=process.cwd()):string[]{const errors:string[]=[],specifications:string[]=[],names:Array=[];this.directory(root,root,errors,specifications,names);this.discovery(root,specifications,names,errors);return errors;} check(root=process.cwd()):void{const errors=this.inspect(root);if(errors.length)throw new Error(errors.join('\n'));} - private discovery(root:string,specifications:string[],errors:string[]):void{ + private discovery(root:string,specifications:string[],names:Array,errors:string[]):void{ const path=join(root,'package.json');if(!existsSync(path))return; const metadata=JSON.parse(readFileSync(path,'utf8')) as {name?:string;scripts?:Record},scripts=metadata.scripts??{}; if(metadata.name==='@treeseed-cordis/platform-verification'){ @@ -15,21 +15,37 @@ export class TestLayout { for(const file of specifications)if(!commands.includes(file))errors.push(file+': exact assembled test must be assigned to a script or CI closure');return; } if(!Object.keys(scripts).some(this.tierScript))return; - for(const tier of ['acceptance','regression','integration','unit']){const command=scripts['test:'+tier];if(!command||!command.includes('tests/'+tier)||command.includes('--test-name-pattern'))errors.push('test:'+tier+': command must discover the complete tier directory');} + for(const tier of ['acceptance','regression','integration','unit']){const command=scripts['test:'+tier],complete=new RegExp('^bun\\s+test\\s+(?:\\./)?tests/'+tier+'/?(?:\\s+(?:--timeout(?:=|\\s+)\\d+|--pass-with-no-tests))*\\s*$');if(!command||!complete.test(command.trim()))errors.push('test:'+tier+': command must directly discover the complete tier directory');} + this.partitions(root,names,errors); } + private partitions(root:string,names:Array,errors:string[]):void{ + const directory=join(root,'.github/workflows');if(!existsSync(directory))return; + for(const file of readdirSync(directory)){ + if(!/\.ya?ml$/.test(file))continue;const source=readFileSync(join(directory,file),'utf8');if(!source.includes('--test-name-pattern'))continue; + try{ + if(!source.includes('matrix.pattern'))throw new Error('Name-filtered CI requires an explicit pattern matrix'); + const patterns:RegExp[]=[];for(const line of source.split('\n')){const entry=line.match(/^\s+(?:-\s+)?pattern:\s*(.+?)\s*$/);if(entry){const value:unknown=JSON.parse(entry[1]!);if(typeof value!=='string')throw new Error('Invalid matrix pattern');patterns.push(new RegExp(value));}} + if(patterns.length===0||names.length===0||names.includes(null))throw new Error('Cannot prove static CI case inventory'); + const hits=patterns.map(this.zero);for(const name of names){let matches=0;for(let index=0;index):void{ for(const entry of readdirSync(directory,{withFileTypes:true})){ if(TestLayout.ignored.has(entry.name))continue; const path=join(directory,entry.name),file=relative(root,path).replaceAll('\\','/'); if(entry.isSymbolicLink()){if(file.startsWith('tests/'))errors.push(file+': test symlink forbidden');continue;} - if(entry.isDirectory()){if(existsSync(join(path,'.git')))continue;this.directory(root,path,errors,specifications);continue;} + if(entry.isDirectory()){if(existsSync(join(path,'.git')))continue;this.directory(root,path,errors,specifications,names);continue;} if(!/\.[cm]?[jt]sx?$/.test(file))continue; const spec=/\.(test|spec)\.[cm]?[jt]sx?$/.test(file),tier=TestLayout.tier.test(file); if(spec)specifications.push(file); if(spec&&!tier)errors.push(file+': specifications require tests/{acceptance,regression,integration,unit}'); if(spec&&/(?:^|[\/-])(?:task|phase)[-_]?\d/i.test(file))errors.push(file+': permanent specifications require capability names'); const source=ts.createSourceFile(file,readFileSync(path,'utf8'),ts.ScriptTarget.Latest,true),visitor=new TestRegistrationVisitor();visitor.inspect(source); + if(spec&&tier)names.push(...visitor.names); if(visitor.registrations&&(!tier||!spec))errors.push(file+': test registrations require a discoverable tier specification'); if(spec&&!visitor.registrations)errors.push(file+': specification has no discoverable test registrations'); } @@ -37,13 +53,14 @@ export class TestLayout { } class TestRegistrationVisitor { registrations=0; + readonly names:Array=[]; private readonly bindings=new Set(); private readonly namespaces=new Set(); inspect(source:ts.SourceFile):void{ for(const node of source.statements){if(!ts.isImportDeclaration(node)||!ts.isStringLiteral(node.moduleSpecifier)||!['bun:test','node:test'].includes(node.moduleSpecifier.text))continue;const clause=node.importClause;if(!clause)continue;if(clause.name)this.bindings.add(clause.name.text);const names=clause.namedBindings;if(names&&ts.isNamespaceImport(names))this.namespaces.add(names.name.text);if(names&&ts.isNamedImports(names))for(const item of names.elements)if(['test','it','describe'].includes((item.propertyName??item.name).text))this.bindings.add(item.name.text);} this.visit(source); } - private visit(node:ts.Node):void{if(ts.isCallExpression(node)&&this.test(node.expression))this.registrations++;ts.forEachChild(node,this.visit.bind(this));} + private visit(node:ts.Node):void{if(ts.isCallExpression(node)&&this.test(node.expression)){this.registrations++;const name=node.arguments[0];this.names.push(name&&ts.isStringLiteralLike(name)?name.text:null);}ts.forEachChild(node,this.visit.bind(this));} private test(expression:ts.Expression):boolean{if(ts.isIdentifier(expression))return this.bindings.has(expression.text);if(ts.isPropertyAccessExpression(expression)){if(ts.isIdentifier(expression.expression)&&this.namespaces.has(expression.expression.text))return ['test','it','describe'].includes(expression.name.text);return this.test(expression.expression);}if(ts.isCallExpression(expression))return this.test(expression.expression);return false;} } if(import.meta.main)new TestLayout().check(); From 0454b6c752e4c16eedd15644e321f02e552a4c1b Mon Sep 17 00:00:00 2001 From: Adrian Webb Date: Tue, 27 Oct 2026 10:12:58 -0400 Subject: [PATCH 9/9] fix: align planning examples with permanent test tiers (#325) --- .agents/skills/writing-plans/SKILL.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.agents/skills/writing-plans/SKILL.md b/.agents/skills/writing-plans/SKILL.md index d8a7a67..48bfe47 100644 --- a/.agents/skills/writing-plans/SKILL.md +++ b/.agents/skills/writing-plans/SKILL.md @@ -102,7 +102,7 @@ owns the code, in that task's own step style.] **Files:** - Create: `packages/cli/src/commands.ts` (inside the independent CLI repo) - Modify: that owner's documented composition class and command contract -- Test: `packages/cli/tests/smoke/cli-program.test.ts` +- Test: `packages/cli/tests/acceptance/cli-program.test.ts` **Interfaces:** - Consumes: [what this task uses from earlier tasks — exact signatures] @@ -146,7 +146,7 @@ Expected: all applicable gates pass; record unavailable environments separately. - [ ] **Step 5: Commit** ```bash -git add tests/smoke/cli-program.test.ts src/commands.ts +git add tests/acceptance/cli-program.test.ts src/commands.ts git commit -m "feat: add specific feature (#N)" ``` ````