Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 44 additions & 1 deletion .agents/skills/release-workflow/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,46 @@ Open PR to `cordis`, verify required CI/acceptance/review, then merge if authori

**Why this order:** documentation written after code describes what *was* built, not what *should* be — it captures implementation detail instead of intent and is perpetually behind. Tests written after code verify the implementation, not the specification. The docs ARE the spec, the tests validate the spec, the code satisfies the tests.

## ⚠️ MANDATORY: Permanent Test Organization

All permanent executable test specifications MUST live in the owning repository under exactly
these tier directories, named for the capability or behavior being verified:

- `tests/acceptance/`: user-observable flows, end-to-end/browser coverage and smoke tests of built programs.
- `tests/regression/`: cases that pin an identified prior defect and prevent recurrence.
- `tests/integration/`: real collaborator, provider, package, filesystem and service boundaries.
- `tests/unit/`: isolated contracts, computations and transformations.

Write tests outside-in in that order, after committing documentation and before implementation.
Split mixed-tier suites. A negative case is classified by its purpose and boundary; it is not
automatically a regression. Permanent test files MUST NOT be flat under `tests/`, colocated with
source, or named after a migration/task/phase number. Do not disguise test suites as helpers.
Shared fixture classes, fixture programs and data may use `tests/support/`, with no test
registrations there. Historical migration evidence, exact composition pins and immutable archives
retain provenance outside executable test suites; durable coverage moves into the proper tier.

Every reorganization preserves assertions, inputs, timeouts, negative cases and discovery/count
parity. Update imports, fixture/build entrypoints, public test-helper exports, package scripts,
partition inventories, CI and current docs together. Verify all four tiers are discovered; an
empty tier must be recorded honestly rather than filled with redundant tests or silently skipped.
Include an executable layout/discovery check in the normal owner verification and CI gates;
exercise rejection of misplaced tests, phase-based names and hidden helper registrations.
For a setup-only repository, install that check with its first implementation, without adding
application scaffolding or dependencies during initialization.

Every injectable seam still needs default production construction coverage, and every user
program needs a built-program acceptance test. Completion requires applicable tier suites,
checks, build, CI and assembled-artifact smoke evidence. Report missing prerequisites or gates
explicitly. A green unit suite or policy prose alone is never sufficient.

The copied development-only checker is `scripts/test-layout.ts` inside this skill. After
installing the owner's declared development dependencies, wire `bun .agents/skills/release-workflow/scripts/test-layout.ts`
into the normal check command and CI. It rejects misplaced or undiscoverable registrations and
phase-based specification paths, and verifies complete tier commands. Platform checks each
assembled specification is explicitly assigned to a script or CI closure because historical
exact artifact selections run in separate jobs. Reusable test contracts with assertions belong
in their actual tier; support holds fixture setup and executable fixture programs. Tier scripts run the complete tier directory directly; an individual-file selector does not qualify. When CI partitions cases with name filters, every static registered case must match exactly one nonempty partition. Missing, overlapping or malformed partitions fail verification.

## ⚠️ MANDATORY: Testing Tiers and the Composition Root

Outside-in testing only works if the **outer** rung exists. Unit tests of injected, isolated
Expand Down Expand Up @@ -409,7 +449,8 @@ gh pr create --base cordis \
- [ ] CI green (<typecheck> + <build> + <tests>)
- [ ] Manual verification: <describe, if user-facing>
- [ ] New tests added for new behavior
- [ ] Existing tests pass unmodified (for refactors)
- [ ] Acceptance, regression, integration and unit tiers are verified; any empty tier or missing prerequisite is explicitly recorded
- [ ] Test layout/discovery check passes; reorganization preserves cases, assertions, arguments and timeouts

## Related
Part of #<issue>
Expand Down Expand Up @@ -536,6 +577,8 @@ Run this every time, immediately before opening the PR (this is the artifact the
- [ ] On a `feature/*` or `fix/*` branch off verified `origin/cordis`, never on `cordis`/`staging`/`main` directly
- [ ] Issue exists and is linked; branch name carries its number
- [ ] Documentation updated **and committed** before code (diff shows the doc files changed)
- [ ] Permanent tests live in the four required tier directories with capability-based names; layout/discovery checks reject misplaced or hidden tests.
- [ ] Reorganizations preserve assertion bodies, inputs, timeouts and discovered case counts; imports, fixtures, exports, scripts, partitions and CI agree.
- [ ] Tests written outside-in and committed per layer; `<test command>` run now → 0 failures
- [ ] `<typecheck command>` and `<build command>` run now → clean / exit 0
- [ ] Seams and user-runnable surfaces have ≥1 test on the **default construction path**, nothing injected (see Testing Tiers and the Composition Root)
Expand Down
66 changes: 66 additions & 0 deletions .agents/skills/release-workflow/scripts/test-layout.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
import ts from 'typescript';
import {readdirSync,readFileSync,existsSync} from 'node:fs';
import {join,relative} from 'node:path';
/** Shared development-only policy. No runtime or target project depends on it. */
export class TestLayout {
private static readonly ignored=new Set(['.git','.agents','.codegraph','node_modules','dist','vendor']);
private static readonly tier=/^tests\/(acceptance|regression|integration|unit)\//;
inspect(root=process.cwd()):string[]{const errors:string[]=[],specifications:string[]=[],names:Array<string|null>=[];this.directory(root,root,errors,specifications,names);this.discovery(root,specifications,names,errors);return errors;}
check(root=process.cwd()):void{const errors=this.inspect(root);if(errors.length)throw new Error(errors.join('\n'));}
private discovery(root:string,specifications:string[],names:Array<string|null>,errors:string[]):void{
const path=join(root,'package.json');if(!existsSync(path))return;
const metadata=JSON.parse(readFileSync(path,'utf8')) as {name?:string;scripts?:Record<string,string>},scripts=metadata.scripts??{};
if(metadata.name==='@treeseed-cordis/platform-verification'){
let commands=Object.values(scripts).join('\n');const workflows=join(root,'.github/workflows');if(existsSync(workflows))for(const name of readdirSync(workflows))if(/\.ya?ml$/.test(name))commands+='\n'+readFileSync(join(workflows,name),'utf8');
for(const file of specifications)if(!commands.includes(file))errors.push(file+': exact assembled test must be assigned to a script or CI closure');return;
}
if(!Object.keys(scripts).some(this.tierScript))return;
for(const tier of ['acceptance','regression','integration','unit']){const command=scripts['test:'+tier],complete=new RegExp('^bun\\s+test\\s+(?:\\./)?tests/'+tier+'/?(?:\\s+(?:--timeout(?:=|\\s+)\\d+|--pass-with-no-tests))*\\s*$');if(!command||!complete.test(command.trim()))errors.push('test:'+tier+': command must directly discover the complete tier directory');}
this.partitions(root,names,errors);
}
private partitions(root:string,names:Array<string|null>,errors:string[]):void{
const directory=join(root,'.github/workflows');if(!existsSync(directory))return;
for(const file of readdirSync(directory)){
if(!/\.ya?ml$/.test(file))continue;const source=readFileSync(join(directory,file),'utf8');if(!source.includes('--test-name-pattern'))continue;
try{
if(!source.includes('matrix.pattern'))throw new Error('Name-filtered CI requires an explicit pattern matrix');
const patterns:RegExp[]=[];for(const line of source.split('\n')){const entry=line.match(/^\s+(?:-\s+)?pattern:\s*(.+?)\s*$/);if(entry){const value:unknown=JSON.parse(entry[1]!);if(typeof value!=='string')throw new Error('Invalid matrix pattern');patterns.push(new RegExp(value));}}
if(patterns.length===0||names.length===0||names.includes(null))throw new Error('Cannot prove static CI case inventory');
const hits=patterns.map(this.zero);for(const name of names){let matches=0;for(let index=0;index<patterns.length;index++)if(patterns[index]!.test(name!)){hits[index]=hits[index]!+1;matches++;}if(matches!==1)throw new Error('Every case must match exactly one CI partition');}
if(hits.includes(0))throw new Error('Empty CI partition');
}catch{errors.push('.github/workflows/'+file+': name partitions must cover every static case exactly once with valid nonempty groups');}
}
}
private zero():number{return 0;}
private tierScript(name:string):boolean{return ['test:acceptance','test:regression','test:integration','test:unit'].includes(name);}
private directory(root:string,directory:string,errors:string[],specifications:string[],names:Array<string|null>):void{
for(const entry of readdirSync(directory,{withFileTypes:true})){
if(TestLayout.ignored.has(entry.name))continue;
const path=join(directory,entry.name),file=relative(root,path).replaceAll('\\','/');
if(entry.isSymbolicLink()){if(file.startsWith('tests/'))errors.push(file+': test symlink forbidden');continue;}
if(entry.isDirectory()){if(existsSync(join(path,'.git')))continue;this.directory(root,path,errors,specifications,names);continue;}
if(!/\.[cm]?[jt]sx?$/.test(file))continue;
const spec=/\.(test|spec)\.[cm]?[jt]sx?$/.test(file),tier=TestLayout.tier.test(file);
if(spec)specifications.push(file);
if(spec&&!tier)errors.push(file+': specifications require tests/{acceptance,regression,integration,unit}');
if(spec&&/(?:^|[\/-])(?:task|phase)[-_]?\d/i.test(file))errors.push(file+': permanent specifications require capability names');
const source=ts.createSourceFile(file,readFileSync(path,'utf8'),ts.ScriptTarget.Latest,true),visitor=new TestRegistrationVisitor();visitor.inspect(source);
if(spec&&tier)names.push(...visitor.names);
if(visitor.registrations&&(!tier||!spec))errors.push(file+': test registrations require a discoverable tier specification');
if(spec&&!visitor.registrations)errors.push(file+': specification has no discoverable test registrations');
}
}
}
class TestRegistrationVisitor {
registrations=0;
readonly names:Array<string|null>=[];
private readonly bindings=new Set<string>();
private readonly namespaces=new Set<string>();
inspect(source:ts.SourceFile):void{
for(const node of source.statements){if(!ts.isImportDeclaration(node)||!ts.isStringLiteral(node.moduleSpecifier)||!['bun:test','node:test'].includes(node.moduleSpecifier.text))continue;const clause=node.importClause;if(!clause)continue;if(clause.name)this.bindings.add(clause.name.text);const names=clause.namedBindings;if(names&&ts.isNamespaceImport(names))this.namespaces.add(names.name.text);if(names&&ts.isNamedImports(names))for(const item of names.elements)if(['test','it','describe'].includes((item.propertyName??item.name).text))this.bindings.add(item.name.text);}
this.visit(source);
}
private visit(node:ts.Node):void{if(ts.isCallExpression(node)&&this.test(node.expression)){this.registrations++;const name=node.arguments[0];this.names.push(name&&ts.isStringLiteralLike(name)?name.text:null);}ts.forEachChild(node,this.visit.bind(this));}
private test(expression:ts.Expression):boolean{if(ts.isIdentifier(expression))return this.bindings.has(expression.text);if(ts.isPropertyAccessExpression(expression)){if(ts.isIdentifier(expression.expression)&&this.namespaces.has(expression.expression.text))return ['test','it','describe'].includes(expression.name.text);return this.test(expression.expression);}if(ts.isCallExpression(expression))return this.test(expression.expression);return false;}
}
if(import.meta.main)new TestLayout().check();
4 changes: 2 additions & 2 deletions .agents/skills/writing-plans/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ owns the code, in that task's own step style.]
**Files:**
- Create: `packages/cli/src/commands.ts` (inside the independent CLI repo)
- Modify: that owner's documented composition class and command contract
- Test: `packages/cli/tests/smoke/cli-program.test.ts`
- Test: `packages/cli/tests/acceptance/cli-program.test.ts`

**Interfaces:**
- Consumes: [what this task uses from earlier tasks — exact signatures]
Expand Down Expand Up @@ -146,7 +146,7 @@ Expected: all applicable gates pass; record unavailable environments separately.
- [ ] **Step 5: Commit**

```bash
git add tests/smoke/cli-program.test.ts src/commands.ts
git add tests/acceptance/cli-program.test.ts src/commands.ts
git commit -m "feat: add specific feature (#N)"
```
````
Expand Down
32 changes: 32 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,38 @@ Why: docs written after code describe what was built, not what should be built.
after code verify the implementation, not the specification. This cycle forces spec-driven
development. See the `release-workflow` skill for full details.

## Permanent Test Organization (MANDATORY)

All permanent executable test specifications MUST live in the owning repository under exactly
these tier directories, named for the capability or behavior being verified:

- `tests/acceptance/`: user-observable flows, end-to-end/browser coverage and smoke tests of built programs.
- `tests/regression/`: cases that pin an identified prior defect and prevent recurrence.
- `tests/integration/`: real collaborator, provider, package, filesystem and service boundaries.
- `tests/unit/`: isolated contracts, computations and transformations.

Write tests outside-in in that order, after committing documentation and before implementation.
Split mixed-tier suites. A negative case is classified by its purpose and boundary; it is not
automatically a regression. Permanent test files MUST NOT be flat under `tests/`, colocated with
source, or named after a migration/task/phase number. Do not disguise test suites as helpers.
Shared fixture classes, fixture programs and data may use `tests/support/`, with no test
registrations there. Historical migration evidence, exact composition pins and immutable archives
retain provenance outside executable test suites; durable coverage moves into the proper tier.

Every reorganization preserves assertions, inputs, timeouts, negative cases and discovery/count
parity. Update imports, fixture/build entrypoints, public test-helper exports, package scripts,
partition inventories, CI and current docs together. Verify all four tiers are discovered; an
empty tier must be recorded honestly rather than filled with redundant tests or silently skipped.
Include an executable layout/discovery check in the normal owner verification and CI gates;
exercise rejection of misplaced tests, phase-based names and hidden helper registrations.
For a setup-only repository, install that check with its first implementation, without adding
application scaffolding or dependencies during initialization.

Every injectable seam still needs default production construction coverage, and every user
program needs a built-program acceptance test. Completion requires applicable tier suites,
checks, build, CI and assembled-artifact smoke evidence. Report missing prerequisites or gates
explicitly. A green unit suite or policy prose alone is never sufficient.

## Testing Tiers and the Composition Root (MANDATORY)

For **every injectable seam**:
Expand Down
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
## [Unreleased]

### Changed

- Organize permanent tests by tier and enforce the shared development/testing strategy (#146).
- Rebuild Admin from exact SDK .16, Identity .7 and reusable UI .16 owner archives (#137).

### Fixed
Expand Down
13 changes: 8 additions & 5 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,15 @@
"LICENSE"
],
"scripts": {
"check": "bun scripts/tooling.ts check",
"check": "bun run check:test-layout && bun scripts/tooling.ts check",
"build": "bun scripts/tooling.ts build && bun dist/src/admin.js --build-assets",
"test:unit": "bun test ./tests --test-name-pattern '^unit:'",
"test:integration": "bun test ./tests --test-name-pattern '^integration:'",
"test:smoke": "bun test ./tests --test-name-pattern '^smoke:'",
"verify": "bun run check && bun run build && bun run test:unit && bun run test:integration && bun run test:smoke"
"test:unit": "bun test ./tests/unit",
"test:integration": "bun test ./tests/integration",
"test:smoke": "bun run test:acceptance",
"verify": "bun run check && bun run build && bun run test:acceptance && bun run test:regression && bun run test:integration && bun run test:unit",
"check:test-layout": "bun .agents/skills/release-workflow/scripts/test-layout.ts",
"test:regression": "bun test ./tests/regression",
"test:acceptance": "bun test ./tests/acceptance"
},
"peerDependencies": {
"@treeseed-cordis/sdk": "0.0.0-cordis.16",
Expand Down
39 changes: 39 additions & 0 deletions tests/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Owner testing strategy


All permanent executable test specifications MUST live in the owning repository under exactly
these tier directories, named for the capability or behavior being verified:

- `tests/acceptance/`: user-observable flows, end-to-end/browser coverage and smoke tests of built programs.
- `tests/regression/`: cases that pin an identified prior defect and prevent recurrence.
- `tests/integration/`: real collaborator, provider, package, filesystem and service boundaries.
- `tests/unit/`: isolated contracts, computations and transformations.

Write tests outside-in in that order, after committing documentation and before implementation.
Split mixed-tier suites. A negative case is classified by its purpose and boundary; it is not
automatically a regression. Permanent test files MUST NOT be flat under `tests/`, colocated with
source, or named after a migration/task/phase number. Do not disguise test suites as helpers.
Shared fixture classes, fixture programs and data may use `tests/support/`, with no test
registrations there. Historical migration evidence, exact composition pins and immutable archives
retain provenance outside executable test suites; durable coverage moves into the proper tier.

Every reorganization preserves assertions, inputs, timeouts, negative cases and discovery/count
parity. Update imports, fixture/build entrypoints, public test-helper exports, package scripts,
partition inventories, CI and current docs together. Verify all four tiers are discovered; an
empty tier must be recorded honestly rather than filled with redundant tests or silently skipped.
Include an executable layout/discovery check in the normal owner verification and CI gates;
exercise rejection of misplaced tests, phase-based names and hidden helper registrations.
For a setup-only repository, install that check with its first implementation, without adding
application scaffolding or dependencies during initialization.

Every injectable seam still needs default production construction coverage, and every user
program needs a built-program acceptance test. Completion requires applicable tier suites,
checks, build, CI and assembled-artifact smoke evidence. Report missing prerequisites or gates
explicitly. A green unit suite or policy prose alone is never sufficient.



Complete tier commands use `bun test ./tests/<tier>` with only non-filtering timeout or
empty-tier reporting options. Individual-file selectors and shell wrappers do not prove
complete discovery. Name-filtered CI matrices must cover each registered case exactly once
and every partition must be nonempty; the shared layout guard verifies that inventory.
Loading
Loading