Skip to content

fix: close the 2026-09-27 follow-up review findings (N01–N08) - #169

Merged
CodeWithJuber merged 5 commits into
masterfrom
claude/forgekit-deep-review-issues-ww6y0h
Sep 27, 2026
Merged

CodeWithJuber merged 5 commits into
masterfrom
claude/forgekit-deep-review-issues-ww6y0h

Conversation

@CodeWithJuber

@CodeWithJuber CodeWithJuber commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner

What & why

This fixes the eight findings of the 2026-09-27 follow-up review of v1.7.1 (N01–N08) and applies its suggestions. Three adversarial reviews of those fixes then found further counterexamples in every area; the third commit closes them. The review package's acceptance scripts both pass on this branch:

  • node reproduce_remaining.mjs <repo> --assert-fixed exits 0: all of N01–N08 are fixed.
  • node original-reproduce.mjs <repo> --assert-fixed still exits 0: the original F01–F16 stay fixed.

Fixes, in the review's priority order:

  • N03, workspace detection. A root test script covered the workspaces whenever -r or --workspaces appeared anywhere, so node -r ./setup.cjs --test hid a failing workspace behind a PASS.
    • The script is now read as shell structure: commands, quotes, wrappers and exit-status flow.
    • Only a known, unfiltered recursive run of every member's test script counts: npm, pnpm, yarn, turbo, lerna or nx, directly or through npm run hops.
    • Membership comes from that tool's own workspace list, ! negations included.
    • Each package's verdict is labelled measured, inferred or declared.
  • N01, exact reuse identity. Exact reuse is now byte-exact (key v3).
    • The exact tier compares a digest of the spec's code units.
    • The near tier's semantic guard now also compares code layout and no longer folds Unicode.
    • Inline code the ledger would rewrite (CRLF, non-NFC text) is refused at mint instead of silently folded.
  • N08, dependency contracts. A dependency's contract is its whole declaration minus the body.
    • The atlas now records each definition's extent (atlas v5), and the declaration is lexed, so destructured keys, defaults and annotations count while comments and formatting don't.
    • The contract is bound to the module the artifact actually imports.
    • An ambiguous or older-format contract is reported unknown, never valid.
  • N02, rule merging. Only exact duplicates merge.
    • Near-duplicates are kept and listed as proposed.
    • A lesson is dropped only when a claim with exactly its text is refuted; one that merely resembles a refuted claim is kept and flagged.
    • forge ledger compact follows the same rule.
  • N06/N07, provenance ingestion.
    • Verifier events carry a derived authenticated flag, and nothing is authenticated when no evidence key is available.
    • The event MAC (contract v2) now covers every field.
    • Outcome provenance is re-derived on every read from authenticated events with a matching verdict, and one run backs one attempt. An edited label counts for nothing.
  • N04, context delivery. A definition is delivered whole or not at all. A span that cuts the body leaves the definition pending and names it under partial.
  • N05, nested repositories. Nested repositories and submodules are bound into the fingerprint by their own state (manifest-v3).
    • Anything still unbound turns a PASS into INCOMPLETE.
    • verify.external declares an intentional boundary, and every verifier event records it.

Adversarial round 2 (third commit). Every counterexample the three re-reviews reported is closed. Each fix leans conservative: what cannot be established reads as INCOMPLETE, unknown, proposed or self-reported.

  • N03:
    • A per-tool option allowlist (--help, --dry-run, --prefi=…, --tag test and anything after -- are no longer credited).
    • Stateful shell builtins (cd, exit, trap, export…) and npm_config_* assignments are refused, and argument-passing hops are not followed.
    • Narrowing config is honoured: .npmrc/env, lerna, nx, turbo.
    • Tool caches must be bypassed: turbo --force, nx/lerna --skip-nx-cache.
    • Membership changes:
      • lists follow the package manager in use;
      • negations are conservative;
      • yarn members need a version;
      • nx project config or a masking member script means a separate run.
    • bun run test replaces bun test.
    • A non-shell script-shell is INCOMPLETE.
    • A recognized-but-refused root command is reported with its reason.
  • N05:
    • Gitlinks without .gitmodules are bound, and .gitmodules is read by git's own parser.
    • A missing registered path is unbound.
    • Nested repos are read under the outer config, and their core.fsmonitor never runs.
  • N01:
    • Near needs a current, verbatim-stored key.
    • The guard adds operators with their operands, typographic and backtick literals, document order, all whitespace, spelling/look-alikes and format characters.
  • N08:
    • Every import form is resolved through the shared resolver: aliases, Python, re-export barrels, and whole-module digests for default, namespace, require and import().
    • Export aliases and alias consts are followed, and a dropped export counts as a change.
    • Decorators and split keywords are read, the contract is lexed on masked text, and a stale dependency file reads as unknown.
  • N04:
    • Extents are now correct for generics, return-type literals, overloads, Go result types, templates, Ruby end, Python strings and continuations, and JSX apostrophes.
    • An untrusted extent is unknown.
    • A stale atlas locates nothing.
  • N02:
    • The statement key folds only edge whitespace and one sentence period.
    • Fact names, lesson triggers and scope are part of identity.
    • A lesson drops only when every exact claim is refuted.
    • Small ledgers still report near-duplicates.
  • N06/N07:
    • Events record their checkout, and outcomes record their code state.
    • A run backs an attempt only in its own checkout and on the same code.
    • Attempt keys cannot alias.
    • Non-finite numbers never authenticate.

Suggestions applied:

  • maxPossibleCost is renamed estimatedCostIfAllAttemptsRun.
  • The verifier event is a versioned contract covering the whole event.
  • New seeded property tests cover quoted whitespace, binding swaps, command-option ambiguity, destructured parameters and a missing evidence key.
  • Coverage is labelled measured, inferred or declared.
  • The claim registry is a release artifact:
    • Every claim records assessed_release and can link review counterexamples.
    • bump.mjs stamps unreleased claims with the version it cuts.
    • With release tags present, claims-status --check fails on a stale unreleased.
    • Twelve stale "master after 1.4.3 (unreleased)" entries now name 1.5.0.
  • The research executive summary marks its 2026-09-26 correction in place.

Docs are updated in the same change: GUIDE, the Mintlify CLI and concept pages, UNIVERSAL_ROUTING, the substrate plan docs, ARCHITECTURE, README and CHANGELOG (with the rendered Mintlify changelog page). The second and fourth commits re-assess the affected claims against the code commits (2699aa0, then a6be83c).

Checklist

  • npm test passes: 1,765 tests, 1,761 pass, 0 fail, 4 platform-gated skips (Node 22)
  • npm run check passes (Biome lint + format; 0 errors)
  • New public functions have a test (recursiveTestInvocation, analyzeRecursiveTestRun, reachesWorkspace, scriptShellProblem, shellCommands, isWorkspaceMember, specDigest, depContract v2, indexedText, literalSpans, layoutFeatures, sameStatement, checkoutId, releaseProblems, stampRelease, …)
  • Conventional commit messages
  • CHANGELOG.md updated under ## [Unreleased]
  • No new runtime dependency
  • Substrate/docs updated: forge verify, forge context, forge reuse, forge ledger compact, forge route outcome and the router all changed, and their docs changed with them

Risk & rollback

  • Risk level: medium. Several formats change and migrate on their own:
    • atlas v5 rebuilds;
    • reuse key v3 (older artifacts never exact-hit, and never reach near);
    • dependency contracts v2: (older ones read as unknown) and new artifact fields moduleDeps/keyVerbatim;
    • fingerprint manifest-v3 (older stamps no longer verify, so re-run forge verify);
    • verifier events v2 with a checkout field (v1 events still verify for their smaller scope, but no longer back verify-event outcomes);
    • outcome rows record codeState, so outcomes backed by older events read as self-reported until forge verify runs again.
  • Every change is conservative, so failures show up as more INCOMPLETE, unknown, proposed or self-reported results, never as a false PASS. Recursive root runs that were credited before may now be refused, for example turbo without --force; the members then run on their own, and the CLI says why.
  • One JSON field is renamed: maxPossibleCost → estimatedCostIfAllAttemptsRun in forge route universal --json. A Bun project's root suite now runs bun run test.
  • Rollback plan: revert the merge commit. No data migration is needed; the old formats are rebuilt or re-verified the same way.

Extra checks (tick if applicable)

  • npm run typecheck passes
  • Input validated at boundaries; errors handled (no swallowing)
  • Authorization/ownership checked (if it touches access): n/a
  • Logs contain no secrets/PII
  • If AI-assisted: I understand it, verified the package APIs, and it has tests

🤖 Generated with Claude Code

https://claude.ai/code/session_01GVVG2VDETWsDxMu6MBWPz2

N03 - a root test script covers the workspaces only when it is a recognized,
unfiltered recursive run of every member's test script whose failure reaches
the exit status; node -r, filters and masked runs cover nothing. Coverage is
labelled measured, inferred or declared.

N01 - exact reuse is byte-exact (key v3: a digest of the spec's code units).
The near tier's semantic guard compares code layout and no longer folds
Unicode; inline code the ledger would rewrite is refused at mint.

N08 - a dependency contract is its whole declaration minus the body (the atlas
records definition extents, v5), bound to the module the artifact imports.

N02 - consolidation and ledger compaction merge or archive exact duplicates
only; near-duplicates are proposed, and only an exact refuted claim drops a
lesson.

N06/N07 - verifier events carry a derived authenticated flag (none without a
key; the v2 MAC covers every field), and outcome provenance is re-derived on
every read from authenticated events with a matching verdict.

N04 - a context span delivers a definition only whole; cut ones stay pending
and are listed under `partial`.

N05 - nested repositories and submodules are bound by their own code state
(manifest-v3); unbound code turns a PASS into INCOMPLETE unless declared in
verify.external.

Also: maxPossibleCost is renamed estimatedCostIfAllAttemptsRun; the claim
registry records assessed_release and review counterexamples, and bump.mjs
stamps unreleased claims; seeded property tests along the semantic
boundaries; linear scans replace two backtracking regexes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GVVG2VDETWsDxMu6MBWPz2
Re-assessed against 2699aa0 (assessed_release "unreleased" until a release
ships it): reuse-exact-identity and phase-p3-reuse-cache (N01, N08),
verify-pass-binding (N03, N05), context-completeness and
phase-p4-context-assembly (N04), route-outcome-provenance (N06, N07), and the
budget contract's renamed cost field. New claim similar-rules-never-merged
records the N02 guarantee. Each links its review counterexamples.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GVVG2VDETWsDxMu6MBWPz2
…N01-N08 fixes

Three adversarial reviews of the N01-N08 fixes found further ways to get a
false PASS, a wrong reuse hit, a merged rule or a borrowed verified label.
Each is fixed conservatively: when something cannot be established, the
result is INCOMPLETE, unknown, proposed or self-reported, never PASS/valid.

N03, workspace coverage:
- per-tool option allowlist; `--` refused
- stateful shell builtins and npm_config_* assignments refused; a hop that
  passes arguments is not followed
- .npmrc/env, lerna, nx and turbo config that narrows the run honoured
- turbo needs --force, nx/lerna --skip-nx-cache
- package-manager-aware lists; conservative negations; yarn members need a
  version; nx project config or masked member scripts run on their own
- `bun run test` replaces `bun test`
- a non-shell script-shell is INCOMPLETE
- refused runs reported with the reason

N05, nested repositories:
- gitlinks from the index are bound; .gitmodules is read by git itself
- missing registered paths are unbound
- nested repos read under the outer config, their committed .gitignore only
- fsmonitor never runs

N01, reuse near tier:
- keys must be current and stored verbatim
- the guard compares symbols with operands, typographic/backtick literals,
  document order, all whitespace, spelling and format characters

N08, dependency contracts:
- every import form resolved through the shared resolver (aliases, Python,
  barrels); whole-module digests (moduleDeps)
- export aliases and alias consts followed; a dropped export is a change
- decorators, split keywords and regex literals read; lexing on masked text
- stale dependency files unknown

N04, definition extents:
- correct for generics, return-type literals, overloads, Go result types,
  templates, Ruby `end`, Python strings/continuations, JSX apostrophes
- untrusted extents (unbalanced, preprocessor branches) unknown
- a stale atlas locates nothing

N02, rule identity:
- statementKey folds only edge whitespace and one sentence period
- fact names, lesson triggers and scope are identity
- drop only when every exact claim is refuted
- small ledgers still report near-duplicates

N06/N07, provenance:
- events record their checkout; outcomes record their code state
- a run backs an attempt only in its checkout, on the same code
- attempt keys cannot alias
- non-finite numbers never authenticate

Docs (GUIDE, Mintlify, routing, plans), CHANGELOG and the rendered changelog
page updated in the same change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GVVG2VDETWsDxMu6MBWPz2
The internal adversarial re-review of 2699aa0 found counterexamples against
seven claims (reuse identity, the reuse cache, PASS binding, context
completeness and assembly, outcome provenance, rule merging); a6be83c repairs
them with regression tests. Each claim is re-assessed against a6be83c, its
notes name what the re-review found and the scope that still applies, and the
generated table is refreshed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GVVG2VDETWsDxMu6MBWPz2
…l-clock bound

The Windows runner failed "the literal scanner, symbols and spelling are
linear on hostile input": 4.0 s against a 4 s bound, about 3× the local time.
The bound was machine-dependent, and the run exposed real super-linear work:
- literalSpans re-scanned the rest of an unbroken line (`indexOf`) from every
  ASCII quote; each line's end is now found once;
- an unmatched backtick run scanned to the end once per distinct run length;
  runs are now indexed by length with a forward-only cursor;
- semanticConflicts compared differing feature lists with a pairwise
  `includes` (quadratic in the feature count) and masked each text twice; it
  now uses Set lookups and analyzes each text once.

The test now measures scaling (4× the input must take well under the 16× a
quadratic scan would), which holds on a fast machine and a slow CI runner
alike, and adds a case with thousands of differing identifiers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GVVG2VDETWsDxMu6MBWPz2
@CodeWithJuber
CodeWithJuber marked this pull request as ready for review September 27, 2026 04:16
@CodeWithJuber
CodeWithJuber merged commit 24cc4b3 into master Sep 27, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants