Skip to content

fix: the fleet-tools test depended on a private SDK field, so it failed on file order - #297

Closed
saucam wants to merge 1 commit into
mainfrom
fix/fleet-tool-test-sdk-internals
Closed

fix: the fleet-tools test depended on a private SDK field, so it failed on file order#297
saucam wants to merge 1 commit into
mainfrom
fix/fleet-tool-test-sdk-internals

Conversation

@saucam

@saucam saucam commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Unblocks #295, whose daemon-check is red. The failure is not #295's fault — it touches no fleet code.

What actually failed

Not an assertion mismatch — a TypeError:

undefined is not an object (evaluating 'Object.keys(server.instance._registeredTools)')
  at src/tests/collaboration.test.ts:2374

_registeredTools is a private field of the MCP SDK's McpServer. Reading it goes wrong two ways, and the second is what bit us:

  1. It breaks on any SDK upgrade that renames or restructures internals.
  2. src/tests/provider-claude.test.ts:56 installs a process-global mock.module("@anthropic-ai/claude-agent-sdk", ...) whose fake createSdkMcpServer returns { instance: { tools } }no _registeredTools. Bun's module mocks leak across files within a run, so whether this test saw the real SDK or that fake depended entirely on which file the runner reached first.

Locally collaboration.test.ts is 4th and provider-claude.test.ts is 14th, so it passes. CI's readdir order differs, and it threw.

Why it's not #294 or #295

It's a latent order-dependent bug that surfaces whenever file ordering shifts — which adding any test file can do. #295 adds src/daemon/guard/repeat-tool.test.ts.

I confirmed the leak mechanism directly: two synthetic files in the same run, one registering that mock and one importing the SDK. Alone the second sees _registeredResources,…,_registeredTools,…; after the mock it sees tools — exactly the CI shape.

The fix

Fix the coupling, not the symptom. buildFleetMcpServer now builds through an exported fleetToolDefinitions() and passes that array through verbatim; the test asserts on those definitions instead of reaching into the SDK.

That still proves what the old test set out to prove — that pick() actually filters what reaches the model, not merely that the constant is well-formed — while being immune to both the SDK's internals and the mock leak.

Adds a second test pinning the seam the split introduces: fleetToolDefinitions would be worthless if the builder stopped calling it. That one asserts only the SDK's public config shape, so it holds under a real or mocked SDK.

Verification

  • previously-failing order (provider-claude first) — 227 pass, 0 fail
  • reverse order — 227 pass, 0 fail
  • full suite — 2311 pass, 0 fail; lint + typecheck clean

Note for #295

Merge main into it after this lands and its daemon-check should go green.

Worth knowing separately

provider-claude.test.ts's mock is still process-global and still shape-unfaithful. Nothing else reads _registeredTools today (I checked — this test was the only consumer), so this PR closes the live issue. But the next thing to reach into a mocked SDK object hits the same trap. Making that mock mirror the real shape would be a reasonable follow-up; I left it out to keep this fix to the failing path.

…ed on file order

`daemon-check` failed on #295 with a TypeError, not an assertion mismatch:

  undefined is not an object (evaluating 'Object.keys(server.instance._registeredTools)')
    at src/tests/collaboration.test.ts:2374

`_registeredTools` is a PRIVATE field of the MCP SDK's McpServer. Two ways that
read goes wrong, and the second is what bit us:

1. It breaks on any SDK upgrade that renames or restructures internals.
2. `src/tests/provider-claude.test.ts` installs a process-global
   `mock.module("@anthropic-ai/claude-agent-sdk", ...)` whose fake
   `createSdkMcpServer` returns `{ instance: { tools } }` — no
   `_registeredTools`. Bun's module mocks leak across files within a run, so
   whether this test saw the real SDK or that fake depended entirely on which
   file the runner reached first. Locally collaboration.test.ts is 4th and
   provider-claude.test.ts is 14th, so it passed; CI's readdir order differs and
   it threw.

Neither #295 nor #294 caused this — #295 touches no fleet code, #294 touches no
fleet.ts, and the test passes on main locally. It is a latent order-dependent
bug that surfaces whenever file ordering shifts, which adding any test file can
do.

Fix the coupling rather than the symptom. `buildFleetMcpServer` now builds its
tool list through an exported `fleetToolDefinitions()` and passes that array
through verbatim; the test asserts on those definitions instead of reaching into
the SDK. That still proves what the old test set out to prove — that `pick()`
actually filters what reaches the model, not just that the constant is
well-formed — while being immune to both the SDK's internals and the mock leak.

Adds a second test pinning the seam the split introduces: `fleetToolDefinitions`
would be worthless if the builder stopped using it. That one asserts only the
SDK's PUBLIC config shape, so it holds under a real or a mocked SDK.

Verified under both file orders (provider-claude first and last) and on the full
suite: 2311 pass, 0 fail.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@saucam

saucam commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator Author

Redundant — #296 landed the same fix (a0bc9a2) about an hour before I started, and my local origin/main was stale when I diagnosed this. That's my error: I should have re-fetched before opening.

#296's approach is also the better one. It derives registeredToolNames from the array handed to createSdkMcpServer (same principle as my fleetToolDefinitions), and keeps the cross-check against the live MCP surface guarded by if (!internals) return — so it degrades to a skip if the SDK's internals move again rather than going red. Mine dropped that cross-check entirely.

Closing. One observation from my diagnosis that may still be worth a follow-up: src/tests/provider-claude.test.ts:56 installs a process-global mock.module("@anthropic-ai/claude-agent-sdk") whose fake createSdkMcpServer returns { instance: { tools } } with no _registeredTools. Bun's module mocks leak across files within a run, so that mock — not only an SDK bundling change — can produce exactly the observed undefined is not an object shape, depending on file order. #296's guard absorbs it either way; making the mock mirror the real shape would remove the trap for whatever reaches into a mocked SDK object next.

@saucam saucam closed this Aug 19, 2026
@saucam
saucam deleted the fix/fleet-tool-test-sdk-internals branch August 19, 2026 03:14

@highflame-oracle highflame-oracle Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔮 Oracle Review

🎯 Start Here

src/daemon/fleet.ts (~15 min) — Logic changes in fleet.ts


📋 PR Summary

What this PR does: Fixes flaky test failures in collaboration.test.ts by removing dependency on a private SDK field (_registeredTools) that was causing failures when Bun's module mocks leaked between test files in different execution orders.

Key changes:

  • Extracted fleetToolDefinitions() as a public API function in fleet.ts
  • Refactored buildFleetMcpServer to use the exported tool definitions instead of accessing private SDK internals
  • Updated tests to assert on public fleetToolDefinitions() instead of SDK's _registeredTools field
  • Added integration test verifying buildFleetMcpServer correctly calls fleetToolDefinitions()

Areas affected: src/daemon/fleet.ts, src/tests/collaboration.test.ts

Testing notes: Verified 227+ tests pass in both problematic and reverse file orderings; full suite 2311 tests pass with lint and typecheck clean.


🔍 Code Review

This is an excellent root-cause fix that properly decouples tests from private SDK internals rather than patching the symptom. The author's thorough investigation identified the Bun module mock leakage mechanism and implemented a clean separation that makes tests robust against SDK changes and execution order.

What's good:

  • ✨ Strong diagnostic work identifying the coupling to private SDK fields as the root cause
  • ✨ Proper architectural fix extracting a stable public API (fleetToolDefinitions) for testing seams
  • ✨ Comprehensive verification across multiple test execution orderings to prevent regression
  • ✨ Clear documentation of the issue mechanism and why adjacent PRs were not responsible

Review Stats: warning:1


Generated by Oracle - Highflame's AI Code Reviewer

Comment thread src/daemon/fleet.ts
@@ -559,10 +589,7 @@ export function buildFleetMcpServer(
const pick = <T extends { name: string }>(tools: T[]): T[] =>

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Avoid as never type assertion

The refactor introduces an as never cast on the tools argument passed to createSdkMcpServer. While this may be necessary to satisfy the compiler after extracting buildFleetTools, it completely bypasses type checking for the tool definitions. If the tool shape returned by buildFleetTools deviates from what the SDK expects, this error will be hidden until runtime. Consider using satisfies or explicit generic typing if possible to maintain type safety.

Suggested fix:

Suggested change
const pick = <T extends { name: string }>(tools: T[]): T[] =>
If strict typing is impossible, consider a comment explaining *why* `as never` is required, or check if the SDK's `createSdkMcpServer` can be typed to accept the inferred return type of `buildFleetTools`.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants