Evidence-first analysis, independent implementations, and reproducible experiments for LLM Agent Harness engineering.
中文说明 · Research plan · Research method · Research scope gate · A0-01 evidence map · A0-03 source audit · Existing research · Design alternatives · A0-04 first controlled experiment · A0-02 observability report · A0-02b public traces · Falsifiable Agent Loop questions · R1 study · R1 iteration 02 · R1 iteration 03 · Model & source provenance · Decision log · Agent Loop analysis · Evidence ledger
Independent research project. Not affiliated with or endorsed by Anthropic. No leaked or proprietary Claude Code source code is redistributed here.
31-item literature audit · priority reading and next research path
Expanded beyond Claude Code v2.1.88 reverse-engineering posts to first-party engineering decisions, independently controlled harness evaluations, cross-scaffold studies, and public real-session evidence. Key synthesis: Loop activation/termination, model decision authority, runtime reentry, session/context persistence, action authorization, and independent task verification are separate research dimensions. Across available studies, harness effects are conditional on workload, model and budgets; no general causal superiority of a Claude Code-style while over Goose-style step is established.
Recommendation: prioritize a paired model-stop vs independent-goal-verification evaluation for task effectiveness; keep reentry/approval as a narrowly scoped runtime-overhead experiment. Do not keep generating toy parity tests and generalize from pass counts.
A0-04-03 benchmark comparison · Reusable adapter and tests · Results · Rungta architecture essay audit
Local execution: 11 unit tests passed; both arms satisfied 21 upstream-derived input vectors and 10 locally authored boundary extensions with an identical scripted policy. This is not an official upstream benchmark run or a Claude Code/Goose product comparison. A separate deterministic two-host stale-snapshot interleaving demonstrated one effect after a competing cancellation; no real concurrency was used. The Rungta essay (2026-02-17) offers conceptual design insights but not original v2.1.88 source verification. Model autonomy and reentrant scheduling are independent axes.
Approval handoff report · Open benchmark inventory · Controller code · Regression tests
Six scripted approval/deny/cancel/reentry tests showed matched results in both arms. The simulator's streaming-style arm reconstructs from a checkpoint; this limits its value as a live-stack vs reentrant architecture comparison. These tests are not a public benchmark, and no real Claude Code, Goose or LLM was run. Next priority: reuse third-party approved-action boundary and fault suites and independently validate effect oracles.
Controlled comparison report · Independent implementation · Results
Four local tests passed, including a real child-process SIGKILL after the turn-2 atomic checkpoint followed by fresh-process resume. Both the streaming-generator and reentrant-step controllers finished with 5 scripted decisions, 4 mock tool effects and matching traces. No Claude/Goose executable or LLM was invoked. This is a deliberately narrow negative finding: with equal checkpointing, reentrancy alone did not improve recovery in this scenario. Mid-tool failures and comparative overhead remain untested.
Research update: Goose's reentrant GDK is already implemented and published in 2026; it must not be described as merely a proposal.
python3 -m unittest discover -s experiments/a0_04_loop_comparison -p 'test_*.py' -vCommunity research reuse · Seven architectural patterns and trade-offs · Falsifiable evaluation plan (no results yet)
Prior art already analyzes Claude Code v2.1.88 extensively. OpenHands, Goose and LangGraph expose alternative event-stepped, reentrant state-machine and graph/functional execution patterns. No reviewed evidence proves that Claude Code's loop representation itself is causally superior. The next study should isolate one mechanism under paired workloads and model/tool parity; do not build a seven-framework benchmark by default.
A0-03 report · source claim registry · next verification steps
We confirmed the historical v2.1.88 source-map exposure and cross-checked two pinned third-party architectural reports. No original leaked source-map artifact was retrieved and authenticated; no function-level claim is labeled source-verified (S). Current module call graphs remain version-scoped third-party evidence (T), not official source facts.
A0-01 report establishes a claim-by-claim distinction between official Claude Code behaviors and third-party source analysis of the reported v2.1.88 snapshot. A0-02 entry criteria specify observable CLI/SDK traces and version checks. Scope gate blocks unrelated runtime infrastructure work. The prior R1 simulations remain exploratory side experiments, not Claude Code behavior verification.
A0-02 report · Safe runbook · Event summarizer
The official CLI supports observable stream-json messages; however this research environment has no Claude Code executable or authentication, so no genuine CLI traces were recorded. A privacy-reduced JSONL summarizer and four synthetic unit tests are ready. Neither the synthetic fixtures nor previous Mock experiments establish Claude Code internal function behavior.
python -m unittest discover -s research/A0-01-agent-loop-path/a0_02 -p 'test_*.py' -vPublic trace evidence & limitations · Dataset manifest · Session JSONL analyzer
A real publicly hosted Claude Code v2.1.177 trace was reviewed through its Hugging Face viewer. The complete raw trace could not be downloaded here, so this project does not claim measured real-file tool-pair counts. The native-session adapter passed five synthetic unit tests; external file processing is documented, not falsely reported as executed.
python3 -m unittest discover -s research/A0-01-agent-loop-path/a0_02 -p 'test_*.py' -vExperiment 001: What happens when a tool commits a side effect but its response is lost?
Run locally with Python 3.10+, without credentials or network:
python -m unittest discover -s experiments/loop_recovery -p 'test_*.py' -v
python experiments/loop_recovery/experiment.pyThis small deterministic mock experiment compares blind retry and stable idempotency-key retry. It tests an independent design, not Claude Code itself; results should not be generalized to production agents.
Five deterministic scenarios produced identical traces and zero invariant violations in both matched variants. This is a mock smoke test, not evidence that State increases reliability. Protocol and limitations.
python -m unittest discover -s experiments/r1_state -p 'test_*.py' -v
python experiments/r1_state/experiment.pyIn a deliberately permissive deterministic toy model, 249/258 short traces accepted at least one disallowed event with either local variables or typed State. Adding a transition validator rejected all specified illegal events (0/258 violating traces). This isolates a rule-enforcement mechanism, not a Claude Code observation or proof that typed State improves reliability. Analysis and limitations.
python -m unittest discover -s experiments/r1_transitions -p 'test_*.py' -v
python experiments/r1_transitions/experiment.pyDeterministic mock crash/cancel event replay (5 scenarios, 4 arms, 5 unit tests). State as a dataclass vs local dictionary changed no results when policies matched. In a deliberately constructed cancel-then-crash sequence, a checkpoint preserved cancellation and blocked redispatch; a volatile variant lost it. No LLM calls were made. Checkpoint alone does not resolve ambiguous committed tool effects. R1-03 report · Source/model attribution.
python -m unittest discover -s experiments/r1_recovery -p 'test_*.py' -v
python experiments/r1_recovery/experiment.pyEvery iteration follows: question → public evidence → architectural interpretation → implementation → baseline comparison → reproducible result → limitations.
Claims are labeled:
- O (official): publicly documented by Anthropic or upstream maintainers.
- T (third-party): published reverse-engineering interpretation requiring cross-checking.
- H (hypothesis): our proposed explanation or mechanism.
- E (experiment): independently obtained measurement with explicit scope.
| Iteration | Question | Status |
|---|---|---|
| 001 · Agent Loop | What happens when a tool result is lost after execution? | Mock experiment available |
| 002 · State and recovery | Separate typed State, guards, and checkpoint effects | R1-01/02/03 exploratory mock studies available |
| 003 · Tool execution | When are parallelism and retries safe? | Planned |
| 004 · Context compaction | How do token savings trade off against constraint retention? | Planned |
| 005 · Harness evaluation | Which mechanisms improve reliability across real tasks? | Planned |
- Source provenance before interpretation. Never present an inference as verified source behavior.
- Compare functionally equivalent baselines.
- Report failures, variance, and limitations alongside results.
- No proprietary source dumps or unlicensed redistributions.
- Keep the loop small; test each engineering mechanism independently.
Contributions with an issue reproduction, evidence correction, or new baseline are welcome. See CONTRIBUTING.md.