Skip to content

Read an agent again once when the node refused its reads through every retry - #332

Merged
RunTimeAdmin merged 1 commit into
mainfrom
fix/guard-read-retry
Nov 16, 2026
Merged

RunTimeAdmin merged 1 commit into
mainfrom
fix/guard-read-retry

Conversation

@RunTimeAdmin

Copy link
Copy Markdown
Owner

At 02:34 UTC on 16 November 2026, the checker's first hourly epoch on 2ed3453 (#330) skipped agent 0x31d08650 with exceeded maximum retry limit. Its node, rpc.quicknode.testnet.arc.io, refused the guard step's first chunk of ten getIdentity reads for about 18 s (02:34:48 to 02:35:06). Nine got through within readWithBackoff's seven tries for a rate limit: 24 waits were over 2 s, so the long schedule ran. One ran out of tries. The guard loop logged it and moved on, so that agent waited an hour. Refund reads at 02:34:35 to 38 had drained the node's budget just before.

What changes (index.js, epoch-policy.js)

  • Guard reads. An agent whose guard read (getAgentInfo and isBonded, just before its write) is still refused with a transient error goes back on the end of the queue once. It is read again after the other agents, and at least REREAD_PAUSE_MS (10 s) after the refusal. No chunk mixes first-pass agents with agents put back. One line is logged when an agent is put back.
  • First reads. Agents whose batched first reads were refused the same way are read once more after the same pause, before the refund reads.
  • Only before writes. Both retries happen before anything is written for the agent, so nothing is repeated. An agent refused a second time waits for the next epoch, as before.
  • Unchanged. The per-agent step after the guards, which writes, behaves as before, including its payer standing reads. RATE_LIMIT_ATTEMPTS stays at 7.
  • Docs. The oracle README's RPC_URL row says this.

Tests (robustness.test.js)

Three end-to-end tests run the real index.js against the stubbed chain. The stub has a new key, refusedReads, which refuses chosen reads of an agent the way ethers reports a 429 it gave up on.

  • A guard read refused once: the agent is read a third time and written after the agent beside it.
  • A guard read refused twice: the agent is put back once and not read a fourth time.
  • A first read refused: the agent is read again, then written.

All three fail on main's index.js and pass here. The harness now takes a preload list, and these tests load fast-retry.js, which cuts the epoch's pauses to milliseconds and leaves the epoch interval alone. Oracle tests: 1549, from one full run.

Deploy

Both operators, checker first. The checker is the one that needs it.

🤖 Generated with Claude Code

…y retry

At 02:34 UTC on 16 November 2026 the checker's node refused the guard step's first chunk of ten
getIdentity reads for about 18 s. Nine got through within readWithBackoff's seven tries for a rate
limit; one ran out, and the guard loop left its agent for the next epoch.

An agent whose guard read is still refused now goes back on the end of the queue once, and is read
again after the others and at least REREAD_PAUSE_MS (10 s) after the refusal. Agents whose batched
first reads were refused are read once more after the same pause. Both come before anything is
written for the agent, so nothing is repeated; refused again, it waits for the next epoch as before.
The per-agent step after the guards, which writes, is unchanged.

Three end-to-end tests run the real index.js against a stub that refuses chosen reads, as ethers
reports a 429 it gave up on; all three fail on main. The harness takes a preload, and these tests
load fast-retry.js to cut the epoch's pauses. Oracle tests: 1549.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@RunTimeAdmin
RunTimeAdmin merged commit 04691ae into main Nov 16, 2026
7 checks passed
@RunTimeAdmin
RunTimeAdmin deleted the fix/guard-read-retry branch November 16, 2026 03:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant