Skip to content

Try a node that refuses for its rate limit for longer, within a bound - #329

Merged
RunTimeAdmin merged 1 commit into
mainfrom
fix/rate-limit-backoff
Nov 16, 2026
Merged

RunTimeAdmin merged 1 commit into
mainfrom
fix/rate-limit-backoff

Conversation

@RunTimeAdmin

Copy link
Copy Markdown
Owner

#318 (finding 7) stopped ethers retrying an HTTP 429 inside each read, so a throttling node could no longer hold /score and /attest for minutes. Retries were left to readWithBackoff: four tries over 3.5 s. That is too short for the checker's node, rpc.quicknode.testnet.arc.io, which answers a burst with HTTP 429 and -32005 rate limit exceeded. Since the deploy the checker has left 2 to 4 agents unread each hourly epoch, and 10 to 34 at a restart: 52 of its retries and 9 of its agent errors were that refusal. The primary's node answers its rate limit inside a 200, which was already retried, and is unaffected.

What changes (read-retry.js, chain.js)

  • A refusal for the rate limit is tried up to 7 times. The wait doubles from 500 ms to at most 8 s, about 24 s at most in all. Each wait is between half and all of its step, so reads refused together do not come back together.
  • Every other failure keeps its four tries and its waits (0.5, 1, 2 s).
  • Still bounded: a read cannot hang for minutes, which is what Fix the oracle's part of the moderate findings from the 15 November performance review #318 fixed.
  • isRateLimited moves from chain.js to read-retry.js unchanged, since read-retry.js cannot load chain.js. chain.js imports it and still exports it.
  • readWithBackoff takes an optional { pause, random }, so the schedule can be tested without waiting.

Tests

chain.test.js adds two tests, both failing on main:

  • a 429 refusal tried 7 times, with waits doubling to 8 s and each at least half its step;
  • a -32005 nested as ethers nests it counts as a rate limit, the schedule ends after 7 tries, and any other failure keeps 4.

keeper.test.js: the keeper's process tests load test-support/fast-retry.js, which cuts waits of 250 ms or more to 10 ms. A keeper whose node refuses every read now fails each run only after the whole schedule, and six such runs no longer fit the test's minute. The production schedule is unchanged.

1,545 oracle tests: the full suite ran once, with one failure, that keeper test, which then passed with the shim (56 keeper tests). The documented 1,544 was one too many, because the minor round's run counted a failed test file as a test.

Deploy

Both operators. The checker is the one that needs it. The services pick it up at their next deploy.

🤖 Generated with Claude Code

#318 stopped ethers retrying an HTTP 429 inside each read, leaving retries to readWithBackoff: four tries over
3.5 s. The checker's node answers a burst with HTTP 429 and -32005, and since the deploy the checker has left 2 to
4 agents unread each hourly epoch. A rate-limit refusal is now tried up to 7 times, the wait doubling to at most
8 s and jittered, about 24 s at most; every other failure keeps four tries. isRateLimited moves to read-retry.js,
unchanged, so readWithBackoff can ask it.

The keeper's process tests load a shim that cuts long waits, so a run that fails only after the whole schedule
does not stretch the test. Oracle tests: 1545 (the documented 1544 counted a failed test file as a test).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@RunTimeAdmin
RunTimeAdmin merged commit caa83a1 into main Nov 16, 2026
7 checks passed
@RunTimeAdmin
RunTimeAdmin deleted the fix/rate-limit-backoff branch November 16, 2026 01:18
RunTimeAdmin added a commit that referenced this pull request Nov 16, 2026
#329's first epoch on the checker still left 2 agents unread, both with "exceeded maximum retry limit". That is
how ethers reports an HTTP 429 once its attempts run out, and rpc-provider.js allows one, so the 429 survives
only in the status of the response ethers escalated. isRateLimited now reads that status and counts the
escalated message as a rate limit, so those reads get the longer schedule. A test drives a real HTTP 429
through rpcProvider from a local endpoint and fails on main. Oracle tests: 1546.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RunTimeAdmin added a commit that referenced this pull request Nov 16, 2026
…ds retry [skip ci] (#331)

The runbook's table had the operators and the watcher on 9a55028. The primary and the checker run
2ed3453 since 01:34 and 01:33 UTC on 16 November, and the watcher and the policy signer 489372c since
23:49 UTC on 15 November, read from each container.

The checklist drops the line saying the publisher had not yet posted through the policy signer's hash
rule. It posts every day, most recently four posts at 00:04 UTC on 16 November, each signed by the
signer. The staking item gains #326's bound on slash evidence and the new Reader that follows it.

The oracle README said readWithBackoff allows four attempts. Since #329 a rate-limit refusal gets
seven, and the RPC_URL row now says how a read gives up and retries. The second-operator runbook's
line about the old scanner is put in the past tense.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant