fix: retry forester epoch processing failures

[sergeytimoshin]

Summary

• retry epoch processing after transient RPC/provider failures
• use exponential backoff from 2 seconds up to a 60-second cap
• stop retries after the epoch report-work deadline and emit retry/recovery events

Root cause

The epoch monitor emits each epoch once. If process_epoch failed during that one attempt—for example, when the RPC provider returned HTTP 500—the spawned task exited and the forester never built that epoch's schedule. Processing remained stuck until the process was restarted.

Impact

Foresters now recover automatically from transient failures during epoch setup instead of silently remaining inactive for the rest of the epoch. Retry tasks are bounded by the report-work deadline.

Validation

• cargo test -p forester test_epoch_processing_retry_delay_is_exponential_and_capped --lib
• cargo check -p forester
• git diff --check

[coderabbitai]

Warning

Review limit reached

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 2 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 1dfc1705-47f4-4653-9280-ad1a3adfb02f

📥 Commits

Reviewing files that changed from the base of the PR and between ad5964f and 3ed4560.

📒 Files selected for processing (1)

• forester/src/epoch_manager.rs

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-08T12:16:23.587707Z	3ed4560	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3ed45606c4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Bound the backoff by the report-work deadline

When an attempt fails with less than retry_delay remaining in the report-work phase—for example, a transient report_work_onchain failure—the unconditional sleep can carry the task past phases.report_work.end. The subsequent process_epoch call then skips the on-chain report and returns Ok(()), so the retry loop logs recovery even though has_reported_work remains false and the epoch's work is omitted from total_work. Cap the sleep to the remaining phase time or otherwise prevent a post-deadline attempt from being treated as successful.

Useful? React with 👍 / 👎.