Skip to content

fix(client): retry rate limits and reduce polling - #38

Merged
gabriel-samfira merged 2 commits into
cloudbase:mainfrom
rm-you:fix/api-rate-limits
Oct 19, 2026
Merged

gabriel-samfira merged 2 commits into
cloudbase:mainfrom
rm-you:fix/api-rate-limits

Conversation

@rm-you

@rm-you rm-you commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

A single HTTP 429 during server creation, deletion, or status polling currently fails the operation. The one-second status polling interval also adds substantial API load when multiple runners are provisioning.

Configure each service client's Gophercloud retry hook for up to 10 retries on rate-limit responses (429 and Gophercloud's existing 498 handling). Honor Retry-After seconds or HTTP dates; otherwise use exponential backoff from 2 seconds with jitter, capped at 60 seconds. Backoff respects context cancellation. Poll server status every five seconds after the previous check completes, retaining the existing operation deadlines.

Tests cover client configuration, create/delete recovery, the 11-request limit, non-rate-limit errors, Retry-After parsing and overflow, cancellation, and polling cadence. Timer tests use Go's virtual clock without changing production intervals. Full vendored tests with the race detector, repository lint, formatting, and dependency consistency checks pass.

The retry policy applies after service-client initialization. Initial authentication, endpoint discovery during construction, and Keystone reauthentication are outside this change; the existing constructor authenticates before exposing the ProviderClient.

rm-you and others added 2 commits September 29, 2026 22:30
A Retry-After value was used as is, so ten retries of a response with
Retry-After: 300 could hold one call for about 50 minutes, longer than
GARM's runner bootstrap timeout. Honor it up to the one-minute ceiling
that the exponential backoff already uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rm-you added a commit to rm-you/garm-provider-openstack that referenced this pull request Oct 14, 2026
Includes the follow-up that caps Retry-After at the one-minute backoff
ceiling.

(cherry picked from commit 1312a86)
(cherry picked from commit 4c3e8bd)

@gabriel-samfira gabriel-samfira left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really nice! LGTM. I was a bit worried about waitFor looping forever but coupled with the waitCtx it is fine. Thanks!

@gabriel-samfira
gabriel-samfira merged commit f2abe80 into cloudbase:main Oct 19, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants