Skip to content

feat(proxy): coalesce concurrent misses for a key - #21

Merged
jamiehdev merged 3 commits into
mainfrom
feat/request-coalescing
Sep 27, 2026
Merged

jamiehdev merged 3 commits into
mainfrom
feat/request-coalescing

Conversation

@jamiehdev

Copy link
Copy Markdown
Owner

When many requests miss on one cache key at once, the proxy now sends one request to the origin and the others wait for it. This stops a burst of origin traffic when a popular entry expires. Stale entries without stale-while-revalidate coalesce too, so the origin gets one conditional request.

Changes

  • A map of in-flight keys, shared by all workers, holds one entry for each key at the origin. The key is the URL plus the request's values for the Vary names the cache knows, so known variants coalesce separately.
  • Only GET and HEAD requests that the cache can serve take part, and only a GET leads. Requests with Authorization, Cookie, a conditional header or request no-cache go to the origin as before.
  • The leader streams its response as before. Its guard drops when the Tee stores the body or gives up the copy, when the origin fails, or when the client disconnects. Followers then read the cache and get X-Shadowstep-Cache: COALESCED.
  • A follower that finds nothing it can use goes to the origin alone, so it never gets the leader's uncacheable response (RFC 9111 section 3).
  • A follower waits for at most the upstream timeout.
  • /health has a coalesced counter, and hit_ratio counts it.

Verification

All twelve tests in tests/integration/coalescing.rs fail on main. Some fail there only on the absent coalesced counter, so deliberate faults test them too. A guard that never drops makes seven tests time out, the disconnect test among them. Coalesced credentials fail the bypass test, and flights that ignore Vary fail the known-variants test. The suite passed five runs in a row.

Risks

Actix does not see a client disconnect while the origin sends nothing. In that case, followers wait for the full upstream timeout.

@jamiehdev
jamiehdev force-pushed the feat/request-coalescing branch from ebe94af to d518329 Compare September 27, 2026 16:46
@jamiehdev
jamiehdev merged commit 950f71c into main Sep 27, 2026
2 checks passed
@jamiehdev
jamiehdev deleted the feat/request-coalescing branch September 27, 2026 16:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant