Repository navigation
fix: fail fast on unreachable Redis instead of queueing every command - #14
Merged
Merged
Conversation
AddCacheWeaveRedis set AbortOnConnectFail = false but left BacklogPolicy at its default, which queues commands while disconnected. With no reachable Redis, a cache-decorated request paid a StringGetAsync on the way in and a StringSetAsync on the way out, each blocking for the full SyncTimeout — ~10-12s of added latency per request, indefinitely, not a one-time connect cost. CacheWeave already converts any cache failure to a miss, so the request returned correct data with a 200 and the degradation was invisible in logs and metrics. Set BacklogPolicy.FailFast alongside AbortOnConnectFail: both settings express one policy — never throw on connect, and never block a request waiting for a connection that isn't there. FailFast is not consulted on the success path, so behaviour against a reachable Redis is unchanged. Evictions are the one case where a dropped command matters: a [CacheWeaveEvict] delete lost during a failover leaves a stale entry served until it expires. RemoveAsync and RemoveByPrefixAsync therefore wait up to 1.5s for the multiplexer to reconnect and retry once; deleting an already-deleted key is a no-op, so replaying a partially applied eviction is safe. Reads and writes stay fail-fast. Also: - Add an Action<ConfigurationOptions> overload, as a separate overload so existing calls still bind. It runs after the defaults, so callers can override them — including back to BacklogPolicy.Default — and reach settings no connection string can express (TLS certs, Sentinel, Azure token auth). - Use TryAddSingleton for IConnectionMultiplexer, matching CacheWeave.Legacy. A multiplexer registered after the call still wins, so the existing re-registration workaround keeps working. - Apply the same fixes to CacheWeave.Legacy (net48), which had the same defect. - Document connection behaviour and the eviction trade-off in the README. Warrants a minor version bump: an app relying on the backlog to ride out a brief blip now takes a cache miss instead of up to SyncTimeout of latency.
The eviction retry window, its poll interval and the SCAN page size were hard-coded constants, reachable only from an internal constructor meant for tests. That repeated the original complaint about this package: a policy the library decided for you with no way to change it. Add RedisCacheOptions, bound through IOptions<T> like SQLiteCacheOptions and DynamoDbCacheOptions already are, and configured via a configureCache callback on every AddCacheWeaveRedis overload — including the one taking an existing IConnectionMultiplexer, which previously offered no provider configuration at all. Defaults are unchanged: 1.5s window, 50ms poll, 250-key batches. EvictionRetryWindow = TimeSpan.Zero now genuinely disables the retry. It did not before: WaitForReconnectAsync returns true immediately when the multiplexer is already connected, so the retry still ran regardless of the window. Caught by the test written for the documented behaviour. Constructor validates the options and throws ArgumentOutOfRangeException on a negative window, a non-positive poll interval, or a scan page size below 1. Applied to CacheWeave.Legacy too. The connection-configure parameter is renamed configureConnection to sit alongside configureCache; both are new in this unreleased branch, so no shipped signature changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
AddCacheWeaveRedissetAbortOnConnectFail = falsebut leftBacklogPolicyat its StackExchange.Redis default, which queues commands while disconnected. With no reachable Redis, a cache-decorated request pays aStringGetAsyncon the way in and aStringSetAsyncon the way out, each blocking for the fullSyncTimeout(5s default).That's ~10–12s of added latency per request, indefinitely — not a one-time connect cost. Every cycle costs the same, because the multiplexer never connects and each command re-enters the backlog.
The README promises "Redis outage degrades to cache-miss behaviour, never a 500". That holds functionally — correct data, HTTP 200 — but not operationally. A service whose cache backend disappears gets an effective outage that looks like success in every log and metric. One downstream service ran like this for nine days without anyone noticing, because it never errored.
Reported with measurements by a consuming team; independently confirmed against this codebase.
The fix
BacklogPolicy.FailFastalongsideAbortOnConnectFail. The two express one policy: never throw on connect, and never block a request waiting for a connection that isn't there.FailFastis never consulted on the success path, so a reachable Redis behaves exactly as before.connectTimeoutalone — the obvious fix — does not help: it only shortens the one-time connect. TuningsyncTimeoutgets to ~2s and then hits a floor that is the backlog policy itself, which no connection string can express.Evictions
Fail-fast is the right trade for reads and writes — a dropped one is just a cache miss. Evictions are different: a
[CacheWeaveEvict]delete lost during a failover leaves the old entry served until it expires.So
RemoveAsyncandRemoveByPrefixAsynccatchRedisConnectionException, wait up to 1.5s for the multiplexer to reconnect, and retry once. Deleting an already-deleted key is a no-op, so replaying a partially applied prefix eviction is safe. A genuine outage costs a write request 1.5s at most, once per eviction, instead ofsyncTimeouton every operation. If the retry doesn't land,CacheWeaveEvictFilterlogs its existing "cache may be stale" warning and the request still succeeds.Raised in review of the first version of this branch, which dropped those deletes silently.
Also in here
Action<ConfigurationOptions>overload. Added as a separate overload rather than an optional parameter, so code compiled against the one-arg method still binds. It runs after the defaults, so callers can override them — including back toBacklogPolicy.Default— and reach what no connection string can express: TLS certs, Sentinel, Azure token auth,SocketManager.TryAddSingletonforIConnectionMultiplexer, matchingCacheWeave.Legacy. A multiplexer registered after the call still wins, so the re-registration workaround consumers currently ship keeps working.CacheWeave.Legacy(net48) had the identical defect and gets the identical fix.Compatibility
FailFastisn't on the success pathSyncTimeout. The filter already converts either to a cache miss, so the outcome is identical — only the latency changesThe one caller who could notice is someone implicitly relying on the backlog to ride out a blip and preserve a cache hit at the cost of up to
SyncTimeout. Hence theconfigureescape hatch — and this should ship as 1.1.0 with a release note, not a patch.Tests
234 passing on net8.0/net9.0/net10.0, Release, with coverage. 13 new here, 5 new in Legacy (builds clean; needs the Windows CI job to run).
RedisCacheProvideris at 100% line and branch coverage.Deterministic assertions — no network, no timing bounds: the defaults land on the resolved options; connection-string timeouts survive;
configureruns after the defaults and can override them; a pre-registered multiplexer survives; the eviction retry fires both immediately and after a reconnect poll, and rethrows when the connection never returns; reads don't retry.Two behavioural tests do use a dead port (127.0.0.1:1) with a 10s bound against a 60s
asyncTimeout— a wide enough margin to be CI-safe while still failing outright under the old policy. Both were mutation-checked: revertingBacklogPolicyfails the first, revertingRemoveAsyncto a direct delete fails the second.Not in here
Two unrelated issues found in the same pass, deliberately left for their own PRs:
LICENSEand every csproj say MIT. A real blocker for anyone doing licence review before adopting. Looks likefeature/switch-to-mit-licensemissed the line.IKeyContextProviderfails open — whenGetContextSegmentAsyncreturns null,DefaultCacheKeyBuildersilently omits the tenant segment, producing a context-free key shared across all tenants. A claim-resolution failure converts a tenant-isolated cache into a shared one, which is the precise leak the feature exists to prevent. Security-relevant; wants its own fix and a decision on whether null should mean skip caching.