Skip to content

fix: fail fast on unreachable Redis instead of queueing every command - #14

Merged
teghoz merged 3 commits into
developfrom
fix/redis-backlog-failfast
Sep 20, 2026
Merged

teghoz merged 3 commits into
developfrom
fix/redis-backlog-failfast

Conversation

@teghoz

@teghoz teghoz commented Sep 19, 2026

Copy link
Copy Markdown
Owner

The defect

AddCacheWeaveRedis set AbortOnConnectFail = false but left BacklogPolicy at its StackExchange.Redis default, which queues commands while disconnected. With no reachable Redis, a cache-decorated request pays a StringGetAsync on the way in and a StringSetAsync on the way out, each blocking for the full SyncTimeout (5s default).

That's ~10–12s of added latency per request, indefinitely — not a one-time connect cost. Every cycle costs the same, because the multiplexer never connects and each command re-enters the backlog.

The README promises "Redis outage degrades to cache-miss behaviour, never a 500". That holds functionally — correct data, HTTP 200 — but not operationally. A service whose cache backend disappears gets an effective outage that looks like success in every log and metric. One downstream service ran like this for nine days without anyone noticing, because it never errored.

Reported with measurements by a consuming team; independently confirmed against this codebase.

The fix

BacklogPolicy.FailFast alongside AbortOnConnectFail. The two express one policy: never throw on connect, and never block a request waiting for a connection that isn't there. FailFast is never consulted on the success path, so a reachable Redis behaves exactly as before.

connectTimeout alone — the obvious fix — does not help: it only shortens the one-time connect. Tuning syncTimeout gets to ~2s and then hits a floor that is the backlog policy itself, which no connection string can express.

Evictions

Fail-fast is the right trade for reads and writes — a dropped one is just a cache miss. Evictions are different: a [CacheWeaveEvict] delete lost during a failover leaves the old entry served until it expires.

So RemoveAsync and RemoveByPrefixAsync catch RedisConnectionException, wait up to 1.5s for the multiplexer to reconnect, and retry once. Deleting an already-deleted key is a no-op, so replaying a partially applied prefix eviction is safe. A genuine outage costs a write request 1.5s at most, once per eviction, instead of syncTimeout on every operation. If the retry doesn't land, CacheWeaveEvictFilter logs its existing "cache may be stale" warning and the request still succeeds.

Raised in review of the first version of this branch, which dropped those deletes silently.

Also in here

  • Action<ConfigurationOptions> overload. Added as a separate overload rather than an optional parameter, so code compiled against the one-arg method still binds. It runs after the defaults, so callers can override them — including back to BacklogPolicy.Default — and reach what no connection string can express: TLS certs, Sentinel, Azure token auth, SocketManager.
  • TryAddSingleton for IConnectionMultiplexer, matching CacheWeave.Legacy. A multiplexer registered after the call still wins, so the re-registration workaround consumers currently ship keeps working.
  • CacheWeave.Legacy (net48) had the identical defect and gets the identical fix.
  • README: a "Redis connection behaviour" section documenting the defaults, the configure hook and bringing your own multiplexer, plus the eviction trade-off. The package README previously documented no connection configuration at all, which is why this took a decompile to diagnose rather than a doc lookup.

Compatibility

Scenario Change
Reachable Redis None. FailFast isn't on the success path
Unreachable Redis Commands fail immediately instead of after SyncTimeout. The filter already converts either to a cache miss, so the outcome is identical — only the latency changes
Brief blip (failover, restart) Reads/writes become misses instead of waiting. Evictions still land, via the retry

The one caller who could notice is someone implicitly relying on the backlog to ride out a blip and preserve a cache hit at the cost of up to SyncTimeout. Hence the configure escape hatch — and this should ship as 1.1.0 with a release note, not a patch.

Tests

234 passing on net8.0/net9.0/net10.0, Release, with coverage. 13 new here, 5 new in Legacy (builds clean; needs the Windows CI job to run). RedisCacheProvider is at 100% line and branch coverage.

Deterministic assertions — no network, no timing bounds: the defaults land on the resolved options; connection-string timeouts survive; configure runs after the defaults and can override them; a pre-registered multiplexer survives; the eviction retry fires both immediately and after a reconnect poll, and rethrows when the connection never returns; reads don't retry.

Two behavioural tests do use a dead port (127.0.0.1:1) with a 10s bound against a 60s asyncTimeout — a wide enough margin to be CI-safe while still failing outright under the old policy. Both were mutation-checked: reverting BacklogPolicy fails the first, reverting RemoveAsync to a direct delete fails the second.

Not in here

Two unrelated issues found in the same pass, deliberately left for their own PRs:

  • Licence mismatch — README says GPL-3.0; LICENSE and every csproj say MIT. A real blocker for anyone doing licence review before adopting. Looks like feature/switch-to-mit-license missed the line.
  • IKeyContextProvider fails open — when GetContextSegmentAsync returns null, DefaultCacheKeyBuilder silently omits the tenant segment, producing a context-free key shared across all tenants. A claim-resolution failure converts a tenant-isolated cache into a shared one, which is the precise leak the feature exists to prevent. Security-relevant; wants its own fix and a decision on whether null should mean skip caching.

AddCacheWeaveRedis set AbortOnConnectFail = false but left BacklogPolicy at
its default, which queues commands while disconnected. With no reachable
Redis, a cache-decorated request paid a StringGetAsync on the way in and a
StringSetAsync on the way out, each blocking for the full SyncTimeout —
~10-12s of added latency per request, indefinitely, not a one-time connect
cost. CacheWeave already converts any cache failure to a miss, so the request
returned correct data with a 200 and the degradation was invisible in logs
and metrics.

Set BacklogPolicy.FailFast alongside AbortOnConnectFail: both settings express
one policy — never throw on connect, and never block a request waiting for a
connection that isn't there. FailFast is not consulted on the success path, so
behaviour against a reachable Redis is unchanged.

Evictions are the one case where a dropped command matters: a [CacheWeaveEvict]
delete lost during a failover leaves a stale entry served until it expires.
RemoveAsync and RemoveByPrefixAsync therefore wait up to 1.5s for the
multiplexer to reconnect and retry once; deleting an already-deleted key is a
no-op, so replaying a partially applied eviction is safe. Reads and writes stay
fail-fast.

Also:
- Add an Action<ConfigurationOptions> overload, as a separate overload so
  existing calls still bind. It runs after the defaults, so callers can
  override them — including back to BacklogPolicy.Default — and reach settings
  no connection string can express (TLS certs, Sentinel, Azure token auth).
- Use TryAddSingleton for IConnectionMultiplexer, matching CacheWeave.Legacy.
  A multiplexer registered after the call still wins, so the existing
  re-registration workaround keeps working.
- Apply the same fixes to CacheWeave.Legacy (net48), which had the same defect.
- Document connection behaviour and the eviction trade-off in the README.

Warrants a minor version bump: an app relying on the backlog to ride out a
brief blip now takes a cache miss instead of up to SyncTimeout of latency.
@teghoz teghoz self-assigned this Sep 19, 2026
The eviction retry window, its poll interval and the SCAN page size were
hard-coded constants, reachable only from an internal constructor meant for
tests. That repeated the original complaint about this package: a policy the
library decided for you with no way to change it.

Add RedisCacheOptions, bound through IOptions<T> like SQLiteCacheOptions and
DynamoDbCacheOptions already are, and configured via a configureCache callback
on every AddCacheWeaveRedis overload — including the one taking an existing
IConnectionMultiplexer, which previously offered no provider configuration at
all. Defaults are unchanged: 1.5s window, 50ms poll, 250-key batches.

EvictionRetryWindow = TimeSpan.Zero now genuinely disables the retry. It did
not before: WaitForReconnectAsync returns true immediately when the multiplexer
is already connected, so the retry still ran regardless of the window. Caught
by the test written for the documented behaviour.

Constructor validates the options and throws ArgumentOutOfRangeException on a
negative window, a non-positive poll interval, or a scan page size below 1.

Applied to CacheWeave.Legacy too. The connection-configure parameter is renamed
configureConnection to sit alongside configureCache; both are new in this
unreleased branch, so no shipped signature changes.
@teghoz
teghoz merged commit 732ceec into develop Sep 20, 2026
4 checks passed
@teghoz
teghoz deleted the fix/redis-backlog-failfast branch September 20, 2026 03:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant