Skip to content

fix(idempotency): retry Redis inconsistent state errors - #8736

Open
coughlmao wants to merge 1 commit into
aws-powertools:developfrom
coughlmao:fix/idempotency-redis-inconsistent-state-retry
Open

coughlmao wants to merge 1 commit into
aws-powertools:developfrom
coughlmao:fix/idempotency-redis-inconsistent-state-retry

Conversation

@coughlmao

Copy link
Copy Markdown

Issue number: closes #8727

Summary

Fix a race condition in the Redis idempotency persistence layer where an existing record expires between a failed SET NX operation and the subsequent GET.

Previously, this caused an IdempotencyItemNotFoundError to escape and be wrapped as an IdempotencyPersistenceLayerError, failing the Lambda invocation instead of retrying.

Changes

  • Catch IdempotencyItemNotFoundError when retrieving a record after a failed Redis SET NX.
  • Raise IdempotencyInconsistentStateError to signal the recoverable inconsistent state.
  • Ensure the existing idempotency retry mechanism can retry the operation.
  • Add regression tests covering the disappearing-record race condition, exception chaining, and successful retry through the public idempotency decorator.

User experience

Before: If an existing Redis idempotency record expired between SET NX and GET, the invocation failed with a persistence-layer error.

After: The Redis persistence layer raises IdempotencyInconsistentStateError, allowing the existing retry mechanism to retry the operation. The invocation succeeds when the subsequent attempt acquires the lock successfully.

Verification

Tests and checks run locally:

  • uv run --locked pytest tests/functional/idempotency/_redis/test_redis_layer.py -k inconsistent_state -v — 3 passed.
  • uv run --locked pytest tests/functional/idempotency/_redis/test_redis_layer.py -q — 33 passed.
  • Idempotency functional tests — 203 passed.
  • uv run --locked ruff check aws_lambda_powertools/utilities/idempotency/base.py aws_lambda_powertools/utilities/idempotency/persistence/redis.py tests/functional/idempotency/_redis/test_redis_layer.py — passed.
  • uv run --locked mypy aws_lambda_powertools/utilities/idempotency/base.py aws_lambda_powertools/utilities/idempotency/persistence/redis.py — passed.
  • git diff --check — passed.
  • Git pre-commit hooks — all passed.

The regression scenario was verified using a mocked Redis client. Real Redis service validation has not been performed.

This contribution was developed with AI assistance. I reviewed the changes and personally ran the tests and checks listed above.


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

@coughlmao
coughlmao requested a review from a team as a code owner November 1, 2026 15:56
@coughlmao
coughlmao requested a review from hjgraca November 1, 2026 15:56
@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Nov 1, 2026
@boring-cyborg boring-cyborg Bot added the tests label Nov 1, 2026
@boring-cyborg

boring-cyborg Bot commented Nov 1, 2026

Copy link
Copy Markdown

Thanks a lot for your first contribution! Please check out our contributing guidelines and don't hesitate to ask whatever you need.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

@sonarqubecloud

sonarqubecloud Bot commented Nov 1, 2026

Copy link
Copy Markdown

@leandrodamascena leandrodamascena added this to the 3.41.0 milestone Nov 1, 2026
@leandrodamascena

Copy link
Copy Markdown
Contributor

Thanks for the PR and for filling in the verification section, that helps. Heads up that the review may take a few days: this goes into 3.41.0 and I review every PR against a real service before approving, one at a time. No need to rebase or ping in the meantime, I'll come back here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/M Denotes a PR that changes 30-99 lines, ignoring generated files. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: Redis layer fails the invocation instead of retrying when a record expires between SET NX and GET

2 participants