Skip to content

validation: retry a retryable abandon in the continue-as-new backoff test - #70

Merged
Affan Dar (affandar) merged 1 commit into
mainfrom
fix/validation-retry-abandon
Oct 11, 2026
Merged

Affan Dar (affandar) merged 1 commit into
mainfrom
fix/validation-retry-abandon

Conversation

@affandar

@affandar Affan Dar (affandar) commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

test_continue_as_new_unregistered_backoff (in src/provider_validation/race_replay.rs, added in #67) failed in CI on the 0.1.32 release PR (#68):

src/provider_validation/race_replay.rs:224: release competing inspection fetch:
ProviderError { operation: "abandon_orchestration_item",
  message: "error returned from database: (code: 6) database is deadlocked", retryable: true }

The test calls the store directly while a runtime fetches from the same store:

Test                                       Runtime (same in-memory SQLite store)
fetch the successor start (to inspect it)
abandon_orchestration_item  ───┐           fetch_orchestration_item
                               └── the two transactions block each other
  → "database is deadlocked", retryable: true
  → test called .expect() → panic          → runtime logged a warning and retried

The validation store is SQLite in-memory with a shared cache (sqlite::memory:?cache=shared). There, two transactions can block each other, and the provider reports a retryable error. The test already retried that error on its own fetch, two lines above, but not on the abandon.

This PR retries the abandon on retryable errors, up to 5 times with a short backoff. instance_locking.rs already does the same for this SQLite error. A failed abandon leaves the lock in place, so the retry is safe.

This code ships in the public provider_validation module. Without the fix, other providers that run the suite could hit the same random failure.

Checklist

  • Tests added/updated (fixes a provider validation test)
  • Regular tests pass locally (cargo nt), run through ./run-tests.sh
  • Doc tests pass locally (cargo test --doc --all-features)
  • Docs updated (not needed)
    • Existing docs updated where behavior changed
    • New doc added under docs/ if introducing a new area
    • Linked from docs/README.md

Validation

Rust 1.97.1, macOS.

Run Result
Before the fix: the test 100 times, 4 loops in parallel 1 failed, same error as CI
With the fix: the test 200 times, 4 loops in parallel 200 passed. The new retry fired 2 times.
This branch: cargo fmt --check, cargo clippy ... -D warnings pass
This branch: sqlite_provider_validations 232 passed
This fix plus the 0.1.32 release edits: ./run-tests.sh all features: 1,217 passed. No features: 1,214 passed.
This fix plus the 0.1.32 release edits: cargo test --doc --all-features 46 passed, 74 ignored

No changelog entry: this test is new in the unreleased 0.1.32.

Links / Design notes

…test

test_continue_as_new_unregistered_backoff releases its competing inspection
fetch with abandon_orchestration_item while a runtime fetches from the same
store. On SQLite's shared-cache in-memory store the two transactions can
block each other. The provider then returns a retryable error ("database is
deadlocked"), and the test panicked on it. CI hit this on the 0.1.32 release
PR. Locally it failed 1 in 100 runs.

Retry the abandon on retryable errors, as the fetch right above it already
does. A failed abandon leaves the lock in place, so the retry is safe. With
the fix, 200 of 200 runs passed and the retry fired twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CuCN2hdY59UrmFYYc7DSBJ
@affandar
Affan Dar (affandar) force-pushed the fix/validation-retry-abandon branch from 7007054 to 8711a3e Compare October 11, 2026 19:07
@affandar
Affan Dar (affandar) merged commit dc75edc into main Oct 11, 2026
2 checks passed
@affandar Affan Dar (affandar) mentioned this pull request Oct 11, 2026
4 of 7 tasks
@affandar
Affan Dar (affandar) deleted the fix/validation-retry-abandon branch October 12, 2026 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant