Deduplicate ResetWorkflowExecution requests - #12042
Qian-Cheng-nju wants to merge 2 commits into
Conversation
| resetMS := resetWorkflow.GetMutableState() | ||
| // Reset has no corresponding history event. Keep an unswept marker without changing | ||
| // CreateRequestId, which must retain the original start request ID for callbacks. | ||
| attachResetRequestID(resetMS, resetRequestID) |
There was a problem hiding this comment.
This marker is mutable-state-only and is lost with legacy event-based XDC (history.enableTransitionHistory=false). The passive reconstructs the reset run from events and generates a new internal request ID, so after failover, retrying the same reset request creates another reset run. Please add failover tests for both transition-history modes and carry the reset request ID in a legacy replication/history artifact.
There was a problem hiding this comment.
Thanks for catching this. I moved the reset request ID into the existing reset history event, with the API addition in temporalio/api#873. Added failover tests for both transition-history modes, including retries after rebuild. These pass locally with generated API bindings. This PR is draft pending the official Go API dependency update.
84fb606 to
cf7e184
Compare
What changed?
Depends on temporalio/api#873 and its generated Go API update. This PR is draft until the official
go.temporal.io/apidependency can be updated; the currently pinned version does not yet containResetRequestId. Local validation used generated bindings with that field, without committing a module replacement.Persist the public reset request ID on the existing
WorkflowTaskFailedevent with causeRESET_WORKFLOW, and rebuild itsRequestIdsentry when replaying history. Only the event whosenew_run_idmatches the run being reconstructed contributes a reset request ID, so resetting a previously reset run does not inherit its ancestor's request identity.CreateRequestIdcontinues to identify the original workflow-start request for scheduler callbacks. Internal resets leave the new field empty. The mutable-state-only marker and its special-case rebuild copying are removed.Why?
Retrying
ResetWorkflowExecutionwith the samerequest_idcan create another reset run and terminate the first. Keeping the request ID only in mutable state also loses it under legacy event-based cross-cluster replication. Recording it in history lets both replication modes and history rebuilds restore deduplication.How did you test it?
All local server checks used generated Go bindings for the companion API field:
service/history/api/resetworkflow,service/history/ndc, andservice/history/workflow; replay tests cover current-run, ancestor-run, missing-ID, and non-reset events.make lint-code-fast GOLANGCI_LINT_BASE_REV=origin/main GOLANGCI_LINT_FIX=false.Before the repair, the XDC regression failed with different run IDs after failover in legacy replication mode, for both explicit and implicit run-ID requests.
Potential risks
Older history does not contain the public reset request ID and cannot recover it retroactively. Older servers ignore the new field and do not gain the new deduplication behavior until upgraded. No database schema migration is required.