Skip to content

Deduplicate ResetWorkflowExecution requests - #12042

Draft
Qian-Cheng-nju wants to merge 2 commits into
temporalio:mainfrom
Qian-Cheng-nju:fix/reset-workflow-request-idempotency
Draft

Qian-Cheng-nju wants to merge 2 commits into
temporalio:mainfrom
Qian-Cheng-nju:fix/reset-workflow-request-idempotency

Conversation

@Qian-Cheng-nju

@Qian-Cheng-nju Qian-Cheng-nju commented Sep 12, 2026 •

Copy link
Copy Markdown

What changed?

Depends on temporalio/api#873 and its generated Go API update. This PR is draft until the official go.temporal.io/api dependency can be updated; the currently pinned version does not yet contain ResetRequestId. Local validation used generated bindings with that field, without committing a module replacement.

Persist the public reset request ID on the existing WorkflowTaskFailed event with cause RESET_WORKFLOW, and rebuild its RequestIds entry when replaying history. Only the event whose new_run_id matches the run being reconstructed contributes a reset request ID, so resetting a previously reset run does not inherit its ancestor's request identity.

CreateRequestId continues to identify the original workflow-start request for scheduler callbacks. Internal resets leave the new field empty. The mutable-state-only marker and its special-case rebuild copying are removed.

Why?

Retrying ResetWorkflowExecution with the same request_id can create another reset run and terminate the first. Keeping the request ID only in mutable state also loses it under legacy event-based cross-cluster replication. Recording it in history lets both replication modes and history rebuilds restore deduplication.

How did you test it?

All local server checks used generated Go bindings for the companion API field:

  • Unit packages: service/history/api/resetworkflow, service/history/ndc, and service/history/workflow; replay tests cover current-run, ancestor-run, missing-ID, and non-reset events.
  • XDC: failover with transition history enabled and disabled, with and without explicit run IDs. Same-ID retries return the original running reset run before and after rebuilding on the new active cluster; a different request ID creates a new run.
  • Functional regressions: immediate and post-rebuild retries, repeated-reset request isolation, scheduler double-reset callbacks with HSM and CHASM, and existing Admin rebuild tests.
  • make lint-code-fast GOLANGCI_LINT_BASE_REV=origin/main GOLANGCI_LINT_FIX=false.

Before the repair, the XDC regression failed with different run IDs after failover in legacy replication mode, for both explicit and implicit run-ID requests.

Potential risks

Older history does not contain the public reset request ID and cannot recover it retroactively. Older servers ignore the new field and do not gain the new deduplication behavior until upgraded. No database schema migration is required.

@Qian-Cheng-nju
Qian-Cheng-nju requested review from a team as code owners September 12, 2026 13:52
@CLAassistant

CLAassistant commented Sep 12, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

resetMS := resetWorkflow.GetMutableState()
// Reset has no corresponding history event. Keep an unswept marker without changing
// CreateRequestId, which must retain the original start request ID for callbacks.
attachResetRequestID(resetMS, resetRequestID)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This marker is mutable-state-only and is lost with legacy event-based XDC (history.enableTransitionHistory=false). The passive reconstructs the reset run from events and generates a new internal request ID, so after failover, retrying the same reset request creates another reset run. Please add failover tests for both transition-history modes and carry the reset request ID in a legacy replication/history artifact.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this. I moved the reset request ID into the existing reset history event, with the API addition in temporalio/api#873. Added failover tests for both transition-history modes, including retries after rebuild. These pass locally with generated API bindings. This PR is draft pending the official Go API dependency update.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants