Skip to content

PDPS-3514: Fix sync timeouts and rate limiting on large kbs - #14

Open
ykulynych wants to merge 2 commits into
mainfrom
PDPS-3514-wordpress-plugin-content-sync-fails
Open

ykulynych wants to merge 2 commits into
mainfrom
PDPS-3514-wordpress-plugin-content-sync-fails

Conversation

@ykulynych

Copy link
Copy Markdown
Contributor

The plugin paginated the resource list with 'from' param, which the API ignores, so it refetched page 0 forever. The status poll and settings saves also scanned the whole KB inside one web request. On sites with many resources this caused 504 timeouts and tripped the 2,400 requests/minute limit.
• Paginate with  page  and stop on  pagination.last 
• Make the status poll read-only, and avoid loading every finished Action Scheduler job while work is pending
• Run sync as a resumable background job on Action Scheduler, started by Sync or by saving settings
• Resolve 409 conflicts with a single  GET /slug/{slug} 
• Report 429 as a rate-limit error
• Add a sync progress banner and rename the button to "Sync", with a spinner and a details view with Cancel
• Add isolated scaling tests with 10,000 resources
• Add a changelog entry

@StrawHat-Dery StrawHat-Dery left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed and validated the synchronization scaling changes. Approving — no blocking issues identified.

Validation completed:

  • PHP lint passed.
  • PHPUnit: 149 tests, 862 assertions passed.
  • Scaling suite: 12/12 tests passed against a simulated 10,000-resource KB.
  • Live WordPress → PAR synchronization completed successfully against a KB with ~1,200 existing resources (3/3 synced, 0 failures).

The pagination changes, background preparation workflow, and resource reconciliation behaved as expected in testing.

Non-blocking follow-ups worth considering:

  • Legacy sync-state recovery: certain pre-upgrade running states may prevent a new synchronization from starting. A documented support recovery procedure would be helpful.
  • Rate limiting: individual slug-verification requests don't currently honor Retry-After, unlike the new catalog pagination requests.
  • Final-state recovery: a narrow interruption window may leave a completed run marked as running until manually cancelled or otherwise reconciled.

None of these findings affected the successful tests or appear to undermine the primary fixes.

Thanks for addressing the pagination and synchronization timeout issues!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants