Remove unconditional setTimeout waits from overlay/consent removal scripts

[SohamKukreti]

Summary

Crawling pages served with a CSP sandbox directive (e.g. raw.githubusercontent.com, huggingface.co/*/raw/*, Jenkins-hosted user content) with remove_overlay_elements=True or remove_consent_popups=True stalled ~30s per page — or hung indefinitely — with the warning Failed to remove overlay elements: Page.evaluate: Execution context was destroyed, most likely because of a navigation.

See discussion #2119 for the issue

Root cause: the CSP sandbox directive disables script timers, so the unconditional await new Promise(r => setTimeout(r, ...)) waits inside the injected overlay/consent-removal scripts never resolve. Chromium tears the execution context down after ~30s (GitHub raw), or never (HuggingFace raw), deadlocking the crawl.

Fix: remove the unconditional in-page waits. The overlay script's trailing settle wait moves to the existing Python-side wait_for_timeout (immune to page CSP); the consent script's mid-script wait now runs only when a consent action (button click or CMP API call) actually fired — which cannot happen on sandboxed raw pages. Total settle budgets before HTML capture are unchanged, so scraping output on normal pages is identical.

Measured: GitHub raw crawls ~31s → ~1.3s; HuggingFace raw hung >130s → ~1.5s. Verified no regression on live CMP sites (BBC, The Verge, The Guardian) and against a synthetic "watchdog" CMP that re-injects its banner if removed before consent registers (both click and vendor-API consent paths).

List of files changed and why

• crawl4ai/js_snippet/remove_overlay_elements.js — removed the trailing 50ms setTimeout wait that hung forever under CSP sandbox; post-click waits kept.
• crawl4ai/js_snippet/remove_consent_popups.js — the mid-script 500ms wait is now conditional on a consent action having fired (accepted || apiCalled); added the apiCalled flag to the Didomi/Cookiebot/Osano/Klaro API branches, which previously didn't record that they acted.
• crawl4ai/async_crawler_strategy.py — overlay settle wait 500→600ms to absorb the removed in-page wait, keeping the pre-capture budget identical.
• tests/regression/test_reg_browser.py — added test_overlay_removal_on_csp_sandbox_page: crawls a commit-pinned raw.githubusercontent.com URL (live CSP sandbox header, immutable content) with both flags on and asserts completion well under the old stall time.

How Has This Been Tested?

• New regression test fails on the unfixed code (~35s stall, exceeds the 20s bound) and passes on the fix (~2s); existing test_remove_overlay_elements still passes.
• Live URLs: GitHub raw / gist raw (was ~31s → ~1.3s), HuggingFace model + dataset raw (was indefinite hang → ~1.5–1.9s), whatwg spec, Wikipedia — all succeed with no warnings.
• Consent regression checks on live CMP sites (BBC, The Verge, The Guardian, Euronews): timings and consent-junk-free markdown identical to the pre-fix code.
• Synthetic pages: three banner types (fixed, animated-dismiss, instant-dismiss) produce identical output before/after; a watchdog CMP that re-injects its banner if removed pre-consent stays removed via both the click path and the vendor-API path.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes