fix: scan deadline grace, callback parse guard, per-scan health (0.8.0b2)

CHANGELOG.md

@@ -2,6 +2,12 @@
 
 This file owns the release history for `ScanLink`.
 
+## v0.8.0b2
+
+- **Scan Deadline Grace**: `ScanWorker` adds `SCAN_TIMEOUT_GRACE_SECONDS` (15s) to the parent's `scan_timeout_seconds` kill limit so the child's own deadline closes TWAIN and reports its error before forced termination; `gui_task_timeout_seconds` is unchanged.
+- **Callback URL Parsing**: `_send_callback` skips URLs that `urlparse` rejects instead of raising after completion.
+- **Per-Scan Transport Check**: `scanlink.js` `startScan()` health-checks before every scan POST, falling back from a stale preferred transport and reporting `helper_unavailable`/`insecure_origin` instead of `scan_request_uncertain` when the helper is unreachable.
+
 ## v0.8.0b1
 
 - **Atomic PDF Completion**: Keep child completion provisional until the process exits and the parent stores result bytes; publish terminal errors with their details.

VERSION

@@ -1 +1 @@
-0.8.0b1
+0.8.0b2

docs/api.md

@@ -222,7 +222,9 @@ on older helpers. Existing HTTP scan jobs remain usable while setup runs.
 
 ## Browser transport retries
 
-The bundled JavaScript client negotiates a healthy transport before its first scan.
+The bundled JavaScript client re-negotiates a healthy transport through `/health`
+before every scan, so a lost transport is reported as `helper_unavailable` or
+`insecure_origin` and a stale preferred transport falls back before the POST.
 A scan POST is sent once on that transport. If the response or its body is lost,
 it rejects with `scan_request_uncertain` instead of resending the request over
 the other transport; the helper may already be scanning. Check ScanLink before

docs/architecture.md

@@ -214,6 +214,27 @@ published by the parent with their error details after process exit.
 
 The parent enforces both `scan_timeout_seconds` and `gui_task_timeout_seconds`;
 the first enabled deadline terminates the child, even if it keeps reporting
-progress. Idle timeout remains an independent progress guard. Downloading frees
+progress. The scan deadline includes a 15-second grace (`SCAN_TIMEOUT_GRACE_SECONDS`
+in `worker.py`) covering child startup, so a child that honours its own
+`scan_timeout_seconds` closes the TWAIN source and reports its error before the
+parent force-terminates it. Unparseable callback URLs are skipped without
+affecting the stored PDF. Idle timeout remains an independent progress guard. Downloading frees
 the PDF immediately and retains consumed-job metadata until the normal job TTL,
 so repeated or racing downloads report `410` while that metadata exists.
+
+## 0.8.0b1 release acceptance
+
+The published x64/x86 installers passed Windows CI and release gates, including
+the native certificate consent prompt, 120 normally verified HTTPS requests per
+architecture and identity reuse after restart. Published installer digests were
+checked through the development project's authenticated distribution endpoints.
+The real worker also passed file-field/PDF-editor compatibility with published
+DjangoLux 1.15.1b1. Scanner hardware was simulated for local job tests; physical
+TWAIN and native browser/enterprise-policy acceptance remain separate checks.
+
+The paired published DjangoLux 1.15.1b2 acceptance passed on 2026-10-26:
+Composer upgrade with full backup, 694 package hashes in web/Celery, published
+client/templates, 300 health checks, 30 scans, 20-way contention, PDF save/download
+and Arabic lost-response guidance with one accepted scan POST. Both installer
+distribution hashes matched. The helper was stopped; physical scanner acceptance
+remains outstanding.

docs/developer-guide.md

@@ -218,7 +218,7 @@ DPAPI-protected keys, trust verification, hourly checks and renewal.
 
 ### Lost scan responses
 
-The JavaScript client health-checks a transport before scanning and never
+The JavaScript client health-checks a transport before every scan and never
 automatically retries a scan POST. Handle `scan_request_uncertain` by asking the
 operator to check the tray app before starting another scan. A lost response can
 mean the request was accepted; retrying automatically could trigger another

static/scanlink.js

@@ -86,7 +86,8 @@ class ScanLink {
         if (options.metadata && typeof options.metadata === "object") payload.metadata = options.metadata;
 
         try {
-            if (!this.preferredUrl && !await this.checkHealth()) {
+            // Re-negotiate every scan so a lost transport is reported before the single POST.
+            if (!await this.checkHealth()) {
                 throw this.connectionError;
             }
             const response = await this._fetchResponse("/scan", {

tests/fixtures/scan_child.py

@@ -20,6 +20,19 @@ def state(status, **extra):
     for count in range(100):
         state("page_scanned", page_count=count)
         time.sleep(0.1)
+elif scenario == "graceful_timeout":
+    request = json.loads((directory / "request.json").read_text())
+    deadline = time.monotonic() + request["scan_timeout_seconds"]
+    count = 0
+    while time.monotonic() <= deadline:
+        count += 1
+        state("page_scanned", page_count=count)
+        time.sleep(0.1)
+    state(
+        "timed_out",
+        error={"code": "scan_timeout", "message": "Child closed the scanner."},
+    )
+    sys.exit(2)
 else:
     (directory / "result.pdf").write_bytes(b"%PDF-1.4\nregression\n")
     state("completed", page_count=1)

tests/test_stress_regressions.py

@@ -76,13 +76,22 @@ def test_completion_requires_pdf_available_and_child_exited(self):
     def test_parent_scan_deadline_applies_despite_child_progress(self):
         self.scenario = "heartbeat"
         self.worker.scan_timeout_seconds = 1
+        self.worker.scan_timeout_grace_seconds = 0
         self.worker.gui_task_timeout_seconds = 3
         started = time.monotonic()
         job = self.finish(self.client.post("/scan", json={}).get_json()["job_id"])
         self.assertEqual(job["status"], "timed_out")
         self.assertEqual(job["error"]["code"], "scan_timeout")
         self.assertLess(time.monotonic() - started, 2)
 
+    def test_child_reports_its_own_timeout_within_parent_grace(self):
+        self.scenario = "graceful_timeout"
+        self.worker.scan_timeout_seconds = 1
+        self.worker.scan_timeout_grace_seconds = 3
+        job = self.finish(self.client.post("/scan", json={}).get_json()["job_id"])
+        self.assertEqual(job["status"], "timed_out")
+        self.assertEqual(job["error"]["message"], "Child closed the scanner.")
+
     def test_parallel_result_downloads_consume_pdf_once(self):
         job_id = self.client.post("/scan", json={}).get_json()["job_id"]
         self.finish(job_id)

tests/test_worker.py

@@ -2,6 +2,7 @@
 import queue
 import time
 import unittest
+from unittest.mock import patch
 
 from worker import BusyError, ScanWorker
 
@@ -75,6 +76,12 @@ def test_consumed_result_keeps_metadata_until_job_ttl(self):
         worker.job_ttl_seconds = 0
         self.assertIsNone(worker.get_job("job-1"))
 
+    def test_unparseable_callback_is_skipped(self):
+        worker = ScanWorker(queue.Queue())
+        with patch("worker.requests") as requests:
+            worker._send_callback({"job_id": "job-1", "callback_url": "http://["}, b"pdf")
+        requests.post.assert_not_called()
+
 
 if __name__ == "__main__":
     unittest.main()

tracker.md

@@ -2,39 +2,42 @@
 
 ## Part 1: Project-specific
 ### Current Snapshot
-- Unpublished candidate 0.8.0b1 complete on `0.8/stress_fixes` (2026-10-25): real worker completion/deadline, public/internal input validation, single POST transport behavior and result-race fixes. Published0.7.5b1 (tagd74bf90) and stable Latest0.7.4 unchanged; no push/tag/publication.
+- Published 0.8.0b1 on `0.8/stress_fixes` (2026-10-25): real worker completion/deadline, public/internal input validation, single POST transport behavior and result-race fixes. Published0.7.5b1 (tagd74bf90) and stable Latest0.7.4 unchanged; owner authorized publication2026-10-25; PR2 merged as a82aab7; v0.8.0b1 published, release40233617433 PASS, x64/x86 asset digests verified; stable Latest stays0.7.4.
 - Windows tray app uses TWAIN/pytwain; scan jobs run through child `main.py --scan-worker-dir` / `scan_runtime.py`.
 - Normal startup serves HTTP :5000 while automatically provisioning trusted localhost HTTPS :5443; failed setup offers native Retry and a tray repair action.
 - `certificates.py`: per-user localhost-only self-signed server identity, Windows CurrentUser Root trust, encrypted PKCS#8 key with DPAPI-protected password; hourly checks/30-day renewal and live TLS context selection.
-- Candidate pairs with DjangoLux1.15.1b2 (`1.15/scanlink_stress`); existing REST clients/helpers remain compatible. Published testbed server stays1.15.1b1; candidate browser assets/template fixture were tested through routing.
+- Paired DjangoLux1.15.1b2 published/accepted2026-10-26 (PR15 explicit owner bypass, fa729074/release40240597894 PASS).8097 web/celery use published b2;694-file checks, published UI/Arabic single-POST recovery and installer distribution PASS. Existing REST APIs remain compatible.
 
 ### Adopted Standards & Rules
-- Owner decision 2026-10-25: the next ScanLink beta targets 0.8.0b1, incorporating any findings from further paired-beta/hardware stress testing. Keep published 0.7.5b1 unchanged; start follow-up implementation on a fresh 0.8 workstream from verified main.
+- Owner selected the0.8.0 line;0.8.0b1 published2026-10-25. Preserve immutable0.7.5b1/0.8.0b1 artifacts; further fixes need a fresh workstream/version.
 - One patch/workstream per release-line branch; VERSION is the version source. Do not tag or publish without owner approval.
 - App data/config/logs live under `%LOCALAPPDATA%\ScanLink`; identity pointer is `https/active.json`, immutable bundles `https/<id>/`, superseded identities `.xclude/https/<id>/`.
 - Certificate covers only localhost/127.0.0.1/::1, CA=False/serverAuth/no certificate-signing usage; no downloaded/shared CA or machine-wide trust. Existing mkcert credentials stay untouched.
 - HTTP remains supported; browser local-network permission, CSP, enterprise trust and an insecure application origin are independent restrictions.
 - Never delete workspace files; relocate under `.xclude/` preserving relative paths. Match executable architecture to the TWAIN driver.
 
 ### Known Bugs & Unsolved Issues
-- Six stress defects fixed locally: premature completion/PDF unavailable, parent deadline ignored, malformed public payload500/callback parsing, non-JSON protocol requests queueing, ambiguous POST retries, consumed results returning404. Candidate Windows CI/hardware remains pending.
+- Six stress defects fixed in the published pair: premature completion/PDF unavailable, parent deadline ignored, malformed public payload500/callback parsing, non-JSON protocol requests queueing, ambiguous POST retries, consumed results returning404. Native Windows CI/release gates PASS; physical hardware remains pending.
 - Windows x64/x86 unit/build/install gates passed, including real certificate consent, DPAPI/trust, 120 verified HTTPS requests per architecture and restart identity reuse. Real Windows browser/scanner and enterprise-policy acceptance remain manual.
 - Target PaperStream fi-6230Z may expose only 32-bit TWAIN; live driver compliance/output quality remain unverified. Back-only and blank-page discard remain unsupported.
 - Organization/browser policies may refuse user-installed trust. A denied setup is reported as failure and retried with the same identity, without repeated automatic dialogs.
+- Callbacks are never delivered (since 36249b6, 2026-04-28): `_process_job` passes the serialized job, which omits `callback_url`, to `_send_callback`. Docs promise delivery; enabling it changes production behavior and needs an owner decision.
 
 ### Active & Pending Tasks
-- [x] Paired deeper stress and real-worker regressions pass; candidate0.8.0b1 prepared locally. No new publication authorized this turn.
+- [ ] 0.8.0b2 on `0.8/stress_followups` (from origin/main + the two unmerged 0.8/stress_fixes docs commits): 15s parent scan-deadline grace, callback parse guard, per-scan `/health` in `scanlink.js`. Committed locally, not pushed; needs Windows CI and a PR. Mirror the per-scan health check in DjangoLux `scan_button.js` if it shares the stale-transport logic. Retire `0.8/stress_fixes` once this merges.
+- [x] Paired deeper stress and real-worker regressions pass; candidate0.8.0b1 prepared locally. Owner authorized next beta; PR2 merged after x86/x64 native CI PASS, release gates PASS, both installers published.
 - [x] Windows x64/x86 installed-app gates and normally verified HTTPS load passed before and during beta publication; stable Latest preserved.
-- [ ] Verify 0.7.5b1 with target scanner hardware; preserve ADF/front/200 DPI/color/paper protection off defaults.
-- [x] Paired DjangoLux 1.15.1b1 published (PR14, 3bda457); installed on testbed-dlux-ocr :8097 through Composer, rollback/reapply and published-file checks passed. Real installers are distributed there.
+- [ ] Verify0.8.0b1 with target scanner hardware; preserve ADF/front/200 DPI/color/paper protection off defaults.
+- [x] Paired1.15.1b2 published/Composer-installed on8097 with full backup IB-esVxV;694 files verified in web/celery. Both0.8.0b1 installers advertised; hashes/auth/100 manifests PASS.
 - [ ] Archive-hosted helper update path and detailed TWAIN capability diagnostics remain separate work.
 ### Completed Recently
 - [x] Automatic setup/repair/renewal and plain native recovery; removed optional mkcert packaging requirement and retained `--generate-ssl` for support compatibility.
 - [x] Extended installer smoke to normal Windows-trusted HTTPS and restart reuse; health timeout/fallback behavior matches the paired browser client.
 - [x] Verified real TLS handshakes, hostname rejection and live identity renewal; setup failure preserves HTTP and Retry recovers HTTPS.
 
 ### Validation & Docs Log
-- 2026-10-25: candidate40 unit tests OK/1 Windows skip; pairedJS24 PASS; real worker1000 health (500 verified TLS)/100 hashed PDFs/50 busy/40 result races PASS; timeouts/crash/cancel/callback/shutdown recovery PASS.8097 UI300 health/30 scans/20 simultaneous, PDF save/Arabic lost-response guidance/one POST PASS. Dlux3064 OK/6skips +79 focused; doctor0error. Published native Windows gates passed previously; candidate native/hardware unverified.
+- 2026-10-30: 0.8.0b2 changes, macOS: 42 unit tests OK/1 Windows skip (scratch venv). New grace/callback tests fail on 0.8.0b1 code; Node harness for `scanlink.js` 4/4 (old: 2/4). Docs: api/architecture/developer guide, CHANGELOG v0.8.0b2, VERSION 0.8.0b2. Windows/native gates not run.
+- 2026-10-25: 40 local unit tests OK/1 Windows skip; Windows CI/release PASS x64+x86 including consent/120 HTTPS/restart; pairedJS24 PASS; real worker1000 health (500 verified TLS)/100 hashed PDFs/50 busy/40 result races PASS; timeouts/crash/cancel/callback/shutdown recovery PASS.8097 UI300 health/30 scans/20 simultaneous, PDF save/Arabic lost-response guidance/one POST PASS. Dlux3064 OK/6skips +79 focused; doctor0error. Native candidate gates now PASS; physical hardware/Windows browser policies unverified.
 - 2026-10-25: API/architecture/developer guide, VERSION0.8.0b1 and CHANGELOG document parent-owned completion, hard deadline, strict payloads, retained metadata and scan_request_uncertain. Local report `.xclude/validation/paired-stress-2026-10-25.md`.
 
 ## Part 2: Global & Handoff
@@ -43,5 +46,5 @@
 - Test stacks belong under `~/Desktop/depy/testbed-*`, use Composer `--dev` and application admin/db123123; preserve testbed-dlux.
 ### Agent Handoff
 - Both patch checkouts are authorized by the owner from the app-mock-pdf chat; preserve that checkout's unrelated uncommitted changes.
-- Deeper stress complete;0.8.0b1/paired1.15.1b2 are local candidates only. Next needs owner-authorized push/release and native Windows candidate gates; prior PR14 bypass applies only to the published beta. Temporary real-worker helper shut down,8097 published server/data preserved (synthetic60 now7 pages);8088/8096 untouched. Preserve app-mock-pdf dirty changes.
+- 2026-10-26: PR15 bypass explicitly authorized;1.15.1b2 published/accepted with0.8.0b1.8097 keeps60 records, full backup IB-esVxV and fixture60 now9 pages/8 history versions. Helper stopped;8088/8096 and app-mock-pdf preserved. Physical TWAIN/native browser-policy acceptance remains manual. Report `.xclude/validation/beta-release-080-1151b2.md`.
 - Evidence: `.xclude/validation/`; paired Chrome evidence is in `pkg-django-lux/scanlink_https/.xclude/validation/chrome-loopback.log`.

worker.py

@@ -22,6 +22,8 @@
 logger = get_logger()
 
 TERMINAL_STATUSES = {"completed", "failed", "cancelled", "timed_out"}
+# Covers child startup and a graceful TWAIN close after the child's own deadline.
+SCAN_TIMEOUT_GRACE_SECONDS = 15
 
 # Optional compatibility hook for future push-based status updates.
 _status_callback = None
@@ -76,6 +78,7 @@ def __init__(self, gui_task_queue):
         self.scan_timeout_seconds = int(config.get("scan_timeout_seconds", 600))
         self.gui_task_timeout_seconds = int(config.get("gui_task_timeout_seconds", 900))
         self.scan_idle_timeout_seconds = int(config.get("scan_idle_timeout_seconds", 45))
+        self.scan_timeout_grace_seconds = SCAN_TIMEOUT_GRACE_SECONDS
 
     def submit_job(self, doc_id=None, callback_url=None, metadata=None, source="api"):
         with self.jobs_lock:
@@ -260,6 +263,17 @@ def _run_scan_subprocess(self, job_id, on_progress):
         last_progress_at = started_at
         last_state_signature = None
         last_state_status = None
+        kill_limits = [
+            limit
+            for limit in (
+                self.scan_timeout_seconds + self.scan_timeout_grace_seconds
+                if self.scan_timeout_seconds > 0
+                else 0,
+                self.gui_task_timeout_seconds,
+            )
+            if limit > 0
+        ]
+        kill_after = min(kill_limits) if kill_limits else None
 
         request_payload = {
             "job_id": job_id,
@@ -304,10 +318,7 @@ def _run_scan_subprocess(self, job_id, on_progress):
                     break
 
                 now = time.monotonic()
-                if any(
-                    limit > 0 and now - started_at >= limit
-                    for limit in (self.scan_timeout_seconds, self.gui_task_timeout_seconds)
-                ):
+                if kill_after is not None and now - started_at >= kill_after:
                     self._terminate_scan_process(process)
                     raise ScanProcessTimeoutError(
                         "scan_timeout",
@@ -391,8 +402,11 @@ def _send_callback(self, job, pdf_bytes):
             logger.warning("requests is not installed; skipping callback delivery for job %s", job["job_id"])
             return
 
-        parsed = urlparse(callback_url)
-        if parsed.scheme not in {"http", "https"} or not parsed.netloc:
+        try:
+            parsed = urlparse(callback_url)
+        except ValueError:
+            parsed = None
+        if not parsed or parsed.scheme not in {"http", "https"} or not parsed.netloc:
             logger.warning("Skipping callback for job %s because callback_url is not absolute: %s", job["job_id"], callback_url)
             return