Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/LLMISER_OPTIMIZATION_PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Updated 29 October 2026. Work only in LLMiser. This is the current continuation

## Current continuation state

PRs #1–#15 are merged. Retained-result recovery, cron cutover, wake coalescing, shared reader statistics, large-preview control, CSV allocation reduction and Excel/final-result memory changes are released; the old frontend remains the static LLMiser fallback and public migration redirects remain live. The 22 October branch/review statements below are historical evidence for that earlier slice, not current release blockers. Use [START_HERE](START_HERE.md) and [PROGRESS](ai-handover/PROGRESS.md) for deployment state.
PRs #1–#17 are merged. Recovery, cron cutover, wake coalescing, reader statistics, large previews, CSV/Excel/final-result memory and shared conversion deadlines are released. Legacy hosting/redirects remain live. Earlier 22 October branch/review statements below are historical; use [START_HERE](START_HERE.md) and [PROGRESS](ai-handover/PROGRESS.md) for current deployment evidence.

Shared reader statistics shipped in PR #9 at `84e510db62ea41f31c131771eb7d32b5e92515f4`: one allocation-bounded helper, one memoized result for reader line/status displays, preserved counting semantics and reproducible before/after measurements. See [reader statistics](LLMISER_READER_STATISTICS.md) for 259-test CI and live verification. Browser profiling follows below; the rest of the queue remains separate. No new production SQL or service retirement was part of this slice.

Expand All @@ -14,7 +14,7 @@ PR #13 released [a source capacity audit and CSV allocation reduction](LLMISER_P

## Verified starting point

Current candidate: [shared conversion deadlines](LLMISER_SHARED_DEADLINES.md), branch `codex/shared-conversion-deadlines`. Backend cooperative checkpoints and remaining-budget propagation are implemented locally; rollout/acceptance remain separate. This is a per-invocation budget and adds no whole-job lifetime, SQL or capacity setting.
Released [shared conversion deadlines](LLMISER_SHARED_DEADLINES.md), PR #17 at `9131a2ac6163f22b630b008b40accb482ed3e0b3`. Optional authenticated absolute deadlines and monotonic cooperative worker checks preserve legacy callers and durable job/lease rules. All seven CI jobs and six Vercel production statuses passed; all six workers reported the exact revision/readiness. Thirty protocol checks, eight workbook/ZIP comparisons, four post-release Storage conversion/recovery jobs and 20 HTTP checks passed. Native preemption, exact-once writes and capacity tuning remain separate. Controller detail pagination is next.

Released [Excel and final-result assembly memory](LLMISER_EXCEL_ASSEMBLY_MEMORY.md), PR #15 at `3db41140521d5cfb1bb3bfd12865feca5a8cc95b`: sequential sheet processing, exact bounded word counts and delayed archive output encoding. Three fresh-process trials per mode preserve complete hashes; local peak reductions are qualified by timing variability/XLS regression. All seven CI jobs passed before/after merge; six production Vercel statuses succeeded; all six N/O/Z workers report the exact revision and readiness. Eight generated multipart probes, two production Storage conversion/recovery checks and 20 HTTP checks passed; all pools are 2/2 ready. No SQL, concurrency, retention or blog changes.

Expand Down Expand Up @@ -47,7 +47,7 @@ Acceptance: expired link, later-part failure, complete Unicode assembly, restart
| --- | --- |
| Shared statistics and large previews | Counting released in PR #9; browser profiling and large-preview control released in PR #11; measure native input/devices before deeper editor/worker changes |
| Backend parser capacity | CSV released in PR #13; Excel/final-result memory released in PR #15 with production XLSX/ZIP Storage acceptance. Profile native/concurrent parsing and whole-result normalization/serialization on representative hosts before queue tuning |
| Shared deadline | Add an optional, backward-compatible remaining-budget protocol, reserving finalization time; test slow failures/cold starts |
| Shared deadline | Released in PR #17; per-invocation propagation/reserves, simulated slow/cold paths and real protocol/OCR/PDF/ZIP Storage acceptance. Preserve cooperative-cancellation and replay-fencing limits |
| Controller detail pagination | Top-level lists already paginate; bound files/tree/events/work-items with stable cursors and truthful totals; query-plan evidence before new indexes |
| Replay-stable object identities | Fence attempt-specific writes before deterministic IDs; SQL attempt guards alone do not prevent a stale Storage overwrite; additive rollout for new roots only |
| Observability | Allowlisted timing, attempts, cleanup age and cron diagnosis; no content/signed links/secrets; notification destination needs user authorization |
Expand Down
15 changes: 12 additions & 3 deletions docs/LLMISER_SHARED_DEADLINES.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Shared conversion deadlines

Updated 29 October 2026. Candidate on `codex/shared-conversion-deadlines`; release evidence is pending. The preceding [Excel/assembly release](LLMISER_EXCEL_ASSEMBLY_MEMORY.md) remains the last verified runtime snapshot.
Updated 29 October 2026. Released in [PR #17](https://github.com/DevBehindYou/LLMiser/pull/17) at runtime revision `9131a2ac6163f22b630b008b40accb482ed3e0b3`. Later documentation-only commits do not change this verified source snapshot. This follows the [Excel/assembly release](LLMISER_EXCEL_ASSEMBLY_MEMORY.md).

## Scope and behavior

Expand All @@ -20,10 +20,19 @@ References: [Node AbortSignal](https://nodejs.org/api/globals.html#static-method

## Validation and rollout

Local/generated checks currently pass: 270 frontend tests, 160 normal/PDF tests, 104 archive tests and 11 focused OCR/capacity tests. Coverage includes slow primary failover, no-time-before-dispatch, body decoding timeout, old-backend failover, expired database calls, queue claim suppression and durable retry recording; authenticated malformed/expired fields, valid/legacy output equality, clamping, duplicate fields, concurrent context isolation, upload-worker propagation, drip reads, slow upload acknowledgement, PDF/ZIP expiry, OCR subprocess budgets and image cleanup. Existing common-package integrity checks remain intact. The prior workstation-wide OCR readiness-suite limitation remains documented in the Excel report; full Linux OCR CI is required for release.
Local/generated checks currently pass: 270 frontend tests, 160 normal/PDF tests, 104 archive tests and 11 focused OCR/capacity tests. Coverage includes slow primary failover, no-time-before-dispatch, body decoding timeout, old-backend failover, expired database calls, queue claim suppression and durable retry recording; authenticated malformed/expired fields, valid/legacy output equality, clamping, duplicate fields, concurrent context isolation, upload-worker propagation, drip reads, slow upload acknowledgement, PDF/ZIP expiry, OCR subprocess budgets and image cleanup. Existing common-package integrity checks remain intact. The prior workstation-wide OCR readiness-suite limitation remains documented in the Excel report; full Linux OCR CI passed before merge and on merged main.

Local lint, production build and dictionary/SQL/blog consistency passed. Before release, eight generated multipart workbook/ZIP requests with future deadline headers passed on all four document workers, confirming old-worker compatibility. Two actual public Storage conversions (XLSX and workbook ZIP), all four output downloads and exact saved-result recovery also passed as baselines; those two generated jobs follow normal retention.

CI, deployment and post-release acceptance are pending. Mixed versions remain compatible during rollout: old workers ignore the additional header, while new workers accept callers without it. Backend cooperative limits are therefore only verified after every worker has the new revision. Do not infer production deployment or savings from local tests.
All seven CI jobs passed on PR #17 and merged main, including 270 frontend tests, full normal/OCR/archive suites, PostgreSQL, edge and extension. All six production Vercel status contexts succeeded. Every N1/N2/Z1/Z2/O1/O2 health response reported exactly `9131a2ac6163f22b630b008b40accb482ed3e0b3` and engine readiness. Mixed-version compatibility was retained during rollout: old workers ignored the added header and new workers accepted legacy callers. No provider setting was changed for this rollout.

No UI/localization/blog/visual, database migration, cron SQL, provider environment, supported-format, file-limit or retention change is included. Operator probes and raw evidence stay ignored under `project-docs/deadline-audit/`.

Post-release generated acceptance on 29 October:

- Thirty direct multipart protocol requests passed across N1/N2/Z1/Z2/O1/O2: each returned 401 for wrong authentication, 504 plus the deadline marker for expired authenticated requests, 400 for malformed deadlines, and 200 with complete output for valid/legacy callers. OCR workers read the generated image text correctly. These requests created no Storage jobs.
- Eight workbook/ZIP requests with future deadlines retained exactly the baseline complete Markdown hashes, character/word counts and engines. MarkItDown remains 0.1.9.
- The public frontend Storage flow passed XLSX and workbook-ZIP conversion, all four output-object downloads and fresh-link recovery, matching the pre-release primary Markdown hashes/counts. Two additional jobs completed image-containing ZIP → OCR → project merge and scanned PDF → OCR → PDF merge, preserving both expected OCR sentences, the ZIP Unicode end marker and exact recovered Markdown. Six real generated Storage jobs in total across pre/post/multi-stage checks follow existing retention; no cleanup was manually invoked.
- Twenty public-page/discovery/health and permanent path/query-preserving redirect checks passed. An uncached wake returned normal/OCR/archive each 2/2 ready; existing legacy API routing was unchanged.

Cold-start/slow-failure boundaries were tested with generated/fake-clock fixtures, not a production stress test or a forced Render sleep. Small-file acceptance is not a maximum-size/concurrency or billing guarantee. Authenticated controller operations, real cleanup execution, hard native-process cancellation and replay-safe Storage fencing remain outside this slice.
4 changes: 2 additions & 2 deletions docs/START_HERE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Updated 29 October 2026. **LLMiser is the sole working repository and the source

## Current state

The app is live at https://llmiser.devbehindyou.com. PRs #1–#15 are merged; the latest verified runtime revision is `3db41140521d5cfb1bb3bfd12865feca5a8cc95b` from PR #15. All seven CI jobs passed before and after merge, including 261 frontend tests; all six production Vercel statuses succeeded. Every N/O/Z worker reports that exact revision and readiness. Eight generated workbook/ZIP multipart checks, two Storage-backed conversion/recovery checks and 20 public HTTP/redirect checks passed after release; all pools are 2/2 ready. See [Excel/assembly release evidence](LLMISER_EXCEL_ASSEMBLY_MEMORY.md), [CSV capacity](LLMISER_PARSER_CAPACITY.md), [large previews](LLMISER_LARGE_PREVIEWS.md) and [retirement evidence](LLMISER_LEGACY_RETIREMENT.md).
The app is live at https://llmiser.devbehindyou.com. PRs #1–#17 are merged; the latest verified runtime revision is `9131a2ac6163f22b630b008b40accb482ed3e0b3` from PR #17. All seven CI jobs passed before and after merge, including 270 frontend tests and Linux OCR; all six production Vercel statuses succeeded. Release checks confirmed that exact revision and readiness on every N/O/Z worker. Thirty protocol checks, eight workbook/ZIP equivalence checks, four post-release Storage conversion/recovery jobs and 20 public HTTP/redirect checks passed; all pools were 2/2 ready. See [shared deadlines](LLMISER_SHARED_DEADLINES.md), [Excel/assembly](LLMISER_EXCEL_ASSEMBLY_MEMORY.md), [CSV capacity](LLMISER_PARSER_CAPACITY.md) and [retirement evidence](LLMISER_LEGACY_RETIREMENT.md).

The full app supports English (US), German, Japanese, Spanish, Russian and Simplified Chinese. The browser remembers the interface language; documents are not translated and OCR languages are unchanged. Blogs remain English. Eight articles, eight banners and forty figures are complete. The Chrome extension source is merged, but native acceptance and store publication remain pending.

Expand All @@ -14,7 +14,7 @@ Released [parser capacity and CSV memory reduction](LLMISER_PARSER_CAPACITY.md)

Released [Excel and final-result memory](LLMISER_EXCEL_ASSEMBLY_MEMORY.md) processes workbook sheets sequentially, bounds temporary word-count allocations and defers ZIP output encoding to the existing upload workers. Local untraced peaks fell about 2% for XLSX, 5% for XLS, 15% for ZIP assembly and 17% for profile finalization, with identical output; timing varies and the XLS median regressed. Production output/recovery equivalence passed. Native/concurrent parser peaks, full normalization/serialization, representative-device input behavior and the shared-deadline queue remain separate work.

Current candidate: [shared conversion deadlines](LLMISER_SHARED_DEADLINES.md), branch `codex/shared-conversion-deadlines`. Retries share each invocation's remaining time; workers receive an optional authenticated deadline and reserve finalization time. Existing job lifetime, queue limits and retention stay compatible. Release checks are pending; the runtime revision above remains the last verified release.
Released [shared conversion deadlines](LLMISER_SHARED_DEADLINES.md) let retries share each invocation's remaining time, account for cold-start/transport time and reserve recording/saving time. Optional authenticated worker headers preserve legacy callers. Expiry cannot be hidden as partial PDF/ZIP success. Production checks include real OCR images, an image-containing ZIP and a scanned PDF with exact saved-result recovery. Native parsing/lock waits remain cooperative; this adds no whole-job lifetime or exactly-once Storage guarantee. Controller detail pagination is the next implementation item in the queue.

## Legacy frontend replacement is deployed

Expand Down
Loading
Loading