Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions docs/LLMISER_EXCEL_ASSEMBLY_MEMORY.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Excel and final-result memory

Updated 29 October 2026. Candidate implementation on `codex/excel-assembly-memory`; production release is not yet claimed. This follows the released [CSV capacity work](LLMISER_PARSER_CAPACITY.md).
Updated 29 October 2026. Released in [PR #15](https://github.com/DevBehindYou/LLMiser/pull/15) at runtime revision `3db41140521d5cfb1bb3bfd12865feca5a8cc95b`. This follows the released [CSV capacity work](LLMISER_PARSER_CAPACITY.md). Later documentation-only commits do not change this verified source snapshot.

## Changes and fidelity

Expand Down Expand Up @@ -56,8 +56,16 @@ Local MarkItDown 0.1.9 suites: 144 normal tests and 101 archive tests passed, wi

MarkItDown 0.1.8 passed all 24 Excel/count checks. With the original local 0.1.7 installation, 13 fallback/count/integrity checks passed and 11 deliberately disabled adapter cases skipped. Small generated authenticated multipart probes passed before release: three workbooks on each normal worker and a ZIP containing all three on each archive worker (eight requests total). All four reported MarkItDown 0.1.9; complete expected tables and word counts matched. These direct probes created no Storage jobs.

Local OCR validation initially lacked `pytesseract`; a no-dependency copy of 0.3.13 was installed only in the ignored audit folder. The full Windows suite then recorded 69 passes, eight orientation skips and one readiness failure claiming the configured binary was unavailable. Isolated readiness and a real text-image conversion both passed with the installed Tesseract 5.4.0. The suite failure remains recorded rather than hidden by weakening assertions. Linux/Python 3.12 OCR CI is required before merge; production Render uses Tesseract 5.5.0.
Local OCR validation initially lacked `pytesseract`; a no-dependency copy of 0.3.13 was installed only in the ignored audit folder. The full Windows suite then recorded 69 passes, eight orientation skips and one readiness failure claiming the configured binary was unavailable. Isolated readiness and a real text-image conversion both passed with the installed Tesseract 5.4.0. The suite failure remains recorded; assertions were not weakened. Linux/Python 3.12 OCR CI passed before merge and again on merged main. Production Render reports Tesseract 5.5.0, `eng` and `osd`.

Before release, the actual public frontend transport also passed generated XLSX and three-workbook ZIP conversions through create/upload/start/advance, Storage downloads and fresh-link retrieval. The single document output and all three ZIP objects downloaded successfully; table bytes, declared object lengths, final Unicode markers and exact recovered Markdown passed. This created two real jobs under normal retention. It validates the preceding production behavior; repeat after deployment to validate the candidate's Storage path.
Before release, the actual public frontend transport also passed generated XLSX and three-workbook ZIP conversions through create/upload/start/advance, Storage downloads and fresh-link retrieval. The single document output and all three ZIP objects downloaded successfully; table bytes, declared object lengths, final Unicode markers and exact recovered Markdown passed. This created two real jobs under normal retention and supplied the baseline for the post-release comparisons below.

CI, deployment and post-release checks are still pending. Do not treat local checks as production release evidence. No blogs/visuals, database schema, retention policy, queue/concurrency settings, supported formats or limits change in this slice. Native parser peaks, retained archive entries, whole-result normalization/serialization, concurrent requests and real Storage/network buffers remain separate boundaries.
PR #15 and merged main passed all seven CI jobs, including 261 frontend tests, normal/OCR/archive regressions, PostgreSQL, edge and extension checks. All six production Vercel status contexts succeeded. A first health sample overlapped rollout and saw three document workers still without the new revision field; after rollout, all six N1/N2/Z1/Z2/O1/O2 health responses reported exactly `3db41140521d5cfb1bb3bfd12865feca5a8cc95b`, with ready responses. Render therefore picked up the shared counting code too.

Post-release generated acceptance on 29 October:

- Eight authenticated multipart workbook/ZIP requests passed on all four document workers, still using MarkItDown 0.1.9. Complete Markdown hashes, character counts, word counts and engines matched the pre-release observations exactly. These requests created no Storage jobs.
- The actual public frontend transport passed generated XLSX and ZIP create/upload/start/advance, all four output-object downloads and fresh-link recovery. The document and combined ZIP Markdown hashes/counts matched before release; the project index hash matched too. Manifests had the same shape/length and three nodes, with different per-job IDs as expected. Before/after Storage tests created four real jobs in total, subject to existing retention; no cleanup was manually invoked.
- All 20 public-page/discovery/health and path/query-preserving 308 redirect checks passed. An uncached wake returned normal/OCR/archive each 2/2 ready. Existing legacy API/unknown-path behavior was unchanged; no new de-indexing action was needed.

No blogs/visuals, database schema, retention policy, queue/concurrency settings, supported formats or limits changed. This acceptance used small generated workbooks and does not certify maximum-size/concurrent production capacity. Native parser peaks, retained archive entries, whole-result normalization/serialization, real HTTP buffers, native browser input and representative devices remain separate boundaries. Live OCR-image/scanned-PDF/admin/cleanup execution was outside this slice. Raw operator evidence, logs and private probes remain ignored in `project-docs/excel-audit/`; no signed URLs or credentials are published.
8 changes: 4 additions & 4 deletions docs/LLMISER_OPTIMIZATION_PLAN.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# LLMiser optimization continuation

Updated 28 October 2026. Work only in LLMiser. This is the current continuation map; copied MDify analysis is historical evidence and proposals, not live deployment instructions. Blog rebranding is complete and blog sources/visuals are outside this implementation slice.
Updated 29 October 2026. Work only in LLMiser. This is the current continuation map; copied MDify analysis is historical evidence and proposals, not live deployment instructions. Blog rebranding is complete and blog sources/visuals are outside this implementation slice.

## Current continuation state

PRs #1–#13 are merged. Retained-result recovery, cron cutover, wake coalescing, shared reader statistics, large-preview control and CSV allocation reduction are released; the old frontend is now the static LLMiser fallback and public migration redirects remain live. The 22 October branch/review statements below are historical evidence for that earlier slice, not current release blockers. Use [START_HERE](START_HERE.md) and [PROGRESS](ai-handover/PROGRESS.md) for deployment state.
PRs #1–#15 are merged. Retained-result recovery, cron cutover, wake coalescing, shared reader statistics, large-preview control, CSV allocation reduction and Excel/final-result memory changes are released; the old frontend remains the static LLMiser fallback and public migration redirects remain live. The 22 October branch/review statements below are historical evidence for that earlier slice, not current release blockers. Use [START_HERE](START_HERE.md) and [PROGRESS](ai-handover/PROGRESS.md) for deployment state.

Shared reader statistics shipped in PR #9 at `84e510db62ea41f31c131771eb7d32b5e92515f4`: one allocation-bounded helper, one memoized result for reader line/status displays, preserved counting semantics and reproducible before/after measurements. See [reader statistics](LLMISER_READER_STATISTICS.md) for 259-test CI and live verification. Browser profiling follows below; the rest of the queue remains separate. No new production SQL or service retirement was part of this slice.

Expand All @@ -14,7 +14,7 @@ PR #13 released [a source capacity audit and CSV allocation reduction](LLMISER_P

## Verified starting point

Current implementation: [Excel and final-result assembly memory](LLMISER_EXCEL_ASSEMBLY_MEMORY.md), branch `codex/excel-assembly-memory`. Sequential sheet processing, exact bounded word counts and delayed archive output encoding are under measurement/validation. Keep production status separate until the release checks finish.
Released [Excel and final-result assembly memory](LLMISER_EXCEL_ASSEMBLY_MEMORY.md), PR #15 at `3db41140521d5cfb1bb3bfd12865feca5a8cc95b`: sequential sheet processing, exact bounded word counts and delayed archive output encoding. Three fresh-process trials per mode preserve complete hashes; local peak reductions are qualified by timing variability/XLS regression. All seven CI jobs passed before/after merge; six production Vercel statuses succeeded; all six N/O/Z workers report the exact revision and readiness. Eight generated multipart probes, two production Storage conversion/recovery checks and 20 HTTP checks passed; all pools are 2/2 ready. No SQL, concurrency, retention or blog changes.

The copied source already contains atomic job startup/finalization, cleanup recovery, upload/active-job/storage-reservation admission, shared archive container checks, bounded PDF page uploads/OCR preparation, lazy Markdown/highlighter loading and current-edit export snapshots. Do not reimplement these based on old “proposed” checklist rows. Their deployment/capacity evidence is distinct from source tests.

Expand Down Expand Up @@ -44,7 +44,7 @@ Acceptance: expired link, later-part failure, complete Unicode assembly, restart
| Work | Next bounded change / dependency |
| --- | --- |
| Shared statistics and large previews | Counting released in PR #9; browser profiling and large-preview control released in PR #11; measure native input/devices before deeper editor/worker changes |
| Backend parser capacity | Source audit and synchronized CSV memory slice released in PR #13. Next measure workbook/native and response/merge memory; preserve fidelity before tuning concurrency |
| Backend parser capacity | CSV released in PR #13; Excel/final-result memory released in PR #15 with production XLSX/ZIP Storage acceptance. Profile native/concurrent parsing and whole-result normalization/serialization on representative hosts before queue tuning |
| Shared deadline | Add an optional, backward-compatible remaining-budget protocol, reserving finalization time; test slow failures/cold starts |
| Controller detail pagination | Top-level lists already paginate; bound files/tree/events/work-items with stable cursors and truthful totals; query-plan evidence before new indexes |
| Replay-stable object identities | Fence attempt-specific writes before deterministic IDs; SQL attempt guards alone do not prevent a stale Storage overwrite; additive rollout for new roots only |
Expand Down
10 changes: 5 additions & 5 deletions docs/START_HERE.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
# LLMiser — start here

Updated 28 October 2026. **LLMiser is the sole working repository and the source for future releases.** Do not depend on an MDify or MDify-Pro checkout: the owner plans to delete those repositories.
Updated 29 October 2026. **LLMiser is the sole working repository and the source for future releases.** Do not depend on an MDify or MDify-Pro checkout: the owner plans to delete those repositories.

## Current state

The app is live at https://llmiser.devbehindyou.com. PRs #1–#13 are merged; the latest verified runtime change is revision `165d2b01ce8e30dfa09d32292ac911264f18e60d`. PR #13 passed all seven CI jobs, including the 261-test frontend suite; all six production Vercel deployment statuses succeeded. Generated live CSV/ZIP probes passed on all four document workers, and 20 public HTTP checks passed. An initial OCR wake check returned 0/2; direct readiness and a subsequent uncached wake check confirmed all three pools 2/2 ready. See [parser-capacity release evidence](LLMISER_PARSER_CAPACITY.md), [large previews](LLMISER_LARGE_PREVIEWS.md), [reader statistics](LLMISER_READER_STATISTICS.md) and [retirement evidence](LLMISER_LEGACY_RETIREMENT.md).
The app is live at https://llmiser.devbehindyou.com. PRs #1–#15 are merged; the latest verified runtime revision is `3db41140521d5cfb1bb3bfd12865feca5a8cc95b` from PR #15. All seven CI jobs passed before and after merge, including 261 frontend tests; all six production Vercel statuses succeeded. Every N/O/Z worker reports that exact revision and readiness. Eight generated workbook/ZIP multipart checks, two Storage-backed conversion/recovery checks and 20 public HTTP/redirect checks passed after release; all pools are 2/2 ready. See [Excel/assembly release evidence](LLMISER_EXCEL_ASSEMBLY_MEMORY.md), [CSV capacity](LLMISER_PARSER_CAPACITY.md), [large previews](LLMISER_LARGE_PREVIEWS.md) and [retirement evidence](LLMISER_LEGACY_RETIREMENT.md).

The full app supports English (US), German, Japanese, Spanish, Russian and Simplified Chinese. The browser remembers the interface language; documents are not translated and OCR languages are unchanged. Blogs remain English. Eight articles, eight banners and forty figures are complete. The Chrome extension source is merged, but native acceptance and store publication remain pending.

Released [large-preview control](LLMISER_LARGE_PREVIEWS.md) pauses automatic formatting above 50,000 characters while retaining complete raw text and exports. Full formatting is available on request. Local/generated browser checks cover editing, document switching and exact full downloads; representative-device/native textarea profiling and the backend parser-capacity audit remain next boundaries. The six interface dictionaries now contain 631 messages each.
Released [large-preview control](LLMISER_LARGE_PREVIEWS.md) pauses automatic formatting above 50,000 characters while retaining complete raw text and exports. Full formatting is available on request. Local/generated browser checks cover editing, document switching and exact full downloads; representative-device/native textarea profiling and deeper native/concurrent parser measurements remain next boundaries. The six interface dictionaries now contain 631 messages each.

Released [parser capacity and CSV memory reduction](LLMISER_PARSER_CAPACITY.md) preserves verified upstream CSV output while reducing retained row allocations in both document pools. The audit distinguishes source limits, local generated measurements and live multipart acceptance. No concurrency increase follows from the benchmark. Workbook/native parser and final response/merge memory remain the next backend investigation.
Released [parser capacity and CSV memory reduction](LLMISER_PARSER_CAPACITY.md) preserves verified upstream CSV output while reducing retained row allocations in both document pools. The audit distinguishes source limits, local generated measurements and live multipart acceptance. Neither it nor the subsequent workbook work justifies a concurrency increase.

Current candidate: [Excel and final-result memory](LLMISER_EXCEL_ASSEMBLY_MEMORY.md) is being measured on `codex/excel-assembly-memory`. It processes workbook sheets sequentially, bounds temporary word-count allocations and defers ZIP output encoding to the existing upload workers. The runtime revision above remains the last verified release until CI/deployment and live checks are recorded.
Released [Excel and final-result memory](LLMISER_EXCEL_ASSEMBLY_MEMORY.md) processes workbook sheets sequentially, bounds temporary word-count allocations and defers ZIP output encoding to the existing upload workers. Local untraced peaks fell about 2% for XLSX, 5% for XLS, 15% for ZIP assembly and 17% for profile finalization, with identical output; timing varies and the XLS median regressed. Production output/recovery equivalence passed. Native/concurrent parser peaks, full normalization/serialization, representative-device input behavior and the shared-deadline queue remain separate work.

## Legacy frontend replacement is deployed

Expand Down
Loading
Loading