Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/LLMISER_OPTIMIZATION_PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,13 @@ Updated 28 October 2026. Work only in LLMiser. This is the current continuation

## Current continuation state

PRs #1–#11 are merged. Retained-result recovery, cron cutover, wake coalescing, shared reader statistics and large-preview control are released; the old frontend is now the static LLMiser fallback and public migration redirects remain live. The 22 October branch/review statements below are historical evidence for that earlier slice, not current release blockers. Use [START_HERE](START_HERE.md) and [PROGRESS](ai-handover/PROGRESS.md) for deployment state.
PRs #1–#13 are merged. Retained-result recovery, cron cutover, wake coalescing, shared reader statistics, large-preview control and CSV allocation reduction are released; the old frontend is now the static LLMiser fallback and public migration redirects remain live. The 22 October branch/review statements below are historical evidence for that earlier slice, not current release blockers. Use [START_HERE](START_HERE.md) and [PROGRESS](ai-handover/PROGRESS.md) for deployment state.

Shared reader statistics shipped in PR #9 at `84e510db62ea41f31c131771eb7d32b5e92515f4`: one allocation-bounded helper, one memoized result for reader line/status displays, preserved counting semantics and reproducible before/after measurements. See [reader statistics](LLMISER_READER_STATISTICS.md) for 259-test CI and live verification. Browser profiling follows below; the rest of the queue remains separate. No new production SQL or service retirement was part of this slice.

PR #11 released [large-preview control](LLMISER_LARGE_PREVIEWS.md) at `0501cd0381b4a41e9a9e94abf791ebd4fd8fd188`. Browser-generated dense Markdown exposed multi-second mount/layout work. Automatic formatting now pauses above 50,000 characters; full formatting is explicitly available, and raw content/exports are complete. Local checks passed 261 tests, the final 20-test selection and production build; all seven CI jobs and all six production deployments passed. The 28 October audit passed 20 HTTP checks with all pools 2/2 ready. Next work should measure native textarea input/representative devices or audit backend parser capacity; a worker alone will not eliminate the observed DOM/layout cost.

The current `codex/csv-parser-memory` candidate contains [a source capacity audit and CSV allocation reduction](LLMISER_PARSER_CAPACITY.md) for N/Z. It preserves output on verified MarkItDown 0.1.8/0.1.9 and falls back unchanged on other versions. Three fresh processes per mode show lower large-fixture peak memory; no concurrency/limit changes are justified by that local evidence. See its PR for release state, then measure workbook/native and final response/merge costs before extending this slice.
PR #13 released [a source capacity audit and CSV allocation reduction](LLMISER_PARSER_CAPACITY.md) for N/Z at `165d2b01ce8e30dfa09d32292ac911264f18e60d`. It preserves output on verified MarkItDown 0.1.8/0.1.9 and falls back unchanged on other versions. Three fresh processes per mode show lower large-fixture peak memory; no concurrency/limit changes are justified by that local evidence. All seven CI jobs and six production deployment statuses passed. Pre/post-release generated CSV/ZIP probes passed on all four document workers; 20 public HTTP checks passed and a follow-up uncached wake confirmed all pools 2/2 after an initial OCR 0/2 result. Next measure workbook/native and final response/merge costs before extending this slice.

## Verified starting point

Expand Down Expand Up @@ -42,7 +42,7 @@ Acceptance: expired link, later-part failure, complete Unicode assembly, restart
| Work | Next bounded change / dependency |
| --- | --- |
| Shared statistics and large previews | Counting released in PR #9; browser profiling and large-preview control released in PR #11; measure native input/devices before deeper editor/worker changes |
| Backend parser capacity | Source audit and synchronized CSV memory slice implemented on the current branch; check its PR for release. Next measure workbook/native and response/merge memory; preserve fidelity before tuning concurrency |
| Backend parser capacity | Source audit and synchronized CSV memory slice released in PR #13. Next measure workbook/native and response/merge memory; preserve fidelity before tuning concurrency |
| Shared deadline | Add an optional, backward-compatible remaining-budget protocol, reserving finalization time; test slow failures/cold starts |
| Controller detail pagination | Top-level lists already paginate; bound files/tree/events/work-items with stable cursors and truthful totals; query-plan evidence before new indexes |
| Replay-stable object identities | Fence attempt-specific writes before deterministic IDs; SQL attempt guards alone do not prevent a stale Storage overwrite; additive rollout for new roots only |
Expand Down
6 changes: 5 additions & 1 deletion docs/LLMISER_PARSER_CAPACITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,10 @@ Local final checks:
- The shared three-pool `common/` integrity checks stay intact; OCR has no new parser dependency. No runtime requirements changed.
- Before release, tiny generated authenticated multipart CSV/ZIP probes returned 200 on **N1, N2, Z1 and Z2**, each preserving the expected widest-row table, pipe escape, interior blank row and final Unicode marker. They created no Storage jobs.

The candidate branch is `codex/csv-parser-memory`. CI/merge/deployment and the matching post-release probe must be checked on its PR before calling it released. Local logs, package copies, raw benchmark JSON and operator probes remain ignored under `project-docs/parser-audit/`; no credentials are included in public evidence.
[PR #13](https://github.com/DevBehindYou/LLMiser/pull/13) merged and deployed runtime revision `165d2b01ce8e30dfa09d32292ac911264f18e60d` on 28 October 2026. All seven CI jobs passed, including the frontend's 261 tests and the normal/archive/OCR regressions. All six production Vercel status contexts succeeded. Post-release generated multipart CSV/ZIP probes passed on N1/N2/Z1/Z2, all still running MarkItDown 0.1.9, preserving exactly the same expected table as before release. They created no Storage jobs; no fresh browser/Storage-backed or cleanup acceptance is claimed.

All 20 public-page/discovery/health and permanent-redirect checks passed. The first bounded public wake check reported OCR 0/2; both OCR readiness endpoints subsequently returned 200/ready with Tesseract 5.5.0, and a fresh uncached wake returned 2/2 for normal/OCR/archive. No OCR, cron or queue configuration was changed to obtain that recovery. Legacy API/unknown-path routing retained its documented behavior.

Local logs, package copies, raw benchmark JSON and operator probes remain ignored under `project-docs/parser-audit/`; no credentials are included in public evidence. Later documentation-only commits do not change the verified parser runtime revision.

Next: profile realistic workbook/native parser memory and full response/merge costs on representative hosts, then evaluate admission/deadlines before any concurrency increase. The CSV reduction alone does not justify raising queue or worker limits.
4 changes: 2 additions & 2 deletions docs/START_HERE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,13 @@ Updated 28 October 2026. **LLMiser is the sole working repository and the source

## Current state

The app is live at https://llmiser.devbehindyou.com. PRs #1–#11 are merged; the latest verified runtime change is revision `0501cd0381b4a41e9a9e94abf791ebd4fd8fd188`. PR #11 passed 261 frontend tests and all seven CI jobs; all six production Vercel deployment statuses succeeded, including the static legacy frontend. A fresh 28 October release audit passed 20 focused HTTP checks and all three pools remained 2/2 ready. See [large-preview release evidence](LLMISER_LARGE_PREVIEWS.md), [reader statistics](LLMISER_READER_STATISTICS.md), the [earlier integrated release](LLMISER_RELEASE_2026-10-26.md) and [retirement evidence](LLMISER_LEGACY_RETIREMENT.md).
The app is live at https://llmiser.devbehindyou.com. PRs #1–#13 are merged; the latest verified runtime change is revision `165d2b01ce8e30dfa09d32292ac911264f18e60d`. PR #13 passed all seven CI jobs, including the 261-test frontend suite; all six production Vercel deployment statuses succeeded. Generated live CSV/ZIP probes passed on all four document workers, and 20 public HTTP checks passed. An initial OCR wake check returned 0/2; direct readiness and a subsequent uncached wake check confirmed all three pools 2/2 ready. See [parser-capacity release evidence](LLMISER_PARSER_CAPACITY.md), [large previews](LLMISER_LARGE_PREVIEWS.md), [reader statistics](LLMISER_READER_STATISTICS.md) and [retirement evidence](LLMISER_LEGACY_RETIREMENT.md).

The full app supports English (US), German, Japanese, Spanish, Russian and Simplified Chinese. The browser remembers the interface language; documents are not translated and OCR languages are unchanged. Blogs remain English. Eight articles, eight banners and forty figures are complete. The Chrome extension source is merged, but native acceptance and store publication remain pending.

Released [large-preview control](LLMISER_LARGE_PREVIEWS.md) pauses automatic formatting above 50,000 characters while retaining complete raw text and exports. Full formatting is available on request. Local/generated browser checks cover editing, document switching and exact full downloads; representative-device/native textarea profiling and the backend parser-capacity audit remain next boundaries. The six interface dictionaries now contain 631 messages each.

The next bounded backend slice is [parser capacity and CSV memory reduction](LLMISER_PARSER_CAPACITY.md), on `codex/csv-parser-memory`. It preserves verified upstream CSV output while reducing retained row allocations in both document pools. The audit distinguishes source limits, local generated measurements and live-worker acceptance; check that branch's PR for merge/deployment state before treating it as released. No concurrency increase follows from the benchmark.
Released [parser capacity and CSV memory reduction](LLMISER_PARSER_CAPACITY.md) preserves verified upstream CSV output while reducing retained row allocations in both document pools. The audit distinguishes source limits, local generated measurements and live multipart acceptance. No concurrency increase follows from the benchmark. Workbook/native parser and final response/merge memory remain the next backend investigation.

## Legacy frontend replacement is deployed

Expand Down
4 changes: 2 additions & 2 deletions docs/ai-handover/HANDOVER.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ LLMiser must remain independently maintainable after MDify and MDify-Pro are del
- Product-focused README with screenshots, recorded language demo, steps, format limits and measured-versus-unmeasured keyword notes.
- Chrome extension 0.2.0 source: one Converter toolbar popup, file picker/drop, manual Paste into detected AI sites, uploaded-job recovery. No public store release or completed native acceptance is claimed.

PRs #1–#11 merged. Latest verified runtime revision: `0501cd0381b4a41e9a9e94abf791ebd4fd8fd188`. [PR #11](https://github.com/DevBehindYou/LLMiser/pull/11) shipped large-preview control after 261 frontend tests and all seven CI jobs passed. All six production Vercel statuses succeeded; a fresh 28 October 20-check HTTP audit passed and all pools were 2/2 ready. See [large-preview release evidence](../LLMISER_LARGE_PREVIEWS.md), [shared statistics](../LLMISER_READER_STATISTICS.md), the [earlier integrated release](../LLMISER_RELEASE_2026-10-26.md) and [retirement acceptance](../LLMISER_LEGACY_RETIREMENT.md).
PRs #1–#13 merged. Latest verified runtime revision: `165d2b01ce8e30dfa09d32292ac911264f18e60d`. [PR #13](https://github.com/DevBehindYou/LLMiser/pull/13) shipped CSV allocation reduction after all seven CI jobs passed, including the 261-test frontend suite. All six production Vercel statuses succeeded; generated multipart CSV/ZIP probes passed on N1/N2/Z1/Z2, and a fresh 20-check HTTP audit passed. OCR initially returned 0/2 in the bounded wake check; direct readiness and a later uncached wake confirmed all pools 2/2. See [parser release evidence](../LLMISER_PARSER_CAPACITY.md), [large previews](../LLMISER_LARGE_PREVIEWS.md), [shared statistics](../LLMISER_READER_STATISTICS.md) and [retirement acceptance](../LLMISER_LEGACY_RETIREMENT.md).

## Repository and private-material boundaries

Expand Down Expand Up @@ -56,7 +56,7 @@ Active local tooling now has zero absolute references to either old checkout. LL

The owner asked to continue the optimization plan after the retirement handover. PR #8 completed that handover; PR #9 released [shared reader statistics](../LLMISER_READER_STATISTICS.md). PR #11 released [large-preview control and reproducible browser profiling](../LLMISER_LARGE_PREVIEWS.md): above 50,000 UTF-16 characters, formatting waits for an explicit request. Editing/document switching resets the request; complete raw text/copy/exports remain available. All six packs contain 631 messages. Local evidence: 261 full tests, final 20 focused tests, lint/consistency checks and production build passed; generated browser fixtures verify full rendering, edit re-pause, identical-text document switching, manual pause and exact current-edit download. CI passed all seven jobs, production deployed revision 0501cd0, the 20-check HTTP audit passed, and normal/OCR/archive were each 2/2 ready. No fresh production conversion/admin/cleanup acceptance is implied by that audit.

The current `codex/csv-parser-memory` branch adds [a parser-capacity audit and bounded CSV memory reduction](../LLMISER_PARSER_CAPACITY.md). N/Z use identical `app/csv_converter.py` modules, outside the three-pool shared `common/` package so OCR gains no MarkItDown dependency. Enable only on verified upstream 0.1.8/0.1.9; unknown versions/missing helper retain the original converter. Never expand the version allowlist without output-equivalence tests. Generated before/after hashes match, including synchronized conversions; queue limits and all other parsers stay unchanged. Inspect its PR for final CI/merge/deployment state.
PR #13 released [a parser-capacity audit and bounded CSV memory reduction](../LLMISER_PARSER_CAPACITY.md) at 165d2b0. N/Z use identical `app/csv_converter.py` modules, outside the three-pool shared `common/` package so OCR gains no MarkItDown dependency. Enable only on verified upstream 0.1.8/0.1.9; unknown versions/missing helper retain the original converter. Never expand the version allowlist without output-equivalence tests. Generated before/after hashes match, including synchronized conversions; queue limits and all other parsers stay unchanged. Local suites passed 118 normal/98 archive tests, 25 tests on 0.1.8, and the expected 0.1.7 fallback checks. CI/deployment and post-release generated multipart probes passed. These direct worker probes created no Storage jobs and do not replace browser/Storage-backed acceptance.

Next investigate native textarea/input behavior and representative devices before workers/editor virtualization, or measure XLSX/native parser and response/merge memory from [the optimization queue](../LLMISER_OPTIMIZATION_PLAN.md). Do not repeat the shipped preview work, silently truncate source/exports, or mistake local parser measurements/multipart probes for Storage-backed acceptance. The optimization plan's 22 October unmerged-branch statements are historical.

Expand Down
7 changes: 4 additions & 3 deletions docs/ai-handover/PROGRESS.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,12 @@ Updated 28 October 2026. This is the current tracker. Earlier phase-specific num
| Six-language web app | PR #5 released; English US, German, Japanese, Spanish, Russian, Simplified Chinese; now 631 messages each after PR #11 |
| Final display-label fixes | PR #6 merged/deployed; preview labels/tooltips, mobile theme and retention labels |
| Public README | Screenshots, recorded demo, steps, formats, profiles, privacy and FAQs; separate keyword evidence |
| Final frontend CI | PR #11: 261 tests passed; lint, dictionaries, SQL/blog consistency and production build passed |
| Final frontend CI | PR #13: 261 tests passed; lint, dictionaries, SQL/blog consistency and production build passed |
| Other CI | Backend N/O/Z, PostgreSQL, edge and extension jobs passed |
| Production deployment | Latest runtime change `0501cd0381b4a41e9a9e94abf791ebd4fd8fd188`; all six Vercel statuses successful, including the legacy static shell |
| Production deployment | Latest runtime change `165d2b01ce8e30dfa09d32292ac911264f18e60d`; all six Vercel statuses successful, including the legacy static shell |
| Shared reader statistics | PR #9 merged/deployed; allocation-bounded counting and shared reader line/status metrics; preserved count semantics; synthetic benchmark and limits documented |
| Large-preview control | PR #11 merged/deployed; generated browser profiling, explicit formatting above 50,000 characters, complete raw/copy/download acceptance, six-language notices |
| CSV parser memory | PR #13 merged/deployed; verified-version output preservation in N/Z, synchronized generated measurements, and successful pre/post-release multipart CSV/ZIP probes on all four document workers |
| Production HTTP audit | 40 passing checks during the integrated release; all 13 sitemap pages; permanent legacy redirects; API noindex/auth boundary |
| Backend readiness | Normal/OCR/archive each 2/2 ready after final release |
| Browser acceptance | All six languages, persistence, English-only blogs, translated policy/controller login; German 390px viewport had no horizontal overflow |
Expand Down Expand Up @@ -50,7 +51,7 @@ See [release record](../LLMISER_RELEASE_2026-10-26.md), [localization](../LLMISE

The owner resumed the implementation plan after the handover was merged in PR #8. [Shared reader statistics](../LLMISER_READER_STATISTICS.md) shipped in PR #9. [Large-preview control](../LLMISER_LARGE_PREVIEWS.md) shipped in PR #11 at 0501cd0 after 261 frontend CI tests/all seven jobs passed. All six production deployments succeeded; the 28 October 20-check HTTP audit passed and all three pools were 2/2 ready. Actual-component generated browser measurements and exact raw/download acceptance support the preview boundary. Representative-device/native textarea profiling and deeper parser-capacity work remain separate.

Current implementation branch: `codex/csv-parser-memory`. [Parser audit/CSV evidence](../LLMISER_PARSER_CAPACITY.md) records existing bounds, verified-version output preservation and three-run memory measurements. The synchronized large untraced fixture used about 25% less peak process memory, with identical output; the small timing sample regressed, so no universal speedup is claimed. Live pre-release multipart CSV/ZIP probes passed on all four document workers. Final regression/CI/deployment evidence must be checked before treating the candidate as released. No worker/queue tuning or Storage-backed conversion acceptance is implied.
[Parser audit/CSV evidence](../LLMISER_PARSER_CAPACITY.md) shipped in PR #13 at 165d2b0. The synchronized large untraced fixture used about 25% less peak process memory, with identical output; the small timing sample regressed, so no universal speedup is claimed. Local normal/archive suites and compatibility checks passed, all seven CI jobs and six production deployments succeeded, and pre/post-release multipart CSV/ZIP probes passed on all four document workers. All 20 public HTTP checks passed. OCR initially returned 0/2; direct readiness and a later uncached public wake confirmed all pools 2/2, without an OCR configuration change. No worker/queue tuning or Storage-backed conversion acceptance is implied.

- Retain the domains and immediate permanent redirects after the completed static frontend cutover.
- Review any additional GitHub-only material the owner needs beyond the completed record exports. Actions artifacts/logs/secrets, provider settings and linked attachments were not exported.
Expand Down
Loading