docs/benchmarks: record COPY FROM STDIN as tried-and-not-faster on laptop - #94
Merged
Merged
Conversation
…ptop The remaining tracked v1.1 follow-up from the restore review chain was `COPY FROM STDIN` as the "next 10× hop" after batched-INSERT. Prototyped 2026-05-24; measurement on the same Docker-Desktop Postgres did NOT show the expected speedup: | N | batched-INSERT | COPY FROM STDIN | Δ | |------|---------------:|----------------:|-----:| | 10K | 102 ms | 104 ms | +2% | | 100K | 1.07 s | 1.06 s | flat | | 1M | 10.34 s | 11.88 s | +15% | Manifest hashes stayed byte-identical — the change was correct, just not faster. Diagnosis: at localhost-IPC scale, per-row write cost (WAL emission + btree insert) dominates protocol overhead, which is what COPY skips. Multi-row INSERT at 1000 rows/statement already amortises the round-trips; COPY's text-encoding hot loop added CPU work without saving anything Postgres-side. Worktree + branch deleted, no PR opened. Recorded here in benchmarks.md so the next person doesn't repeat the experiment expecting a different result on the same hardware. The doc now points at the next viable path: binary COPY (skips text-encoding CPU) on a managed-Postgres instance (where RTT + group commit change the balance). Cloud-class hardware remains the unambiguous §9-meeting path. Three sites updated in benchmarks.md: - Targets-vs-measured headline row (no longer claims COPY is the next 10× hop). - §"the §9-shaped row" section's framing of the remaining gap. - §"Coverage gaps" — the COPY bullet now records the negative result, the diagnosis, and the conditions under which a revisit might land differently.
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the architecture benchmark documentation to record that a COPY FROM STDIN prototype (as a potential next restore speedup after batched multi-row INSERT) was measured on the same laptop setup and did not improve performance.
Changes:
- Reframes the rollback/restore benchmark notes to indicate
COPY FROM STDINwas tried (2026-05-24) and wasn’t faster on the laptop setup. - Adds measured COPY vs batched-INSERT numbers (10K/100K/1M) plus a brief diagnosis and guidance for future revisits (e.g., binary COPY, managed Postgres).
- Removes prior wording that implied COPY was the expected “next 10× hop” on this hardware.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| | `commit` (memory snapshot of N-row pgvector) | < 2 s @ 1M rows + 100 deltas | **5 ms @ 10K / 5.6 ms @ 100K / 18.81 ms @ 1M** (laptop) | text-only `episodes` schema; no embeddings, no deltas. ✓ comfortably within target. | | ||
| | `commit` (prompts-only, 16-byte system prompt, no memory) | n/a — degenerate input | **2.3 ms median** (Criterion) | ✓ no obvious blocker; not a §9 measurement | | ||
| | `rollback` (memory restore of N-row pgvector) | < 5 s @ 1M rows + 10 commits | **102 ms @ 10K / 1.07 s @ 100K / 10.34 s @ 1M** (laptop, post-batched-INSERT) | ⚠ 1M still ~2× over §9 on this hardware shape, but linear and predictable. Multi-row INSERT closed the 12× gap from the per-row path; the next 10× hop is `COPY FROM STDIN` (tracked, v1.1). | | ||
| | `rollback` (memory restore of N-row pgvector) | < 5 s @ 1M rows + 10 commits | **102 ms @ 10K / 1.07 s @ 100K / 10.34 s @ 1M** (laptop, batched-INSERT) | ⚠ 1M still ~2× over §9 on this hardware shape, but linear and predictable. Multi-row INSERT closed the 12× gap from the per-row path. `COPY FROM STDIN` was prototyped 2026-05-24 and **did not help on this hardware** (see "Coverage gaps" below for measured numbers and reasoning). A representative cloud-class machine remains the unambiguous §9-meeting path. | |
Comment on lines
+99
to
+101
| Diagnosis: on Docker-Desktop Postgres at localhost, the bottleneck for restore is per-row write cost (WAL emission + btree insert), not the frontend/backend protocol overhead COPY skips. Multi-row INSERT at 1000 rows/statement already amortises the protocol round-trip; COPY's text-encoding pass added CPU work in the hot loop without saving anything Postgres-side. Manifest hashes stayed byte-identical, so the change was correct — just not faster. | ||
|
|
||
| What the measurement does NOT rule out: COPY winning on a representative cloud-class instance where (a) network RTT is higher than localhost IPC, (b) WAL fsync overhead is amortised by group commit, or (c) the binary COPY format (not yet tried) skips the text-encoding CPU cost. If a future revisit happens, start with binary COPY and run against managed Postgres, not Docker-Desktop. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The remaining tracked v1.1 follow-up from the restore review chain was `COPY FROM STDIN` as the "next 10× hop" after batched-INSERT. Prototyped 2026-05-24; the measurement on the same Docker-Desktop Postgres did NOT show the expected speedup.
Measured
Manifest hashes stayed byte-identical — the change was correct, just not faster.
Diagnosis
At localhost-IPC scale, per-row write cost (WAL emission + btree insert) dominates the protocol overhead COPY skips. Multi-row INSERT at 1000 rows/statement already amortises the round-trips; COPY's text-encoding hot loop added CPU work without saving anything Postgres-side.
What the measurement does not rule out: COPY winning on a representative cloud-class instance where network RTT is higher than localhost IPC, WAL fsync is amortised by group commit, or where binary COPY (skipping the text-encoding pass) is used.
Action taken
Test plan