Skip to content

server: genes (LoRA adapters) page between the device and host RAM (stacked on #70) - #72

Open
joelteply wants to merge 3 commits into
fable/ram-pagesfrom
fable/gene-pages
Open

server: genes (LoRA adapters) page between the device and host RAM (stacked on #70)#72
joelteply wants to merge 3 commits into
fable/ram-pagesfrom
fable/gene-pages

Conversation

@joelteply

@joelteply joelteply commented Oct 24, 2026

Copy link
Copy Markdown

Stacked on #70 (slot RAM pages). The mechanism half of continuum's gene working set (SERVING-ENGINE.md, the genes module; one memory budget, §3): the core decides by temperature which loaded genes hold device memory; these verbs move them.

What

  • llama: llama_adapter_lora_page_out / page_in / is_paged_out / n_bytes. page_out copies the tensors to one host vector and frees the device buffers; page_in re-allocates from each context's buffer type and copies back. The adapter keeps its id and pointer, so slot adapter lists, KV computed under it and saved pages stay valid.
  • server-genes.{h,cpp}: POST /lora-adapters/page_out and /page_in {"id"}, idempotent; refuses an unknown id, the learning optimizer's adapter, one active by default, one a processing slot applies. Reports resident, n_bytes, n_moved, t_ms. Drop is the existing /lora-adapters/unload (load is the existing /lora-adapters/load).
  • A task (or /learn) applying a host-resident gene pages it in on its own path and counts the miss (lora_page_in_on_demand_total): never a decode on freed memory.
  • GET /lora-adapters lists resident + n_bytes; /metrics: lora_page_{out,in,in_on_demand,failed}_total, lora_{device,host}_bytes; /props lora_page_version: 1.

Tests

tools/server/tests/unit/test_lora_page.py (4, stories15M_MOE + its Shakespeare LoRA): page out/in is identical to never paging; a task applying a paged-out gene pages it in first (same output, miss counted); typed refusals + idempotent repeats; unload of a paged-out gene frees its host copy. test_lora.py and test_slot_page.py still pass (11 passed, 2 skipped).

Measured

M5 Metal, qwen35-0.8b.gguf, synthetic rank-16 LoRA over 204 projections (50.4 MB), build 3fa5844, 2026-10-24:

  • page_out median 2.3 ms, page_in median 4.6 ms (5 rounds); output identical after an on-demand page-in.
  • Two minds decoding together (aggregate tok/s, median of 3): base 212; same gene set 115; different gene sets 27; same genes at different scales 39. can_batch_with requires equal adapter lists, so minds with different gene sets serialize, and each switch re-reserves the scheduler. The scheduler must group minds by gene set; the core's composer holds scales bit-identical while a mind's set holds.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

@joelteply @claude
server: a slot pages to named host RAM and back; checkpoints keep a b…
971962d 
…yte budget; a page carries its draft KV

A mind switch costs a memcpy instead of a 66-100 s disk save or a 70-160 s re-prefill
(continuum SERVING-ENGINE.md, slice S1, plan line 3b).

- /slots/{id}?action=page_out&page=<name>: the slot's target KV, draft (MTP) KV,
  speculative tail state and context checkpoints move into a server-owned RAM page store,
  and the slot is freed. page_in puts them into any slot with its tokens, so the next
  request's prefix matches; the page leaves the store. spill writes a page in the slot-save
  format (.bin + .ckpt + .lora + new .dft) on the HTTP thread, off the engine thread;
  "drop": true then frees that page's generation only. page_drop frees a page.
  page_out/page_in defer while the slot processes, as restore does. Errors are typed
  (404 unknown page, 503 over --page-ram-bytes, 400 empty slot); a failed page_out leaves
  the slot intact, a failed page_in leaves the page in the store. /props slot_page_version 1.
- save writes the draft state to `.dft` and counts it in its quote; restore reads it, so
  MTP is warm after a restore (no `.dft`: the draft seq is cleared and the cold start logged).
- --ctx-checkpoints-bytes: one budget across all slots in place of the per-slot count; the
  slot holding the most bytes drops its oldest checkpoint after its first. Every eviction
  logs and counts.
- /metrics: page_store_bytes/pages, ctx_checkpoint_bytes, page_{out,in,failed}_total,
  ctx_checkpoints_{evicted,skipped}_budget_total.
- llama_state_seq_save_file_from_data (llama-ext.h): libllama stays the only owner of the
  sequence-file format; the sidecar writers move from the save arm into server-pages.cpp,
  shared by save and spill.

Measured (Qwen3.5 0.8B hybrid, M5 Metal, 19,794 tokens, 2 checkpoints, median of 5):
page_out 10.4 ms, page_in 7.8 ms, 304 MB; save 134 ms, restore 37 ms (page-cache warm),
304 MB on disk; spill 101 ms. The paged slot's next completion processed 6 prompt tokens
and its output matched a slot that never paged.
Tests: tools/server/tests/unit/test_slot_page.py (5, hybrid case via
LLAMA_SERVER_TEST_HYBRID_GGUF) + test_slot_save.py (11): 16 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
joelteply added a commit to CambrianTech/continuum that referenced this pull request Oct 24, 2026
@joelteply @claude
Genome residency: one node owner pages genes device <-> host RAM insi…
4cdf184 
…de the KV's budget; the unwired working-set stack and the TS-era LRU are deleted

Serves her learning on any hardware (goal 8, SERVING-ENGINE §3 one budget): a node's genes
hold device memory only while minds run them, without a second budget beside the KV.

- genome::gene_residency: ONE node-wide owner. A DeviceTenant of the lanes' pool
  (inference-lanes): its device bytes count there and the PressureBroker relieves it first
  (a gene pages in ms, a lane costs a mind her KV); LRU-2 temperature (second-last use by any
  mind, no decay constant); genes in a mind's current phenotype are pinned; after each turn a
  coalesced reconcile pages pinned host genes back in, off the request path. It holds each
  mind's phenotype, which is the composer's previous (one copy). Probes genome.page,
  genome.page.failed.
- CoordinatorResourcePool::with_tenant: usage = lanes + tenants, evict tenants before lanes.
- llama_server::gene_pages: the engine's /lora-adapters catalog with residency and the
  page_out / page_in verbs (CambrianTech/llama.cpp#72); unknown bytes stay None.
- gene_trial::Standing (Pending / Adopted) on her_genes; Standing::may_run is the one rule
  the examiner's verdict owns (Cormac's condition); the composer asks it.
- Deleted (Cormac agreed on airc): genome/{local_manager (+848-line tests), manager, bus,
  store, blob, working_set}, tier.rs down to TierRole (recall uses it); persona/genome_paging
  (+tests) and PersonaCognition::with_budget; the six cognition/genome-* commands and
  sync-domain-classifier (no caller in any client; COMMAND-CENSUS marks it for deletion);
  the channel tick's enrollment step (read an engine nothing ever filled);
  DomainClassifier::sync_from_adapters; their ts-rs bindings; the dead stress-tests feature.

Tests: residency 3, tenant 1, gene_pages 1; genome 323, engine 17, mind 68, core cognition
104, source-hygiene 49; continuum-core --tests checks clean with and without stress-tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
@joelteply joelteply mentioned this pull request Oct 24, 2026
Merged
joelteply added a commit to CambrianTech/continuum that referenced this pull request Oct 24, 2026
@joelteply @claude
Genome: many genes per turn, weighted by distance; genes paged under …
4ac5e64 
…the KV budget; the old paging stacks deleted (−6.8k) (#5367)

* Genome: many genes per turn, weighted by distance, held while the set holds (GENE-REUSE-FORK-MINT §3d)

Serves her learning (goal 8): a mind's genes follow what her mind is on, several at
once, instead of every gene at full scale on every turn (the live path) or one gene by
max_by (rung 0, a command nothing called).

- genome::selection is the one owner of APPLIED: every gene whose region the turn's
  focused line is inside, a convex mix weighted by depth in its own region
  ((sim - leave) / (1 - leave)), enter at r and leave at 1.5 r, weights held
  bit-identical while the set holds: the fork clears a slot's KV on ANY scale change
  (are_lora_equal compares scales exactly), so re-weighting per turn would re-prefill
  her context every turn. Unsigned genes run at their trained scale (as before).
- GeneSignature files its region at mint (10th-percentile similarity of its own corpus),
  floored at the space's measured null + 2 sigma; old signatures still load.
- Both turn sites ask genome::selection::live_genes with recall's focused_query (one
  owner of "what her mind is on"); the genes stay on her cycle until the next turn, so
  nothing between turns flips her slot's adapters.
- Probes: genome.select (each gene's similarity, edges, scale), genome.select.set_changed.
- Deleted: gene_trial::live_genes, GenomeRestore (restoring the pre-turn genome now
  flips her adapters twice a turn), model_selection rung 0 + NeedEmbedding + its two
  constants, the select-model command's signature join.

Tests: 6 selection contracts + 1 region test; genome 387 pass, model_selection 14 pass,
source-hygiene 47 pass; continuum-core --tests checks clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

* Genome composer: the phenotype, a = clamp((sim - f)/(1 - f)), top-k, a / max(1, sum a); the receipt carries it

Per the coordinator's learning-loop design: genome::selection becomes genome::composer, the ONE
composer of a turn's phenotype (gene_trial::genes_for_turn is now her_genes, the candidate set
at trained scale; the composer sets the scale).

- weight a = clamp((sim - f) / (1 - f), 0, 1) from each gene's floor; the `capacity` deepest
  kept; engine scale a / max(1, sum a); hysteresis and held-while-the-set-holds unchanged.
- compose(persona, perception) -> Phenotype; Phenotype::without(g) is the examiner's P minus g
  (g off, every other scale unchanged).
- GenerationReceipt carries gene_scales aligned with genes (PartialEq only, f64); turn sites
  probe persona.genome.turn_genes with each gene's scale.

Tests: composer 8 (adds capacity and P minus g), genome 389, provenance 3, mind 38,
source-hygiene 47; continuum-core --tests checks clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

* Genome residency: one node owner pages genes device <-> host RAM inside the KV's budget; the unwired working-set stack and the TS-era LRU are deleted

Serves her learning on any hardware (goal 8, SERVING-ENGINE §3 one budget): a node's genes
hold device memory only while minds run them, without a second budget beside the KV.

- genome::gene_residency: ONE node-wide owner. A DeviceTenant of the lanes' pool
  (inference-lanes): its device bytes count there and the PressureBroker relieves it first
  (a gene pages in ms, a lane costs a mind her KV); LRU-2 temperature (second-last use by any
  mind, no decay constant); genes in a mind's current phenotype are pinned; after each turn a
  coalesced reconcile pages pinned host genes back in, off the request path. It holds each
  mind's phenotype, which is the composer's previous (one copy). Probes genome.page,
  genome.page.failed.
- CoordinatorResourcePool::with_tenant: usage = lanes + tenants, evict tenants before lanes.
- llama_server::gene_pages: the engine's /lora-adapters catalog with residency and the
  page_out / page_in verbs (CambrianTech/llama.cpp#72); unknown bytes stay None.
- gene_trial::Standing (Pending / Adopted) on her_genes; Standing::may_run is the one rule
  the examiner's verdict owns (Cormac's condition); the composer asks it.
- Deleted (Cormac agreed on airc): genome/{local_manager (+848-line tests), manager, bus,
  store, blob, working_set}, tier.rs down to TierRole (recall uses it); persona/genome_paging
  (+tests) and PersonaCognition::with_budget; the six cognition/genome-* commands and
  sync-domain-classifier (no caller in any client; COMMAND-CENSUS marks it for deletion);
  the channel tick's enrollment step (read an engine nothing ever filled);
  DomainClassifier::sync_from_adapters; their ts-rs bindings; the dead stress-tests feature.

Tests: residency 3, tenant 1, gene_pages 1; genome 323, engine 17, mind 68, core cognition
104, source-hygiene 49; continuum-core --tests checks clean with and without stress-tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

* Examiner: her genome for a sitting is every gene she may run (her_genes + Standing::may_run), at trained scale

gene_trial::live_genes was replaced by genome::composer; the exam holds the genome fixed so
the arms differ only by the gene.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
joelteply and others added 2 commits October 24, 2026 04:46
@joelteply @claude
server: page RAM ceiling counts every live page; page files publish w…
e08b11e 
…hole generations

Review fixes on the RAM page verbs (Codex on #70):

- One count: server_page_bytes is what page_out admits (measured before any copy,
  speculative tail state included) and what the store charges; put re-checks the
  ceiling on exactly the page it stores, so no pre-count can get a page past it.
- The ceiling counts live ownership: a stored page carries a charge released when
  its LAST owner drops it, so a page a spill still holds after page_in, page_drop,
  replacement or a model unload keeps counting. Admission subtracts a replaced page
  only when the store is its sole owner.
- page_out moves the slot's prompt only into a stored page: put inserts first and
  moves the prompt back on any refusal, so a refused or failed page_out leaves the
  slot's sequence intact.
- Page files (server-page-files.{h,cpp}, moved out of server-pages.cpp): save and
  spill publish `<file>` + .ckpt/.lora/.dft as one generation. Every file is
  written to `..page-staging/<pid>-<n>/` and synced, a commit record is renamed
  into place, then the files move over the committed ones and a sidecar the
  generation lacks is removed. A failure before the record leaves the committed
  generation untouched and nothing staged; restore reads through
  server_page_reader, which completes a publication cut short after its record,
  removes stages no live publication owns, and holds commits off while it reads.
  A save whose sidecar cannot be written now fails instead of leaving a mixed set.
- /slots page results report n_bytes_spec.

Tests: tests/test-server-pages.cpp (admission = charge with speculative state,
spill-held pages stay counted, refusal keeps the incumbent, failed and interrupted
publications keep the committed generation and leave nothing staged);
test_slot_page.py adds a crashed-publication restore and an MTP page at an exact
ceiling (hybrid GGUF).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
@joelteply @claude
server: a gene (LoRA adapter) pages between the device and host RAM; …
a719414 
…a task applying a paged-out one pages it in first

The core's gene working set decides, by temperature and within one memory budget, which of a
node's loaded genes hold device memory (continuum SERVING-ENGINE.md, the `genes` module). This
is the mechanism, beside the slot RAM pages:

- llama: llama_adapter_lora_page_out / page_in / is_paged_out / n_bytes. page_out copies the
  tensors to one host vector and frees the device buffers; page_in re-allocates from each
  context's buffer type and copies back. The adapter keeps its id and pointer, so slot adapter
  lists, KV computed under it and saved pages stay valid.
- server-genes.{h,cpp}: POST /lora-adapters/page_out and /page_in {"id"} (idempotent; refuses
  an unknown id, the learning optimizer's adapter, one active by default, one a processing
  slot applies), reporting resident, n_bytes, n_moved, t_ms. Drop is the existing unload.
- A task (or /learn) that applies a host-resident gene pages it in on its own path and counts
  the miss: never a decode on freed memory.
- GET /lora-adapters lists resident + n_bytes; /metrics: lora_page_{out,in,in_on_demand,failed}_total,
  lora_{device,host}_bytes; /props lora_page_version 1.

Tests: unit/test_lora_page.py (4): page out/in identical to never paging, on-demand page-in,
typed refusals + idempotent repeats, unload of a paged-out gene; test_lora + test_slot_page pass.

Measured (M5, Metal, qwen35-0.8b, synthetic rank-16 LoRA over 204 projections, 50.4 MB,
2026-10-24): page_out median 2.3 ms, page_in median 4.6 ms (5 rounds); on-demand page-in
gives identical output. Two minds decoding together: same gene set 115 tok/s aggregate,
different sets 27 tok/s, same genes at different scales 39 tok/s, base 212 tok/s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

server

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@joelteply