Bug: reembed_backfill starves behind extract_chunk on the shared LLM gate, so new chunks wait 7-12 min for vectors

[YellowSnnowmann]

TinyAgents version or commit

tinycortex 8dab3d2 (main, v0.1.2). Reproduced through tinymemory v1.14.1, which vendors tinycortex cb1c163; both have the same queue code.

Rust toolchain

rustc 1.96.1 (aarch64-apple-darwin)

Reproduction

1. Admit a few hundred chunks quickly with the default QueueConfig (llm_permits = 1). In the field this was a 500-message Gmail sync through openhuman (tinymemory module 1.14.1).
2. Inspect mem_tree_jobs: ~450 extract_chunk rows ready, and exactly one reembed_backfill row ready whose last_error is llm concurrency gate busy, with available_at_ms re-bumped every few seconds.
3. Inspect mem_tree_chunk_embeddings.created_at: vectors arrive in bursts 7–12 minutes apart, every row of a burst stamped the same second (one batch, one transaction).

Observed on 2026-09-04: bursts at 14:17 (24 rows), 14:24:55 (327 rows), 14:36:48 (340 rows) while extraction completed 10–20 jobs/min at 3–5 s each. Chunks written between bursts sat vector-less for up to 12 minutes (mean lag 354 s, max 720 s over 694 chunks).

Expected behavior

A newly admitted chunk gets its vector within about a minute of being written while the queue is busy. The backfill job should take the LLM permit as soon as it frees, embed one bounded batch, defer for REEMBED_BACKFILL_REVISIT_MS (750 ms) and hand the permit back to extraction.

Actual behavior

• Since handle_extract no longer embeds inline (src/memory/queue/handlers.rs, "extract no longer embeds inline"), the deduped reembed_backfill job is the only path that writes chunk vectors.
• JobKind::ReembedBackfill is is_llm_bound() (src/memory/queue/types.rs:80), so it shares the single-permit gate with every extract_chunk.
• claim_next (src/memory/queue/store.rs:134-141) ranks only seal / flush_stale / append_buffer; everything else, including the backfill, sits in the ELSE bucket ordered by available_at_ms.
• When a claim finds the gate busy, run_once_with_gate calls mark_deferred(now + LLM_GATE_RETRY_MS) (50 ms, src/memory/queue/worker.rs). With ~450 extract rows doing the same, the backfill row is round-robined behind the whole extraction backlog: one embed batch per full rotation, i.e. per ~450 extract completions.

Net effect: embeddings trail extraction by the length of the backlog instead of leading it, and the openhuman Sources row reports hundreds of chunks "pending" for minutes after every large sync (tinyhumansai/openhuman#6025).

Additional context

Proposed fix (small): give the backfill claim priority right after seal in claim_next:

CASE kind
  WHEN 'seal'             THEN 1
  WHEN 'reembed_backfill' THEN 2
  WHEN 'flush_stale'      THEN 3
  WHEN 'append_buffer'    THEN 4
  ELSE 5
END ASC, available_at_ms ASC

Keep it LLM-gated: the single permit also stops a local Ollama embedder and a local extraction model from running side by side on small machines, which is presumably why it was gated. With priority, each permit release goes to the backfill first when it has work (≤1000 rows, one call), it defers 750 ms, extraction runs in between, and once coverage is complete the job settles Done, so extraction is never starved in return. Add a store_tests.rs case: an older ready extract_chunk and a ready reembed_backfill → the backfill is claimed first. types_tests.rs:140 stays as is.

Optional, separate, needs measurement: LLM_GATE_RETRY_MS 50 → 250–500 ms. Today four workers spin claim → gate busy → defer across all ready LLM rows; in the field every one of the 452 ready rows had available_at_ms inside a 3-second window, i.e. hundreds of row rewrites per second on the chunk store while ingest is also writing to it.

Downstream: tinymemory bumps vendor/tinycortex and cuts a patch release, then openhuman re-pins the module. Parent issue with the full RCA: tinyhumansai/openhuman#6025.