Using .Q.pm to give us an in-memory index (attribute) on the live partition

[picoDoc]

The idea

Ok, so in the first two blog posts in what I'm now calling my "rdbs are shit" trilogy, I argued that rdbs are not so necessary in kdb+ architectures, but they do still win for anything that really benefits from attributes. And we can't have attributes on disk for live data because they are not appendable.

I've spent a while banging my head against this (for years really), and now talking to AI about it too. Fable recently pointed out .Q.pm to me, and while not being mentioned in the docs at all, it basically unlocks the whole thing.

So what is .Q.pm? .Q.MAP[] leaves it behind, and it's basically a dictionary of the mapped tables and their columns. so if you have a normal looking hdb with one table .Q.pm might look like:

q).Q.pm
trade | `u#((`:.;2026.08.04);(`:.;2026.08.05))!(+`time`sym`price`size!...

and something I really did not expect about this: you can just modify the entries!

.Q.pm[`trade]:@[.Q.pm`trade;enlist(`:.;2026.08.05);@[;`sym;:;`g#`sym?`A`B`C`D`E]]]

so that just drops an in-memory column in place of the one it originally mapped from disk, with an attribute. So long as that column is the right length everything else just seems to work. The engine just sees a column with an attribute and does not care where it came from. So you get plain q-sql, by sym, aj, all that stuff. And it composes with history: p# serves the older partitions, and g# the live one. And it's appendable now.

So really this lets you build something like a "normal" database: an in-memory index over bulk data on disk.

This PR specifically

Adds an experimental, opt-in, per-process flag that gives the live partition an in-memory index. Default off, mapped mode only. With it off, everything is as it was

Two important notes:

• The in-memory copy is derived from the disk file itself on each flush, appending only that flush's delta (n _ d, so it reads the new rows, not the whole column). So no messing around with needing an IPC feed/subscription etc., and it's pretty cheap as it's only ever tailing the files from disk
• Config comes from sort.csv, the same file EOD already uses to place p#, mirroring .sort.sorttab's precedence, so live and historical indexing shoudl stay lined up

Performance

This is in the README now too, but worth putting here:

query	1 RDB	1 mapped DB	1 mapped DB + usememattr	4 DBs in parallel
selective lookup, indexed col (sym, ~5k rows)	0.5	392	0.6	101
selective lookup, un-indexed col (tradetime, ~5k rows)	124	118	136	33
aggregate (avg/sum by sym)	590	660	576	181
filtered aggregate	1247	1153	1184	296

So it basically goes to full rdb performance on everything. Even ajs (not shown here).

It does obviously mean, if you switch this on, that each db process is no longer "free". They each have whatever the attributed columns are in-memory. but that's a lot less than an rdb. And you can opt in or out of it on a process by process basis.

I've had claude batter as many queries as it could think of against this and it can't find anything where splicing data in like this causes any issues, but it is still basically an undocumented feature. We should maybe ask KX about it.

I've got a blog post sitting here that explains the same stuff in a more blog-posty way, but it's not quite done yet. Any and all thoughts and feedback appreciated here.