Feature merge

[alowrydi]

On-disk partition-segment merge module, di.merge

Extracts TorQ's code/common/merge.q into a standalone kdb-x module - on-disk partition-segment
merging for the write-down (WDB) flow, called by both wdb.q and tickerlogreplay.q in legacy
TorQ: whole-partition, column-by-column, or a size-driven hybrid of the two, chosen per partition
by a configurable row-count or byte-size limit, plus partition-size tracking to drive that
decision. The last unresolved hard dependency for di.wdb.

Trello ticket - https://trello.com/c/HAJdcl9V/111-kdb-x-merge

---

Files created

File	Description
di/merge/init.q	Loads merge.q, defines export of 14 functions
di/merge/merge.q	Full implementation - module state, internal helpers, public API, init
di/merge/merge.md	Full module documentation - see below
di/merge/test.csv	190 k4unit test assertions
di/merge/VERSION	Module version

---

How to test

k4unit:use`di.k4unit
k4unit.moduletest`di.merge

2026.08.21T14:46:17.418 start
2026.08.21T14:46:17.418 :.../di/merge/test.csv 190 test(s)
2026.08.21T14:46:17.456 end
Test results:
...
All tests passed

190/190 assertions passing.

Coverage includes: dependency validation, the requireinit guard on every exported function,
config application (row-count vs byte-size batching, partlimit splitting), partition-size
tracking and cross-process sync, both re-init and failure-isolation design decisions with
regression coverage, version/getapimeta shape, and end-to-end mergebypart/mergebycol/
mergehybrid against real on-disk segments.

---

Design decisions

1. No hard module dependencies; parted columns supplied by the caller - Legacy merge.q read a
table's parted column(s) from .sort.params (populated from sort.csv) via
getextrapartitiontype. That coupling was removed on extraction: the parted column(s)
(extrapartitiontype) are passed in by the caller instead, so di.merge doesn't depend on
di.sort - confirmed against the plan's own "Hard dependency tree", which lists di.merge under
STANDALONE. Only log is injected, via init.

2. syncpartsizes - a real receive-side API for a legacy raw-IPC pattern - Legacy wdb.q fans
partition-size state out to sort-worker processes over raw async IPC with no symmetric
receive-side function: a receiving process evaluated the raw (upsert;.merge.partsizes;y)tuple directly, which only worked if it had already loadedmerge.qso the table existed with the right schema - an undocumented, load-order-dependent contract.syncpartsizes[t]gives the receive side a real,requireinit`-guarded function to go through instead.

3. init preserves tracked-but-unmerged partition sizes across a re-init - Calling init again
with valid deps (e.g. a live config reload) does not wipe .z.m.partsizes; segments tracked since
the last clearpartsizes[] survive. partsizes is orthogonal to the log/mergebybytelimit/
partlimit deps a re-init is typically changing, and silently discarding tracked-but-unmerged
segment sizes is a worse failure mode than leaving them alone. init logs explicitly when it
preserves state, so the decision is visible rather than something a future debugger has to
discover by reading source.

4. mergebypart isolates each segment's read; mergebycol deliberately does not protect its
column read - Not equivalent failure modes, so making them match wouldn't obviously be the safer
choice. mergebypart now reads each segment in a batch individually and protected - a
missing/corrupt segment is error-logged and dropped without disturbing its batch-mates, which
still merge. mergebycol merges one column at a time into the same destination; a swallowed
failure partway through would leave some columns reflecting the new data and others silently
stale - a genuinely worse, silently-inconsistent partition, not just a delayed merge. So
mergebycol's column read stays intentionally unprotected.

5. The parted attribute is applied once, to the whole destination, only via mergehybrid - A
full write-down-and-merge smoke test against real segments and real di.log (not mocks) found
that neither mergebypart nor mergebycol alone can guarantee dest ends up with the `p#
attribute genuinely set: upsert appends raw values onto an on-disk column without persisting an
in-memory attribute, and no single batch/column write can guarantee the whole destination stays
grouped once several have all appended to it. mergehybrid closes this with one final
read-resort-reattribute-rewrite pass over the complete destination, after every batch and column
has merged - a deliberate, documented departure from the module's memory-flat design goal for that
one step, since a destination that's silently never truly parted is worse. mergebypart/
mergebycol called standalone, bypassing mergehybrid, do not get this guarantee automatically -
documented in merge.md; di.wdb (in progress) will always route through mergehybrid for this
reason.

6. A real-logger smoke test, not just the k4unit mock suite - Mock loggers only confirm a
message was logged at the right level; they don't process message content, so they can't catch a
malformed message - a list where a flat string was expected, for instance. That gap is exactly
what a real-logger pass found: checkenumerabletype built its error message with string applied
directly to a symbol list rather than ", " sv string ..., producing a malformed nested value
that crashed the moment a non-enumerable parted column was checked - the one case the function
exists to catch. Fixed to match checkpartitiontype's already-correct pattern, with a regression
test that asserts on the message's structure (10h=type), not just that it fired.

---

Checklist

• 190/190 k4unit test assertions passing
• Follows consistency.md and style.md
• Follows dependency injection guidelines
• merge.md documents all exported functions, config, the requireinit guard, all design
  decisions, cross-process partition-size sync, and a usage example
• No hard dependencies on other di.* modules - standalone

---

Documentation

See merge.md for full reference including the dependency contract, config keys, exported
function documentation, the requireinit guard, all six design decisions, cross-process
partition-size sync, and a usage example.

[DI-Software-Engineering]

In mergeonecol, the get segcol is performed on line 96 (destdata:get segcol: sv segment,col) before the protected .[upsert;...]on the following lines. A missing or unreadable column file will throw *outside* the protected apply, propagating an uncaught error out ofmergeonecoland then out ofmergebycol/mergehybrid. The merge.md intentionally documents this as a deliberate design decision for mergebycol, but the same unprotected read is used inside mergebycol's column loop, not just at the top level, so the specific column that fails is NOT named in the error that propagates – the caller cannot tell which column caused the failure. This is consistent with the documented intent, but the .[upsert;...]guard around the save side is misleading since the read side is unprotected; a save failure cannot happen without a prior successful read, so the protected upsert wrapper provides no additional safety over the unprotected read. Consider removing the protected upsert inmergeonecol` to avoid implying partial protection, or documenting explicitly that the save-side guard is only for I/O errors on the write (not the read).

[DI-Software-Engineering]

In mergehybrid, the condition if[(count overlimit)<>count partdirs enters the whole-partition merge path when the number of over-limit partitions is not equal to the total number of partitions, i.e. it runs whenever at least one partition is not over-limit. But it also runs when there are zero over-limit partitions (since 0 <> count partdirs as long as there is at least one partition). Inside the branch, partdirs:partdirs except overlimit is correct (strips the over-limit ones), but the condition should be if[0<count partdirs except overlimit (or equivalently if[not all partdirs in overlimit). As written, if all partitions are over-limit, count overlimit = count partdirs, the condition is false, and mergebypart is correctly skipped. That part is fine. The actual bug is the opposite edge: when overlimit is empty (all partitions are under-limit), partdirs except overlimit equals partdirs and mergebypart is called correctly too. So the logic accidentally produces correct results in both extremes, but the variable partdirs is reassigned inside the if-block, meaning the original partdirs is no longer available. The subsequent if[0<>count overlimit block then correctly uses overlimit, so the bug does not surface in normal use. However partdirs after the first if block now refers to the pruned list, and any code added after these two if blocks that references partdirs would see the pruned value rather than the full original. This is a latent correctness issue; the mergehybrid final re-sort step uses dest not partdirs so it is not affected today. Fix by capturing the pruned list in a separate name: underlimit:partdirs except overlimit and testing if[0<count underlimit;...].

[DI-Software-Engineering]

getfirstcharpartitions uses raze each value (...) on the result of a group call, but the two functional-select expressions inside – the distinct;first extrapartitiontype select and the {first each string x};(distinct;first extrapartitiontype) select – each operate on tablename (a symbol naming an in-scope table). When tablename is a global table symbol this works, but the function takes tablename as a parameter; if the caller passes a symbol that names a table local to their scope (or a file-path hsym), the functional ?[tablename;...] call will fail or return unexpected results because ? resolves the symbol in the q namespace, not in the caller's local scope. This matches the legacy TorQ behaviour but is worth flagging as a known limitation: the function only works with globally-scoped table names.

[DI-Software-Engineering]

DIReview Summary

1 critical | 2 warning(s) | 0 suggestion(s)

⚠️ Spec check skipped — tracker lookup failed (NO_REF_FOUND). Standards axis only.