Feature tickerplant

[ascottDI]

di.tickerplant

Ships together with di.tplog. This module depends on the new di.tplog (injected-log init and
1-arg check); its test suite exercises the two against each other, so the two modules should be
reviewed and merged as a pair.

Summary

Adds di.tickerplant, the core tick-capture process for the modular TorQ world: it receives updates
from feeds, stamps them, writes them to a tickerplant log for recovery, and publishes them to
subscribers, rolling the log at end of day. It is the modular replacement for TorQ's
code/processes/tickerplant.q.

It orchestrates three hard dependencies — di.pubsub (subscribe/publish), di.eodtime (roll timing)
and di.tplog (log check/repair) — with an injected logger and timer.

Motivation

TorQ's tickerplant.q hand-rolls the whole capture loop — .z.* handling, the .u publish/subscribe
machinery, log writing and rolling, and end-of-day scheduling — in one process file. The modular
framework already provides those pieces as standalone modules (di.pubsub, di.eodtime, di.tplog,
di.timer), so this extraction re-expresses the tickerplant as an orchestration layer over them:
capture, stamp, log, publish, roll — delegating pub/sub, roll timing and log recovery to the modules
that own them.

Design

Root tables and the upd contract

The captured tables live at root, not in .z.m: a tickerplant owns its tables, feeds insert into
them, and di.pubsub reads them by name, so they cannot be module-local. This is the one deliberate
root-state exception; all other mutable state is module-local. di.torq wires the process's root
upd to tickerplant.upd so feeds publish to it.

Capture modes

• Batch (default) — upd inserts into the root table; a timer job publishes the accumulated rows
  every batchperiod and clears them.
• Zero-latency — upd publishes each update immediately and does not buffer.

Both modes stamp the update (prepending a data timestamp unless one is already present) and log every
message.

End of day

The roll fires when the current time passes di.eodtime's next roll timestamp, checked on every upd
and every timer tick. endofday flushes the buffer, notifies subscribers, rolls the log to the next
day, and refreshes the roll time and data-timestamp offset from di.eodtime.

Use of di.tplog

di.tplog is used only for check/repair on recovery: when openlog finds a pre-existing log it runs
it through tplog.check, repairing a corrupt one. The tickerplant opens for append and rolls the
log itself, rather than using di.tplog's open/roll, because those replay the log through upd —
which a tickerplant must not do to its own log. (This module was updated to the new di.tplog
contract: it now calls tplog.init during its own init, and tplog.check is 1-arg.)

No di.handlers dependency

Subscriber-disconnect cleanup is handled by di.pubsub's own .z.pc, so this module takes no handler
dependency. di.pubsub should migrate to di.handlers so .z.* is not assigned outside the central
registry — tracked separately, out of scope here.

Dependencies

Dependency	Kind	Description
di.pubsub	hard (use)	subscribe / publish / roll notifications
di.eodtime	hard (use)	trading date, roll scheduling, data-timestamp offset
di.tplog	hard (use)	log check/repair on recovery
log	injected	`info`warn`error dict of {[ctx;msg]} functions
timer	injected	di.timer's exports (must expose addjob)

Hard deps are declared in deps.q; the injected log and timer are validated by di.depcheck's
core-contract check, not declared there. init validates every dependency strictly and signals
immediately if any is missing or malformed.

Public API

Function	Signature	Description
init	[deps]	Wire log + timer, init the dep modules, materialise the schemas as root tables (`g# on sym), open today's log, and schedule the batch/roll timer job. Idempotent.
upd	[table;data]	Feed entry point: stamp, then buffer+log (batch) or publish+log (zero-latency).
subscribe	[tables;filters]	Register a subscriber (delegates to di.pubsub).
endofday	[]	Flush, notify subscribers, roll the log, advance the end-of-day state.
getcounts	[]	`i`j`d — messages published, messages logged, trading date.
gettables	[]	The tables this tickerplant captures.
version	—	Module version ("0.1.0").

upd validates its arguments and routes failures through a log-then-signal helper. getapimeta[]
exposes the callable API for central registration with di.api; the framework plumbing (init,
getapimeta, version) is intentionally excluded.

init also accepts optional config keys: batch (1b), batchperiod (timespan), logdir (string,
"" disables logging), logname (prefix, default "tp"), subtables (symbol list), and the
di.eodtime keys (rolltimezone / datatimezone / rolltimeoffset) forwarded verbatim.

Testing

test.csv / test.q (k4unit), 18 checks, run against the real di.pubsub, di.eodtime,
di.tplog and di.timer — no dependencies are mocked. The timer is used without init (so no live
.z.ts; its job is exercised through endofday), and a capturing logger is shared across the modules
so their output is assertable.

Coverage: the metadata/version contract; strict init dependency validation (a fail row per guard);
init materialising the root tables and scheduling the timer job; batch and zero-latency upd;
endofday flushing and rolling; upd input validation; and the two di.tplog integration points — a
tickerplant-written log replaying through di.tplog, and rolling into a corrupt log repairing it via
tplog.check.

k4unit:use`di.k4unit
k4unit.moduletest`di.tickerplant

Because two tests replay/repair through di.tplog, the suite passes once di.tplog and
di.tickerplant are present together (as they merge) — it is not runnable against an older di.tplog.

Files

di/tickerplant/init.q          entry point, version read, export list
di/tickerplant/tickerplant.q   implementation
di/tickerplant/tickerplant.md  module documentation
di/tickerplant/VERSION         module version (single source of truth)
di/tickerplant/deps.q          dependency manifest (di.pubsub, di.eodtime, di.tplog)
di/tickerplant/test.csv        k4unit tests (18)
di/tickerplant/test.q          test fixtures and helpers

The module version lives in a plain-text VERSION file, read in init.q (version:trim first read0\:::VERSION) and exported for di.depcheck, matching the convention used by di.servers, di.clienttrackinganddi.tplog`.

[DI-Software-Engineering]

DIReview Summary

1 critical | 2 warning(s) | 0 suggestion(s)

⚠️ Spec check skipped — tracker lookup failed (NO_REF_FOUND). Standards axis only.

[DI-Software-Engineering]

DIReview Summary

0 critical | 3 warning(s) | 0 suggestion(s)

⚠️ Spec check skipped — tracker lookup failed (NO_REF_FOUND). Standards axis only.

[DI-Software-Engineering]

pubsub.subscribe returns a bare error symbol when no tables match, but the partial-match branch checks type first r — when r is a symbol atom, first of a symbol atom is the atom itself (not its first element), so -11h=type first r is 1b for BOTH the all-fail case and a partial-match case where the first element of the returned pair happens to be a symbol. The all-fail guard (-11h=type r) is evaluated first, so the all-fail path is handled correctly, but if pubsub.subscribe ever returns a pair whose first element is a symbol atom (e.g. an error symbol paired with partial results), the partial-match branch would silently log and continue rather than raise. More concretely: after the all-fail guard the only shapes remaining are (tables;schemas) and (errmsg;(tables;schemas)). In the partial case first r is an error symbol atom, so -11h=type first r is correct — but the code relies on first of a 2-element list returning the first element, which is fine. The real bug is that pairs:flip $[partial;last r;r] uses last r when partial, but pubsub.subscribe on a partial match returns (errmsg;(tables;schemas)) — so last r is (tables;schemas), and flip of that is (tables list; schemas list), which is correct. However when all tables match, r is (tables;schemas) directly, and flip r yields the same shape. This path is actually consistent. The real issue: in the all-none case the guard correctly signals, but for ZERO matched tables where pubsub.subscribe instead returned a (tables;schemas) pair with empty lists, pairs[;0] would be an empty list and nms!0^.z.m.rowcounts nms would be an empty dict — silently returning a dict that looks like success with no tables. Verify that pubsub.subscribe always returns a bare symbol (not an empty pair) when nothing matches.

[DI-Software-Engineering]

logfilelist checks if[null .z.m.logpath;:()] but openlog sets .z.m.logpath:`` (the null symbol) when logging is disabled (0=count .z.m.logdir). The nullcheck on a symbol atom is correct per q semantics. However on the very first call tosubdetailsbefore anyinit(i.e. beforeopenloghas run),.z.m.logpathdoes not exist yet and the protected-apply ininitialised[]guards against that for the public entry points — butlogfilelistis an internal helper called fromsubdetailswhich itself callsrequireinit. So the ordering is safe only because requireinitruns first. This is fine as written, but iflogfilelistis ever called from a path that skipsrequireinit, it will throw '.m.di.0tickerplant.logpath`. No fix needed unless the call graph changes, but worth noting the implicit coupling.

[DI-Software-Engineering]

resetmodule calls tp[\teardown][]and then directly manipulates.m.di.0tickerplantstate with@[{if[.m.di.0tickerplant.logfile>0i;hclose .m.di.0tickerplant.logfile]};::;{[e] :(::)}]followed by![`.m.di.0tickerplant;();0b;`schemas`scheduled]. But teardownalready sets.z.m.scheduled:0b(which is.m.di.0tickerplant.scheduled), so the manual delete of scheduledis redundant. More critically,teardowndoes NOT close the log handle — it deliberately leaves module state intact per the design. The explicithcloseimmediately afterteardownis therefore the only place the log is closed during reset. Ifteardownthrows (e.g. because it internally errors before thescheduled:0bwrite), thehcloseis still attempted via the protected apply, which is correct. However, deletingschemasfrom.m.di.0tickerplantafter ateardownthat has already returned meansinitialised[]will correctly probe false — this is the intended mechanism. The sequence is fragile: iftickerplant.qever adds a new name thatinitialised[]probes,resetmodule silently stops working. The test comment acknowledges this (initialised[] probes), but the fix — exporting a test-only reset or making initialised[]probe a dedicated flag — is documented as a deliberate decision. Not blocking, but the brittleness is real: adding a second probe variable toinitialised[]intickerplant.q` would silently break the test suite.

[DI-Software-Engineering]

publishbuffer computes .z.m.rowcounts:.z.m.rowcounts+.z.m.tabs!count each . .z.m.tabsbefore callingpubsub.pubclear, which publishes and then empties each table. If pubsub.pubclearthrows (e.g. a subscriber handle is broken and the publish errors out internally), the row counts have already been incremented but the rows may not have been cleared from the buffer or fully published. On the next tick,pubclearwould attempt to publish the same rows again androwcountswould be incremented a second time, producing double-countedrowcountsand potentially duplicate delivery. The safer order is to incrementrowcountsafterpubclearsucceeds. Consider wrapping the whole sequence or at minimum moving therowcountsupdate afterpubclear`.

[DI-Software-Engineering]

The readport helper uses a while loop with system"sleep 0.05" — a busy-wait that the project style guide explicitly prohibits (do not use the do, while, and for functions). While this is test/integration code rather than production code, it violates the documented project convention. Replace with a recursive converge or a timed \ iteration, or at minimum acknowledge the exception in a comment.

[DI-Software-Engineering]

DIReview Summary

0 critical | 5 warning(s) | 0 suggestion(s)

⚠️ Spec check skipped — tracker lookup failed (NO_REF_FOUND). Standards axis only.