Repository navigation
Execution pool recruits every core regardless of work size: 2.7x the CPU for 1.2x the speed #599
Description
Summary
The execution pool splits a query across every available worker regardless of
how much work there is. Past a handful of workers the coordination cost grows
faster than the saving, so total CPU rises steeply while wall time barely
improves. A service that is not latency-bound pays several times the CPU it
needs and has no way to ask for the cheaper trade.
Reproducer
200 inner-joins of a 3-row key set against a 130,000-row table, the shape a
market-data service runs continuously:
(set N 130000)
(set syms (map (fn [i] (parse (format "INST%" i))) (til 773)))
(set t (table [venue instrument session message_id px]
(list (take ['BFU] N) (take syms N) (take ['S1] N) (til N) (til N))))
(set ks (select {from: t by: [venue instrument] n: (count px)}))
(set k3 (select {from: ks take: 3}))
(set n 0)
(times 200 (set n (+ n (count (inner-join [venue instrument] t k3)))))/usr/bin/time on a 16-core machine, 2.6.2.r229.ge7770412:
| cores | user CPU | wall | speedup | CPU vs -c 1 |
|---|---|---|---|---|
-c 1 |
0.71 s | 0.71 s | 1.0x | 1.00x |
-c 2 |
0.71 s | 0.36 s | 2.0x | 1.00x |
-c 4 |
0.80 s | 0.22 s | 3.2x | 1.13x |
-c 16 |
1.93 s | 0.18 s | 3.9x | 2.72x |
-c 2 is free: half the wall time for the same CPU. -c 4 costs 13% for a
3.2x speedup. -c 16 costs 172% more CPU to go 1.2x faster than -c 4.
Small queries are unaffected — 20,000 joins over a 10-row table cost 0.05 s at
every core count, so nothing is dispatched for those. The effect starts where
the pool starts splitting.
Impact in production
A depth-of-book service of ours, default pool on a 16-core host, against an
identical instance on the same live feed with -c 2, in lock-step on every
state counter throughout:
default (16c) 50% 13% 23% 44% 57% 64%
-c 2 25% 10% 15% 23% 27% 31%
Half the CPU, identical throughput, identical progress. A second run comparing
-c 1 with -c 4 was monotonic in the same direction. The service's own batch
takes 2.5-5.7 ms against a ~26 ms inter-batch gap, so the latency the pool buys
is worth nothing to it — and .sys.querylog reports effective parallelism of
~0.0002 for a typical batch, while perf put 73% of samples in worker_loop.
Notably our other Rayforce services on the same host and the same feed sit at
1-2% CPU: they do not run joins large enough to dispatch, so they never meet
this. It is specific to whoever does the table-sized work.
What we are asking for
- Scale the worker count to the work, not to the machine — a morsel-count
or row-count heuristic, so a 130k-row join does not recruit 16 threads for a
1.2x gain over 4. The measurements above suggest the knee is low. - Failing that, a throughput mode — a way to say "minimise CPU, I am not
latency-bound".-cis that knob today, but it is a blunt global: it also
caps the genuinely parallel work a process may want elsewhere.
We have set -c 2 and consider the immediate problem solved for us; this is
filed because the default costs every non-latency-bound service the same
multiple, silently.
Measured on rayforce 2.6.2.r229.ge7770412, 16-core host.
Activity
singaraiona commentedon Sep 21, 2026
Reproduced, and the cause is not what either of us assumed. Thank you for the
CPU-vs-wall table — it is the measurement that made this findable.
Reproduced
Your reproducer on an i7-6700 (4 physical cores, 8 logical), release build:
-c 1-c 1-c 2-c 4-c 8Same shape as yours: past a point the CPU climbs steeply and the wall stops
improving. Note where that point is on this machine —
-c 8is where the poolstarts using hyperthread siblings rather than more cores.
It is SMT oversubscription, not coordination overhead
The pool defaults to every online logical CPU.
src/core/pool.ccallsray_thread_count()(sysconf(_SC_NPROCESSORS_ONLN)), commented "Default toevery online logical CPU". Meanwhile the tree already contains
ray_physical_core_count(), whose own comment says why that is wrong — two SMTsiblings share one core's load/store machinery, so the second thread adds
contention rather than throughput. It was being used only to estimate a cache
size, never to size the pool.
Changing that one call, nothing else:
Half the CPU, and the wall time does not regress. That is the bulk of what you
measured, and on your 16-logical host it is very likely the same story.
What I can tell you about the trade
I have not shipped that default change, because it is not free everywhere. On
the full ClickBench suite (10M rows, 43 queries × 3) on this box:
regressing >10%, worst +36%
So it halves CPU and improves end-to-end time while costing a few percent on the
hot per-query numbers that get published. This box is also the least favourable
case for it — 4 physical / 8 logical means giving up half the threads — and the
in-tree note records ClickBench running 11% faster on physical cores on a
5950X (16 physical / 32 logical). Your host is closer to that. It needs a
decision against a multi-machine benchmark run, so it is filed separately rather
than slipped into a patch release.
What this means for you
Your
-c 2is sound, and-c <physical core count>is probably better —here
-c 4gave 3.6x the throughput of-c 1for 9% more CPU, while-c 8bought nothing for another 1.9x. On a 16-logical host, try
-c 8(or whateverlscpureports asCore(s) per socket × Socket(s)): you may get most of yourlatency back at close to the CPU you have now.
Two corrections to the framing in the issue, for the record:
Coordination overhead is not the dominant cost. At 50k rows and
-c 4,the parallel split costs 24% more CPU for 2.6x the wall — a good trade. I
tried the sizing heuristic you asked for (request 1), gating the join probe on
the engine's own
RAY_PARALLEL_THRESHOLD, and it made things worse: 2 tasksinstead of 16 regressed both CPU and wall at
-c 4(2.59 s / 0.67 s →3.11 s / 1.58 s). The cost curve is U-shaped, because the main thread's
completion spin burns a full core waiting on a straggler, so under-splitting is
expensive too. I have dropped that approach.
The join probe does still lack a work-size gate — it dispatches from 8193
rows up, 8x under the threshold 16 other dispatch sites respect. Real, but
closing it costs ~2.6x wall on sub-threshold joins for ~44% CPU. That is your
preferred trade, not a safe default, which is what request 2 was really
asking for.
On request 2, a throughput mode: I would rather not add a knob for it. If the
pool sizes itself correctly there is much less left for the knob to buy, and
-calready covers the blunt case. Happy to be argued out of that if physical-core
sizing still leaves you paying for latency you do not want.
One small thing landed out of this: #605 makes a dispatch wake only the workers
it can keep busy, instead of the whole pool. It does not help your shape
(your probe makes 16 tasks against 8 workers, so the clamp does not bind —
measured 5.19 s → 5.18 s), but it removes real waste for dispatches narrower
than the machine.
Leaving this open and tracking the default in the new issue.
vbmithr commentedon Sep 21, 2026
Your diagnosis holds on a 16-logical host, and I have a measurement that removes
a confound both of our tables contain. Short version: ship the physical-core
default. Also, the framing in my issue body was wrong and I withdraw it.
The confound: user CPU is not a safe metric here
My original table and yours both report user CPU. That is thread-seconds, and on
any part that drops clocks as more cores go active, identical work inflates.
I re-ran your sweep on an 8-physical / 16-logical AMD Ryzen AI 7 350 and user CPU
rose smoothly with no knee at 8 — which appears to contradict the SMT
explanation:
Measuring instructions and cycles instead — both frequency-independent — tells a
different and much sharper story:
-c 1-c 2-c 4-c 8-c 12-c 16Cycle counts are means of 3; the
-c 8and-c 16figures varied by under 4%.Two things fall out of that
1. Instructions are flat: +0.7% from 1 to 16 cores, while cycles double.
The extra CPU is not extra work. It is stall cycles — IPC collapses from 6.5 to
3.2. So the framing in my issue body is wrong and I withdraw it: I wrote
"the coordination cost grows faster than the saving", which implies work being
added. It is not. This is direct evidence for your explanation instead — the
main thread's completion spin burning a core waiting on a straggler — and it is
consistent with your finding that under-splitting made things worse.
2. The split lands almost entirely on the SMT step, as you argued.
1.70x the cycles for exactly zero wall improvement. Your hypothesis, on the
topology you said you lacked, with the throttling confound removed. The residual
sub-SMT overhead is only 1.21x across a 4.5x wall improvement — that is a good
trade and I do not think it is worth chasing.
Worth noting this box is heterogeneous (Zen5 + Zen5c, different clocks), so
straggler variance should be systematically worse here than on either of your
homogeneous test machines. It still shows the effect concentrated at the SMT
boundary rather than smeared across core counts, which I think strengthens the
case rather than weakening it.
On the ClickBench trade
Your 4-physical/8-logical box gives up half its threads and is the least
favourable case; the in-tree note has a 5950X running 11% faster on physical
cores. Ours sits with the 5950X: 41% of cycles recovered for no wall regression
at all. For whatever it is worth as a third data point, I would take a 2.7%
regression on published hot per-query numbers over a 1.7x cycle cost that buys
nothing, and I would ship #606.
Your
-c 8suggestion: right in general, wrong for usIt is well supported by the table above —
-c 8gets all the available wallbenefit (0.16 s, identical to
-c 16) at 59% of the cycles. Anyonelatency-sensitive on an SMT host should do this today.
We will stay at
-c 2, because the service is not latency-bound: its batch takes2.5-5.7 ms against a ~26 ms inter-batch gap, so 0.38 s versus 0.16 s buys nothing
we can use, and
-c 2costs 3.33 G cycles against 4.04 G.On request 2, where I would push back gently
I accept that a "throughput mode" is the wrong shape, and I am not asking for a
mode. But I do not think
-ccovers the blunt case, for a reason your owncorrection 2 identifies.
-cconflates two separate things: how much parallelism a process may use,and how eagerly work is split. We want the pool to stay available for a rare
whole-table operation (an invalidation over every book) while never splitting the
steady small joins.
-ccannot express that — which is exactly why our unit filecarries
-c 2rather than-c 1, with a comment explaining we are keeping onespare worker for an event that happens a few times a day.
You measured the gate already: the join probe dispatches from 8,193 rows, 8x
under the threshold 16 other sites respect, and closing it costs ~2.6x wall for
~44% CPU — "your preferred trade, not a safe default". I agree it is not a safe
default. That is an argument for making the dispatch floor settable, not for
dropping it: one tunable that decouples split-eagerness from pool size, left at
today's value by default. That is a smaller surface than a mode and it is the
thing
-cgenuinely cannot do.One caveat on my own issue body
The production comparison I cited (50% -> 25% across state counters) was measured
on an older revision of that service. It has since been rewritten to avoid the
joins that were doing the dispatching, and on #596 I have just measured that the
join in question is no longer executed in its steady state at all. So please do
not weight that particular table heavily — the reproducer above and the cycle
counts are the part I stand behind.
v2.10.0 (#650)
v2.11.0 (#711)