Repository navigation
mux: withBatch logic - #5474
Open
coot wants to merge 16 commits into
Open
mux: withBatch logic#5474coot wants to merge 16 commits into
coot wants to merge 16 commits into
Conversation
…a shared token bucket, lanes, a slice and rotation A node-wide token bucket (Network.Mux.Egress.Bucket) shared by every bearer: the muxer waits for the socket to be writable, then for a grant of its batch, so a stalled peer holds at most one batch. Service order is (rank, arrival): equal ranks share equally, distinct ranks are strict priority, and within a tier the place is dealt from a seed and re-dealt every rotation period. The state is the instant the bucket is full again, a pure core the tests replay concurrent schedules through exactly. A scheduled bearer has three lanes, a muxer thread each. Direct carries the node's own requests, which never wait for tokens and are charged to the budget on credit once written; Slice is a reserved share with its own bucket that borrows the budget's idle capacity; Scheduled is what the node serves, ranked in the budget's bucket. The writability wait and the batch write time out after one SDU timeout, and makeSocketBearerWith sets TCP_NOTSENT_LOWAT. Diffusion builds the buckets from dcEgressScheduling, seeds the rotation from its RNG and hands the policy to the connection handler, which schedules every node-to-node mux and leaves node-to-client ones unscheduled. The default is 250 Mb/s with TxSubmission's initiator on a 15% slice and its responder's requests direct, since there the responder asks; a Slice lane without a slice bucket is scheduled. The diffusion simulations run with scheduled egress two times in three, shrinking towards egress off, and a dedicated group checks tx chain integrity and the invariants at trickle budgets. The mux-bucket-demo shows peers fetching one block at once, with and without the bucket. Squashes leios-prototype commits 09f1945, adc849c, 2c8eeac, 78e87de and ed62adf. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…roots first, benchmarks The buckets count what they grant in the transactions that grant it: bytes and batches per lane, the bytes the slice borrowed and the direct lane took on credit, the time grants waited for tokens and at the writability gate with a histogram of the waits, and the budget's busy periods, from a first bearer queued until none is, with their durations and widths. The node-to-node egress snapshot carries them with the token level and the queue lengths, the slice's batches and waits apart from the ranked queue's; a write timeout at the gate is counted by the lane that timed out. Diffusion ranks each scheduled mux as the connection handler creates it: tier 0 for a resolved local root, tier 1 for anyone else, so when the budget binds a local root at the head of the queue is served first. The rank is traced as TraceEgressRank. A peer's source port is its listening port, so an inbound local root matches its configured address. The socket read/write benchmarks gain the tier decision, bearers contending for one budget with and without rotation, and the 0 ms poll muxes twinned on a scheduled lane. Squashes leios-prototype commits ac838fb, 12db4db, d51c47f and 3960621. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… the governor, credit buckets by pool A bearer's rank is a rule the bucket asks in the transaction that queues it, so a reloaded topology or a changed standing takes effect the next time a bearer queues. The ranks: 0 a local root; 1 a partner, a duplex peer hot both ways that we have kept hot for esTenureThreshold, two deadline churn rounds at their longest plus the demotion timeout; 2 a big pool's relay with credit; 3 a stranger with credit; 4 the rest. The governor records when it promoted each active peer and exports it as hotUpstreamSince, the connection handler reads off the mux whether the peer runs any hot responder and knows the connection's data flow, and the rule joins them; it takes the policy and the rule as one. The scheduled lane reports every batch it writes, bytes per mini-protocol, to a ChargeSink installed beside the rule; direct and slice batches are never charged. Ouroboros.Network.Diffusion.PoolAllowances keeps one credit bucket per big-ledger pool, known by its relay list so that it keeps its bucket from poll to poll whatever its stake or position, in integer arithmetic over a monotone reading of fresh bytes: credit is κ·fresh less what was charged, κ 3/2, capped at 2·κ·freshMax, and a dry bucket charges nothing. Ouroboros.Network.Diffusion.PoolRelays resolves every big pool's relays every 1871 s on its own DNS semaphore and keeps an index from address, without its port, to the pool's bucket. A connection's rule and sink look its pool's buckets up at every use, or its own bucket when it belongs to no pool, and charge every scheduled byte except the esUnchargedProtocols, keep-alive by default. The budget counts what it grants by tier, and the pool relays thread is traced. The allowance arithmetic and the rank rule's decision table are checked against exact oracles, and the relay thread's index against a model of the ledger and DNS. Squashes leios-prototype commits 6664ca9, 2b4aa83, 737d207, 980b143, e6e2cd6, d70af1e, aad730d and c70c63d. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…rns at the rest tier Strict priority starves every tier above one that can fill the budget. A budget now carries a floor: a small bucket, esFloorPercent of the budget's rate, for the credited bearers the strict order does not reach, shared fairly in bytes across three classes, partners, pool relays and strangers, by start-time fair queuing in Network.Mux.Egress.Floor, and in turns within a class. The budget is charged on credit, so the floor redistributes it and never adds to it. Within a tier the order is a dealt place, right for the credited tiers and wrong for the rest, where a flood of newcomers left all but one with nothing: from roTurnsFrom up, the rest tier, a request queues behind the tier's current waiters and backlogged bearers rotate one batch at a time. A stranger's own bucket opens only after esStrangerLock, 887 s, so a peer shed for spending it cannot reconnect and spend it again. A pool new to the list starts with an empty bucket and a re-registration with other relays makes it a newcomer, so a pool cannot buy a fresh allowance by touching its registration; the resolver looks names up on two workers of its own, patches the index with each answer, and keeps what it last learned of a name across a transient failure. The tiers are written down once, as Tier in Network.Mux.Types, which the rule, the floor's classes and the metric names all read. The floor's laws run in IOSim, the lock and the rebuild rule against the allowance oracle, and the relay thread against its model. Squashes leios-prototype commits c531d26, 5dcffb3, 759ccac, 486b098 and a786648. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…bursts of credited demand A queued bearer asks its rule again every tierRefresh and once more as it is granted, moving to its new tier's place while keeping its order of arrival, so credit that refills while it waits reaches it without a new request; the scheduled lane's charge runs in the transaction that grants the batch, so connections sharing an allowance see each other's spending at the next grant. The budget remembers each bearer's last grant per tier, and the counters report the bearers each tier served within the window; the fast path asks the rule at every grant. A burst is credited demand: from the first grant of at least 4 KiB to a credited bearer after a gap to the last before the next, with grants at rest rank and small grants as filler, closed at its last grant and capped at 60 s. The counters loop traces one record per burst, with bytes and batches by tier, the bearers, the peak queue and the waits; the busy-period counters take the same discount. The bucket's state and the mux counters hold no thunks, in IOSim and in IO. The served record, the refresh and the burst detector are replayed against oracles. Squashes leios-prototype commits 136755d, 3eabe2c, 976bd3a, dab1d32 and df50884. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ocal roots, a protocol done at its last batch The partner tier is gated by the connection's own bucket: a partner is served at tier 1 while that bucket has credit and falls through to its pool's rank, a stranger's or rest otherwise, and the sink charges the bucket that gave the rank. A waiting bearer takes the place of each new rotation period instead of the one it queued with. Local roots take turns, one batch each, whatever the turns threshold says, so a producer's small reply never waits a whole period behind a sibling's backlog. A mini-protocol was reported complete as soon as its send buffer was empty, before the batch holding its last bytes was granted and written, so a peer demoted to cold could lose its last message. A send buffer now counts the batches a muxer holds some of its bytes in, on the transactions the muxer already runs, and a protocol completes once its buffer is empty and no batch holds it; the shutdown drain waits for the same. The schedule replay models the re-deal exactly, and a demoted peer receives its whole last message in every schedule. Squashes leios-prototype commits e086ce4, 30d1e8e, dc0a920 and 1d7c642. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… the lock in wall-clock time, the slowed counters The credit buckets refilled by a clock standing in for the chain at hard-coded maxima. Diffusion now takes both readings from consensus: a monotone count of the bytes the chain has freshly announced, daFreshBytes, and the most one announcement can carry under the current parameters, daFreshMaxBytes, read at every use so a parameter change takes effect at once. esFreshMaxBytes, esFreshBytesPerSecond and the clock are gone. A stranger's lock is wall-clock time from the connection rather than progress of the reading, so a quiet chain does not keep it shut for hours. The diffusion simulations run their fresh clock only with egress on. Each bearer has a standing at its tier, how long its batches waited for tokens against how long they were its own, decayed over the served window, and the counters report per tier the bearers whose batches took at least 2, 4, 8 and 16 times their own time, as egress.tier.<tier>.slowed.<n>x. Properties: no credit before the lock however fast the reading moves, a smaller allowance caps a charge at once, and the standings match a model in IOSim. Squashes leios-prototype commits a01d468, 26eb362, 46940ed and 33281da. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The muxer merged a per-batch map into a per-mux map after every write, 416 bytes a batch. A mini-protocol's held-batches count and its bytes written now live in one variable per mini-protocol and direction, kept with its ingress queue and updated in the release the muxer already does after a write, and the mux counters read those variables, as does the shutdown drain, so the muxers share no held count; the muxer takes no counter argument. The write-accounting property now covers batches carrying several mini-protocols. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
floorGrant removed the request from the queue before telling the burst detector of the grant, so the detector saw an empty queue: a floor grant after a wait longer than the burst gap split a burst queued throughout, and the first floor grant recorded a peak of zero queued. The grant is now noted first, as the budget's grant path does. The burst property's oracle now notes a grant before its leave, and its runs include stalls served by the floor while direct traffic holds the budget in debt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 4103fd2)
The scheduled lane handed its batch to the charge sink one mini-protocol at a time, and an allowance stops charging once its credit is spent, so the bytes after the protocol that spent it went uncharged and the peer regained its credited rank early. ChargeSink now takes the batch's bytes per mini-protocol, and mkEgressRule's sink charges the charged protocols' sum once; a bucket dry at the grant still accrues no debt. The rule's property generates batches spanning protocols, and the write property checks the sink is handed each batch once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit ba07565)
After a batch's first SDU, buildBatch dequeued the next demand in one transaction and took its SDU, requeueing a fragment's remainder, in another. A producer could fill the freed slot in between, and the muxer, the queue's only consumer, then blocked for good on the requeue. Both now happen in one transaction, as for the batch's first demand. A new property drains full queues with producers blocked on them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 1a249b8)
A short slice borrows the budget's bytes only while nobody waits on the budget. One that found the budget busy slept until its own bucket refilled, even when the budget went idle with the bytes meanwhile. A short attempt now records whether the budget was busy, and such a slice also wakes when the budget's queue empties; waking on that change only keeps it from spinning on an idle budget that is short too. The schedule replay now models the wake. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 618d092)
…k names The charge sink decided what to charge from pool membership and the rule's last answer, so a local root that belongs to a pool spent the pool's shared allowance, and a rank fixed with setEgressRank charged by a stale answer. ChargeSink and the awaitGrantCharged callback now take the granted rank, and mkEgressRule's sink charges by it alone: the own bucket at the partner and stranger tiers, the pool's buckets at the pool tier, nothing otherwise. Only the rule opens the own bucket. The rule's model now grants at given ranks and changes standing, and a new property changes standing or fixes the rank while a grant is queued. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit c2cc02d)
…s for scheduled egress The fragments the series carried described changes to an API main never had. One fragment per package now describes scheduled egress as it lands. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The `GrantAt` type makes the logic easier to understand.
coot
force-pushed
the
coot/egress-scheduler-review
branch
from
November 13, 2026 14:26
50ac156 to
88e4154
Compare
coot
force-pushed
the
coot/egress-scheduler-review
branch
from
November 13, 2026 14:33
88e4154 to
1603687
Compare
karknu
force-pushed
the
karknu/egress_scheduler.main
branch
2 times, most recently
from
November 14, 2026 06:04
e273932 to
5f20d4e
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Checklist
Quality
Maintenance
ouroboros-networkproject.