Skip to content

feat: serve traffic reports from the sync operator app - #111

Open
salindne wants to merge 14 commits into
mainfrom
feat/sync-operator-traffic-report-v2
Open

salindne wants to merge 14 commits into
mainfrom
feat/sync-operator-traffic-report-v2

Conversation

@salindne

@salindne salindne commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Refs ChainSafe/canton-extending-mainnet#152

Replaces #103, which carried #101's history from before the squash.

Summary: GET /v0/traffic-report returns the traffic charged to participants per 10-minute interval, summed across participants as #87 asks, 100 intervals per page. For each interval it reads every member's consumed traffic from the sequencer's GetLsuTrafficControlState at the start and end. If the sequencer can't serve an interval (not over yet, pruned, or before the upgrade to it), the call fails with a 400 that says which range to ask for instead. It needs a JWT, so auth is a new required config field, and the image and Helm chart pass it through.

A successor sequencer takes over the counters at an upgrade but not their history, so readings up to the announced upgrade time come from the current sequencer and later ones from the successor.

  • GetLsuTrafficControlState with a timestamp is documented for disaster recovery; is a page of up to 101 calls per request OK? One call took 6 to 30 ms locally.

Signed-off-by: Sebastian Lindner <33971232+salindne@users.noreply.github.com>
@salindne
salindne marked this pull request as ready for review November 16, 2026 10:19
Nothing prunes the dedicated synchronizer's sequencer, so its journal
already has every interval. Read both ends of each requested interval
from it instead of storing totals, which drops the store, the
migration and the trigger.

Signed-off-by: Sebastian Lindner <33971232+salindne@users.noreply.github.com>
A page is 100 intervals, so a request makes at most 101 sequencer
reads. The response says where the next page starts until it reaches
the requested end.

Signed-off-by: Sebastian Lindner <33971232+salindne@users.noreply.github.com>
The interval length is now a config field, 10 minutes by default, so
the LSU test can shorten it and see the interval that spans the upgrade
end. It checks that interval is read from both sequencers.

Signed-off-by: Sebastian Lindner <33971232+salindne@users.noreply.github.com>
Paymasters rely on 10-minute intervals, so only tests should change it.

Signed-off-by: Sebastian Lindner <33971232+salindne@users.noreply.github.com>
The reads already stop at the first refusal, so checking the app clock
first only saved one call per request.

Signed-off-by: Sebastian Lindner <33971232+salindne@users.noreply.github.com>
…affic-report-v2

Signed-off-by: Sebastian Lindner <33971232+salindne@users.noreply.github.com>
`auth` is now required, so the image reads the audience and JWKS URL
from env vars that the chart sets from `auth.audience` and `auth.jwksUrl`.

Signed-off-by: Sebastian Lindner <33971232+salindne@users.noreply.github.com>
@salindne
salindne requested a review from isegall-da November 17, 2026 18:06
synchronize = Some(NonNegativeDuration.ofSeconds(60)),
)

clue("the interval that is not over is Undetermined") {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I like this UX. Can't we just reject a request that includes an interval that's not over? We don't need real-time data here, just forcing the user query a few minutes back seems fine.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done, the call now fails with a 400 that says which range to ask for instead, and the per-interval status is gone

Instead of a status per interval, the request fails with a 400 that says which range to ask for instead.

Signed-off-by: Sebastian Lindner <33971232+salindne@users.noreply.github.com>
The route setup moves into `SyncOperatorApp.routes`, so a unit test covers the app's own wiring: the auth the integration test checked, and the 400 for a range the sequencer can't serve.

Signed-off-by: Sebastian Lindner <33971232+salindne@users.noreply.github.com>
The check used `Map.empty` in place of the successor's reading before the upgrade; it now reads the successor there.

Signed-off-by: Sebastian Lindner <33971232+salindne@users.noreply.github.com>
Without a server there is no request timeout, so the directive warned and failed checkErrors. The app now wraps `SyncOperatorApp.routes` in the error handler, and the test uses only its exceptions directive.

Signed-off-by: Sebastian Lindner <33971232+salindne@users.noreply.github.com>
@salindne
salindne requested a review from isegall-da November 20, 2026 10:16
/** The first interval that starts after the sequencer's pruning lower bound. Pruning keeps only
* each member's latest journal row up to that bound, so readings at or before it are wrong.
* Canton prunes the journal before it moves the bound, so a request at the edge of the retention
* period can still race a prune and report a wrong total.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we query firstReadable again after we create the report, and confirm that at that time it hasn't been invalidated yet? That way, we risk only a false-negative if you're querying close to the pruning window, which seems better than risking reporting wrong numbers.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good call, changed it.

it reads the pruning bound again after the page and rejects if it moved past the start.

}

val acrossStart = trafficReport.alignDown(upgradeTime)
val acrossEnd = acrossStart.plus(trafficReport.intervalLength)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't there a small flakiness risk here if the upgradeTime falls exactly on the end of the interval? Might be safer to just always add 2xintervalLength?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, but then alignDown is exactly upgradeTime.. So I think it's fine.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed!


def notOver(start: CantonTimestamp, end: CantonTimestamp): String =
s"The interval [$start, $end) is not over yet, or the sequencer has not sequenced past its " +
s"end. Request a range that ends at or before $start."

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
s"end. Request a range that ends at or before $start."
s"end. Request a range that ends at or before $start (or wait for long enough after $end and retry)"

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

applied in 166cfcb, thanks!

Canton prunes the traffic journal before it moves the pruning bound, so a prune during a read could change the readings without the first check noticing. The reader now checks the bound again afterwards and rejects the range if it moved past the start.

Signed-off-by: Sebastian Lindner <33971232+salindne@users.noreply.github.com>
Besides asking for an earlier range, the caller can wait until the sequencer is past the interval and retry.

Signed-off-by: Sebastian Lindner <33971232+salindne@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants