Skip to content

docs: scalability guide with the 10k figures and the connection model - #691

Merged
aojea merged 1 commit into
google:mainfrom
aojea:scalability-guide-10k
Oct 13, 2026
Merged

aojea merged 1 commit into
google:mainfrom
aojea:scalability-guide-10k

Conversation

@aojea

@aojea aojea commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator

The guide's numbers came from the 3200-member run, and its model counted a router's connections as its members' sessions. The 10,000-member run (#277) measured otherwise: each router held 3200 to 5100 sessions and about 9000 connections, because the members attached to the other routers keep a DHT connection to this one most of the time. The kad client refreshes its table every two minutes against every router it knows (its table of six routers is below kad-dht's refresh threshold), and the router closes the connection at the auth deadline in between.

What changes in the guide:

  • The connection model gets that term: connections ≈ F × K/N + 0.8 × F × (N − K)/N, about 0.9 × F at six routers. Adding a router divides the sessions and leaves the connections, so what bounds the fleet on one router is libp2p's connection cap of twice the high watermark, and the --high-watermark rule gains a second bound, 0.55 × F. The testnet example is recomputed: 10,000 on six routers at 6000 held about 9000 connections per router of a cap of 12,000, about the same with one router out.
  • Memory per connection: 190 KiB and six goroutines, 1.7 GiB at 9000 connections. What stays after a drain (720 MiB with 16 connections hours later) and the two upstream fixes it waits on: fix: release the writer context when a peer's writer exits libp2p/go-libp2p-pubsub#735 and fix(pstoremem): expire addrs downgraded from a connected TTL libp2p/go-libp2p#3559.
  • Join times by mesh size: ready p50/p95 0.4/0.7 s on an empty mesh, 3.5/5.5 s onto 9000 resident, control plane p99 from 25 ms to 2.5 s.
  • Service figures: 1000 callers per step against two providers, every first call completed, 1.0 to 1.4 attempts per member; the failed attempts were the providers' 400-connection trim.
  • What a middlebox sees: about 3700 new connections a minute per router at 10k.

The router reference no longer says that the connection and member counts stay close, and points to the guide for the bound.

The guide's numbers came from the 3200-member run and its model counted
a router's connections as its members' sessions. The 10,000-member run
measured otherwise: each router held 3200 to 5100 sessions and about
9000 connections, because the members attached to the other routers
keep a DHT connection to this one most of the time (the kad client
refreshes its table every two minutes against every router, and the
router closes the connection at the auth deadline in between).

State the model with that term: connections ≈ F × K/N + 0.8 × F ×
(N − K)/N, about 0.9 × F at six routers, bounded by the connection cap
of twice the high watermark rather than by the watermark, so the
high-watermark rule gains a second bound (0.55 × F). Update memory per
connection (190 KiB, six goroutines; 1.7 GiB at 9000), the retention
after a drain with the two upstream fixes it waits on, the join times
by mesh size, the service figures (1000 callers per step, every first
call completed, 1.0 to 1.4 attempts), and the churn a middlebox sees
(3700 new connections a minute per router at 10k). The router
reference no longer says the two counts stay close.
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@aojea
aojea merged commit 31ac5ce into google:main Oct 13, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant