add SWIP-39: balanced neighbourhood registry aka smart neighbourhood management

[zelig]

UPDATED AUGUST 2026

First: the PR was heavily restructured on 2026-07-27 (two-transaction join, no
reservation held, identity-keyed registrations, concurrent departures, ICBT as a
separate container contract) — several of the points below are answered in the text
now, so please re-read the current version before diving into the old one.
Point-by-point:

1. Why do we need on-chain topology?

Because assignment must be litigable. The redistribution game checks, at claim
time, that a node plays in the neighbourhood it was assigned — that check has to run
in the contract, against state the contract trusts. Off-chain assignment can be
neither enforced (nothing stops a node self-selecting) nor verified (the contract
has no source of truth to check an overlay against). The same goes for the two
enforcement events: forfeiting an expired registration and slashing a defaulted
donor — both need the assignment state on-chain to be adjudicable. And grind-proof
randomness (stake locked before entropy known, seed from a committed block height)
only means anything if the commit itself is on-chain. What stays off-chain is
everything that can: target-prefix computation is a read-only call, mining is local,
and the join costs exactly two transactions.

2. Sorted ring + sparse buckets — formalize and compare.

Happy to add a comparison subsection to the SWIP; here is the summary. The layout
(doubly-linked list in overlay order + mapping(prefix at staking depth => [first node, count])) is good at what rings are good at: O(1)-write
insertion/removal once the position is known, O(1) neighbourhood membership query,
cheap ordered iteration. It is weak exactly where this SWIP lives:

• Uniform random selection. The core operation is "pick uniformly among the
  $2^{d+1}-N$ free neighbourhoods" (and among $N-2^d$ donor pairs). A ring+buckets
  layout either scans buckets — O(2^d) — or maintains hierarchical per-subtree
  counts to support O(log N) rank selection. The moment you add the counts
  hierarchy (your "buckets hierarchy?" bullet concedes this), you have rebuilt the
  ICBT: the trie is the counting structure, with the ring's information implicit
  in it.
• Depth transitions. Buckets keyed by "prefix at staking depth" must be re-keyed
  when $d$ changes — at $N=2^{d+1}$ every key changes, an O(N) migration or a
  lazy-migration scheme with its own bookkeeping. The ICBT never re-keys: $d$ is
  derived, indexes are stable, a depth transition is zero writes.
• Write cost is a wash. ICBT: O(log N) counter updates per activation/departure
  (path to root, ~30 slots at a million nodes). Ring: O(1) link writes + O(log N)
  anyway for whatever counting structure supports selection. We are comparing
  log-vs-log; the constant matters less than the re-keying cliff above.

Where the ring genuinely wins: iterating a neighbourhood's members in overlay order
(we never need this — one node per leaf by construction) and finding the successor
of an arbitrary address (the ICBT does it in O(log N) via nodeFor, good enough for
a view function). So: formal comparison in the SWIP yes, layout change no.

3. High-level operations.

Now in the SWIP: join = register + activate (§Join protocol), departure =
deregister (+ donor redraw path) (§Departure and rebalancing), neighbourhood
queries = getPrefix / nodeFor (read-only). Redistribution eligibility is one
prefix check against the assignment record — the staking contract calls
getPrefix(identity) and compares against the claimed neighbourhood. If a specific
operation list is wanted verbatim in the issue's terms, point me at it.

4. Random/balanced assignment without A and R.

Reading "A and R" as the ordinal mapping and the reservation from the old draft:
the restructured version already dropped both. There is no reservation
(target prefix is a read-only computation, nothing locked, activation revalidates
against current state) and no request IDs / ordinal indirection (registrations are
keyed by staking identity; the seed is H(domain ‖ identity ‖ blockhash)). If A and R
meant something else, tell me what and I'll answer that instead.

5. Comparison with compacted binary trie.

A compacted (path-compressed) trie saves storage when keys are sparse and clustered
— but our key population is dense by construction: the invariant forces every
prefix at depth $d$ or $d+1$ to be occupied, i.e. the trie is always complete to
within one level. There is nothing to compact — path compression on a complete tree
adds skip-pointers that must be maintained on every split/collapse and saves zero
levels. The implicit heap layout additionally removes all pointer storage: parent,
children, sibling are arithmetic on the index, so a "node" is just its counter
slots. Compaction pays off for arbitrary key sets; balanced assignment is precisely
the regime where it cannot. Will add this as a paragraph to the comparison
subsection.

6. Depth transitions.

Walked through in the SWIP (§Balance invariant: preservation under
insertion/removal; §Counting: the $N=2^D$ boundary cases for splitCount/donorCount).
The short version: transitions are emergent, not an event — no stored $d$, no
migration, the counters at the root already equal $2^{d+1}-N$ and $N-2^d$ and both
hit the boundary values exactly at powers of two. If a worked $N=2^D$ example would
help, I can add one next to the existing worked examples.

7. Complexity / gas.

§Gas and performance analysis: selection O(d) reads, activation/departure O(d)
writes, storage O(N), registration/expiry O(1) amortized (monotonic queue head,
bounded per call — no unbounded iteration anywhere). Mining is the real cost and it
is off-chain: expected $2^\ell$ hashes, $\ell \approx \log_2 N + 1$. Benchmarks are
listed as a reference-implementation deliverable; concrete numbers per depth once
there is a contract to measure.

8. The earlier notes (shrink, queue, gas of updates).

• Shrink/withdrawals: the departure path is now first-class — direct collapse
  completes immediately; the donor path holds the departing node active until a
  donor lands, with forfeiture + redraw on default. Withdrawal of stake itself is
  the staking contract's business (strict separation in the SWIP).
• Queue: yes — two commit queues ($C_R$, $C_D$), append-only with monotonic
  block heights, head-advancing expire bounded per call.
• Gas of data-structure updates: O(log N) slot writes per structural change; see
  point 2 for why the proposed alternative doesn't beat it once selection is
  accounted for.

this comment was added here #74 (comment)

[Copilot]

Pull Request Overview

This PR introduces SWIP-39, a smart neighbourhood management system for decentralized service networks. The proposal aims to solve the "one operator, one node in a neighbourhood" problem through a balanced assignment mechanism that ensures fair load distribution and prevents sybil attacks.

Key changes include:

• A comprehensive specification for balanced neighbourhood registry with random assignment
• Smart contract implementation for managing node registration and neighbourhood assignments
• Mathematical formulations for neighbourhood depth calculation and overlay address validation

[0xCardiE]

Thanks for the well-thought-out SWIP — the design is elegant and clearly addresses Sybil resistance and balanced assignment. I had a few questions and points I’d like to discuss for further clarity and robustness:

• Overlay mining and gaming – how is the off-chain mining step protected against precomputation attacks or delay tactics to gain strategic placement?
• Expired stake handling – have you considered slashing (partial or full) instead of just locking the stake on expiry?

[0xCardiE]

Could _upgradeDepth() become too expensive to execute as the number of assigned nodes grows?

The _upgradeDepth() function doubles the assignment and remaining lists, copies all existing nodes to their new positions using bitwise logic, and clears/rebuilds state — all in a single call. If the number of nodes reaches high volumes (e.g. 1,000+ or 10,000+), this could approach or exceed the block gas limit, making the function fail or stall the system.

Proposed solutions:
• Consider breaking _upgradeDepth() into incremental steps, e.g., by:
• Using checkpointing to process part of the upgrade each time it is triggered.
• Spreading the reassignment of existing entries across multiple assign() calls.
• Alternatively, lazy-initialize entries in the assign() function, only when a neighbourhood is actually being filled.

[0xCardiE]

Can the committers array become inefficient with a large number of registrants (e.g. 20k nodes)?

The committers[] array is iterated over in _expire(), _findEntryFor(), and _removeCommitter() using for loops. If a large number of nodes register, or if expired entries are not promptly cleared, the gas cost of these operations can grow linearly and become prohibitively expensive.

Proposed solutions:
• Maintain a mapping from address to index to allow constant-time lookup and removal.
• Use a circular buffer or a “head” pointer to avoid shifting array entries when removing expired ones.
• Alternatively, mark entries as expired/inactive with a boolean flag instead of removing them from the array immediately.

[zelig]

Can the committers array become inefficient with a large number of registrants (e.g. 20k nodes)?

I did not consider it realistic, since each registrant entry expires in max 256 blocks, that is in a matter of <4 game rounds and they lose their deposit if they refuse to pay, so likely all the potential players may organically wait out.

The committers[] array is iterated over in _expire(), _findEntryFor(), and _removeCommitter() using for loops. If a large number of nodes register, or if expired entries are not promptly cleared, the gas cost of these operations can grow linearly and become prohibitively expensive.

Proposed solutions: • Maintain a mapping from address to index to allow constant-time lookup and removal. • Use a circular buffer or a “head” pointer to avoid shifting array entries when removing expired onesr . • Alternatively, mark entries as expired/inactive with a boolean flag instead of removing them from the array immediately.

but they need to be removed at some point.... and I am not sure how a mapping that needs to be reindexed after every entry removed, will solve this.

[zelig]

Can the committers array become inefficient with a large number of registrants (e.g. 20k nodes)?

well, maybe. To be honest, there is also another way. We do not need to allow, just any length of the committer list. The length represents the queue, and the length of valid entries are the ones in the queue you can skip. This effectively quantifies the tries that you got (effectively mining) but also the realistic probability that that someone will come in and change the neighbourhood you (thought you were) assigned to. If this probability is high (there is a lot of nodes that can submit mined overlays), then it can easily happen, that whenever an assigned neighbourhood is read off, nodes will frontrun. So it would just make sense to limit this skip queue to a fix constant number. But this means that the committers list should effectively have a limited length. Now if we siply reject registrations beyond this limit, then the shorter this length, the harder it is for the same amount of currently aspiring nodes to commit. Now in order to avoid that the registration tx needs to be continuously retried (due to it most likely be frontrun by competing resistrants), we should introduce another proper FIFO queue (that is unlimited but does not need iteration). In this case the validity period starts when you enter the limited queue.

Proposed solutions: • Maintain a mapping from address to index to allow constant-time lookup and removal. • Use a circular buffer or a “head” pointer to avoid shifting array entries when removing expired ones. • Alternatively, mark entries as expired/inactive with a boolean flag instead of removing them from the array immediately.

Not sure I get how these structures would be useful: index needs reindexing or keeps inactive entries, head pointer just delays the problem and so does the inactive flag.

[significance]

Thanks for the well-thought-out SWIP — the design is elegant and clearly addresses Sybil resistance and balanced assignment. I had a few questions and points I’d like to discuss for further clarity and robustness:

• Overlay mining and gaming – how is the off-chain mining step protected against precomputation attacks or delay tactics to gain strategic placement?
• Expired stake handling – have you considered slashing (partial or full) instead of just locking the stake on expiry?

i. the mining step is just offloading computation rather than POW, strategic placement is prevented by random allocation, economic disincentives to be quantified forwith
ii. in this model the stake as discussed if always burned here could otherwise be seen as a fee which perhaps should be considered as recirculating to rewards and/or burned

[significance]

Could _upgradeDepth() become too expensive to execute as the number of assigned nodes grows?

The _upgradeDepth() function doubles the assignment and remaining lists, copies all existing nodes to their new positions using bitwise logic, and clears/rebuilds state — all in a single call. If the number of nodes reaches high volumes (e.g. 1,000+ or 10,000+), this could approach or exceed the block gas limit, making the function fail or stall the system.

Proposed solutions: • Consider breaking _upgradeDepth() into incremental steps, e.g., by: • Using checkpointing to process part of the upgrade each time it is triggered. • Spreading the reassignment of existing entries across multiple assign() calls. • Alternatively, lazy-initialize entries in the assign() function, only when a neighbourhood is actually being filled.

agree with this, some discussion around implementing binary trie or similar datastructure which will ensure uniform gas usage while providing for the necessary functionality

[significance]

very good swip, a few thoughts for discussion and expansion in the doc:

1. it is important to ensure currency of nodes, methods must be added that record recent activity and penalise nodes not taking part in this by at least removal from the allocated nodes pool, thus preventing squat attacks 🏋️
2. a thorough quantisation and parameterisation of economic disincentives should be performed, including costs arising from capital illiquidity
3. a withdrawal queue or delay should be considered to improve network integrity promises
4. likewise an node onboarding processes which perhaps could include proof of well behaved protocol adherance prior to admission into rewards pool
5. it would improve ui/security to provide facility to decouple keys: network key, withdrawal address similar to eth, maybe also a nominated admin address for web ui too
6. a rigorous approach to nomenclature at this point would be prudent, given that the tree depths discussed here are distinct from those in the storage network itself

[awmacpherson]

depth has various levels just as in the physical property, so to some extent interchangeable yes

If this is a response to whether "depth" and "level" are interchangeable, then I'm afraid I don't understand. To what extent are they interchangeable? What does "has various levels" mean?

as positions in the tree become available after deregistration, those causing the greatest imbalance could be best weighted in the randomised allocation, it is important to find the right tradeoff here and maybe the crux of the problem. an easy solution could be achieve by simply picking at random from the best $n$ addresses to achieve growth and uniformity, where $P = 1 / n$ for some desired probability (composed of the deregistrations and next logical)

Here "the tree" means the tree of all bitstrings (of length <= 256)? What does it mean for a position in the tree to become available? The way the proposal is written suggests that only bitstrings of length $d_{39}$, where $d_{39}$ is globally defined at any one time, are assigned at any one time. Is the intention actually that nodes corresponding to prefixes of different lengths can be assigned at the same time?

Here is my attempt to make sense of this: given a set $S$ of overlay addresses, each address $a$ has a shortest prefix $p(a)$ not shared by any other address in the set. Take the subtree $T(S)$ of the tree of all bitstrings spanned by the set of prefixes $p(a)$. Then it makes sense to ask if $T(S)$ is balanced as a binary tree. It sounds as though this is the type of "balancing" you might be after. One can then cook up a metric measuring how far $T(S)$ is from being balanced and always prefer to assign addresses that reduce this distance.

Note that assigning addresses uniformly at random already has a weak version of this property, which is that if $x$ and $y$ are leaves and $\ell(x) &lt; \ell(y)$, then $x$ is more likely to be assigned than $y$ because it corresponds to a larger address block.

[0xCardiE]

Made this, might be useful to add it to SWIP. If its correct :) let me know If its needed to change or add something

[0xCardiE]

Some semantics, for a term “Ether address” that is mentioned in SWIP multiple times, its technically incorrect term, needs to be “Ethereum address” as Ether is currency and address doesn’t belong to Ether but to network.

[significance]

Here is my attempt to make sense of this: given a set S of overlay addresses, each address a has a shortest prefix p ( a ) not shared by any other address in the set. Take the subtree T ( S ) of the tree of all bitstrings spanned by the set of prefixes p ( a ) . Then it makes sense to ask if T ( S ) is balanced as a binary tree. It sounds as though this is the type of "balancing" you might be after. One can then cook up a metric measuring how far T ( S ) is from being balanced and always prefer to assign addresses that reduce this distance.

Note that assigning addresses uniformly at random already has a weak version of this property, which is that if x and y are leaves and ℓ ( x ) < ℓ ( y ) , then x is more likely to be assigned than y because it corresponds to a larger address block.

this is correct i believe. for the second part: yes, but i think it it is too weak and that we must pursue an onboarding/off-boarding queue approach

cc: @zelig 👁️

[significance]

Here is my attempt to make sense of this: given a set S of overlay addresses, each address a has a shortest prefix p ( a ) not shared by any other address in the set. Take the subtree T ( S ) of the tree of all bitstrings spanned by the set of prefixes p ( a ) . Then it makes sense to ask if T ( S ) is balanced as a binary tree. It sounds as though this is the type of "balancing" you might be after. One can then cook up a metric measuring how far T ( S ) is from being balanced and always prefer to assign addresses that reduce this distance.

Note that assigning addresses uniformly at random already has a weak version of this property, which is that if x and y are leaves and ℓ ( x ) < ℓ ( y ) , then x is more likely to be assigned than y because it corresponds to a larger address block.

this is correct i believe. for the second part: yes, but i think it it is too weak and that we must pursue an onboarding/off-boarding queue approach

cc: @zelig 👁️

[zelig]

First: the PR was heavily restructured on 2026-07-27 (two-transaction join, no
reservation held, identity-keyed registrations, concurrent departures, ICBT as a
separate container contract) — several of the points below are answered in the text
now, so please re-read the current version before diving into the old one.
Point-by-point:

1. Why do we need on-chain topology?

Because assignment must be litigable. The redistribution game checks, at claim
time, that a node plays in the neighbourhood it was assigned — that check has to run
in the contract, against state the contract trusts. Off-chain assignment can be
neither enforced (nothing stops a node self-selecting) nor verified (the contract
has no source of truth to check an overlay against). The same goes for the two
enforcement events: forfeiting an expired registration and slashing a defaulted
donor — both need the assignment state on-chain to be adjudicable. And grind-proof
randomness (stake locked before entropy known, seed from a committed block height)
only means anything if the commit itself is on-chain. What stays off-chain is
everything that can: target-prefix computation is a read-only call, mining is local,
and the join costs exactly two transactions.

2. Sorted ring + sparse buckets — formalize and compare.

Happy to add a comparison subsection to the SWIP; here is the summary. The layout
(doubly-linked list in overlay order + mapping(prefix at staking depth => [first node, count])) is good at what rings are good at: O(1)-write
insertion/removal once the position is known, O(1) neighbourhood membership query,
cheap ordered iteration. It is weak exactly where this SWIP lives:

• Uniform random selection. The core operation is "pick uniformly among the
  $2^{d+1}-N$ free neighbourhoods" (and among $N-2^d$ donor pairs). A ring+buckets
  layout either scans buckets — O(2^d) — or maintains hierarchical per-subtree
  counts to support O(log N) rank selection. The moment you add the counts
  hierarchy (your "buckets hierarchy?" bullet concedes this), you have rebuilt the
  ICBT: the trie is the counting structure, with the ring's information implicit
  in it.
• Depth transitions. Buckets keyed by "prefix at staking depth" must be re-keyed
  when $d$ changes — at $N=2^{d+1}$ every key changes, an O(N) migration or a
  lazy-migration scheme with its own bookkeeping. The ICBT never re-keys: $d$ is
  derived, indexes are stable, a depth transition is zero writes.
• Write cost is a wash. ICBT: O(log N) counter updates per activation/departure
  (path to root, ~30 slots at a million nodes). Ring: O(1) link writes + O(log N)
  anyway for whatever counting structure supports selection. We are comparing
  log-vs-log; the constant matters less than the re-keying cliff above.

Where the ring genuinely wins: iterating a neighbourhood's members in overlay order
(we never need this — one node per leaf by construction) and finding the successor
of an arbitrary address (the ICBT does it in O(log N) via nodeFor, good enough for
a view function). So: formal comparison in the SWIP yes, layout change no.

3. High-level operations.

Now in the SWIP: join = register + activate (§Join protocol), departure =
deregister (+ donor redraw path) (§Departure and rebalancing), neighbourhood
queries = getPrefix / nodeFor (read-only). Redistribution eligibility is one
prefix check against the assignment record — the staking contract calls
getPrefix(identity) and compares against the claimed neighbourhood. If a specific
operation list is wanted verbatim in the issue's terms, point me at it.

4. Random/balanced assignment without A and R.

Reading "A and R" as the ordinal mapping and the reservation from the old draft:
the restructured version already dropped both. There is no reservation
(target prefix is a read-only computation, nothing locked, activation revalidates
against current state) and no request IDs / ordinal indirection (registrations are
keyed by staking identity; the seed is H(domain ‖ identity ‖ blockhash)). If A and R
meant something else, tell me what and I'll answer that instead.

5. Comparison with compacted binary trie.

A compacted (path-compressed) trie saves storage when keys are sparse and clustered
— but our key population is dense by construction: the invariant forces every
prefix at depth $d$ or $d+1$ to be occupied, i.e. the trie is always complete to
within one level. There is nothing to compact — path compression on a complete tree
adds skip-pointers that must be maintained on every split/collapse and saves zero
levels. The implicit heap layout additionally removes all pointer storage: parent,
children, sibling are arithmetic on the index, so a "node" is just its counter
slots. Compaction pays off for arbitrary key sets; balanced assignment is precisely
the regime where it cannot. Will add this as a paragraph to the comparison
subsection.

6. Depth transitions.

Walked through in the SWIP (§Balance invariant: preservation under
insertion/removal; §Counting: the $N=2^D$ boundary cases for splitCount/donorCount).
The short version: transitions are emergent, not an event — no stored $d$, no
migration, the counters at the root already equal $2^{d+1}-N$ and $N-2^d$ and both
hit the boundary values exactly at powers of two. If a worked $N=2^D$ example would
help, I can add one next to the existing worked examples.

7. Complexity / gas.

§Gas and performance analysis: selection O(d) reads, activation/departure O(d)
writes, storage O(N), registration/expiry O(1) amortized (monotonic queue head,
bounded per call — no unbounded iteration anywhere). Mining is the real cost and it
is off-chain: expected $2^\ell$ hashes, $\ell \approx \log_2 N + 1$. Benchmarks are
listed as a reference-implementation deliverable; concrete numbers per depth once
there is a contract to measure.

8. The earlier notes (shrink, queue, gas of updates).

• Shrink/withdrawals: the departure path is now first-class — direct collapse
  completes immediately; the donor path holds the departing node active until a
  donor lands, with forfeiture + redraw on default. Withdrawal of stake itself is
  the staking contract's business (strict separation in the SWIP).
• Queue: yes — two commit queues ($C_R$, $C_D$), append-only with monotonic
  block heights, head-advancing expire bounded per call.
• Gas of data-structure updates: O(log N) slot writes per structural change; see
  point 2 for why the proposed alternative doesn't beat it once selection is
  accounted for.

[zelig]

@significance Dan — your Aug/Sep comments all predate the 2026-07-27 restructure. Several of them are now addressed in the text: the binary trie you endorsed is in as the ICBT (implicit complete binary trie, with a layout-comparison subsection), de/registration runs through commit queues with bounded expiry, and there is now a proper Terminology section separating these tree depths from storage depth. Still open from your list: activity/liveness tracking (squat-attack eviction), quantified economic disincentives (explicitly deferred to deployment/staking spec), key decoupling, and onboarding adherence proofs.

Please re-read the current version and leave a proper GitHub review (approve / request changes with the open points) rather than comments — it would help move this toward Accepted.

[Copilot]

🟡 Changes recommended

The SWIP text still contains an unresolved (?) ambiguity and it contradicts the PR description about appended/generated Solidity material.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 1/1 changed files
• Comments generated: 2
• Review effort level: Lite