Repository navigation
Directory hardening, allowed_publish deny by default, one router sync loop - #719
Merged
Merged
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a new allowed_publish field to PolicyRole to control which services a node is authorized to serve. It updates the directory protocol to support Withdraw operations and adds validation logic (including ServiceOfRecord and requirePublish) to ensure that providers are authorized to serve the services they announce. The control plane and router components are updated to sync these policies and enforce them during MCP session establishment and egress proxying. I have one piece of feedback regarding a potential runtime error in the Python SDK's string splitting logic.
…w them
The router kept members' records in kad-dht's provider store and ran
lookups as kad queries. That store cannot forget a record before its
TTL, merges every announced address into the peerstore for the TTL, and
cannot be replaced (kad-dht does not take a custom ProviderStore), so a
service a node stopped stayed discoverable for 15 minutes and a circuit
through a router the node left lingered as long.
The router now keeps records in a store of its own, by key and provider,
each with the addresses of the last announcement and an expiry at the
mesh's record TTL; announcing again replaces the addresses, a sweep drops
what expired. The DHT is left with what the directory needs from it: the
routers' table, from which the routers closest to a key are read. A
lookup from a member merges this router's records with the answers of
those routers, asked at once over the directory protocol, each within
five seconds; a lookup from a peer router is answered from this router's
records alone, so a lookup is one hop. A provider the router holds is
answered at the addresses it sees for it.
DirectoryWithdraw (api/agentmesh.proto) removes a record the same way a
provide stores one: from the router asked and from the routers closest
to the key, with the provider named only between routers. A node
withdraws a service's records when it unregisters it (the record by type
stays while another service of the type is registered) and every record,
its own included, when it tears down.
Metrics: agentmesh_router_directory_records, _records_expired_total,
_lookups_total{result}; op labels withdraw and lookup. Bindings
regenerated; go-cid is no longer imported directly.
A member took the first router's answer to a lookup, empty or not. The router answers from its own records and the routers closest to the key; one cut off from those answers from what it can reach, and the member holds another router that may not be. A lookup that comes back empty is now put to the next held router before it is empty; an announcement or a withdrawal one router took is still not repeated. The same in the Go node and both SDKs.
An SDK member published no address, so the directory knew it only through the router that held it: a find_peer from any other router found nothing, and a caller on other routers fell back to circuits through its own routers, which refuse it. The rollout probe on the testnet hit this: a node reaching the JS agent by peer ID got PERMISSION_DENIED from every circuit it tried. Both SDKs now announce their relayed addresses under their peer key to a router they hold: after join, when the addresses change (a router replaced) and every five minutes, agentmesh-node's reprovide interval; a member that reserves no relay announces nothing, there is nothing to reach it at. A member leaving withdraws the record. The announcement is best effort: a router that does not take it is a warning the next one retries. The Python directory client asks only routers it is connected to, as the JS one does: a request goes over the session the handshake opened, never over a dial of its own, which left the reservation loop facing a connection it had not authenticated. top_up_routers runs one at a time: the reservation loop and a go-away both call it, and two picking the same router at once held it twice, or shunned it while holding it.
…is its A member could announce any key: a record under a key it had no business with, a service name it does not serve, another member's peer key. The router stored whatever multihash came with addresses of the sender. DirectoryProvide now names the service a record is about, "<type>://<name>", and the member's router refuses a key that is not that service's by name or by type (api.ServiceOfRecord); a record naming no service must be the sender's own peer key. A placed copy is not checked again: the router that holds the member did. This is also what a publish grant will be checked against: the service by name, not a hash. The node's Directory.Provide takes the service; the SDKs announce only their peer key, which names none.
…ault
A role said what a member may call (allowed_services) and where it may
send (allowed_targets), but nothing said what a member may serve. Any
enrolled node could publish mcp://github or inference://gpt-4 and
callers that asked the directory would be routed to it: a member of the
mesh could stand in for any service, which the record check in the
previous commit does not prevent, it only ties a key to the name the
record itself claims.
PolicyRole gains allowed_publish, the same patterns as allowed_services
("*", "mcp://*", "mcp://github", "mcp://team-*"), empty by default: a
member publishes nothing until a role it holds grants the service. The
control plane validates and stores the list, and mints it into the
authority block as granted_publish_* facts (api.BuildPublishDatalogFacts)
beside the granted_* facts of allowed_services, with the same budget. A
member of a served egress destination gets egress://<name> granted at
mint from the destination's served_by (api.WithEgressPublish), so the
egress policy is the single place an operator names who serves it.
Two verifiers enforce the grant, from the same Datalog allow policies
(api.PublishPolicies, exported to the SDK artifact as publish_policies):
* The router that holds a member keeps its credential from the auth
handshake and refuses a DirectoryProvide whose service the
credential does not grant (requirePublish; the peer record is always
allowed); the refusal is a "Record denied" warning and the "denied"
result of agentmesh_router_directory_lookups_total.
* The caller checks the provider it is about to use: node
(VerifyProvider at the egress proxy, A2A, egress tunnel, OpenAI
facade and MCP connect, 403 "provider is not granted <svc>"), JS and
Python (requirePublish / require_publish on the provider's verified
biscuit before the MCP session or egress stream carries a request).
The catalog service system://mesh.catalog needs no grant.
The default policies grant "*" to the node role where every node is the
operator's (standalone, the testnets, the integration fixtures); the
chart takes bootstrap.nodePublish, empty by default, and the container
e2e mesh grants exactly the services its nodes run.
A publish grant was minted into the credential and nowhere else, so a
change to allowed_publish reached the router and the callers when the
member's credential was minted again: up to nineteen hours after the
operator saved it, with nothing for them to do but wait or re-enroll.
Every other grant already travels a second way, as the Datalog text the
control plane renders (granted_*(...) <- role(...)) and every member adds
to its authorizer at its next sync or at once on POLICY_UPDATE.
Publish grants now travel the same way. BuildPolicyRules renders
granted_publish_*(...) <- role("<role>") for each role's allowed_publish,
and BuildEgressServingRules renders granted_publish_exact("egress", name)
beside granted_service_exact for each served_by selector. Every verifier
adds the synced rules to the authorizer that decides the grant
(identity.RequirePublish takes them): the node in checkProvider, the JS
and Python sessions in requirePublish, which every session now syncs the
policy for, callers included, not only A2A endpoints. The router, which
read no policy before, pulls /policies with its own credential at start,
every PolicySyncInterval and on POLICY_UPDATE, and evaluates it at
provide. A node whose policy text changed announces its records again,
so a record the directory refused under the previous policy is in it
within the sync interval instead of the reprovide interval.
GetMeshPolicy reads roles, permissions and bindings in a fixed order, so
the rendered text only changes when the policy does; it was built from a
map, and two reads of one policy rendered the same rules in a different
order.
Tests: the rendering (api), RequirePublish over rules (identity), the
router and node checks with rules, both SDKs' MCP opens with rules, a
control plane validation test for allowed_publish, and an integration
test that runs the whole thing: a node serving a service its role does
not grant is not in the directory and is refused by peer ID, then the
policy grants it and, with nothing restarted, the service is found and
called.
The router held five conversations with the control plane, each in its own goroutine on its own clock: the lease every --lease-renew-interval, /info every 30 seconds, /keys every --keys-sync-interval, /policies every 15 minutes, and a credential expiry check every 10 minutes. A node has had one pass for all of this since the start (SyncControlPlane). The router now has one too. Every --control-plane-sync-interval (5 minutes, what the lease was renewed at) and at once on a KEY_ROTATION or POLICY_UPDATE event, a pass re-reads /keys first, so a credential the control plane no longer signs for is refreshed before it is presented, then renews the lease, re-reads /info and /policies, and refreshes the router's own credential when less than a fifth of its lifetime is left. While /info names a router this one has not authenticated with, the federation step alone is retried every 2 seconds, as before; a dead listed router costs the control plane no more than it did. What has to be quick does not wait for the interval: a new router reads /info on its own first pass and handshakes with every listed router, a lookup that finds nothing re-reads /info (refederateIfStale), and an event runs the pass now. The control plane sees four requests per router per interval. --keys-sync-interval and --lease-renew-interval are gone, replaced by --control-plane-sync-interval, the node's flag name; agentmesh-one's --router-keys-sync-interval and --router-lease-renew-interval by --router-control-plane-sync-interval. Tests that set either now set the one interval to the shorter of the two.
aojea
force-pushed
the
directory-hardening
branch
from
October 14, 2026 18:17
ffa8294 to
0a29755
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The directory #704 introduced, hardened for what a mesh does over time: routers that restart or partition, members that stop or move, and members that announce what they may not serve. With it, two things an rc should not ship without: a service a member may publish is a policy decision, deny by default, and the router talks to the control plane in one loop.
Directory
internal/router/directory_store.go). A record lives for the mesh's record TTL after its last announcement, an announcement replaces the one before, a sweep expires the rest. A member withdraws its records when it stops a service or leaves (DirectoryWithdraw; the node onUnregisterand at teardown, the SDKs on close), and the router withdraws them from the routers it placed them on. Lookups ask the routers closest to the key at once (op=lookup, bounded bydirectoryLookupTimeout). Metricsagentmesh_router_directory_records,_records_expired_total,_lookups_total{result}.DirectoryProvide.service,api.ServiceOfRecord): the router refuses a key that is not that service's by name or by type, and a record naming no service must be the sender's own peer key. Before, any multihash with the sender's addresses was stored.Publish authorization, deny by default
PolicyRole.allowed_publish: the same patterns asallowed_services, empty by default. A member publishes nothing until a role it holds grants the service. The control plane validates and stores it, mints it asgranted_publish_*facts, and renders it into the policy text asgranted_publish_*(...) <- role(...), so a change reaches every verifier at its next sync or at once onPOLICY_UPDATE, not at the member's next credential. A node serving an egress destination getsegress://<name>fromserved_by, both minted and rendered; the egress policy stays the one place that names who serves it.Two verifiers, from the same Datalog policies (
api.PublishPolicies, exported to the SDK artifact aspublish_policies):result="denied", aRecord deniedwarning);agentmesh-nodeat the egress proxy, A2A, egress tunnel, OpenAI facade and MCP connect (403provider is not granted <svc>), the JS and Python SDKs before an MCP session or an egress stream carries a request.system://mesh.catalogneeds no grant.The default policies grant
"*"to the node role where every node is the operator's (standalone, the testnets, the integration fixtures); the chart takesbootstrap.nodePublish, empty by default; the container e2e mesh grants exactly the services its nodes run.GetMeshPolicyreads in a fixed order so the rendered text changes only when the policy does (it was built from a map).One control plane loop on the router
The router held five conversations with the control plane on five clocks (lease,
/info,/keys,/policies, credential expiry). It now runs one pass every--control-plane-sync-interval(5 minutes, the lease cadence) and at once on a mesh event:/keysfirst, then the lease,/info,/policies, and its own credential when less than a fifth of its lifetime is left. The 2 s federation catch-up stays a federation-only retry. Four requests per router per interval, fewer than before.Breaking changes
allowed_publishis deny by default. A mesh upgrading to this version must grant its node roles what they serve ("*"keeps today's behaviour) or its nodes publish nothing. The grant takes effect at the next policy sync without re-enrolling.agentmesh-router --keys-sync-intervaland--lease-renew-intervalare replaced by--control-plane-sync-interval;agentmesh-one --router-keys-sync-intervaland--router-lease-renew-intervalby--router-control-plane-sync-interval./mesh/directory/1.0.0gainswithdraw,lookupandDirectoryProvide.service; an older SDK that sends no service is still accepted for its peer record and publishes nothing else.Validation
make,make lint(0 issues),hack/verify-generated.sh,hack/verify-sdk-generated.sh,go test ./...(integration suite green on the rebased tree), JS 133 and Python 169 SDK tests, SDK conformance integration tests with both SDKs built,make e2e-test32/32, chart tests 60/60. New integration testTestUngrantedServiceIsNeitherPublishedNorServed: a node serving a service its role does not grant is not in the directory and is refused by peer ID; the policy then grants it and, with nothing restarted, the service is found and called.Follows #704. Refs #277.