Improve meshnet robustness to cluster changes - #743
Merged
Conversation
In pod pairs, one expects the other to do the veth setup work. If the lazy half restarts, it continues to expect this and things never get fixed. This updates the code to fix even in this case.
When a link is dynamically removed from the topology, clean up its associated meshnet resources
A pod's IP can change on restart, so make sure IP is updated when the pod resyncs
When a pod is rescheduled elsewhere, the type of link it needs may change, so clean up and handle that case properly when needed
#### 1. In-Memory Topology Cache (TopologyCache)
• Location: controller.go
• Maintains a thread-safe in-memory cache of all Topology CRs, populated on controller startup and kept updated in real-
time via Kubernetes watch events and local status updates (handler.go).
• Resolves handler.go, controller.go, and controller.go from memory in O(1) time with fallback, eliminating the kube-
apiserver List/Get request storm.
#### 2. Inverted Peer Dependency Indexing & Targeted Reconciliation
• Location: controller.go, controller.go
• TopologyCache maintains an inverted dependency index (peerPod -> []dependentLocalPods).
• When a pod X starts, restarts, or updates:
• If X is on this node, X is queued for reconciliation.
• All local pods Y that link to X (GetDependents) are identified in O(1) and queued for reconciliation.
• Unrelated pods and nodes in the rest of the 1500-node cluster do zero reconciliation work.
#### 3. Protection Against Post-Configure State Decay & Restarts
• Peer Pod Restart / NetNS Re-creation: If peer pod B restarts (updating its net_ns, src_ip, or container_id), the watch
event for B immediately triggers reconciliation for all dependent local pods (such as A). The veth link or gRPC wire is
re-plumbed against B's new namespace.
• Node Rescheduling: Node IP changes (same-node ↔ remote) trigger the transition logic to tear down stale links and
create the new link type.
• Safety Net Periodic Resync: A 60-second periodic resync runs in the background as a safety net against any missed
watch events or delayed state convergence.
#### 4. Coalescing and Debouncing Work Queue (ReconcileQueue)
• Location: controller.go
• Debounces incoming pod arrival bursts (50ms window) so that simultaneous pod starts coalesce into a single batched
pass.
### 1. Per-RPC Timeout on Remote Handshakes
• Locations: controller.go & controller.go
• Changes: Wrapped remoteClient.AddGRPCWiresRemoteBatch and remoteClient.GRPCWireDownRemote with a 5*time.Second context timeout (context.
WithTimeout(ctx, 5*time.Second)).
• Benefit: If a remote worker node is slow, booting, or dropping packets, the call quickly times out and fails gracefully rather than
blocking the single-threaded local reconciler loop indefinitely.
──────
### 2. Prevent Duplicate RecvFrmLocalPodThread Spawn
• Locations: gwire_rpc_handlers.go & handler.go
• Changes:
1. Made gwire_rpc_handlers.go check if the wire is already in memory before creating a new TAP handle or spawning a reader thread.
2. Updated gwire_rpc_handlers.go to return an isNew boolean flag.
3. In AddGRPCWireRemote and AddGRPCWiresRemoteBatch, only spawn go RecvFrmLocalPodThread if isNew == true.
• Benefit: When both peer nodes reconcile simultaneously and exchange remote wire creation RPCs, existing TAP handles will not have
multiple competing reader goroutines spawned on /dev/net/tun.
──────
### 3. Mitigate Thundering Herd Polling in CNI cmdAdd
• Location: meshnet.go
• Changes:
1. Batch NetNS Lookup: Discovers all container interfaces in a single netns switch per polling tick instead of executing a netns switch
per link.
2. Peer Node IP Cache: Caches the peer pod node IP to avoid querying meshnetClient.Get on every 100ms tick for each link.
3. Exponential Backoff: Replaced the fixed 100ms ticker with an exponential backoff (starting at 100ms and backing off to 1s).
4. Nil-check: Added a peerPod != nil guard in cmdDel to prevent panics during cascading teardowns.
1. Added streamMgr.Send(topoNs, peerIP, pkt) to route packets to the multiplexed stream corresponding to (topoNs, peerIP). 2. In grpcwire.go, when a peer pod is reassigned to a different node (wire.PeerNodeIP != newPeerIP), the old stream reference is cleanly released with streamMgr.ReleaseStream and the new stream is registered with streamMgr.GetOrCreateStream. 3. In grpcwire.go, packets dynamically read the current wire.PeerNodeIP under lock, ensuring outgoing packets instantly route to the new node without blackholing traffic or corrupting stream refcounts.
A. Concurrent Local vs. Remote Wire Creation Race
Root Cause: When two nodes (Node A and Node B) reconcile a link simultaneously, Node A calls CreateGRPCWireLocal while sending an AddGRPCWiresRemoteBatch RPC to Node B, and Node B does the inverse. If CreateGRPCWireLocal and CreateUpdateGRPCWireRemoteTriggered execute concurrently on Node B for the same link UID, both check GetWireByUID / UpdateWireByUID before either inserts into wires.wires, resulting in duplicate TAP creation and clobbered map entries.
Fix: Added synchronization (createWireMu sync.Mutex) in
gwire_rpc_handlers.go
across checking if a wire already exists and creating/inserting a new wire.
B. Missing Stale NetNS Wire Cleanup in AddInMemNDataStore
Root Cause: Commit
81dbf4a
added cleanup for old wires with stale LocalPodNetNS paths (when containers restart) to wires.AddInMem. However, gRPC RPC triggers and local reconciliation use AddInMemNDataStore, which omitted that cleanup loop—leaving old netNS TAP handles and reader goroutines running indefinitely when pods restarted in a new netNS.
Fix: Extracted a helper method
cleanupOldWireLocked
and invoked it in both AddInMem and AddInMemNDataStore. Added unit test
TestAddInMemNDataStore_StaleWireCleanup
to cover this scenario.
C. Stream Reference Count Leak on Late Peer Discovery
Root Cause: In
RecvFrmLocalPodThread
, defer func() { ... ReleaseStream(...) }() was registered inside if peerIP != "". When CreateGRPCWireLocal spawns the thread before the peer pod is scheduled (peerIP == ""), the defer was never scheduled. When PeerNodeIP was updated later via wire.UpdateWire, GetOrCreateStream incremented the refcount, but thread termination never released it.
Fix: Moved the defer cleanup block outside if peerIP != "" so that currIP := wire.PeerNodeIP is checked upon thread exit and any acquired stream reference is released.
D. Reader Thread Leak on Teardown of Non-Ready Wires
Root Cause: In
RemoveWireAcrosAll
and WireDownByUID, wire.StopC was only closed if wire.IsReady. If a wire was locally initiated (IsReady = false) and the pod was deleted before the remote peer completed the handshake, StopC was never closed.
Fix: Updated wire teardown methods to unconditionally close wire.StopC (if non-nil) regardless of wire.IsReady.
… obsolete CNI code
1. Dead-Wire Recovery on WireDownByUID:
- Updated WireDownByUID in grpcwire.go to invoke RemoveWireAcrosAll(wire, true)
instead of leaving a dead wire in the active map with closed channels and dead
reader goroutines. This ensures future peer pod reconnects and reconciliations
cleanly re-create TAP handles and reader threads.
- Added unit test TestWireDownByUID_CleansUpAndRemovesFromMemory.
2. gRPC Wire Cleanup on Topology CR Deletion:
- Added grpcwire.DeletePodWires(ns, name) to the watch.Deleted event handler in
controller.go to clean up gRPC wires, reader goroutines, and TAP handles when
a local Topology CR is deleted directly.
3. Prevent gRPC Conn Leaks and Timeouts in CNI MakeGRPCChanDown:
- Added defer remote.Close() to ensure gRPC connections are closed on exit.
- Added a 5-second context timeout to GRPCWireDownRemote RPC calls to avoid
hanging CNI deletions when remote peers are unreachable.
4. Clean Up Obsolete Legacy Code:
- Removed unused CreatGRPCChan and legacy retry constants from grpcwires-plugin.go,
as link creation is now handled by the daemon reconciler.
Cross-Node Wire Teardown Resolution (third_party/meshnet/daemon/grpcwire/gwire_rpc_handlers.go):
Updated GRPCWireDownRemoteTriggered to check wireDef.LocalPodNetNs first and fallback to wireDef.TopoNs. This ensures remote wire teardowns across nodes correctly match and destroy the remote wire end even when the caller sends its own local container netns path.
Added unit test TestGRPCWireDownRemoteTriggered_RemoteNetNSFallback in third_party/meshnet/daemon/grpcwire/gwire_map_test.go.
CNI Plugin Cleanup & String Trimming (third_party/meshnet/plugin/meshnet.go):
Removed the obsolete and redundant meshnetClient.SkipReverse calls in cmdDel to prevent unnecessary Kubernetes API conflict retries on pod deletion.
Trimmed whitespace on interNodeLinkType using strings.TrimSpace and fixed the log typo in SetInterNodeLinkType ("iner" -> "inner").
1. Standard Kubernetes Informer Integration (third_party/meshnet/daemon/meshnet/controller.go)
• Integrated DynamicSharedInformerFactory in RunControllerLoop.
• Cache sync, event streaming, and watch reconnection are now managed by standard client-go Informer infrastructure. Watch event callbacks update
topoCache and trigger debounced reconciliation in RAM with 0 unnecessary API server List calls.
2. CLI Initialization Fail-Fast Timeout (topo/node/node.go)
• Updated GetCLIConn to apply a 5-second WithTimeoutOps during early boot connection attempts (d.Open()).
• When pods enter PodRunning while router OS containers (JunOS, SR Linux, cEOS) are still booting up, d.Open() fails in 5 seconds instead of
stalling on prompt-matching timeouts for 30–60 seconds per attempt. Once ready, full operational driver settings are restored.
3. Eliminated Arista Status() Watch Thrashing (topo/node/arista/arista.go & arista_test.go)
• Replaced continuous short-lived Watch() stream creation in arista.go Status() with direct single-pod Get() queries via n.Pods(ctx).
• Updated arista_test.go to test Get reactors.
These variables are initialized to a non-zero default, so the check was never successful
Don't hang and fail if the close sequence can't be sent (already closed.) Don't return error if the close message can't be sent.
bstoll
approved these changes
Aug 20, 2026
1. Apply mutex hat idiom in GRPCWire struct (third_party/meshnet/daemon/grpcwire/grpcwire.go):
• Reordered GRPCWire fields to place sync.Mutex directly above the mutable fields it synchronizes (IsReady,
WireIfaceIDOnPeerNode, PeerNodeIP, StopC).
2. Handle DeletedFinalStateUnknown tombstones (third_party/meshnet/daemon/meshnet/controller.go):
• Unwrapped cache.DeletedFinalStateUnknown in toUnstructured and informer DeleteFunc to prevent missed topology
deletion events during watch reconnects.
3. Make remote wire teardown non-blocking (third_party/meshnet/daemon/meshnet/controller.go):
• Dispatched GRPCWireDownRemote RPCs asynchronously in cleanupRemovedPodLinks to prevent blocking the main
reconciliation loop when peer nodes are unreachable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Meshnet's algorithm is really written only for initial config. If pods restart, get reassigned to other nodes, etc., it doesn't keep the mesh current.
This introduces a number of changes to monitor K8s state post-initialization and keep the mesh properly configured.