Restarting Tailscale silently isolated a k3s node: fixing a flannel VXLAN bug upstream A developer diagnosed and fixed an upstream flannel bug that silently isolates k3s nodes after a VPN restart. When tailscaled recreates the external interface, the kernel drops the VXLAN device's routes, ARP and FDB entries, and flannel's reCreateVxlan() rebuilds flannel.1 without replaying them because handleSubnetEvents() only fires on lease changes, leaving the node Ready but unable to reach Pods on other nodes. The fix, filed as flannel issue #2617, owns the LeaseWatcher in the Run loop, removes an unsynchronized write to nw.dev, and replays known leases as EventAdded events, with a new netns test TestRecreatedDeviceGetsKnownLeases added to CI. This is an English version of my original Japanese article on Zenn https://zenn.dev/ryan0215/articles/flannel-vxlan-lease-replay , translated with the help of AI. tailscaled left one node unable to reach Pods on any other node. The node stayed Ready and nothing alerted. flannel.1 , but the kernel had already dropped the routes, ARP and FDB entries that went through it, and flannel only rewrites those when a remote lease k3s / k3s-agent after a VPN restart and alert on the number of Issue: vxlan: routes/ARP/FDB not restored after the external interface is recreated 2617 https://github.com/flannel-io/flannel/issues/2617 Who this affects: anyone running the flannel VXLAN backend including k3s's embedded flannel with --iface flannel-iface in k3s set to a VPN interface such as Tailscale, NetBird or WireGuard, on flannel v0.28.10 or earlier. flannel-iface: tailscale0 k3s also has a Tailscale integration https://docs.k3s.io/networking/distributed-multicloud integration-with-the-tailscale-vpn-provider-experimental --vpn-auth , experimental ; flannel runs on tailscale0 in that case as well. After restarting tailscaled or a Tailscale package upgrade , the k3s log showed: Interface flannel.1 deleted vxlan device missing, attempting to recreate... external interface tailscale0 not found, retrying in 1s VXLAN device flannel.1 recreated successfully flannel.1 came back, so it looked fine. But on that node: ip route | grep -c 'dev flannel.1' dropped from 3 the number of other nodes to ip neigh show dev flannel.1 and bridge fdb show dev flannel.1 were empty upstream connect error ... connection timeout recreated successfully It did not recover after about 2.5 minutes. Restarting k3s-agent brought the routes back within about a second. Following pkg/backend/vxlan/vxlan network.go , two things combine: tailscale0 disappears, the kernel removes reCreateVxlan recreates the device itself, but routes, ARP and FDB entries are written only by handleSubnetEvents , which runs Recreating the interface does not change any remote lease, so the entries are never written again and the node stays isolated until flannel restarts. Older releases v0.28.4, for example panicked on this path in IPv4-only setups 2433 https://github.com/flannel-io/flannel/issues/2433 , which restarted k3s and, as a side effect, restored the routes. Once that panic was fixed, the recreate path completed "successfully" and the node quietly stayed isolated instead. The PR makes three changes: lease.LeaseWatcher owned by the Run loop. nw.dev / nw.v6Dev from a goroutine. EventAdded , which rewrites routes, ARP and FDB entries. // knownLeaseEvents returns an EventAdded for every known remote lease. func nw network knownLeaseEvents lease.Event { batch := make lease.Event, 0, len nw.knownLeases.Leases for , l := range nw.knownLeases.Leases { batch = append batch, lease.Event{Type: lease.EventAdded, Lease: l} } return batch } The second change also removes an unsynchronized write to nw.dev from another goroutine. Using ns.SetUpNetlinkTest , as pkg/backend/route network test.go does, the new test TestRecreatedDeviceGetsKnownLeases creates a dummy interface in a network namespace and: Against the old code it fails with no route to 198.18.2.0/24 via flannel.1 after flannel.1 was recreated ; with the fix it passes, both shown in CI. pkg/backend/vxlan was not in make test 's TEST PACKAGES at the time, so following review feedback I added it in the same PR. The node stays Ready , so you need another signal. node exporter's network route collector listed under " Disabled by default https://github.com/prometheus/node exporter disabled-by-default "; enable it with --collector.network route exports the number of routes per interface as node network routes{device="..."} . With the VXLAN backend, a healthy node has as many flannel.1 routes as there are other nodes. When all routes disappear the series disappears too, so the rule also catches a missing series: - alert: FlannelVxlanRoutesMissing expr: | max by node node network routes{job="node-exporter", device="flannel.1"} < scalar count kube node info - 1 or max by node node uname info{job="node-exporter"} unless on node node network routes{job="node-exporter", device="flannel.1"} and on count kube node info 1 for: 3m node label on node-exporter series commonly added by relabeling in kube-prometheus-stack and kube node info from kube-state-metrics. node network route info series per route. With Tailscale that grows with the number of peers, so I dropped it with metricRelabelings and kept only the counts. master go.mod ubuntu-latest runner. I wrote up the cause the test CA certificate was not accepted as a CA in a separate issue I used generative AI Claude Code and Codex for the investigation and the implementation. Roughly: I did not ship the AI's code or explanations as they were: I confirmed in CI that the test fails before the fix and passes after it before opening the PR. Checking a root-cause hypothesis against tests and the kernel's actual state, rather than against an AI's explanation, is an order I intend to keep. Reproducing a problem from your own environment, finding the cause, and sending it upstream with a test helps everyone running the same setup, and lets you remove the workaround from your own operations. I plan to keep sending what I hit upstream.