cd /news/ai-infrastructure/restarting-tailscale-silently-isolat… · home › topics › ai-infrastructure › article
[ARTICLE · art-173002] src=dev.to ↗ pub= topic=ai-infrastructure verified=true sentiment=· neutral

Restarting Tailscale silently isolated a k3s node: fixing a flannel VXLAN bug upstream

A developer diagnosed and fixed an upstream flannel bug that silently isolates k3s nodes after a VPN restart. When tailscaled recreates the external interface, the kernel drops the VXLAN device's routes, ARP and FDB entries, and flannel's reCreateVxlan() rebuilds flannel.1 without replaying them because handleSubnetEvents() only fires on lease changes, leaving the node Ready but unable to reach Pods on other nodes. The fix, filed as flannel issue #2617, owns the LeaseWatcher in the Run loop, removes an unsynchronized write to nw.dev, and replays known leases as EventAdded events, with a new netns test TestRecreatedDeviceGetsKnownLeases added to CI.

recommendation

Enable node_exporter's network_route collector and alert on flannel.1 route counts dropping, and upgrade flannel past v0.28.10 to get lease replay on VXLAN device recreation.

more in playbook: Running AI in production
by read4 min views6 publishedNov 20, 2026

This is an English version of my original Japanese article on Zenn, translated with the help of AI.

tailscaled left one node unable to reach Pods on any other node. The node stayed Ready and nothing alerted.flannel.1, but the kernel had already dropped the routes, ARP and FDB entries that went through it, and flannel only rewrites those when a remote lease k3s / k3s-agent after a VPN restart and alert on the number of Issue: vxlan: routes/ARP/FDB not restored after the external interface is recreated (#2617)

Who this affects: anyone running the flannel VXLAN backend (including k3s's embedded flannel) with --iface (flannel-iface in k3s) set to a VPN interface such as Tailscale, NetBird or WireGuard, on flannel v0.28.10 or earlier.

flannel-iface: tailscale0) k3s also has a Tailscale integration (--vpn-auth, experimental); flannel runs on tailscale0 in that case as well.

After restarting tailscaled (or a Tailscale package upgrade), the k3s log showed:

Interface flannel.1 deleted
vxlan device missing, attempting to recreate...
external interface tailscale0 not found, retrying in 1s
VXLAN device flannel.1 recreated successfully

flannel.1 came back, so it looked fine. But on that node:

ip route | grep -c 'dev flannel.1' dropped from 3 (the number of other nodes) to ip neigh show dev flannel.1 and bridge fdb show dev flannel.1 were emptyupstream connect error ... connection timeout) recreated successfully It did not recover after about 2.5 minutes. Restarting k3s-agent brought the routes back within about a second.

Following pkg/backend/vxlan/vxlan_network.go, two things combine:

tailscale0) disappears, the kernel removes reCreateVxlan() recreates the device itself, but routes, ARP and FDB entries are written only by handleSubnetEvents(), which runs Recreating the interface does not change any remote lease, so the entries are never written again and the node stays isolated until flannel restarts.

Older releases (v0.28.4, for example) panicked on this path in IPv4-only setups (#2433), which restarted k3s and, as a side effect, restored the routes. Once that panic was fixed, the recreate path completed "successfully" and the node quietly stayed isolated instead.

The PR makes three changes:

lease.LeaseWatcher owned by the Run loop.nw.dev / nw.v6Dev from a goroutine.EventAdded, which rewrites routes, ARP and FDB entries.

// knownLeaseEvents returns an EventAdded for every known remote lease.
func (nw *network) knownLeaseEvents() []lease.Event {
    batch := make([]lease.Event, 0, len(nw.knownLeases.Leases))
    for _, l := range nw.knownLeases.Leases {
        batch = append(batch, lease.Event{Type: lease.EventAdded, Lease: l})
    }
    return batch
}

The second change also removes an unsynchronized write to nw.dev from another goroutine.

Using ns.SetUpNetlinkTest, as pkg/backend/route_network_test.go does, the new test TestRecreatedDeviceGetsKnownLeases creates a dummy interface in a network namespace and:

Against the old code it fails with no route to 198.18.2.0/24 via flannel.1 after flannel.1 was recreated; with the fix it passes, both shown in CI.

pkg/backend/vxlan was not in make test's TEST_PACKAGES at the time, so following review feedback I added it in the same PR.

The node stays Ready, so you need another signal. node_exporter's network_route collector (listed under "Disabled by default"; enable it with --collector.network_route) exports the number of routes per interface as node_network_routes{device="..."}.

With the VXLAN backend, a healthy node has as many flannel.1 routes as there are other nodes. When all routes disappear the series disappears too, so the rule also catches a missing series:

- alert: FlannelVxlanRoutesMissing
  expr: |
    (
      (
        max by (node) (node_network_routes{job="node-exporter", device="flannel.1"})
          < scalar(count(kube_node_info) - 1)
      )
      or
      (
        max by (node) (node_uname_info{job="node-exporter"})
          unless on (node) node_network_routes{job="node-exporter", device="flannel.1"}
      )
    )
    and on () (count(kube_node_info) > 1)
  for: 3m

node label on node-exporter series (commonly added by relabeling in kube-prometheus-stack) and kube_node_info from kube-state-metrics.node_network_route_info series per route. With Tailscale that grows with the number of peers, so I dropped it with metricRelabelings and kept only the counts.master go.mod`` ubuntu-latest runner. I wrote up the cause (the test CA certificate was not accepted as a CA) in a separate issue ( I used generative AI (Claude Code and Codex) for the investigation and the implementation. Roughly:

I did not ship the AI's code or explanations as they were: I confirmed in CI that the test fails before the fix and passes after it before opening the PR. Checking a root-cause hypothesis against tests and the kernel's actual state, rather than against an AI's explanation, is an order I intend to keep.

Reproducing a problem from your own environment, finding the cause, and sending it upstream with a test helps everyone running the same setup, and lets you remove the workaround from your own operations. I plan to keep sending what I hit upstream.

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @flannel 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/restarting-tailscale…] indexed:0 read:4min 2026-11-20 · —