{"slug": "restarting-tailscale-silently-isolated-a-k3s-node-fixing-a-flannel-vxlan-bug", "title": "Restarting Tailscale silently isolated a k3s node: fixing a flannel VXLAN bug upstream", "summary": "A developer diagnosed and fixed an upstream flannel bug that silently isolates k3s nodes after a VPN restart. When tailscaled recreates the external interface, the kernel drops the VXLAN device's routes, ARP and FDB entries, and flannel's reCreateVxlan() rebuilds flannel.1 without replaying them because handleSubnetEvents() only fires on lease changes, leaving the node Ready but unable to reach Pods on other nodes. The fix, filed as flannel issue #2617, owns the LeaseWatcher in the Run loop, removes an unsynchronized write to nw.dev, and replays known leases as EventAdded events, with a new netns test TestRecreatedDeviceGetsKnownLeases added to CI.", "body_md": "*This is an English version of [my original Japanese article on Zenn](https://zenn.dev/ryan0215/articles/flannel-vxlan-lease-replay), translated with the help of AI.*\n\n`tailscaled` left one node unable to reach Pods on any other node. The node stayed `Ready` and nothing alerted.`flannel.1`, but the kernel had already dropped the routes, ARP and FDB entries that went through it, and flannel only rewrites those when a remote lease `k3s` / `k3s-agent` after a VPN restart and alert on the number of Issue: [vxlan: routes/ARP/FDB not restored after the external interface is recreated (#2617)](https://github.com/flannel-io/flannel/issues/2617)\n\n**Who this affects:** anyone running the flannel VXLAN backend (including k3s's embedded flannel) with `--iface` (`flannel-iface` in k3s) set to a VPN interface such as Tailscale, NetBird or WireGuard, on flannel v0.28.10 or earlier.\n\n`flannel-iface: tailscale0`)\nk3s also has a [Tailscale integration](https://docs.k3s.io/networking/distributed-multicloud#integration-with-the-tailscale-vpn-provider-experimental) (`--vpn-auth`, experimental); flannel runs on `tailscale0` in that case as well.\n\nAfter restarting `tailscaled` (or a Tailscale package upgrade), the k3s log showed:\n\n```\nInterface flannel.1 deleted\nvxlan device missing, attempting to recreate...\nexternal interface tailscale0 not found, retrying in 1s\nVXLAN device flannel.1 recreated successfully\n```\n\n`flannel.1` came back, so it looked fine. But on that node:\n\n`ip route | grep -c 'dev flannel.1'` dropped from 3 (the number of other nodes) to `ip neigh show dev flannel.1` and `bridge fdb show dev flannel.1` were empty`upstream connect error ... connection timeout`)` recreated successfully`\nIt did not recover after about 2.5 minutes. Restarting `k3s-agent` brought the routes back within about a second.\n\nFollowing `pkg/backend/vxlan/vxlan_network.go`, two things combine:\n\n`tailscale0`) disappears, the kernel removes `reCreateVxlan()` recreates the device itself, but routes, ARP and FDB entries are written only by `handleSubnetEvents()`, which runs Recreating the interface does not change any remote lease, so the entries are never written again and the node stays isolated until flannel restarts.\n\nOlder releases (v0.28.4, for example) panicked on this path in IPv4-only setups ([#2433](https://github.com/flannel-io/flannel/issues/2433)), which restarted k3s and, as a side effect, restored the routes. Once that panic was fixed, the recreate path completed \"successfully\" and the node quietly stayed isolated instead.\n\nThe PR makes three changes:\n\n`lease.LeaseWatcher` owned by the Run loop.`nw.dev` / `nw.v6Dev` from a goroutine.`EventAdded`, which rewrites routes, ARP and FDB entries.\n\n```\n// knownLeaseEvents returns an EventAdded for every known remote lease.\nfunc (nw *network) knownLeaseEvents() []lease.Event {\n    batch := make([]lease.Event, 0, len(nw.knownLeases.Leases))\n    for _, l := range nw.knownLeases.Leases {\n        batch = append(batch, lease.Event{Type: lease.EventAdded, Lease: l})\n    }\n    return batch\n}\n```\n\nThe second change also removes an unsynchronized write to `nw.dev` from another goroutine.\n\nUsing `ns.SetUpNetlinkTest`, as `pkg/backend/route_network_test.go` does, the new test `TestRecreatedDeviceGetsKnownLeases` creates a dummy interface in a network namespace and:\n\nAgainst the old code it fails with `no route to 198.18.2.0/24 via flannel.1 after flannel.1 was recreated`; with the fix it passes, both shown in CI.\n\n`pkg/backend/vxlan` was not in `make test`'s `TEST_PACKAGES` at the time, so following review feedback I added it in the same PR.\n\nThe node stays `Ready`, so you need another signal. node_exporter's `network_route` collector (listed under \"[Disabled by default](https://github.com/prometheus/node_exporter#disabled-by-default)\"; enable it with `--collector.network_route`) exports the number of routes per interface as `node_network_routes{device=\"...\"}`.\n\nWith the VXLAN backend, a healthy node has as many `flannel.1` routes as there are other nodes. When all routes disappear the series disappears too, so the rule also catches a missing series:\n\n```\n- alert: FlannelVxlanRoutesMissing\n  expr: |\n    (\n      (\n        max by (node) (node_network_routes{job=\"node-exporter\", device=\"flannel.1\"})\n          < scalar(count(kube_node_info) - 1)\n      )\n      or\n      (\n        max by (node) (node_uname_info{job=\"node-exporter\"})\n          unless on (node) node_network_routes{job=\"node-exporter\", device=\"flannel.1\"}\n      )\n    )\n    and on () (count(kube_node_info) > 1)\n  for: 3m\n```\n\n`node` label on node-exporter series (commonly added by relabeling in kube-prometheus-stack) and `kube_node_info` from kube-state-metrics.`node_network_route_info` series per route. With Tailscale that grows with the number of peers, so I dropped it with `metricRelabelings` and kept only the counts.`master` `go.mod`` ubuntu-latest` runner. I wrote up the cause (the test CA certificate was not accepted as a CA) in a separate issue (\nI used generative AI (Claude Code and Codex) for the investigation and the implementation. Roughly:\n\nI did not ship the AI's code or explanations as they were: I confirmed in CI that the test fails before the fix and passes after it before opening the PR. Checking a root-cause hypothesis against tests and the kernel's actual state, rather than against an AI's explanation, is an order I intend to keep.\n\nReproducing a problem from your own environment, finding the cause, and sending it upstream with a test helps everyone running the same setup, and lets you remove the workaround from your own operations. I plan to keep sending what I hit upstream.", "url": "https://wpnews.pro/news/restarting-tailscale-silently-isolated-a-k3s-node-fixing-a-flannel-vxlan-bug", "canonical_source": "https://dev.to/ryan0215/restarting-tailscale-silently-isolated-a-k3s-node-fixing-a-flannel-vxlan-bug-upstream-1p0n", "published_at": "2026-11-20 04:33:06+00:00", "updated_at": "2026-11-20 04:43:40.208747+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "developer-tools"], "entities": ["flannel", "k3s", "Tailscale", "NetBird", "WireGuard", "node_exporter", "GitHub"], "also_reported_by": [], "takeaway": "On flannel v0.28.10 or earlier with --iface pointed at a VPN interface, restarting tailscaled recreates flannel.1 but never rewrites routes, ARP or FDB entries, so nodes stay Ready while isolated; monitor node_network_routes{device=\"flannel.1\"} since no default alert fires.", "recommendation": {"advice": "Enable node_exporter's network_route collector and alert on flannel.1 route counts dropping, and upgrade flannel past v0.28.10 to get lease replay on VXLAN device recreation.", "playbook": "production-ops", "playbook_url": "https://wpnews.pro/playbooks/production-ops"}, "alternates": {"html": "https://wpnews.pro/news/restarting-tailscale-silently-isolated-a-k3s-node-fixing-a-flannel-vxlan-bug", "markdown": "https://wpnews.pro/news/restarting-tailscale-silently-isolated-a-k3s-node-fixing-a-flannel-vxlan-bug.md", "text": "https://wpnews.pro/news/restarting-tailscale-silently-isolated-a-k3s-node-fixing-a-flannel-vxlan-bug.txt", "jsonld": "https://wpnews.pro/news/restarting-tailscale-silently-isolated-a-k3s-node-fixing-a-flannel-vxlan-bug.jsonld"}}