kube-vip for k3s Homelab: Control Plane HA with a Virtual IP

You applied a DaemonSet YAML in the k3s installation walkthrough. You updated your kubeconfig to a VIP. You ran kubectl get nodes through that VIP and it worked. Then — maybe out of curiosity, maybe because you're smart — you asked: HOW does that VIP float between physical machines? What's happening at the packet level when Node 1 fails and kubectl keeps working?

Nobody explains this part.

Most kube-vip k3s homelab guides say "it provides a VIP" and move on. The k3s installation post was the "apply the thing" post. You now have kube-vip running. This post is the "here's what you actually built" — the ARP mechanics, the leader election, the DaemonSet design rationale, and the pool separation rules that keep kube-vip and MetalLB from fighting over IPs. When something breaks — and it will — you'll know where to look because you'll know how the thing works.

Key Takeaways

• kube-vip floats a VIP between control plane nodes using gratuitous ARP — a mechanism defined in RFC 826 (1982) that works on every Ethernet switch ever made, including your $30 homelab switch
• kube-vip VIP must sit outside the MetalLB pool range — they must not collide. Traefik gets its IP from inside the MetalLB pool by design — that's the one intentional overlap.
• The --tls-san flag you set during the k3s installation is PERMANENT. The VIP is baked into the API server's TLS certificate at install time. Forget it and you're reinstalling k3s on every node — there is no k3s cert update command that can fix this
• kube-vip auto-detects the network interface when vip_interface is left blank — on single-interface homelab nodes, you don't need to set it at all

Table of Contents

• The Problem — What Breaks Without a Control Plane VIP
• How ARP Makes a Virtual IP Float
• DaemonSet vs Static Pod — Why It Matters
• IP Planning — Pool Separation Rules
• kube-vip vs MetalLB — The Layer Distinction Everyone Gets Wrong
• The --tls-san Gotcha — Why It's Permanent
• Frequently Asked Questions
• Next: Why Bootstrap Order Matters

The Problem — What Breaks Without a Control Plane VIP

Without kube-vip, your kubectl connects to one node's IP — usually Node 1, because that's what you put in your kubeconfig after install. When that node goes down, the API server on that node stops responding. kubectl can't reach the cluster. New deployments fail. ArgoCD can't sync. Your cluster is running — pods are alive, CoreDNS is resolving, workloads are serving traffic — but you can't interact with it. You are locked out of your own cluster by a single point of failure that has nothing to do with etcd, nothing to do with your workloads, and everything to do with a hardcoded IP address.

I ran without kube-vip for two weeks because "I'll set it up later." Node 1 rebooted for an unattended kernel update while I was SSH'd into a different machine. kubectl stopped working. The cluster was fine — all 18 apps still serving. But I couldn't deploy, couldn't check logs, couldn't do anything. kube-vip got deployed that evening. "Just Do Something" — and that something should be kube-vip, before you deploy anything else.

Your kubeconfig contains a server: field — a single IP address. If that IP belongs to a dead node, every tool relying on that kubeconfig (kubectl, helm, ArgoCD CLI, Lens, k9s) stops working simultaneously. Workloads keep running — kubelet doesn't need the API server to keep containers alive — but nothing NEW can happen. No deployments. No scaling. No config changes. No troubleshooting.

The manual workaround: SSH into Node 2, run sudo k3s kubectl get nodes, edit your kubeconfig to point at Node 2, repeat every time a node reboots. That's what you're automating with kube-vip. The workaround proves the cluster is healthy — it's just the access path that's broken.

How ARP Makes a Virtual IP Float

kube-vip works through ARP — Address Resolution Protocol, the same mechanism your switch uses to map IP addresses to MAC addresses for every device on your network. ARP is defined in RFC 826, published in 1982. Forty-plus years later, it's still the foundation of Ethernet networking — and it's how your homelab cluster achieves control plane HA without BGP, without VRRP, and without enterprise hardware.

Here's how it works. Every device on an Ethernet network has a MAC address (hardware) and an IP address (logical). ARP answers the question "which MAC address has IP X?" A device sends an ARP request: "Who has 192.168.1.201?" The device with that IP replies: "I do — my MAC is aa:bb:cc." The asking device caches this in its ARP table. This is how your laptop finds your router. It's also how kube-vip directs API traffic to the control plane leader.

The magic is gratuitous ARP — an ARP reply sent without anyone asking. It announces "I am 192.168.1.201 — update your ARP tables." When kube-vip's leader election determines which node owns the VIP, it sends a gratuitous ARP. Every device on the subnet updates its ARP table. Traffic to the VIP now goes to the new leader's physical interface. When that leader fails, the new leader sends its own gratuitous ARP. The VIP "moves" to a different physical machine without changing IP, without DNS updates, without reconfiguration. Your switch handles the redirect automatically.

The leader election uses the Kubernetes Lease API (coordination.k8s.io/v1). Each kube-vip pod tries to acquire a lease. The pod holding the lease is the leader — it configures the VIP on its node's interface and sends gratuitous ARP (Kubernetes API docs, 2025). If the leader pod stops renewing (node failure, pod crash, network partition), another pod acquires the lease within seconds and becomes the new leader.

What failover looks like in practice: leader node fails → leader lease expires → new pod acquires lease → new leader adds VIP to its interface → new leader sends gratuitous ARP → switch updates MAC table → kubectl reconnects. Total time: 5–15 seconds. kubectl retries automatically; you might see one "connection refused" error before traffic flows to the new leader.

Why not BGP? kube-vip supports BGP mode for true load distribution. BGP requires a BGP-capable router — something your $30 Gigabit switch absolutely is not. ARP mode works on every Ethernet switch manufactured in the last 30 years. For homelab, ARP mode is the only mode that works. The tradeoff: all control plane traffic goes to ONE node (no load balancing). But control plane traffic is minimal — kubectl commands, API calls, controller loops. For a 3-node homelab, this is completely fine.

DaemonSet vs Static Pod — Why It Matters

kube-vip can run as a static pod (manifest file on each node's filesystem, managed outside Kubernetes) or as a DaemonSet (Kubernetes resource, managed via kubectl). You deployed it as a DaemonSet in the k3s installation walkthrough. Here's WHY that's correct.

The static pod method has a chicken-and-egg problem most guides ignore. Static pods are defined by YAML files placed in /etc/kubernetes/manifests/ (or the flavor equivalent). kubelet watches this directory and runs whatever it finds. The problem: you need to place those files on every node BEFORE k3s starts — SSH to every node, create directories, write manifests, then install k3s. Add a node later? Remember to place the manifest before joining. This is manual, error-prone, and exactly the kind of per-node configuration GitOps exists to eliminate.

The DaemonSet lives inside the cluster it protects. Apply once with kubectl, Kubernetes ensures one pod runs on every control plane node. Add a node → pod scheduled automatically. Pod crashes → restarted automatically. This is self-healing infrastructure managed through the same API as your applications. The full manifest is in the homelab-k8s repo at bootstrap/kube-vip/ — the same one you applied in the k3s install walkthrough.

kube-vip requires hostNetwork: true and privileged access. WHY: it binds the VIP as a secondary IP on the host's physical interface and sends ARP packets. Without hostNetwork, the VIP would exist inside a virtual ethernet pair in the pod's network namespace — invisible to your physical switch. MetalLB, Cilium, and Calico all require the same for the same reason. Not a kube-vip quirk — a constraint of any network-level Kubernetes component.

The design is deliberately minimal: vip_address and vip_interface are the only required env vars. vip_interface auto-detects when left blank — on single-interface homelab nodes, don't set it. kube-vip's philosophy: Kubernetes already has leader election and scheduling — use those instead of reinventing them. Compare to keepalived (VRRP) which requires a separate config file with priority values, authentication, and virtual router IDs. kube-vip is two env vars. That's elegance.

IP Planning — Pool Separation Rules

Your homelab subnet has a finite number of IPs, and three Kubernetes components need reserved addresses. The rules are simple but critical:

The key rule: kube-vip VIP and MetalLB pool must be separate ranges. Traefik gets its IP from inside the MetalLB pool — that's the one intentional overlap and it is correct. If DHCP hands out an address inside your MetalLB pool, a random device steals the IP Traefik expects — connections break intermittently, and the failure looks like "Traefik is down" when it's actually "your kid's tablet stole Traefik's IP."

If you haven't done your IP planning yet, read the Homelab Kubernetes IP Planning guide — it walks through the full process with DHCP exclusion instructions for pfSense, OPNsense, OpenWRT, and consumer routers.

Here's a concrete example. My homelab subnet is 192.168.5.0/24. DHCP hands out .100–.200 for devices. I carved .201–.220 as the static block:

• .201 = kube-vip (outside MetalLB pool)
• .202–.210 = MetalLB pool
• .202 = Traefik (first pool IP, inside MetalLB pool)
• .211–.220 = reserved for future static assignments

Your numbers will be different. The principle is the same: pick a block at the top of your subnet, exclude it from DHCP, place kube-vip outside the MetalLB pool range.

For pfSense/OPNsense: Services → DHCP Server → LAN → set the Address Pool Range so it doesn't include your static block. For consumer routers (Asus, TP-Link): LAN → DHCP Server → IP Pool range — set the pool end before your static block starts. If your router can't configure DHCP exclusions, shrink the DHCP pool. Same effect.

[INTERNAL-LINK: How MetalLB L2 mode assigns service IPs → future MetalLB deep dive]

kube-vip vs MetalLB — The Layer Distinction Everyone Gets Wrong

kube-vip and MetalLB both use ARP to make IPs available on your network. But they serve completely different layers. Confusing them is the #1 community support question — and running them with overlapping IP ranges causes ARP conflicts where both daemonsets fight to answer ARP requests for the same IPs.

The ARP conflict scenario: kube-vip has a --services flag that makes it ALSO act as a ServiceLB. If you enable this AND deploy MetalLB, both daemonsets respond to ARP requests for the same service IPs. Result: intermittent connectivity — sometimes kube-vip answers, sometimes MetalLB answers, the switch's ARP table flaps, connections drop, and nothing in the logs tells you why. The only symptom is "sometimes it works, sometimes it doesn't." The fix: never enable kube-vip --services. kube-vip for control plane ONLY. MetalLB for services ONLY.

What happens if you skip kube-vip: no control plane VIP. kubectl points at one node. That node fails, you can't manage the cluster. Services still work (MetalLB handles service IPs), but you can't deploy, scale, or troubleshoot. Worst failure mode: everything looks fine to users, but the operator is locked out.

What happens if you skip MetalLB: LoadBalancer services stay <pending> forever. Pods run, but there's no external IP to reach them. Traefik specifically needs a LoadBalancer IP for ingress traffic.

[INTERNAL-LINK: MetalLB L2 mode on k3s → future MetalLB setup guide]

The --tls-san Gotcha — Why It's Permanent

You set --tls-san <VIP> during the k3s installation. This flag adds the VIP to the API server's TLS certificate as a Subject Alternative Name. And it is FOREVER.

Here's WHY it can't be changed: k3s generates the API server's TLS certificate at install time. The certificate is a signed X.509 document — the SAN list is baked into the certificate's signed data. Changing it requires generating a NEW certificate with a new signature, which requires restarting the API server, which means reinstalling k3s on every node. There is no k3s cert update command because there's no mechanism to hot-swap the API server's serving certificate in a running cluster.

If you forget --tls-san, kubectl connects through the VIP and fails with: x509: certificate is valid for 127.0.0.1, 192.168.1.10, not 192.168.1.201. The VIP isn't in the certificate — TLS verification fails. The fix: uninstall k3s on every node (/usr/local/bin/k3s-uninstall.sh), reinstall with --tls-san <VIP>, rejoin nodes. This is NOT a "fix it later" situation.

Set it on EVERY node during install — any node can become the leader, and every node's API server certificate must include the VIP. It accepts multiple values: --tls-san 192.168.5.201 --tls-san k3s.taegost.com.

The k3s install walkthrough emphasized this flag for exactly this reason. If you followed it, you're good. If you skipped it... you know what to do.

Frequently Asked Questions

Can I use kube-vip on a single-node cluster?

Technically yes. kube-vip's entire purpose is to float a VIP between multiple nodes so kubectl survives a node failure. On a single-node cluster, if the node fails, there's nowhere for the VIP to float to — and your cluster is down regardless. With that being said, if you plan on adding more nodes in the future, or if you're not sure, then set it up anyway.

Why does kube-vip need hostNetwork and privileged access?

kube-vip manipulates the host's network stack directly: it adds the VIP as a secondary IP on the physical interface and sends ARP packets. Both operations require host network access (not a virtual pod network) and elevated privileges. This is expected for any network-level daemonset — MetalLB, Cilium, and Calico all require similar access.

What happens if I forgot --tls-san during k3s install?

You're reinstalling. The API server certificate is generated at install time and the SAN list is baked in permanently. There is no k3s cert update command. Uninstall k3s on every node, reinstall with --tls-san <VIP>, and rejoin nodes. This is why the k3s install walkthrough emphasized the flag — get it right the first time.

How do I know if kube-vip is working?

Three checks: (1) kubectl get pods -n kube-system -l app=kube-vip — one pod per control plane node, all Running. (2) kubectl logs -n kube-system -l app=kube-vip --tail=20 — should show "starting kube-vip" with your VIP. (3) Point your kubeconfig's server: at the VIP and run kubectl get nodes — it should work. Bonus: ip addr show on the leader node — the VIP appears as a secondary IP.

Can I change the kube-vip IP after setup?

Technically yes, unlike --tls-san, the VIP can be changed. Update the vip_address env var in the DaemonSet, restart the pods, and update your kubeconfig. BUT: if the new VIP wasn't in --tls-san at install time, TLS verification will fail. Plan your VIP before k3s install. Change it only if you included a range of potential VIPs in --tls-san.

Next: Why Bootstrap Order Matters

You now understand what kube-vip does at the packet level. You know why ARP mode is the right choice for homelab. You know where your VIP sits relative to the MetalLB pool. And you know why --tls-san is the most important flag in your k3s install command.

kube-vip is step one of six. The full chain is: kube-vip → Sealed Secrets → cert-manager → MetalLB → Traefik → ArgoCD. Each step exists because the NEXT component needs something the PREVIOUS one provides.

That's the next post: k3s Bootstrap Order: Why Sequence Matters — GitOps and ArgoCD Primer. It covers the full dependency chain and introduces GitOps as the destination. No more "apply this, then that" — you'll understand the WHY behind the order.

What's your experience with kube-vip? Did you hit the --tls-san gotcha? Let me know in the comments.