Replacing Three Tools with One: Migrating to Cilium on Talos

My homelab cluster originally ran Flannel for pod networking, kube-proxy for service routing, and MetalLB for giving services real LAN IPs. Three separate components, three sets of configuration, three things that can break independently. MetalLB broke first.

The MetalLB Fight

MetalLB gives Kubernetes services real IPs on your LAN instead of NodePort random high ports. Simple concept. On Talos Linux, not a simple install.

The first problem was PodSecurity. MetalLB's speaker DaemonSet needs NET_ADMIN, NET_RAW, and SYS_ADMIN capabilities plus hostNetwork=true. Talos enforces the baseline PodSecurity standard by default, which blocks all of these:

Error creating: pods "metallb-speaker-xxx" is forbidden:
  violates PodSecurity "baseline:latest":
  non-default capabilities (containers "speaker", "frr" must not include
  "NET_ADMIN", "NET_RAW", "SYS_ADMIN")
  host namespaces (hostNetwork=true)

The fix was labeling the namespace as privileged:

kubectl label namespace metallb-system \
  pod-security.kubernetes.io/enforce=privileged --overwrite

The speaker started. Then the FRR (Free Range Routing) container went into CrashLoopBackOff:

ZEBRA: NB_OP_CHANGE: oper_walk_done: ERROR:
  Error notifying for datastore path:
  /frr-interface:lib/interface[name="bond0"]/state: generic error

FRR was looking for a bond0 interface that doesn't exist on Talos. MetalLB defaults to FRR mode for BGP routing, but I only needed L2 mode. Reinstalling with --set speaker.frr.enabled=false timed out again.

At this point I'd spent an hour fighting a tool that should have taken five minutes. The question became obvious: why am I maintaining three networking tools when one can do it all?

Enter Cilium

Cilium replaces all three components. It handles CNI (pod networking), replaces kube-proxy with eBPF (faster than iptables), and provides L2 load balancing. One Helm chart. One set of values. Here's what the migration looked like.

Step 1: Remove the Old Stack

# Remove MetalLB
helm uninstall metallb -n metallb-system
kubectl delete namespace metallb-system

# Tell Talos to stop managing Flannel and kube-proxy
talosctl patch machineconfig --nodes 192.168.50.10 --patch '
cluster:
  network:
    cni:
      name: none
  proxy:
    disabled: true
'

# Clean out the old pods
kubectl delete daemonset kube-flannel -n kube-system
kubectl delete daemonset kube-proxy -n kube-system

Setting cni.name: none tells Talos "don't install any CNI. I'll bring my own." Setting proxy.disabled: true tells Talos "don't run kube-proxy. Cilium handles service routing."

Step 2: Install Cilium (First Attempt: Failed)

cilium install \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=192.168.50.10 \
  --set k8sServicePort=6443 \
  --set l2announcements.enabled=true \
  --set externalIPs.enabled=true \
  --set devices=enp55s0u1

The operator and envoy came up fine, but the Cilium agent crashed with the same class of error as MetalLB. Container capabilities blocked by Talos:

error during container init: unable to apply caps:
  can't apply capabilities: operation not permitted

Here's the thing about Talos that will bite you repeatedly: labeling a namespace as privileged fixes the Kubernetes PodSecurity admission check, but Talos's containerd configuration has its own restrictions at the runtime level. You must explicitly declare every Linux capability a container needs.

Step 3: Install Cilium (The Fix)

cilium install \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=192.168.50.10 \
  --set k8sServicePort=6443 \
  --set l2announcements.enabled=true \
  --set externalIPs.enabled=true \
  --set devices=enp55s0u1 \
  --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
  --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
  --set cgroup.autoMount.enabled=false \
  --set cgroup.hostRoot=/sys/fs/cgroup \
  --set ipam.mode=kubernetes

The Talos-specific flags that matter:

• securityContext.capabilities.ciliumAgent: every Linux capability the Cilium agent needs, explicitly listed. Normal Linux distros grant these automatically for privileged pods. Talos requires them declared.
• securityContext.capabilities.cleanCiliumState: same for the init container that was crashing.
• cgroup.autoMount.enabled=false: Talos mounts cgroups its own way. Telling Cilium "don't try to mount cgroups yourself" avoids conflicts.
• cgroup.hostRoot=/sys/fs/cgroup: where Talos put the cgroup filesystem.

Everything came up green in about 30 seconds:

    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    OK
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

Step 4: Configure L2 Load Balancing

Cilium's equivalent of MetalLB's IPAddressPool + L2Advertisement is two CRDs:

apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
  name: servers-vlan-pool
spec:
  blocks:
    - start: 192.168.50.200
      stop: 192.168.50.250
---
apiVersion: cilium.io/v2alpha1
kind: CiliumL2AnnouncementPolicy
metadata:
  name: l2-policy
spec:
  interfaces:
    - enp55s0u1
  externalIPs: true
  loadBalancerIPs: true

The IP pool gives Cilium 51 addresses to hand out. The L2 announcement policy tells Cilium to respond to ARP requests for those IPs on the physical interface. This is how other devices on the LAN discover where to send traffic.

Step 5: Verify End-to-End

# Create a test deployment + LoadBalancer service
kubectl create deployment test-lb --image=nginx --replicas=1
kubectl expose deployment test-lb --port=80 --type=LoadBalancer

# Check for external IP
kubectl get svc test-lb
# NAME      TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)
# test-lb   LoadBalancer   10.98.228.214   192.168.50.200   80:32699/TCP

# Hit it from macOS
curl http://192.168.50.200
# Welcome to nginx!

192.168.50.200: first IP from the pool, assigned instantly, reachable from the Mac over the LAN.

The Scorecard

What I Learned

If you're running Talos, just use Cilium from the start. MetalLB + PodSecurity + FRR on an immutable OS is a debugging trifecta you don't want. Cilium replaces Flannel + kube-proxy + MetalLB in one component, and once you have the Talos-specific capability declarations right, it just works.

Talos restricts capabilities at the runtime level, not just admission. Labeling a namespace as privileged fixes the Kubernetes PodSecurity check, but Talos's containerd config has its own restrictions. You must explicitly declare capabilities in Helm values for any pod that needs elevated permissions.

Keep a copy of the Talos-specific Cilium values. You'll need them every time you upgrade. The critical ones: explicit capability lists, cgroup.autoMount.enabled=false, cgroup.hostRoot=/sys/fs/cgroup, and ipam.mode=kubernetes.

The counter-argument to "just use Cilium from the start" is that fighting MetalLB first taught me why Cilium is the better choice. I understand PodSecurity enforcement, namespace labels, FRR mode, and container capability restrictions because I hit every one of those walls. Sometimes the scenic route is the educational one.

But next time, I'm taking the highway.