Subscribe

Adding a Worker Node to a Single-Node Talos Kubernetes Cluster

How I joined an Intel NUC as a worker node to a single-node Talos Linux cluster: secrets extraction, Cilium fixes, Longhorn HA, and the gotchas you won't find in the docs.

Adding a Worker Node to a Single-Node Talos Kubernetes Cluster

My homelab started as a single Dell laptop running Talos Linux, 15.6 GB of RAM, one node doing everything: control plane, workloads, storage. It worked, but at 77% memory overcommit and ~59% actual usage, I was one ambitious deployment away from OOM kills. Time to add a second node.

I picked up an Intel NUC with 32 GB RAM, roughly tripling my cluster's available memory. Here's how I joined it as a worker node to an existing single-node Talos cluster, entirely via the command line.

The Starting Point

  • Existing cluster: Single Dell laptop, Talos Linux v1.12.1, Kubernetes v1.35.0
  • GitOps: Flux reconciling from a GitHub repo every 10 minutes
  • CNI: Cilium with kube-proxy replacement, VXLAN tunnel mode, L2 announcements
  • Storage: Longhorn (single replica, because single node)
  • New hardware: Intel NUC, 32 GB RAM, 512 GB NVMe

The goal: join the NUC as a worker, steer heavy workloads to it with soft affinity, enable Longhorn HA with 2 replicas across nodes, and keep the Dell as the sole control plane.

Phase 1: Extracting Cluster Secrets

Talos clusters are built from a secrets.yaml: the root of trust containing CA certificates and bootstrap tokens. If you still have yours from the initial cluster creation, great. I didn't, so I extracted it from the running control plane.

# Extract the machine config from the running control plane
talosctl -n 192.168.50.10 get machineconfig -o yaml \
  | python3 -c "
import sys, yaml
docs = list(yaml.safe_load_all(sys.stdin))
with open('/tmp/running-controlplane.yaml', 'w') as f:
    f.write(docs[0]['spec'])
"

# Generate a secrets bundle from the running config
talosctl gen secrets \
  --from-controlplane-config /tmp/running-controlplane.yaml \
  -o ~/.talos/secrets.yaml

Keep secrets.yaml safe: this is your cluster's root of trust. Anyone with this file can generate valid node configs.

Phase 2: Building the Installer Image

Talos is immutable: you can't install packages after boot. Extensions like iscsi-tools (required for Longhorn) need to be baked into the installer image. The Talos Image Factory makes this easy.

# Define the extensions you need
cat > /tmp/worker-extensions.yaml <<'EOF'
customization:
  systemExtensions:
    officialExtensions:
      - siderolabs/iscsi-tools
      - siderolabs/util-linux-tools
EOF

# Submit to Image Factory — returns a schematic ID
curl -sS -X POST --data-binary @/tmp/worker-extensions.yaml \
  https://factory.talos.dev/schematics
# {"id":"613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245"}

The schematic ID is a content-addressed hash of your extension set. Use it to download a boot ISO or reference the installer image directly in your machine config:

factory.talos.dev/installer/613e1592...61245:v1.12.1

I flashed the ISO to a USB drive and booted the NUC from it. Talos enters maintenance mode: no OS installed yet, just an API endpoint waiting for configuration.

Phase 3: Identifying the Hardware

With the NUC in maintenance mode (DHCP IP: 192.168.1.215), I queried its hardware:

# Find the NIC name
talosctl get links --insecure --nodes 192.168.1.215
# Result: eno1 (up, ether, 1c:69:7a:08:f9:4d)

# Find the disk
talosctl get disks --insecure --nodes 192.168.1.215
# Result: /dev/nvme0n1 (512 GB, PCIe SSD)

Two key values: NIC is eno1, disk is /dev/nvme0n1. These go into the machine config patch.

Phase 4: Generating the Worker Config

Talos machine configs are generated with talosctl gen config. I wrote a patch file with NUC-specific settings:

# patches/worker-nuc.yaml
machine:
  network:
    hostname: nuc-worker-01
    interfaces:
      - interface: eno1
        dhcp: false
        addresses:
          - 192.168.50.11/24
        routes:
          - network: 0.0.0.0/0
            gateway: 192.168.50.1
    nameservers:
      - 192.168.1.1
  install:
    disk: /dev/nvme0n1
    image: factory.talos.dev/installer/613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245:v1.12.1
  kubelet:
    extraMounts:
      - destination: /var/lib/longhorn
        type: bind
        source: /var/lib/longhorn
        options: [bind, rshared, rw]
  kernel:
    modules:
      - name: nbd
      - name: iscsi_tcp
  nodeLabels:
    node.kubernetes.io/instance-type: talos
    homelab.naviauxlab.com/workload-tier: primary

Key decisions:

  • Static IP on the servers VLAN (192.168.50.0/24), same subnet as the Dell
  • Longhorn mount at /var/lib/longhorn, required for Longhorn to manage disks
  • Kernel modules nbd and iscsi_tcp: Longhorn's iSCSI backend
  • Node labels: workload-tier: primary for steering heavy workloads here later

Then generated the full worker config:

talosctl gen config homelab https://192.168.50.10:6443 \
  --with-secrets ~/.talos/secrets.yaml \
  --talos-version v1.12.1 \
  --kubernetes-version 1.35.0 \
  --output-types worker \
  --config-patch-worker @patches/worker-nuc.yaml \
  --output nuc-worker-01.yaml

The HostnameConfig Gotcha

talosctl gen config appends a second YAML document: a HostnameConfig with auto: stable. This conflicts with machine.network.hostname in the main config, causing a validation error:

configuration validation failed: static hostname is already set in v1alpha1 config

Fix: Strip the second document before applying:

python3 -c "
with open('nuc-worker-01.yaml') as f:
    content = f.read()
parts = content.split('\n---\n')
with open('nuc-worker-01.yaml', 'w') as f:
    f.write(parts[0] + '\n')
"

Phase 5: Joining the Cluster

One command:

talosctl apply-config --insecure \
  --nodes 192.168.1.215 \
  --file nuc-worker-01.yaml

The NUC partitions the NVMe, downloads the installer image from Image Factory, installs Talos, and reboots. After ~3 minutes, it comes up at 192.168.50.11 and kubelet registers with the control plane.

$ kubectl get nodes -o wide
NAME            STATUS   ROLES           AGE   VERSION   INTERNAL-IP     ...
nuc-worker-01   Ready    <none>          48s   v1.35.0   192.168.50.11
talos-1ue-4vm   Ready    control-plane   47d   v1.35.0   192.168.50.10

Don't forget to update your talosctl config to include both endpoints:

talosctl config endpoint 192.168.50.10 192.168.50.11
talosctl config node 192.168.50.10 192.168.50.11

Phase 6: Fixing Cilium

The NUC's Cilium pod immediately CrashLooped:

unable to determine direct routing device. Use --direct-routing-device to specify it

The issue: Cilium was deployed with devices: enp55s0u1 (the Dell's NIC name). The NUC has eno1. Cilium couldn't find the device and refused to start.

Since Cilium is Helm-managed in my cluster (installed during Talos bootstrap), the fix was a Helm upgrade:

helm upgrade cilium cilium/cilium -n kube-system \
  --version 1.18.5 \
  --reuse-values \
  --set "devices=en+"

en+ is a Cilium device wildcard that matches any interface starting with en, covering both enp55s0u1 (Dell) and eno1 (NUC). After deleting the crashed pod, the DaemonSet recreated it with the new config.

Phase 7: GitOps Changes

With the node joined, I updated the Git repo so Flux would reconfigure everything for multi-node operation.

Cilium L2 Announcement Policy

The L2 policy had a hardcoded interface name. Changed to a regex:

# Before
spec:
  interfaces:
    - enp55s0u1

# After
spec:
  interfaces:
    - ^enp

Longhorn: HA Replicas

Bumped replica count from 1 to 2 for cross-node redundancy:

defaultReplicaCount: 2      # was 1
csi:
  attacher:
    replicas: 2              # was 1
  provisioner:
    replicas: 2
  resizer:
    replicas: 2
  snapshotter:
    replicas: 2
longhornUI:
  replicas: 2

Existing volumes stay at 1 replica until manually upgraded. Longhorn doesn't retroactively change them.

Workload Affinity

I added soft node affinity to all heavy workloads (>256Mi memory limit), preferring the NUC:

affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 80
        preference:
          matchExpressions:
            - key: homelab.naviauxlab.com/workload-tier
              operator: In
              values: ["primary"]

Why soft (preferred) instead of hard (required)? If the NUC goes down, workloads fall back to the Dell. The cluster degrades gracefully rather than leaving pods unschedulable.

Workloads steered to the NUC: Prometheus (768Mi), Authentik server (768Mi), Authentik worker (512Mi), Grafana (512Mi), Ghost (384Mi), Loki (256Mi), Life Hub (256Mi), Audiobookshelf (256Mi).

The Result

$ kubectl top nodes
NAME            CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
nuc-worker-01   2015m        25%      706Mi           2%
talos-1ue-4vm   1693m        21%      13953Mi         92%

After Flux reconciled and I upgraded all 17 Longhorn volumes to 2 replicas (cross-node replication), then restarted the heavy workloads:

NAME            CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
nuc-worker-01   706m         8%       2109Mi          6%
talos-1ue-4vm   680m         8%       9703Mi          64%

Dell dropped from 92% → 64%. Authentik server + worker (1.3 Gi combined) migrated to the NUC. The NUC still has 30 GB free.

What Scheduled Automatically (DaemonSets)

  • Cilium agent (CNI)
  • Alloy (log collector)
  • node-exporter (metrics)
  • Longhorn manager + CSI plugin

No configuration needed: DaemonSets do their thing.

What Migrated to the NUC

  • Authentik server (768Mi limit), no PVC of its own
  • Authentik worker (512Mi limit), no PVC of its own
  • Longhorn UI: stateless

What Stayed on the Dell (and Why)

Prometheus, Grafana, Ghost, Loki, Life Hub, and Audiobookshelf all stayed on the Dell despite having soft affinity for the NUC. The reason: Longhorn PV node affinity is immutable.

When Longhorn provisions a PV via CSI, it stamps a required node affinity matching the node where the volume was first created. Kubernetes won't schedule a pod to a node that doesn't satisfy the PV's required affinity. And since this field is immutable on bound PVs, you can't just patch it.

The data is replicated to the NUC (all 17 volumes now have 2 replicas), but the scheduler can't place pods there because the PVs say "Dell only."

Options to migrate PVC-bound workloads in the future:

  1. Longhorn backup + restore: back up the volume, create a new PVC (which provisions on the NUC), restore into it
  2. Delete and recreate PVCs: destructive, requires data backup first
  3. New workloads just work: any new PVCs will respect affinity and potentially land on the NUC

For a homelab, this is fine. The memory relief from moving Authentik alone freed 1.3 Gi on the Dell, and new workloads will naturally schedule to the NUC.

What Didn't Go Smoothly

  1. talosctl gen config appends a conflicting HostnameConfig document: had to strip it manually before applying
  2. Cilium devices was hardcoded to the Dell's NIC name: the NUC has a different name (eno1 vs enp55s0u1), causing CrashLoopBackOff until I changed it to a wildcard
  3. Longhorn PV node affinity is immutable: even with data replicated across both nodes, existing PVCs keep pods pinned to the original node

All fixable, but they're the kind of thing you don't find in the docs until you hit them.

Phase 8: Updating the Dashboard

With two nodes, the single-node Grafana dashboard was useless: cluster-wide averages hide which node is under pressure. I rewrote the Homelab Overview dashboard to show per-node breakdowns.

The key PromQL pattern for per-node metrics:

METRIC * on(instance) group_left(nodename) node_uname_info

node_uname_info has value 1 for every node, with a nodename label containing the hostname. Multiplying any instance-keyed metric by it effectively joins in the friendly name. Use legendFormat: "{{nodename}}" and Grafana shows nuc-worker-01 instead of 192.168.50.11:9100.

Changes:

  • CPU, Memory, Disk: replaced single gauges with per-node bar gauges showing both nodes side by side
  • Network I/O, Disk I/O: split by node (nuc-worker-01 RX/TX, talos-1ue-4vm RX/TX)
  • Load Average: per-node stat panel
  • New stat panels: Node count, Total Pods, Longhorn attached volumes
  • Tables: added Node column to PVC Usage and Pod Restarts

The dashboard is a ConfigMap labeled grafana_dashboard: "1". Grafana’s sidecar picks it up automatically. No manual import needed, and it stays in sync via GitOps.

Lessons Learned

  1. Save your secrets.yaml from the initial cluster creation. Extracting it later works, but it's an unnecessary step.
  2. Use device wildcards in Cilium from day one. Hardcoding NIC names works for single-node but breaks the moment you add hardware with different naming.
  3. Soft affinity > hard affinity for homelabs. You want workloads to prefer the beefier node, not require it. Hardware fails.
  4. Longhorn replica counts aren't retroactive. New volumes get the new default, but existing volumes need manual patching.
  5. Longhorn PV node affinity locks volumes to their original node. Data replication doesn't mean pod migration. Plan for this when adding nodes: you may need to backup/restore volumes to truly move workloads.
  6. Talos maintenance mode is your friend. Being able to query hardware (get links, get disks) before committing to a config prevents guessing.
  7. Stateless workloads migrate instantly. If your app doesn't have its own PVC (like Authentik server/worker), soft affinity works perfectly. Design for this: externalize state when possible.