Subscribe

Building the Platform Layer: Traefik, Longhorn, and cert-manager on Talos

Building the infrastructure between 'I have a cluster' and 'I can deploy services': ingress routing, persistent storage, and automatic TLS on Talos Linux.

Building the Platform Layer: Traefik, Longhorn, and cert-manager on Talos

With Cilium handling networking and load balancing, the next step was building the platform layer: the infrastructure that sits between "I have a Kubernetes cluster" and "I can deploy services." Three components make up this layer: Traefik for ingress routing, Longhorn for persistent storage, and cert-manager for automatic TLS certificates.

Traefik: One IP to Rule Them All

Without an ingress controller, every service needs its own LoadBalancer IP. With 51 IPs in the Cilium pool and 15+ services planned, you'd burn through IPs fast. More importantly, you'd need to remember which IP goes to which service.

Traefik solves this. One IP handles everything. DNS records point subdomains to Traefik's IP, and Traefik routes based on the hostname: uptime.homelab.local goes to Uptime Kuma, grafana.homelab.local goes to Grafana, and so on.

helm install traefik traefik/traefik \
  --namespace traefik \
  --create-namespace \
  --set service.type=LoadBalancer \
  --set providers.kubernetesCRD.enabled=true \
  --set providers.kubernetesCRD.allowCrossNamespace=true \
  --set providers.kubernetesIngress.enabled=true

The key flag is allowCrossNamespace=true. Without it, Traefik can only route to services in its own namespace. Since every app runs in its own namespace (good practice), Traefik needs to reach across all of them.

kubectl get svc -n traefik
# NAME      TYPE           EXTERNAL-IP      PORT(S)
# traefik   LoadBalancer   192.168.50.200   80:32522/TCP,443:32511/TCP

Traefik grabbed 192.168.50.200 and is listening on both HTTP and HTTPS. A curl to that IP returns 404. Correct, because no routes are configured yet.

Longhorn: The iSCSI Detour

Longhorn provides persistent storage: the thing that makes databases survive pod restarts. Without it, every PersistentVolumeClaim stays in Pending forever because there's no StorageClass to provision volumes.

My first install attempt failed immediately:

Error starting manager: failed to check environment,
  please make sure you have iscsiadm/open-iscsi installed on the host

Longhorn uses iSCSI to attach storage volumes to pods. On Ubuntu, you'd apt install open-iscsi. On Talos, you can't install anything: the OS is immutable. System extensions get compiled into the OS image itself.

The Talos Image Factory Upgrade

The process for adding iSCSI support to Talos:

  1. Go to Talos Image Factory
  2. Select your Talos version (must match exactly), architecture, and add the siderolabs/iscsi-tools extension
  3. The factory generates a custom installer image with a schematic ID
  4. Upgrade your node to that image
talosctl upgrade \
  --nodes 192.168.50.10 \
  --image factory.talos.dev/installer/c9078f94....:v1.12.1 \
  --preserve --wait --force

The --preserve flag keeps your data. The --force was needed because etcd's advertised URL was still pointing at an old IP from a previous VLAN migration. The health check failed even though the node was healthy.

After the reboot, iSCSI tools were installed:

talosctl get extensions --nodes 192.168.50.10
# ID   NAME          VERSION
# 0    iscsi-tools   v0.2.0

Post-Reboot: The Taint Problem

After the upgrade reboot, every pod except Cilium was stuck in Pending. The cause: the control-plane NoSchedule taint came back, plus two transient taints from the upgrade.

On a single-node cluster, the control-plane taint blocks everything that doesn't explicitly tolerate it. The permanent fix:

talosctl patch machineconfig --nodes 192.168.50.10 --patch '
cluster:
  allowSchedulingOnControlPlanes: true
'

This should be in your machine config from day one on a single-node cluster. Without it, every reboot requires manually removing the taint before anything can schedule.

Longhorn: Second Attempt

With iSCSI tools installed, Longhorn's manager pod came up. But 12 CSI pods (csi-attacher, csi-provisioner, csi-resizer, csi-snapshotter) were all Pending:

0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector.

I'd set a node selector during install (node.kubernetes.io/instance-type: talos) but never labeled the node. Quick fix:

kubectl label node talos-1ue-4vm node.kubernetes.io/instance-type=talos

All 12 pods started immediately. Longhorn became the default StorageClass, backed by the 512 GB NVMe.

cert-manager: Automatic TLS

Every HTTPS service needs a TLS certificate. Without automation, that means manually generating, submitting, proving domain ownership, installing, and renewing every 90 days. cert-manager does all of it through Kubernetes-native resources.

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true \
  --set crds.keep=true

I created three ClusterIssuers, each representing a different way to get certificates:

# Self-signed — works immediately, great for .local domains
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}
---
# Let's Encrypt staging — rate-limit-friendly testing
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-staging-key
    solvers:
      - http01:
          ingress:
            ingressClassName: traefik
---
# Let's Encrypt production — real browser-trusted certs
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
      - http01:
          ingress:
            ingressClassName: traefik

For a homelab on the LAN, selfsigned is fine: the connection is encrypted, your browser just shows a warning because it doesn't recognize the CA. Real Let's Encrypt certificates come later with Cloudflare Tunnel and DNS-01 validation.

The Proof: Deploying Uptime Kuma

Uptime Kuma was the first real service, chosen because it touches every layer of the stack. If it works, everything works.

One set of manifests creates everything: namespace, persistent storage, deployment, service, and Ingress with TLS.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: uptime-kuma-data
  namespace: monitoring
spec:
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: uptime-kuma
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: uptime-kuma
  template:
    metadata:
      labels:
        app: uptime-kuma
    spec:
      containers:
        - name: uptime-kuma
          image: louislam/uptime-kuma:1.23.17
          ports:
            - containerPort: 3001
          volumeMounts:
            - name: data
              mountPath: /app/data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: uptime-kuma-data
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: uptime-kuma
  namespace: monitoring
  annotations:
    cert-manager.io/cluster-issuer: selfsigned
spec:
  ingressClassName: traefik
  tls:
    - hosts: [uptime.homelab.local]
      secretName: uptime-kuma-tls
  rules:
    - host: uptime.homelab.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: uptime-kuma
                port:
                  number: 80

Everything came up on the first try. Pod running, PVC bound to a Longhorn volume, Ingress routed through Traefik with a self-signed TLS cert.

When you hit https://uptime.homelab.local, here's the full chain:

  1. DNS: /etc/hosts resolves the hostname to 192.168.50.200
  2. Cilium L2: ARP resolution finds the Dell's MAC address
  3. Traefik: matches the hostname to the Ingress rule
  4. cert-manager: generated the TLS cert that encrypts the connection
  5. Kubernetes Service: routes to the Uptime Kuma pod
  6. Longhorn: provides the persistent volume via iSCSI on the NVMe

Seven components working together, installed from scratch in one evening. The platform layer was done.

Gotchas Worth Remembering

Check talosctl get extensions before installing storage providers. Longhorn needs iscsi-tools. If it's not there, you need a Talos Image Factory upgrade. Plan for a reboot.

PostgreSQL on Longhorn volumes needs a PGDATA subdirectory. Longhorn creates a lost+found directory in new volumes. PostgreSQL refuses to initialize in a non-empty directory. Fix: PGDATA=/var/lib/postgresql/data/pgdata.

Always verify node labels match your selectors. Setting a nodeSelector in Helm values without labeling the node results in pods stuck in Pending with no obvious error until you read the events.

allowSchedulingOnControlPlanes: true is essential for single-node. Without it, every Talos reboot re-applies the control-plane taint and nothing can schedule.