Migrating PVC-Bound Workloads Across Kubernetes Nodes

Where We Left Off

In the previous post, I joined an Intel NUC (32 GB RAM) to my single-node Talos cluster as a worker node. Stateless workloads like Authentik server and worker migrated to the NUC immediately via soft node affinity. The Dell dropped from 92% to 64% memory. Victory.

Except seven PVC-bound workloads refused to move. Prometheus, Grafana, Ghost, Loki, Life Hub, Audiobookshelf, Linkding: all pinned to the Dell despite having preferredDuringSchedulingIgnoredDuringExecution affinity for the NUC. The data was replicated across both nodes (all 17 Longhorn volumes at 2 replicas), but the scheduler would not place pods on the NUC. The reason turned out to be one of those sharp edges that Longhorn's documentation mentions in passing but does not emphasize enough.

The Problem: Immutable PV Node Affinity

When Longhorn provisions a PersistentVolume via CSI, it stamps a required node affinity on the PV object matching the node where the volume was originally created. Here is what that looks like:

# kubectl get pv pvc-a1b2c3d4 -o yaml (excerpt)
spec:
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - talos-1ue-4vm    # Dell — stamped at creation, immutable

This field is immutable on bound PVs. You cannot kubectl edit it. You cannot patch it. The Kubernetes scheduler sees required node affinity and refuses to place the pod anywhere else, regardless of what your Deployment's preferredDuringScheduling says. Pod affinity is a suggestion. PV node affinity is a hard constraint.

The data is physically on both nodes (Longhorn replicas), but the scheduler only looks at the PV spec, not the data distribution. This is by design. Longhorn needs to know which node should attach the volume. But the practical effect is that "add soft affinity and let the scheduler figure it out" does not work for existing volumes.

To move these workloads, the PVs had to be destroyed and recreated. Which meant I needed a plan that would not lose data.

The Fix: WaitForFirstConsumer

Before migrating anything, I needed to fix how new PVCs would be provisioned. The default Longhorn StorageClass uses volumeBindingMode: Immediate: the PVC binds and provisions the moment it is created, regardless of which node the pod will eventually run on. On a single-node cluster, this does not matter. On a multi-node cluster with affinity-based scheduling, it means new PVCs might provision on the wrong node.

The fix is WaitForFirstConsumer. With this mode, the PVC stays in Pending state until a pod actually mounts it. At that point, Kubernetes has already scheduled the pod (honoring affinity preferences), and Longhorn provisions the volume on whatever node the scheduler chose. Combined with soft affinity preferring the NUC, new PVCs now land on the NUC automatically.

The Longhorn HelmRelease change is two lines:

# infrastructure/longhorn/helmrelease.yaml (excerpt)
persistence:
  defaultClassReplicaCount: 2
  volumeBindingMode: WaitForFirstConsumer

There is a catch: volumeBindingMode is immutable on an existing StorageClass. Kubernetes will not let you update it in place. The StorageClass has to be deleted and recreated.

# volumeBindingMode is immutable — must delete SC before Helm can recreate
kubectl delete storageclass longhorn

# Force Flux to reconcile — Helm recreates SC with WaitForFirstConsumer
flux reconcile helmrelease longhorn -n longhorn-system --with-source

This sounds scary, but it is safe. Existing bound PVCs are unaffected by StorageClass deletion: they already have their PV and do not reference the SC at runtime. Only new PVCs use the StorageClass, and once Flux recreates it, they get the new binding mode.

The Migration Procedure

With WaitForFirstConsumer in place, the migration for each app follows the same pattern:

1. Backup: pipe data out of the running pod to local storage
2. Scale to zero: stop the deployment
3. Delete PVC and PV: remove the old volume with its Dell node affinity
4. Scale to one: Flux reconciles the PVC (it is declared in the kustomization), WaitForFirstConsumer keeps it Pending
5. Pod schedules to NUC: soft affinity places it there, PVC binds and provisions on NUC
6. Restore: pipe backup data back in
7. Verify: check the app works, confirm PV node affinity now says nuc-worker-01

Budget 5-10 minutes per app. Not glamorous, but reliable.

Backup via kubectl exec Pipe

My first instinct was to spin up a temporary pod with an NFS volume mount to write backups directly to Unraid. That failed immediately: Pod Security Standards in restricted mode block NFS volumes (they require privileged or baseline). I could have relaxed the PSS, but that felt like the wrong move.

The simpler approach: kubectl exec to pipe a tar archive from the running pod straight to local disk. No special pods, no PSS exceptions, no NFS mounts.

# Backup: pipe tar from the running pod to local storage
kubectl exec -n ghost deploy/ghost -- \
  tar czf - -C /var/lib/ghost/content . > /tmp/ghost-backup.tar.gz

This works for any app with a single data directory. The pod is still running, so the data is consistent (for SQLite databases, the WAL is flushed on read). For PostgreSQL, you need pg_dump instead. More on that later.

Restore via kubectl exec Pipe

After the PVC is recreated and the pod is running on the NUC with a fresh empty volume:

# Restore: pipe tar back into the new PVC
kubectl exec -i -n ghost deploy/ghost -- \
  tar xzf - -C /var/lib/ghost/content < /tmp/ghost-backup.tar.gz

# Restart to pick up restored data
kubectl rollout restart deployment/ghost -n ghost

Note the -i flag on restore: it connects stdin so the tar archive streams into the pod.

Clearing Stuck PV Finalizers

When you delete a PVC, Kubernetes deletes the associated PV (assuming reclaimPolicy: Delete). Longhorn's controller handles the actual volume cleanup via a finalizer. Normally this takes a few seconds. Sometimes it hangs: the Longhorn controller is busy, or the volume is in a transitional state.

If a PV deletion hangs for more than 30 seconds:

# Clear the finalizer — PV deletes immediately
kubectl patch pv <pv-name> --type json \
  -p '[{"op":"remove","path":"/metadata/finalizers"}]'

Longhorn will garbage-collect the orphaned volume data on its own. I hit this twice across 13 migrations. Not common, but good to know.

Phase 1: Five PVC-Bound Apps

I started with five apps that use simple file-based storage (SQLite databases, flat files, config directories):

All five followed the exact same procedure: backup, scale down, delete PVC, scale up, wait for NUC scheduling, restore, verify. The pattern became mechanical after the second app. The only variable is the data directory path and the PVC name.

Ghost took the longest because the content directory is the largest (SQLite database, uploaded images, and installed themes add up to several hundred megabytes). The others were trivially small: Linkding's bookmarks database is under 10 MB. After Phase 1, the Dell dropped from ~68% to ~59% memory, and the NUC climbed from ~2% to ~9%.

One thing I tracked carefully: after each migration, I verified the new PV's node affinity with kubectl get pv -o yaml. Every recreated volume showed nuc-worker-01 in its nodeSelectorTerms. The WaitForFirstConsumer change was working exactly as intended: the scheduler picked the NUC (soft affinity), then Longhorn provisioned the volume there.

Phase 2: Eight More Workloads

With Phase 1 proven out, I moved on to the remaining workloads: a mix of stateless apps, SQLite-backed services, and one PostgreSQL database.

Stateless Apps: Homepage and IT-Tools

These have no PVCs at all. Adding soft affinity to the deployment and deleting the pod is the entire migration. The scheduler recreates the pod on the NUC within seconds. Thirty seconds of work per app.

SQLite Apps: Uptime Kuma, AutoKuma, ntfy, Actual Budget

Same backup/delete/recreate procedure as Phase 1. Nothing new here. The pattern is mechanical at this point. AutoKuma (the CRD-based monitor manager for Uptime Kuma) has its own small PVC for state tracking. Uptime Kuma is the most visible service in the cluster (it monitors everything else), so I migrated it last in this batch to minimize the monitoring gap.

Vaultwarden: The High-Stakes Migration

Vaultwarden is the password manager. Its data directory contains db.sqlite3 (the vault database) and rsa_key.pem / rsa_key.pub.pem (the RSA keypair used to encrypt vault entries). Lose either of these and every password is gone.

The procedure was the same as every other SQLite app, but the verification step was more thorough:

# Backup
kubectl exec -n vaultwarden deploy/vaultwarden -- \
  tar czf - -C /data . > /tmp/vaultwarden-backup.tar.gz

# Verify the backup contains the critical files BEFORE deleting anything
tar tzf /tmp/vaultwarden-backup.tar.gz | grep -E '(db.sqlite3|rsa_key)'
# ./db.sqlite3
# ./rsa_key.pem
# ./rsa_key.pub.pem

# ... (scale down, delete PVC, scale up, restore) ...

# Post-restore: verify files exist and vault is accessible
kubectl exec -n vaultwarden deploy/vaultwarden -- ls -la /data/db.sqlite3 /data/rsa_key.pem
# Then log in via the web UI and confirm entries are intact

You do not rush the password manager migration. Verify the backup before you destroy anything. Verify the restore before you declare success. This is the difference between engineering and hoping.

Miniflux + PostgreSQL: pg_dump, Not tar

Miniflux is the one app in the cluster backed by PostgreSQL instead of SQLite. You cannot just tar up a running Postgres data directory. The result is likely corrupt. PostgreSQL has its own backup tool.

# Backup: pg_dump produces a clean SQL dump
kubectl exec -n miniflux deploy/miniflux-postgres -- \
  pg_dump -U miniflux miniflux | gzip > /tmp/miniflux-backup.sql.gz

# ... (scale down both miniflux and miniflux-postgres, delete PVCs, scale up) ...

# Restore: pipe the SQL dump back into the fresh database
gunzip -c /tmp/miniflux-backup.sql.gz | \
  kubectl exec -i -n miniflux deploy/miniflux-postgres -- \
  psql -U miniflux -d miniflux

Note the difference: pg_dump produces a logical backup (SQL statements), not a physical backup (raw files). The dump is portable across nodes, Postgres versions (within major version), and even storage backends. It is the only correct way to migrate a running PostgreSQL database.

Miniflux also has a separate PVC for its daily pg_dump CronJob backups. That PVC needed the same delete/recreate treatment, but since it just holds compressed SQL files, a simple tar backup worked fine.

The key with Miniflux is ordering: scale down Miniflux (the application) first, then the Postgres deployment. On restore, bring Postgres up first, run the psql restore, then scale Miniflux back up. Miniflux will not start if its database is empty: it expects the schema to exist.

Re-enabling Mealie and Stirling PDF

Before the NUC joined the cluster, I had disabled Mealie and Stirling PDF to free memory on the Dell. With 30 GB of headroom on the NUC, I re-enabled both by adding them back to apps/kustomization.yaml.

Stirling PDF came up cleanly. Mealie did not.

The Mealie Crash Loop

Mealie crash-looped on first deploy. The pod would start, run for about 25 seconds, get killed by the liveness probe, restart, and repeat forever. The logs told the story: on a cold start, Mealie downloads NLTK language data, runs database migrations, and builds search indexes. This takes 60-90 seconds, well past the 30-second liveness probe deadline I had originally configured.

Kubernetes was killing the pod before it finished starting, which triggered a restart, which started the download again, which got killed again. Classic liveness probe misconfiguration.

# BEFORE: 30s initialDelay — too short for cold start
livenessProbe:
  httpGet:
    path: /api/app/about
    port: 9000
  initialDelaySeconds: 30
  periodSeconds: 10

# AFTER: 90s initialDelay — accounts for NLTK download + DB init
livenessProbe:
  httpGet:
    path: /api/app/about
    port: 9000
  initialDelaySeconds: 90    # NLTK download + DB init takes 60-90s on cold start
  periodSeconds: 10
readinessProbe:
  httpGet:
    path: /api/app/about
    port: 9000
  initialDelaySeconds: 30
  periodSeconds: 5

The lesson: liveness probe initialDelaySeconds must account for worst-case cold start behavior, not average steady-state startup. An app that normally starts in 5 seconds but occasionally needs 90 seconds for a first-time initialization will crash-loop until you accommodate that worst case. A startup probe would also solve this, but bumping the initial delay is simpler for apps with a known cold-start ceiling.

The Result

After both phases, every application workload runs on the NUC. The Dell is a lean control plane.

Seventeen Longhorn volumes, all healthy with 2 replicas across both nodes. Every PV now has nuc-worker-01 in its node affinity. The architecture that emerged:

• Dell: true control plane: kube-apiserver, etcd, Cilium, Longhorn manager, Traefik, Authentik, Prometheus, Grafana, Loki, cert-manager, Flux, Cloudflare tunnel. Infrastructure only.
• NUC: all application workloads: Ghost, Miniflux, Navidrome, Audiobookshelf, Vaultwarden, Uptime Kuma, ntfy, Actual Budget, Linkding, Homepage, IT-Tools, Mealie, Stirling PDF, Life Hub, AutoKuma.

Soft affinity (weight 80, preferredDuringSchedulingIgnoredDuringExecution) means this is a preference, not a mandate. If the NUC goes down, every workload falls back to the Dell. The cluster degrades gracefully: slower, memory-tight, but functional. The Slumlord would hardcode everything to one node and call it a day. The Architect builds a system that survives a hardware failure at 2 AM without a phone call.

What I Learned

Longhorn PV node affinity is the sharpest edge in multi-node Longhorn. It is by design, it is immutable, and it will silently prevent pod migration. If you are adding a node to an existing Longhorn cluster, plan for this before you start. The backup/delete/recreate procedure is straightforward but entirely manual. There is no longhorn migrate-volume command.

Change volumeBindingMode to WaitForFirstConsumer before you need it. On a single-node cluster, Immediate is fine because there is only one node to provision on. The moment you add a second node, you want the scheduler to pick the node first and Longhorn to follow. Changing the binding mode retroactively requires deleting and recreating the StorageClass, which is safe but feels wrong.

kubectl exec pipe is better than dedicated backup pods. It works with any PSS level, it does not require additional RBAC or PVC mounts, and it is a single command. For homelabs, this is the right tool. At enterprise scale, you would use Velero or Longhorn's built-in backup-to-S3. For 13 volumes on a Saturday afternoon, a tar pipe is perfect.

PostgreSQL needs pg_dump, full stop. Do not tar a running Postgres data directory. The result is a pile of WAL segments and half-written pages that will not recover cleanly. Use the tool the database gives you.

Liveness probes must account for worst-case cold start. Mealie's 90-second NLTK download is not a bug. It is a legitimate first-time initialization. The bug was my 30-second liveness probe that did not account for it. Every app has a cold start ceiling. Know yours.

Verify your critical backups before destroying anything. Listing the contents of the Vaultwarden tar before deleting the PVC took five seconds. Not doing it risks every password in the vault. The Architect verifies. The Slumlord yolos.

What's Next

With workloads distributed across two nodes and 17 healthy Longhorn volumes replicated across both, the cluster is in a good place. But good infrastructure is invisible only when you can see what it is doing. How do I know the Dell is at 59% and the NUC is at 15%? How do I know all 17 volumes are healthy? How do I get woken up at 2 AM if a volume degrades or a pod starts crash-looping?

The next post covers how I built the observability stack that watches all of this: Prometheus for metrics, Loki and Alloy for log aggregation, Alertmanager piped to ntfy for notifications on my phone, and a custom Grafana dashboard with per-node breakdowns so I know which node is under pressure before something breaks.