Longhorn Deep Dive: Distributed Storage on Two Nodes

Distributed storage on a two-node cluster is a contradiction that works. You cannot have quorum with two nodes. You cannot survive a simultaneous dual-node failure. You cannot run Ceph or MinIO with any reasonable performance. But you can run Longhorn, and for a homelab, it is enough.

What Longhorn Does

Longhorn is a distributed block storage system for Kubernetes. It turns local disk space on each node into replicated PersistentVolumes. When you create a PVC with storageClassName: longhorn, Longhorn carves out a volume on the local NVMe, creates a replica on the other node, and synchronously writes to both. If a node dies, the surviving replica serves reads and writes immediately.

It is not the fastest storage system. It is not the most feature-rich. But it runs reliably on two nodes with minimal configuration, and that is exactly what a homelab needs.

The Two-Node Reality

My cluster has two nodes: a Dell laptop (control plane + worker) and an Intel NUC (worker). Both have NVMe drives. Longhorn runs on both, with a replica count of 2: one replica per node.

apiVersion: longhorn.io/v1beta2
kind: StorageClass
metadata:
  name: longhorn
parameters:
  numberOfReplicas: "2"
  staleReplicaTimeout: "2880"
  fromBackup: ""
provisioner: driver.longhorn.io
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true

With two replicas on two nodes, every write goes to both NVMe drives synchronously. The write latency is roughly 2x a single-node write (network round-trip to the second node), but for SQLite databases and application configs, this is imperceptible.

The limitation is obvious: if both nodes die simultaneously, both replicas are lost. There is no third node to provide a tiebreaker or a surviving copy. For a homelab, this is an acceptable risk. The NFS backups on Unraid provide the off-cluster recovery path.

How Replication Works

Longhorn uses a replicated block device model. Each volume has an engine (the primary I/O path) and one or more replicas (secondary copies). Writes are synchronous: the engine waits for all healthy replicas to acknowledge before returning success to the pod.

The replication is at the block level, not the file level. Longhorn does not know or care that the volume contains a SQLite database or a PostgreSQL data directory. It replicates raw blocks, which means any filesystem and any application works without modification.

When a node goes offline:

1. Longhorn marks the replica on the failed node as unhealthy
2. The engine continues serving I/O from the surviving replica
3. Kubernetes reschedules the pod to the surviving node (if it was on the failed node)
4. The pod mounts the volume via the surviving replica
5. When the failed node comes back, Longhorn rebuilds the stale replica automatically

Step 5 is where the magic happens. The rebuild is incremental: Longhorn tracks which blocks changed while the replica was offline and only syncs the delta. For a lightly-used homelab, the rebuild typically completes in seconds to minutes.

PVC Migration: The Live Drill

When I added the NUC as a second node, every existing PVC was single-replica on the Dell laptop. Migrating to two replicas was my first real test of Longhorn's operational model.

The migration process for each volume:

1. Update the Longhorn volume's replica count from 1 to 2 via the Longhorn UI or API
2. Longhorn schedules a new replica on the NUC
3. The engine begins copying all blocks from the existing replica to the new one
4. During the copy, the volume remains online: pods continue reading and writing normally
5. Once the copy completes, the volume has two healthy replicas

No downtime. No pod restarts. The migration happened in the background while services continued running. For the larger volumes (Ghost's PostgreSQL data, about 2 GB), the copy took a few minutes. For smaller volumes (Life Hub's SQLite, about 2 MB), it was nearly instant.

The experience convinced me that Longhorn's operational model is solid for homelab use. The live migration capability means adding a node and upgrading storage redundancy is a non-event.

Snapshot Scheduling

Longhorn supports recurring snapshots: point-in-time copies of a volume that can be used for rollback. Snapshots are space-efficient because they use copy-on-write: only blocks that change after the snapshot consume additional space.

apiVersion: longhorn.io/v1beta2
kind: RecurringJob
metadata:
  name: daily-snapshot
  namespace: longhorn-system
spec:
  cron: "0 3 * * *"
  task: snapshot
  groups:
  - default
  retain: 7
  concurrency: 1

Daily snapshots at 3 AM, retaining the last 7. This gives me a week of rollback points for every Longhorn volume. If an application corrupts its data (bad migration, botched update, accidental deletion), I can restore from the most recent clean snapshot.

Snapshots are not backups. They live on the same physical storage as the volume. If the NVMe drive fails, the snapshots are gone too. That is why the NFS backup CronJob exists: it copies data off-cluster to a physically separate machine.

Volume Expansion

The allowVolumeExpansion: true in the StorageClass is essential. I have expanded volumes live (without pod restart) several times:

• Ghost's PostgreSQL PVC: 1Gi to 5Gi when blog content grew
• Life Hub's data PVC: 1Gi to 5Gi when the activity_events table passed 10,000 rows
• Grafana's PVC: 1Gi to 2Gi when Prometheus retention grew

The expansion process: edit the PVC's spec.resources.requests.storage to the new size, and Longhorn handles the rest. The filesystem is expanded online. The pod does not restart. This is one of those Kubernetes features that sounds boring in documentation but is excellent in practice.

Performance Characteristics

Longhorn is not fast. Let me be specific about what that means.

Sequential write throughput on a single-replica volume: approximately 150-200 MB/s (limited by the NVMe and the Longhorn engine overhead).

Sequential write throughput on a two-replica volume: approximately 80-120 MB/s (limited by the network round-trip to the second node).

Random 4K write IOPS: approximately 5,000-8,000 (single replica) and 3,000-5,000 (two replicas).

For comparison, raw NVMe can do 500,000+ IOPS. Longhorn adds significant overhead because every I/O goes through the userspace engine and, for replicated volumes, across the network.

For my workloads (SQLite databases under 100 MB, PostgreSQL with a few thousand rows, application config files), this performance is more than adequate. I have never observed Longhorn as a bottleneck. The collector CronJob ingests data in milliseconds. SQLite queries return in microseconds. The Longhorn overhead is invisible at this scale.

Where Longhorn would struggle: large database workloads (hundreds of GB), high-throughput media transcoding to disk, or any application that needs tens of thousands of IOPS. For those, local-path provisioner (single-node, no replication) or a dedicated NFS mount is more appropriate.

What Happens When a Node Dies

I have tested this both deliberately (draining the NUC for maintenance) and accidentally (the Dell kernel-panicked during a Talos upgrade).

Deliberate drain:

kubectl drain nuc-worker-01 --ignore-daemonsets --delete-emptydir-data

Pods migrate to the Dell. Longhorn marks NUC replicas as unhealthy. All volumes continue operating on Dell replicas. When the NUC comes back: kubectl uncordon nuc-worker-01, pods reschedule, replicas rebuild. Total impact: a few minutes of single-replica operation.

Accidental failure: The Dell's kernel panic during a Talos upgrade was unplanned. The NUC continued running, but since the Dell was the control plane, the Kubernetes API was unavailable. Existing pods on the NUC kept running (kubelet operates independently), but no new scheduling could happen. Once the Dell recovered (Talos rebooted successfully after the panic), the control plane came back, and Longhorn rebuilt the stale replicas.

The key lesson: Longhorn's data plane (I/O to volumes) operates independently of the Kubernetes control plane. Even when the API server is down, volumes that are already mounted continue working. This is an important property for a two-node cluster where the control plane runs on one of the data nodes.

Longhorn vs Alternatives

Rook/Ceph is the production-grade answer, but it requires at least 3 nodes for a functioning CRUSH map and consumes significant CPU and memory for the OSD daemons. On a two-node homelab, it is not viable.

Local-path provisioner is the zero-overhead option. No replication, no snapshots, no live migration: just a directory on the node's local disk. I use this for ephemeral volumes where data loss is acceptable (Prometheus WAL during bootstrapping, temporary build caches).

Longhorn occupies the sweet spot: replication and snapshots without requiring three nodes or heavy resource consumption.

Operational Tips

Set staleReplicaTimeout appropriately. The default is 2880 minutes (48 hours). This is how long Longhorn waits before marking a disconnected replica as permanently failed and removing it. For a homelab where node reboots are common, 48 hours is reasonable. If you set it too low, Longhorn will delete and rebuild replicas on every reboot, wasting I/O.

Monitor replica health. The Longhorn UI (or API) shows replica status for every volume. A volume with one degraded replica works fine but has lost its redundancy. I check this weekly and after every node maintenance.

Do not put databases on NFS. This bears repeating. SQLite and NFS do not mix. NFS's file-locking semantics are incompatible with SQLite's locking protocol. Use Longhorn for databases, NFS for media and backups.

Right-size your PVCs. Start small (1Gi) and expand as needed. Longhorn expansion is online and non-disruptive. Over-provisioning wastes NVMe space that cannot be reclaimed without recreating the volume.

Separate snapshot retention from backup retention. Snapshots are fast rollback points (on the same disk). Backups are disaster recovery (on different hardware). They serve different purposes and should have different retention policies.

Lessons

Two replicas on two nodes is enough for a homelab. You will not survive simultaneous dual-node failure. Accept that, plan for it (NFS backups), and enjoy the benefit of automatic failover for single-node failures, which are the scenarios that actually happen.

Longhorn's operational model is intuitive. Volumes have replicas. Replicas live on nodes. When a node disappears, the replica is rebuilt when it comes back. This is easy to reason about, easy to monitor, and easy to explain to someone else.

Performance is not the bottleneck. For homelab workloads (small databases, application configs, write-ahead logs), Longhorn's performance overhead is invisible. Optimize for reliability and operational simplicity, not throughput.

Live migration is the killer feature. Adding a node, expanding a volume, rebuilding a replica: all of these happen without downtime. For a homelab that you cannot take offline during business hours (because your family uses it), this matters more than raw performance numbers.