Subscribe

Disaster Recovery for a Homelab That Actually Matters

Most homelab DR posts are theoretical. This one covers tested backup pipelines, actual RPO/RTO numbers, and what happens when your nodes die, because mine did.

Disaster Recovery for a Homelab That Actually Matters

Most homelab disaster recovery documentation is theoretical. It lists backup strategies in the abstract, describes what you should do, and stops before the hard part: what actually happens when hardware fails.

I run a two-node Kubernetes cluster. A Dell laptop as the control plane and primary worker. An Intel NUC as the second worker. An Unraid NAS for bulk storage and offsite backups. When I added the NUC, I migrated PVCs live and tested what happens when a node disappears. When the Dell kernel-panicked during a Talos upgrade, I found out which parts of my DR plan actually worked.

This post covers the real DR story: what's backed up, how often, where it goes, and what fails.

The Storage Stack

Understanding DR starts with understanding where data lives.

Longhorn provides replicated block storage across both nodes. Every PVC gets 2 replicas, one on each node. When a node fails, the surviving replica serves reads and writes immediately. No manual failover required.

NFS on Unraid serves two purposes: read-only media libraries (music, audiobooks) and read-write backup targets. The NAS is a separate machine on the same network, not part of the Kubernetes cluster.

SQLite with WAL mode is the database engine for Life Hub. WAL (Write-Ahead Logging) allows concurrent reads during writes and makes point-in-time backups possible without locking the database.

The Backup Architecture

Life Hub has two independent backup systems that overlap intentionally.

System 1: Application-Level Backups (Every 15 Minutes)

The collector CronJob triggers POST /api/v1/backup/run before doing anything else. The backup endpoint:

  1. Runs sqlite3.backup() in a thread executor for a WAL-safe copy
  2. Gzip compresses the copy (1-2 MB → 100-200 KB)
  3. Writes atomically: compress to .tmp, rename to final .db.gz
  4. Prunes old files: keeps 7 daily + 4 weekly (Sundays)
  5. Updates SyncState: source="sqlite-backup", status="ok"
async def create_backup(db_path: str, backup_dir: str) -> dict:
    # WAL-safe backup via sqlite3.backup() in executor
    raw_path = await asyncio.get_event_loop().run_in_executor(
        None, _do_sqlite_backup, db_path, tmp_path
    )
    
    # Atomic gzip compression
    gz_path = raw_path + ".gz"
    with open(raw_path, "rb") as f_in:
        with gzip.open(gz_path + ".tmp", "wb") as f_out:
            shutil.copyfileobj(f_in, f_out)
    os.rename(gz_path + ".tmp", gz_path)  # Atomic rename

The retention policy keeps approximately 11 files at any time: 7 daily backups plus 4 weekly snapshots from Sundays. Total storage: about 2.2 MB. The 10 GB NFS allocation has 9.998 GB of headroom.

System 2: Kubernetes CronJob Backup (Daily at 2:45 AM)

A separate Kubernetes CronJob runs an Alpine container that copies the raw SQLite files (database + WAL + SHM) and tars them to the NFS-backed PVC:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: life-hub-backup
spec:
  schedule: "45 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: alpine:3.19
            command: ["/bin/sh", "-c"]
            args:
            - |
              cp /data/life-hub.db* /tmp/
              tar czf /backups/life-hub_20260303_234052.tar.gz \n                -C /tmp life-hub.db*
              ls -t /backups/life-hub_*.tar.gz | tail -n +8 | xargs rm -v
            volumeMounts:
            - name: data
              mountPath: /data
              readOnly: true
            - name: backups
              mountPath: /backups

This backup writes to an NFS PV on Unraid at /mnt/user/backups. It keeps 7 files and prunes older ones. The readOnly: true mount on the data volume ensures the backup process can't accidentally modify the live database.

Why Two Systems?

The application-level backup runs every 15 minutes but stores files on the same Longhorn volume as the database. If Longhorn fails, both the database and its backups are gone.

The Kubernetes CronJob runs daily but stores files on NFS, which is a physically separate machine. It survives total cluster failure.

Together: 15-minute RPO for most scenarios, daily RPO for total cluster loss, and the NFS copy provides geographic separation from the cluster.

Failure Scenarios

Scenario 1: Worker Node Dies (NUC Fails)

This is the easy one. Longhorn has replicas on both nodes. The Dell laptop still has a complete copy of every volume. Kubernetes reschedules pods to the surviving node. The pod restarts, the readiness probe passes, traffic resumes.

  • RPO: 0 (Longhorn replicates synchronously)
  • RTO: ~2-3 minutes (pod reschedule + readiness probe)
  • Data loss: None
  • Manual intervention: None. Watch it happen in Uptime Kuma.

Scenario 2: Control Plane Dies (Dell Laptop Fails)

This is worse. The Dell runs the Kubernetes control plane (etcd, API server, scheduler). When it dies, the cluster can't schedule new pods, can't process API requests, and Longhorn's control plane components stop running. Existing pods on the NUC keep running until they need to be rescheduled.

  • RPO: ~15 minutes (last collector CronJob backup)
  • RTO: ~30 minutes (rebuild control plane from Talos config, restore from Longhorn snapshot or NFS backup)
  • Data loss: At most 15 minutes of collector data
  • Manual intervention: Rebuild Talos control plane, apply Flux manifests, verify Longhorn volume recovery

I tested a version of this during a Talos upgrade that went wrong. The control plane was unreachable for about 10 minutes. Longhorn volumes came back automatically once the node recovered. No data loss. The 15-minute CronJob picked up where it left off.

Scenario 3: Both Nodes Die (Total Cluster Failure)

This is the scenario that separates real DR from theoretical DR. Both Longhorn replicas are gone. Every in-cluster volume is lost.

What survives:

  • NFS backups on Unraid: The daily tar.gz files from the backup CronJob
  • Git repository: All manifests, all application code, all SOPS-encrypted secrets
  • Container images: GHCR (GitHub Container Registry) has every built image

Recovery steps:

  1. Rebuild the Talos cluster from controlplane.yaml + worker.yaml (stored in Git, secrets encrypted with SOPS + Age)
  2. Bootstrap Flux, which reconciles all manifests from Git
  3. Restore Life Hub database from NFS: tar xzf life-hub_latest.tar.gz -C /data/
  4. Collectors resume on next CronJob cycle
  • RPO: ~24 hours (daily NFS backup) for Life Hub data, zero for infrastructure (Git)
  • RTO: ~1 hour (full cluster rebuild + app restoration)
  • Data loss: Up to 24 hours of Life Hub data. All infrastructure config recoverable from Git.

Scenario 4: Unraid NAS Dies

The backup target is gone, but the live cluster is fine. Media libraries (music, audiobooks) become unavailable because they're served via NFS.

  • RPO: N/A (live data unaffected)
  • RTO: Depends on Unraid recovery
  • Impact: Navidrome and Audiobookshelf lose their media mounts. Life Hub continues operating. Backups stop landing on NFS until recovery.
  • Manual intervention: Restore or replace Unraid. Backups resume automatically on next CronJob cycle.

Secrets Recovery

Secrets deserve special attention because they're the one thing that can't be reconstructed from source code.

All Kubernetes secrets are encrypted with SOPS + Age and stored in Git. The Age private key is the single root secret: everything else can be derived from it. I keep the Age key in multiple locations outside the cluster: password manager, encrypted USB drive, printed QR code in a fireproof safe (yes, really).

The recovery chain: Age key → decrypt SOPS secrets → apply to cluster → all services authenticate.

Without the Age key, every secret (API keys, database passwords, OAuth client secrets) would need to be regenerated manually. That's the true RPO=infinity scenario, and it's why the Age key gets the paranoid treatment.

Network Policies for Backup Isolation

The backup CronJob has its own network policy that restricts egress to DNS only:

- podSelector:
    matchLabels:
      app: life-hub-backup
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector: {}
    ports:
    - protocol: UDP
      port: 53

The backup job reads from a local PVC and writes to an NFS PVC. It doesn't need internet access. Restricting egress means that even if the backup container were compromised, it couldn't exfiltrate data or phone home. Defense in depth for a process that handles the crown jewels.

Monitoring the Backup Pipeline

Backups that aren't monitored aren't backups. They're hopes.

The backup system reports through three channels:

  1. SyncState table: Updated on every backup run with status (ok/error), timestamp, and file path. Queryable via GET /api/v1/backup/status and the MCP get_backup_status() tool.
  2. ntfy notifications: Failure notifications are always sent (priority: warning). Success notifications are optional (BACKUP_NOTIFY_SUCCESS=true).
  3. Uptime Kuma: Monitors the Life Hub health endpoint. If the app is down, backups aren't running.

The MCP integration means I can ask Claude "what's the backup status?" and get a direct answer from the database. No log diving required.

What I'd Change

The NFS backup gap bothers me. The daily CronJob at 2:45 AM means the worst-case RPO for total cluster failure is ~24 hours. I should run it more frequently: every 6 hours would drop the worst case to 6 hours with minimal additional storage.

Off-site backup is missing. Unraid is on the same network as the cluster. A house fire takes everything. Pushing encrypted backups to B2 or S3 would close this gap for under /month at current data sizes.

Longhorn snapshot scheduling could be more aggressive. The default snapshot interval covers most scenarios, but explicit hourly snapshots would give finer-grained recovery points for the Longhorn-survives scenarios.

Recovery testing needs a schedule. I've recovered from real failures, but I haven't done a planned, full-cluster-rebuild drill. That's the kind of thing you should do quarterly, and I haven't.

The Honest Assessment

For a homelab running personal data (not a business, not customer data), this DR setup is solid. The RPO ranges from 0 (node failure) to 24 hours (total cluster loss). The RTO is under an hour for any scenario. The backup storage cost is effectively zero.

The gap is geographic redundancy. Everything is in one location. For personal data at this scale, that's a calculated risk I'm comfortable with. The Age key backup is the one piece that's truly distributed, and it's the one piece that matters most.

If this were production infrastructure serving other people, I'd add off-site encrypted backups and quarterly DR drills. For a homelab that "actually matters" (it matters to me, it tracks my health data, my reading, my finances), the current setup gives me confidence that I can recover from anything short of a house fire. And even then, the Git repo and container images survive in the cloud.

That's disaster recovery for a homelab that actually matters. Not theoretical. Tested.