Subscribe

Production Hardening a Homelab: Secrets, Limits, Probes, and Pinned Images

Production Hardening a Homelab: Secrets, Limits, Probes, and Pinned Images

There's a temptation in homelab culture to take shortcuts. Skip the health probes. You'll just restart it manually. Use :latest tags. You want the newest version anyway. Store passwords in plain text config files. It's only your home network.

Every one of these shortcuts becomes a bad habit. And bad habits built in a homelab follow you to production.

This post covers the production hardening I applied to my single-node Talos Linux cluster: SOPS-encrypted secrets, resource requests and limits, health probes, pinned image versions, security contexts, and self-signed TLS. None of this is theoretical: it's all running on a Dell laptop with 16 GB of RAM.

Secrets: SOPS + Age Encryption

The first rule of GitOps is that everything lives in Git. The first rule of security is that secrets don't live in Git. SOPS (Secrets OPerationS) resolves this contradiction by encrypting secret values in-place while leaving metadata readable.

The setup is simple. A .sops.yaml file at the repo root tells SOPS which files to encrypt and which key to use:

creation_rules:
  - path_regex: .*\.sops\.yaml$
    age: age10jfr5zf2vq8ptqq4j6p47de2l0mx2qekdjjmydvpylr3stsdkgaswydr0s

Any file matching *.sops.yaml gets encrypted with my age public key. The naming convention is the enforcement mechanism: if you name a secret file my-secret.sops.yaml, SOPS knows to encrypt it.

Here's what an encrypted secret looks like in Git:

apiVersion: v1
kind: Secret
metadata:
  name: grafana-credentials
  namespace: monitoring
stringData:
  admin-password: ENC[AES256_GCM,data:SkIsBs+sRoXdaaxODD1m7JHB3nY=,...]
  admin-user: ENC[AES256_GCM,data:YUGyymc=,...]
sops:
  age:
    - recipient: age10jfr5zf2vq8ptqq4j6p47de2l0mx2qekdjjmydvpylr3stsdkgaswydr0s
  lastmodified: "2026-02-15T07:19:42Z"
  encrypted_regex: ^(data|stringData)$
  version: 3.11.0

Notice what's encrypted and what isn't. The metadata block (name, namespace, labels) stays in plain text. You can see what secrets exist, where they're deployed, and when they were last modified, without seeing the actual values. This is critical for GitOps: Flux needs to read the metadata to know where to apply the secret, but the values stay encrypted until decryption happens inside the cluster.

Flux has native SOPS support. On bootstrap, you create a single age key secret in the flux-system namespace:

kubectl create secret generic sops-age \
  --namespace=flux-system \
  --from-file=age.agekey=~/.config/sops/age/keys.txt

From that point forward, Flux automatically decrypts any *.sops.yaml file it encounters during reconciliation. No plugins, no operators, no external vault: just a private key in a Kubernetes secret and encrypted files in Git.

The existingSecret Pattern

Most Helm charts support injecting credentials via an existing Kubernetes secret rather than plain text values. This is the existingSecret pattern, and it's how you keep secrets out of HelmRelease manifests.

For example, Grafana's admin credentials in my kube-prometheus-stack HelmRelease:

grafana:
  admin:
    existingSecret: grafana-credentials
    userKey: admin-user
    passwordKey: admin-password

The grafana-credentials secret is a separate SOPS-encrypted file that Flux decrypts and applies independently. The HelmRelease never sees the actual password. It just references the secret by name. This decouples credential management from chart deployment, which means you can rotate secrets without touching the HelmRelease.

Resource Requests and Limits

On a 16 GB single-node cluster, every megabyte matters. Kubernetes itself, Cilium, and Longhorn consume roughly 3-4 GB before you deploy a single application. That leaves about 12 GB for everything else, and "everything else" includes 15+ applications, Prometheus, Grafana, Alertmanager, and their supporting infrastructure.

Resource requests and limits serve different purposes:

  • Requests tell the scheduler what a pod needs to start. They're a floor: the scheduler won't place the pod on a node that can't satisfy the request.
  • Limits tell the kubelet when to kill a pod. They're a ceiling: exceed your memory limit and you get OOMKilled.

The goal is to set requests low enough that pods can schedule, and limits high enough that they don't get killed during normal operation. Here's a representative deployment (Ghost CMS):

resources:
  requests:
    cpu: 50m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 384Mi

How did I arrive at these numbers? By running the application, watching actual usage via Prometheus, and setting limits at 2-4x the observed peak. Ghost typically runs at about 150-200 MB of memory. A 384 MB limit gives it headroom for spikes without wasting cluster capacity.

Some apps surprised me. Mealie (a Python/FastAPI recipe manager) runs at ~415 MB despite being a relatively simple app. Python's memory overhead is real. Grafana needs 512 MB (it was OOM-killing at 384 MB). Prometheus with 30 days of retention needs 768 MB. Right-sizing took multiple iterations and a lot of kubectl top pods.

The alternative (not setting limits) means a single misbehaving app can consume all available memory and take down the entire node. On a single-node cluster, that's a complete outage.

Pinned Image Versions

Every container image in my cluster uses an explicit version tag:

image: ghost:6.19.2-alpine
image: binwiederhier/ntfy:v2.17.0
image: ghcr.io/mealie-recipes/mealie:v2.8.0
image: ghcr.io/bigboot/autokuma:2.0.0

Never :latest. Never an unpinned tag.

The :latest tag is a lie. It doesn't mean "the latest version". It means "whatever was most recently pushed with this tag." Two pulls of the same :latest tag can return completely different images. In a GitOps workflow, this is catastrophic: your Git repo says one thing, but the cluster might be running something different depending on when a pod was last restarted.

Pinned versions give you:

  • Reproducibility: the manifest in Git exactly describes what's running
  • Rollback: change the tag back and push; Flux deploys the old version
  • Audit trail: the Git log shows exactly when each version was deployed
  • Intentional upgrades: upgrades happen when you decide, not when an upstream maintainer pushes

To keep versions current without manual tracking, I use Renovate for automated dependency updates. It opens PRs when new versions are available, grouped by risk level: patch versions auto-merge, major versions require review. The version in Git is always the version in the cluster.

Liveness and Readiness Probes

Without probes, Kubernetes considers a pod healthy as long as the process is running. A process can be running and completely broken: stuck in a deadlock, out of memory, unable to serve requests. Probes tell Kubernetes how to actually check if the application is working.

Liveness probes answer: "Is this container alive?" If it fails, Kubernetes restarts the container. Use this to recover from deadlocks and unrecoverable states.

Readiness probes answer: "Can this container serve traffic?" If it fails, Kubernetes removes the pod from Service endpoints. Use this to prevent routing traffic to a pod that's still starting up or temporarily unable to handle requests.

Here's the Ghost CMS probe configuration:

livenessProbe:
  httpGet:
    path: /ghost/api/admin/site/
    port: 2368
  initialDelaySeconds: 60
  periodSeconds: 10
readinessProbe:
  httpGet:
    path: /ghost/api/admin/site/
    port: 2368
  initialDelaySeconds: 30
  periodSeconds: 5

The timing parameters matter:

  • initialDelaySeconds: How long to wait before the first probe. Ghost takes 30-60 seconds to start. Probe too early and Kubernetes kills it before it's ready.
  • periodSeconds: How often to probe. 5-10 seconds is typical. Too frequent wastes resources; too infrequent means slow failure detection.

The probe endpoint should be lightweight but meaningful. Ghost's /ghost/api/admin/site/ endpoint returns basic site info: it exercises the application stack (Node.js, SQLite, routing) without doing heavy work. For Mealie, I use /api/app/about. For ntfy, /v1/health. Each app has its own health endpoint, and using it means Kubernetes can distinguish between "the process is running" and "the application is actually working."

Security Contexts

Every deployment in my cluster runs with a restricted security context:

spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: ghost
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]

This does several things:

  • runAsNonRoot: Kubernetes refuses to start the container if it would run as root. This is a guardrail: even if the image defaults to root, the pod won't start.
  • runAsUser/runAsGroup: Explicit UID/GID. No relying on image defaults.
  • seccompProfile: RuntimeDefault: Restricts available system calls to a sane default set. Blocks most privilege escalation vectors.
  • allowPrivilegeEscalation: false: Prevents the process from gaining more privileges than its parent. Closes setuid/setgid attack vectors.
  • capabilities: drop ALL: Linux capabilities are fine-grained root privileges. Dropping all of them means the container has no special OS-level permissions.

Not every app can run with all restrictions. Mealie, for instance, requires root internally (the PUID/PGID environment variables don't actually work). For those cases, I apply what I can (seccompProfile: RuntimeDefault and drop: ["NET_RAW"] at minimum) and document why the full restrictions aren't applied.

Self-Signed TLS on the LAN

Every service in my cluster is accessible over TLS, even on the local network. The internal domain (*.homelab.local) uses self-signed certificates issued by cert-manager:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}

Any Ingress resource annotated with cert-manager.io/cluster-issuer: selfsigned gets a TLS certificate automatically. cert-manager creates the certificate, stores it as a Kubernetes secret, and Traefik picks it up for TLS termination.

For external access via Cloudflare Tunnel, TLS is handled at the tunnel edge: Cloudflare terminates TLS and forwards plain HTTP to Traefik. The external domain (*.naviauxlab.com) uses Cloudflare's certificates. For production use or if I wanted trusted internal certs, I have Let's Encrypt issuers ready:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - dns01:
          cloudflare:
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: api-token

The Cloudflare API token is, naturally, a SOPS-encrypted secret.

Why Bother?

None of these practices make my homelab faster. Most of them make it slightly slower to deploy new things. Writing security contexts, setting resource limits, encrypting secrets, configuring probes: it's all friction.

But it's intentional friction. Every hardening measure I apply in my homelab is a pattern I can apply in production without thinking twice. SOPS encryption becomes second nature. Resource limits become automatic. Security contexts become default. Probes become non-negotiable.

The alternative is building habits around shortcuts. And shortcuts in a homelab become shortcuts in production, where they become incidents.

This post is part of a series documenting my homelab build on Talos Linux. Next up: zero-trust networking with Cilium NetworkPolicies. You can find the full series on the blog index.