Observability for a Two-Node Kubernetes Homelab

This is the fourth and final post in a series about building a production-grade Kubernetes homelab on Talos Linux. In Post 1, I right-sized resource budgets to stop my single Dell laptop from OOM-killing itself. In Post 2, I added an Intel NUC worker node to triple available memory. In Post 3, I migrated PVC-bound workloads to the new node and got the cluster into its final two-node shape.

Now comes the part that separates a cluster you can trust from one you just hope is working: observability. Metrics, logs, alerts, and dashboards, configured so that when something breaks at 2 AM, your phone buzzes before you notice anything is down.

The Budget

Observability is expensive. On a resource-constrained homelab, every component needs to justify its memory footprint. Here is what the full stack costs:

Two gigabytes for full observability across a two-node cluster. That is the cost of actually knowing what your cluster is doing, versus hoping nothing is on fire. Worth every megabyte.

Layer 1: Metrics with kube-prometheus-stack

The foundation is kube-prometheus-stack, a Helm chart that bundles Prometheus, Grafana, Alertmanager, node-exporter, kube-state-metrics, and the Prometheus Operator into one release. It is managed by Flux like everything else in this cluster: a HelmRelease in the monitoring namespace that reconciles every 30 minutes.

Key configuration choices:

• 30-day retention with a 5 GB size cap: enough history for trend analysis without burning through NVMe storage.
• 60-second scrape interval: the default 15s is overkill for a homelab. Quadrupling the interval cuts storage churn and CPU usage without losing meaningful resolution.
• Right-sized limits: Prometheus gets 768 Mi (actual usage sits around 500-600 Mi), Grafana gets 512 Mi (it genuinely needs ~410 Mi at steady state and will OOM-kill at 384 Mi).

On Talos Linux, several upstream kube-prometheus-stack components do not apply. Talos manages its own control plane, so kubeControllerManager, kubeScheduler, kubeProxy (Cilium replaces it), and kubeEtcd are all disabled:

kubeControllerManager:
  enabled: false
kubeScheduler:
  enabled: false
kubeProxy:
  enabled: false
kubeEtcd:
  enabled: false

This saves memory and eliminates false-positive alerts about targets that will never exist.

Custom Alert Rules

The stock kube-prometheus-stack alerts cover generic Kubernetes failures: KubePodCrashLooping, KubeNodeNotReady, and so on. But a GitOps homelab has infrastructure-specific failure modes that the defaults do not understand. I added a PrometheusRule covering three categories.

Flux GitOps Alerts

If Flux cannot reconcile, nothing deploys. These are the "your cluster is deaf" alerts:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: homelab-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  groups:
    - name: flux.rules
      rules:
        - alert: FluxKustomizationNotReady
          expr: gotk_reconcile_condition{type="Ready",status="False",kind="Kustomization"} == 1
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "Flux Kustomization {{ $labels.name }} not ready"
            description: >-
              Kustomization {{ $labels.name }} in namespace {{ $labels.exported_namespace }}
              has been not-ready for more than 15 minutes.

        - alert: FluxHelmReleaseNotReady
          expr: gotk_reconcile_condition{type="Ready",status="False",kind="HelmRelease"} == 1
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "Flux HelmRelease {{ $labels.name }} not ready"

        - alert: FluxReconciliationStalled
          expr: time() - gotk_reconcile_condition{type="Ready",status="True"} > 1800
          for: 30m
          labels:
            severity: warning
          annotations:
            summary: "Flux {{ $labels.kind }} {{ $labels.name }} reconciliation stalled"

The 15-minute for duration avoids noise during normal Helm upgrades. The stalled alert catches a subtler failure: Flux reports Ready, but the last successful reconciliation was over 30 minutes ago. Something is silently stuck.

Longhorn Storage Alerts

With two-replica Longhorn volumes across two nodes, a degraded volume means one replica is down. A faulted volume means data is at risk:

    - name: longhorn.rules
      rules:
        - alert: LonghornVolumeDegraded
          expr: longhorn_volume_robustness == 2
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Longhorn volume {{ $labels.volume }} is degraded"

        - alert: LonghornVolumeFaulted
          expr: longhorn_volume_robustness == 3
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: "Longhorn volume {{ $labels.volume }} is FAULTED"

        - alert: LonghornStorageUsageHigh
          expr: >-
            (longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Longhorn storage usage above 85% on {{ $labels.node }}"

The faulted alert fires after just 1 minute with severity critical. When a volume is faulted, you are in data-loss territory. That is not something you want to sleep through.

Cert-Manager Alerts

TLS certificates that silently expire are the kind of failure that makes you look like you do not care about your infrastructure:

    - name: certmanager.rules
      rules:
        - alert: CertificateExpiringSoon
          expr: >-
            certmanager_certificate_expiration_timestamp_seconds - time() < 14 * 24 * 3600
          for: 1h
          labels:
            severity: warning
          annotations:
            summary: "Certificate {{ $labels.name }} expiring within 14 days"

        - alert: CertificateRenewalFailed
          expr: certmanager_certificate_ready_status{condition="False"} == 1
          for: 30m
          labels:
            severity: warning
          annotations:
            summary: "Certificate {{ $labels.name }} is not ready"

Fourteen days gives you time to investigate and fix without panic. The renewal failure alert catches ACME/issuer problems before expiration math even becomes relevant.

The Alert Pipeline

Writing alert rules is the easy part. Getting those alerts to your phone reliably is where most homelab setups fall apart. Here is the pipeline:

Prometheus / Loki ruler → Alertmanager → ntfy-alertmanager bridge → ntfy → phone notification

Alertmanager receives firing alerts from both Prometheus (metric alerts) and the Loki ruler (log-based alerts). An AlertmanagerConfig CRD routes everything to a webhook: the ntfy-alertmanager bridge running as a sidecar deployment in the ntfy namespace:

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: homelab-notifications
  namespace: monitoring
  labels:
    alertmanagerConfig: homelab
spec:
  route:
    receiver: ntfy
    groupBy: ["alertname", "namespace"]
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 4h
    matchers:
      - matchType: "!="
        name: alertname
        value: Watchdog
      - matchType: "!="
        name: alertname
        value: InfoInhibitor
  receivers:
    - name: ntfy
      webhookConfigs:
        - url: "http://ntfy-alertmanager.ntfy.svc.cluster.local:80"
          sendResolved: true

The Watchdog and InfoInhibitor exclusions are important: Watchdog is a meta-alert that fires continuously to prove Alertmanager is alive, and InfoInhibitor is an internal mechanism. Neither should hit your phone. The sendResolved: true means you get a follow-up notification when an alert clears, so you know the problem is fixed without having to check manually.

The ntfy bridge converts Alertmanager webhooks into ntfy push notifications. ntfy itself uses the upstream ntfy.sh service for push delivery to iOS and Android. The entire pipeline is in-cluster except for the last-mile phone push: no external dependencies for the alerting logic itself.

Layer 2: Logs with Loki + Alloy

Metrics tell you what is broken. Logs tell you why. Loki provides log aggregation, and Alloy handles log collection, both from the Grafana ecosystem, both ruthlessly trimmed to fit the homelab's memory budget.

Loki: Single-Binary, No Frills

Loki 3.6.5 runs in single-binary mode with filesystem storage. No object store, no microservice deployment, no distributed state. One pod, one 5 Gi PVC, 256 Mi memory limit:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: loki
  namespace: loki
spec:
  interval: 30m
  chart:
    spec:
      chart: loki
      version: "6.x"
      sourceRef:
        kind: HelmRepository
        name: grafana
        namespace: flux-system
  values:
    deploymentMode: SingleBinary

    loki:
      auth_enabled: false
      commonConfig:
        replication_factor: 1
      storage:
        type: filesystem
      schemaConfig:
        configs:
          - from: "2024-01-01"
            store: tsdb
            object_store: filesystem
            schema: v13
            index:
              prefix: loki_index_
              period: 24h
      limits_config:
        retention_period: 168h
      compactor:
        retention_enabled: true
        working_directory: /var/loki/compactor
        compaction_interval: 10m
        delete_request_store: filesystem

    singleBinary:
      replicas: 1
      persistence:
        enabled: true
        size: 5Gi
        storageClass: longhorn
      resources:
        requests:
          cpu: 50m
          memory: 128Mi
        limits:
          cpu: "1"
          memory: 256Mi

    # Disable everything that costs memory
    gateway:
      enabled: false
    read:
      replicas: 0
    write:
      replicas: 0
    backend:
      replicas: 0
    lokiCanary:
      enabled: false
    test:
      enabled: false
    monitoring:
      selfMonitoring:
        enabled: false
      lokiCanary:
        enabled: false
    chunksCache:
      enabled: false
    resultsCache:
      enabled: false

That bottom section is doing a lot of work. Each of those disabled components (gateway, canary, self-monitoring, chunk/results caches) would cost 64-128 Mi of memory. On a cluster where 2 Gi buys you the entire observability stack, leaving these enabled would nearly double the cost of log aggregation alone. A production multi-tenant Loki deployment needs them. A two-node homelab does not.

Seven-day retention with the compactor keeps disk usage predictable. Logs older than a week are automatically purged.

Loki 6.x Gotchas

The Loki Helm chart has some sharp edges that cost me hours. Documenting them here so they do not cost you the same.

Gotcha 1: delete_request_store is mandatory. When retention is enabled in Loki 6.x, the compactor requires delete_request_store: filesystem in its config. Without it, Loki fails config validation at startup with a cryptic error. This was not required in earlier versions.

Gotcha 2: Two canary toggles. The Loki chart has two separate flags for the canary: a top-level lokiCanary.enabled and a nested monitoring.lokiCanary.enabled. You must disable both. Miss one and you get a canary pod consuming memory and generating noise for no reason.

Gotcha 3: First startup is slow. Loki single-binary takes approximately 10 minutes on first start. The compactor waits for ring stability before becoming ready, and the readiness probe returns 503 until then. Do not panic and start killing pods. It is working.

Alloy: The Log Collector

Grafana Alloy runs as a DaemonSet (one pod per node), reading pod logs from /var/log/pods/ and shipping them to Loki. The config is straightforward River syntax:

alloy:
  configMap:
    content: |-
      discovery.kubernetes "pods" {
        role = "pod"
      }

      discovery.relabel "pod_logs" {
        targets = discovery.kubernetes.pods.targets

        rule {
          source_labels = ["__meta_kubernetes_namespace"]
          target_label  = "namespace"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_name"]
          target_label  = "pod"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_container_name"]
          target_label  = "container"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
          target_label  = "app"
        }
      }

      loki.source.kubernetes "pod_logs" {
        targets    = discovery.relabel.pod_logs.output
        forward_to = [loki.write.default.receiver]
      }

      loki.write "default" {
        endpoint {
          url = "http://loki.loki.svc.cluster.local:3100/loki/api/v1/push"
        }
      }

Four labels: namespace, pod, container, app. Enough to filter and query effectively, few enough to keep Loki's cardinality manageable.

Important: The Alloy namespace needs pod-security.kubernetes.io/enforce: privileged because Alloy mounts the host's /var/log/pods/ directory. Without the privileged PSA label, the pod will be rejected. This is one of the few legitimate uses of the privileged security level: you genuinely need hostPath access to read container logs.

Log-Based Alerting with the Loki Ruler

Prometheus alerts trigger on metrics. But some failures only show up in logs: panics, segfaults, runtime errors. That is where the Loki ruler comes in: it evaluates LogQL queries on a schedule and fires alerts to the same Alertmanager that handles metric alerts.

The ruler is configured inline in the Loki HelmRelease:

loki:
  rulerConfig:
    alertmanager_url: http://alertmanager-operated.monitoring.svc.cluster.local:9093
    enable_api: true
    ring:
      kvstore:
        store: inmemory
    storage:
      type: local
      local:
        directory: /var/loki/rules

The actual alert rules are defined in a separate ConfigMap and mounted into the Loki pod:

apiVersion: v1
kind: ConfigMap
metadata:
  name: loki-ruler-rules
  namespace: loki
data:
  rules.yaml: |
    groups:
      - name: container-log-alerts
        rules:
          - alert: ContainerLogPanic
            expr: |
              sum by (namespace, pod, container) (
                count_over_time(
                  {namespace=~".+", namespace!="loki"}
                    |~ `(panic:|runtime error:|OOMKilled|Segmentation fault|SIGABRT)`
                  [5m]
                )
              ) > 0
            for: 1m
            labels:
              severity: warning
            annotations:
              summary: "Crash pattern detected in {{ $labels.namespace }}/{{ $labels.pod }}"

The ConfigMap is mounted via extraVolumes and extraVolumeMounts in the singleBinary section:

singleBinary:
  extraVolumes:
    - name: ruler-rules
      configMap:
        name: loki-ruler-rules
  extraVolumeMounts:
    - name: ruler-rules
      mountPath: /var/loki/rules/fake
      readOnly: true

Why /var/loki/rules/fake/?

The mount path looks odd, but it is required. Loki ruler expects rules at <storage_dir>/<tenant_id>/. When auth_enabled: false, the tenant ID defaults to fake. So the rules must land at /var/loki/rules/fake/rules.yaml. Miss this path convention and the ruler silently loads zero rules.

Why Not Use ruler.directories?

The Loki Helm chart has a ruler.directories value that is supposed to handle ConfigMap creation and mounting automatically. It does not work in SingleBinary mode. The chart simply does not create the ConfigMaps or mount them. I wasted time discovering this. The explicit ConfigMap + extraVolumes approach is the reliable path.

Ruler Gotchas

Self-referential false positive. This one is subtle. The ContainerLogPanic alert searches for patterns like panic: and runtime error:. The Loki ruler logs the LogQL query string itself as part of normal operation. If the ruler's own logs are indexed by Loki (which they are, since Alloy collects from all pods), the ruler's log lines contain the search patterns, matching the alert query, firing the alert. An alert that triggers itself.

The fix: namespace!="loki" in the LogQL selector. The ruler's own namespace is excluded from the search.

Avoid (?i)fatal. My first draft used a case-insensitive regex for "fatal" as a crash indicator. Postgres logs FATAL: role "root" does not exist every 5 seconds from failed probe connections, a known harmless message from default health checks. This pattern generated hundreds of false positives per hour. Stick to specific crash signatures: panic:, runtime error:, OOMKilled, Segmentation fault, SIGABRT.

Multi-Node Grafana Dashboards

When I added the NUC worker node in Post 2, every single-node dashboard panel broke. Metrics that assumed one node suddenly had two data series with different instance labels. Worse, the instance label is an IP:port string like 192.168.50.10:9100, useless for human identification.

The fix is a PromQL join pattern using node_uname_info:

(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes))
  * on(instance) group_left(nodename) node_uname_info * 100

The * on(instance) group_left(nodename) node_uname_info portion joins the metric with node_uname_info on the instance label, pulling in the nodename label (which contains the human-readable hostname). Every per-node panel uses this pattern: CPU, memory, disk, load average, network I/O, disk I/O.

The dashboard itself is a ConfigMap with the grafana_dashboard: "1" label. Grafana's sidecar container watches for ConfigMaps with this label across all namespaces and auto-loads them. No manual import, no Grafana UI clicking:

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboard-homelab-overview
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
data:
  homelab-overview.json: |
    { ... }

The homelab overview dashboard has 16 panels:

• Per-node bar gauges: CPU, Memory, Disk usage with color thresholds (green/yellow/red)
• Stat panels: Node count, Total Pods (running), Not Ready count, Longhorn Volumes (attached)
• Time series: CPU by Namespace, Memory by Namespace (stacked), Top 10 Pods by Memory, CPU Throttling by Container
• Split I/O panels: Network I/O by Node (RX/TX), Disk I/O by Node (Read/Write)
• Load Average: 1m and 5m load per node
• Tables: Pod Restarts (last hour) with node column, PVC Usage with node/namespace/capacity

Everything is defined as code in the Git repo. If Grafana's PVC disappears tomorrow, the dashboard comes back automatically on the next reconciliation. That is the difference between an architect and a slumlord: the slumlord configures dashboards by hand and loses them when the disk dies.

Network Policy Complexity

Observability components are inherently cross-cutting: Prometheus scrapes every namespace, Alloy collects logs from every node, Grafana queries both Prometheus and Loki, the Loki ruler pushes to Alertmanager in a different namespace. In a cluster with default-deny network policies on every namespace, this creates a web of required allow rules.

The monitoring namespace alone has 14 network policy resources. Here is a simplified view of the cross-namespace flows that need explicit policies:

Each application namespace also needs an ingress rule allowing Prometheus to scrape its metrics endpoint. Forget one and that app becomes a blind spot.

The Cilium + Flux SSA Gotcha

This one cost me a full evening of debugging. CiliumNetworkPolicy resources define egress rules as a list of entries. When you have multiple egress entries with similar toEndpoints structure, Flux's server-side apply (SSA) merges them. Two separate egress rules targeting different services on different ports get collapsed into one rule with a single port. Your policy silently loses connectivity.

The symptom: intermittent connection failures between services that should be allowed. The cause: Flux SSA's list-item merge strategy treats CiliumNetworkPolicy egress entries as mergeable if their structure looks similar enough.

The fix: use separate CiliumNetworkPolicy resources per destination service. Each resource is its own Kubernetes object. SSA cannot merge fields across distinct objects. It means more YAML files, but it means your policies actually work.

Grafana as Loki Frontend

Loki does not have its own query UI. Grafana is the frontend, connected via an additional datasource in the kube-prometheus-stack HelmRelease:

grafana:
  additionalDataSources:
    - name: Loki
      type: loki
      access: proxy
      url: http://loki.loki.svc.cluster.local:3100
      jsonData:
        maxLines: 1000

With this in place, Grafana's Explore view lets you run LogQL queries against the full 7-day log history. Combined with metric dashboards, you can correlate a memory spike in Prometheus with the actual error logs in Loki, all from one UI.

What This Gets You

With all of this in place, here is what a typical failure looks like from detection to notification:

1. A Longhorn volume loses a replica (node reboot, disk issue, network partition).
2. Prometheus evaluates longhorn_volume_robustness == 2 on its 60-second scrape cycle.
3. After 5 minutes in degraded state, LonghornVolumeDegraded fires.
4. Alertmanager receives the alert, groups it with any other active alerts, waits the 30-second group wait.
5. The webhook fires to ntfy-alertmanager, which formats the alert and pushes it to ntfy.
6. My phone buzzes with the volume name, severity, and a description of what is wrong.
7. When the replica rebuilds and the volume returns to healthy, a resolved notification follows.

Total time from failure to phone notification: approximately 6 minutes. Total manual intervention to set this up after the initial deploy: zero. Flux reconciles everything from Git.

The Full Picture

Across this four-part series, the cluster went from a single overcommitted laptop to a two-node, fully observable, GitOps-managed platform. Here is the final state:

• Two nodes: Dell laptop (control plane, 15.6 GB) + Intel NUC (worker, 32 GB)
• Storage: Longhorn with 2 replicas across nodes, WaitForFirstConsumer binding
• GitOps: Flux reconciling every 10 minutes, with alerts if reconciliation breaks
• Metrics: Prometheus with 30-day retention, custom alert rules for Flux, Longhorn, cert-manager
• Logs: Loki + Alloy with 7-day retention, log-based crash detection via the ruler
• Alerts: Prometheus + Loki ruler → Alertmanager → ntfy → phone
• Dashboards: 16-panel Grafana overview, defined as code, auto-loaded via ConfigMap sidecar
• Network policies: Default-deny everywhere, explicit allows for every cross-namespace flow
• Observability cost: ~2 Gi total, 4% of available cluster memory

Every piece of configuration lives in Git. Every component is declarative. Every failure mode has an alert. If I lose a node, workloads fail over via soft affinity. If I lose a disk, Longhorn has a replica on the other node. If something crashes, a log-based alert fires to my phone. If Flux breaks, a metric alert fires to my phone.

This is what it means to build infrastructure like an architect. Not because a two-node homelab needs production-grade observability, but because the habits you build on your own infrastructure are the habits you carry into production. Cut corners here and you will cut corners there. Build it right here and building it right everywhere else becomes the default.

The entire repository is public at github.com/snaviaux/home-ops. Every YAML snippet in this series is running in production right now.