Observability on 16GB: Monitoring a Single-Node Kubernetes Cluster

Until this point, the only way to know something was wrong with my cluster was to manually run kubectl get pods or notice a service was down. That's fine for a weekend project. It's not fine for something you want to rely on. I needed metrics, dashboards, and alerting: the same stack enterprise teams deploy on day one.

The Stack: kube-prometheus-stack

Rather than assembling Prometheus, Grafana, and Alertmanager individually, I used kube-prometheus-stack, a single Helm chart that bundles everything:

• Prometheus: scrapes numeric metrics from every pod, node, and Kubernetes component every 30 seconds. CPU, memory, disk, network, pod restarts, request latency, all stored as time-series data.
• Grafana: turns Prometheus data into dashboards with graphs, gauges, and tables. Comes with pre-built Kubernetes dashboards out of the box.
• Alertmanager: fires notifications when thresholds are breached. Handles deduplication and silencing.
• node-exporter: hardware/OS metrics: CPU per core, memory breakdown, disk usage, network stats.
• kube-state-metrics: Kubernetes object state: replica counts, pending pods, PVC status.

Talos-Specific Tuning

Several Prometheus components try to scrape Kubernetes internals that Talos handles differently. Without disabling these, Prometheus shows persistent scrape errors for targets that don't exist:

# These don't work on Talos — disable them
kubeControllerManager:
  enabled: false     # Talos manages internally
kubeScheduler:
  enabled: false     # Not exposed as regular pods
kubeProxy:
  enabled: false     # Replaced with Cilium
kubeEtcd:
  enabled: false     # Talos manages directly

This is a common Talos gotcha. The actual metrics from these components are still available through Talos's own endpoints, but the standard kube-prometheus-stack exporters can't reach them.

Resource Allocation for 16GB

The monitoring stack itself needs resources. On a single node with 16GB total and 20+ pods competing for memory, I had to tune for a low footprint:

Prometheus gets the most because it's constantly scraping, storing, and querying. Grafana needs less because it only works when someone loads a dashboard. The exporters are lightweight collectors.

Prometheus retention is set to 7 days with a 10Gi PVC. For a homelab, a week of history is plenty for debugging. Enterprise would keep 30-90 days and ship to long-term storage.

The Monitoring Stack Almost OOMed Itself

Here's the irony. After deploying the monitoring stack, I noticed Grafana was running at 282Mi against a 256Mi memory limit. The monitoring system was about to OOM-kill itself.

I only caught this because I was looking at the very dashboards it provides. Without monitoring, I wouldn't know the monitoring was about to fail. I bumped the limit to 384Mi.

The lesson: always check the monitoring components in your own dashboards. They need resources too, and running near limits means one dashboard refresh could trigger the OOM killer.

Custom Dashboard via GitOps

The pre-built dashboards show raw numbers in millicores and bytes, useful for engineers who stare at Prometheus daily, but not great for a quick health check. I built a custom "Homelab Overview" dashboard managed via GitOps.

Grafana supports a sidecar pattern: any ConfigMap in the monitoring namespace with the label grafana_dashboard: "1" is automatically imported as a dashboard. The dashboard definition is a JSON blob in the ConfigMap: version-controlled, deployed via Flux, recoverable from git clone.

The PromQL Bug

The "Pods Not Ready" panel initially showed 141, wildly wrong. The query count(kube_pod_status_phase{phase!="Running",phase!="Succeeded"}) was counting zero-value time series. Every pod emits a series for every phase (Running, Pending, Failed, Unknown, Succeeded), with value 1 for the active phase and 0 for all others. With ~47 pods times 3 excluded phases, you get 141.

The fix: add == 1 to filter for actually-active phases only. A subtle Prometheus behavior that's documented but easy to miss.

Longhorn Snapshots

The last piece of the reliability layer: automated backups for persistent data. Without snapshots, if someone drops a database table or an app corrupts its data, the only recovery is "hope you have a manual backup somewhere."

Longhorn snapshots are point-in-time copies of volumes. They're fast (instant) and cheap (only store the delta/changes). Two RecurringJob CRDs applied to the default group, so all volumes get them automatically:

# Daily snapshot — 2:00 AM, keep last 7 days
apiVersion: longhorn.io/v1beta2
kind: RecurringJob
metadata:
  name: daily-snapshot
  namespace: longhorn-system
spec:
  cron: "0 2 * * *"
  task: snapshot
  retain: 7
  groups: ["default"]
---
# Weekly snapshot — 3:00 AM Sunday, keep last 4 weeks
apiVersion: longhorn.io/v1beta2
kind: RecurringJob
metadata:
  name: weekly-snapshot
  namespace: longhorn-system
spec:
  cron: "0 3 * * 0"
  task: snapshot
  retain: 4
  groups: ["default"]

Important limitation: these snapshots live on the same NVMe as the data. They protect against accidental deletion and app-level corruption, but not against drive failure. If the NVMe dies, the snapshots die with it. True disaster recovery needs an offsite target.

What You Get Out of the Box

kube-prometheus-stack comes with pre-configured alert rules that fire when thresholds are breached:

• KubePodCrashLooping: pod restarted more than 5 times in 15 minutes
• KubePodNotReady: pod hasn't been ready for 15 minutes
• NodeFilesystemSpaceFillingUp: disk predicted to fill within 24 hours
• NodeMemoryHighUtilization: memory above 90%

These alerts fire to Alertmanager. I later wired Alertmanager to ntfy for push notifications to my phone: every cluster alert lands as a notification within seconds.

The Most Useful Dashboard

The most useful view turned out to be the Node Exporter dashboard showing memory pressure. On a 16GB single-node cluster running 20+ pods, memory is the first constraint. Watching the available memory trend over a few days showed exactly how much headroom remained for new services, and whether the monitoring stack itself was slowly eating into it.

You can't fix what you can't see. Now I can see everything.