How I Freed 26 GiB of RAM by Auditing My Homelab Kubernetes Cluster

My homelab Kubernetes cluster has been running on two nodes for a few months now: a Talos Linux VM (16 GiB) and an Intel NUC (32 GiB). Last week, Prometheus fired two alerts: CPUThrottlingHigh on a Grafana sidecar and NodeMemoryHighUtilization on the NUC. The NUC was sitting at 95% memory usage. On a 32 GiB machine running a homelab, that felt wrong.

What followed was a deep audit that uncovered a chain of issues: OOMKilling pods, network policies blocking sidecars, missing resource limits, and (the big one) a Longhorn instance-manager process holding 26 GiB of RAM due to Prometheus snapshot chain bloat.

The Starting Point

Here’s what kubectl top nodes showed:

NAME            CPU(cores)   MEMORY(bytes)   MEMORY(%)
nuc-worker-01   1428m        29937Mi         95%
talos-1ue-4vm   646m         9845Mi          65%

95% memory on a 32 GiB node. Something was very wrong. kubectl top pods immediately pointed the finger:

longhorn-system/instance-manager-...   232m   26254Mi

A single Longhorn instance-manager pod was consuming 25.6 GiB. But I didn’t start there. I found this after peeling back several layers.

Layer 1: The Obvious Fires

Miniflux OOMKilled (226 Restarts)

Miniflux, my RSS reader, had been OOMKilled 226 times. The memory limit was set to 128Mi, too tight for a Go process managing hundreds of RSS feeds. Fix was straightforward:

# apps/miniflux/deployment.yaml
resources:
  requests:
    cpu: 50m
    memory: 128Mi   # was 64Mi
  limits:
    cpu: 300m
    memory: 256Mi   # was 128Mi

Loki Sidecar Crash Loop (30 Restarts)

The loki-sc-rules sidecar (a kiwigrid/k8s-sidecar container that watches ConfigMaps for alerting rules) was crashing in a loop. Logs showed:

ConnectTimeoutError: Connection to 10.96.0.1 timed out

10.96.0.1 is the Kubernetes API server ClusterIP. The sidecar needs API access to list and watch ConfigMaps, but my CiliumNetworkPolicy only allowed egress to DNS and Alertmanager. The fix was one stanza:

# infrastructure/loki/network-policies.yaml (CiliumNetworkPolicy)
egress:
  # Sidecar watches ConfigMaps via K8s API
  - toEntities:
      - kube-apiserver
  # ... existing DNS and Alertmanager rules

After applying this and restarting the Loki pod: zero restarts, sidecar synced immediately.

Grafana Sidecar CPU Throttling

The Grafana dashboard sidecar (grafana-sc-dashboard) was hitting its 200m CPU limit during dashboard discovery across all namespaces. Bumped to 500m. Alert cleared.

Layer 2: Missing Resource Limits

I ran an audit of every container in the cluster and found several sidecars and config-reloaders running without any resource limits:

• Loki’s loki-sc-rules sidecar
• Alloy’s config-reloader
• Prometheus and Alertmanager config-reloader sidecars

Each of these was using 10–70 MiB. Not dangerous individually, but containers without limits are a ticking time bomb under memory pressure. The tricky part was finding the correct Helm value paths. Each chart has its own opinion about where sidecar resources go:

# Loki chart: top-level sidecar key
sidecar:
  resources:
    limits: { cpu: 100m, memory: 128Mi }

# Alloy chart: top-level configReloader key (NOT under alloy.)
configReloader:
  resources:
    limits: { cpu: 100m, memory: 64Mi }

# kube-prometheus-stack: under prometheusOperator
prometheusOperator:
  prometheusConfigReloader:
    resources:
      limits: { cpu: 100m, memory: 64Mi }

Lesson: always check helm show values for the actual schema. Don’t guess at nesting.

Layer 3: The 26 GiB Elephant

After fixing the obvious issues, the NUC was still at 95%. Time to dig into Longhorn.

The instance-manager is Longhorn’s supervisor process. It runs as a single pod per node and manages all the replica and engine processes for every volume on that node. I had 17 replicas running on nuc-worker-01. The child processes were using 25–90 MiB each (normal). But the parent Go process (PID 7) had ballooned:

$ cat /proc/7/smaps_rollup
Rss:            25522508 kB
Pss_Anon:       25513972 kB
Private_Dirty:  25513972 kB

25.5 GiB of anonymous dirty memory in a single Go process. It had been running for 12 days without a restart.

Root Cause: Prometheus Snapshot Chain Bloat

I checked volume actual sizes versus spec sizes:

Volume                    Spec    Actual   Ratio
pvc-95b89bc0 (Prometheus) 10.0G   35.4G    3.5x  !!!
pvc-efcabd3d (Loki)        5.0G    1.7G    0.3x
pvc-c2f4be48 (Grafana)     2.0G    1.4G    0.7x
... all others under 1.5G ...

Prometheus was the only outlier. The reason: TSDB compaction. Prometheus periodically rewrites its time-series blocks, which means every Longhorn snapshot captures massive diffs. My snapshot config was retaining 7 daily + 4 weekly snapshots. The weekly snapshots alone were 7–10 GiB each:

weekly-s-ae7ab0de  Mar 01   9.6 GiB
weekly-s-8332fe7d  Feb 22   7.2 GiB
daily-sn-05bc8881  Mar 06   6.9 GiB

10 snapshots totaling ~35 GiB of snapshot chain on a volume spec’d for 10 GiB. The instance-manager had to track all of this in memory. Over 12 days, the Go runtime accumulated allocations and never returned them to the OS (MADV_FREE behavior in Go’s GC).

The Fix

Four actions, applied in sequence:

1. Exclude ephemeral volumes from recurring snapshots. Prometheus, Loki, Alertmanager, Grafana, ntfy, and autokuma volumes don’t need Longhorn snapshots: their data is either ephemeral or rebuilt from scrapes/configs.

# Remove from the default snapshot group
kubectl label volume.longhorn.io/pvc-95b89bc0... \
  -n longhorn-system \
  recurring-job-group.longhorn.io/default-

2. Purge all Prometheus snapshots.

kubectl get snapshots.longhorn.io -n longhorn-system \
  -l longhornvolume=pvc-95b89bc0... -o name | \
  xargs -I{} kubectl delete {} -n longhorn-system --wait=false

3. Reduce snapshot retention for the remaining volumes (5 daily / 2 weekly, down from 7/4).

4. Restart the instance-manager to force a fresh Go process.

kubectl delete pod -n longhorn-system instance-manager-96c62ec7... --grace-period=30

This causes a brief (~30 second) disruption to all volumes on the node while Longhorn rebuilds the engine processes. Every pod on nuc-worker-01 momentarily lost disk access. They all recovered automatically.

The Result

BEFORE:
nuc-worker-01   30038Mi   95%
instance-manager: 26254Mi

AFTER:
nuc-worker-01   4770Mi    15%
instance-manager: 497Mi

From 95% to 15% memory usage. 26 GiB freed. The cluster now has 27 GiB of headroom on the NUC.

Lessons

Don’t snapshot Prometheus volumes with Longhorn. TSDB compaction creates enormous diffs between snapshots. If you need Prometheus backup, use Thanos sidecar or periodic TSDB snapshots to object storage, not block-level snapshots.

Every container needs resource limits. Even “tiny” sidecars. Not because they’ll use a lot of memory today, but because under pressure, a container without limits can consume everything. Audit with:

kubectl get pods -A -o json | python3 -c "
import json, sys
for pod in json.load(sys.stdin)[items]:
    for c in pod[spec].get(containers, []):
        if not c.get(resources, {}).get(limits, {}):
            print(f\"{pod[metadata][namespace]}/{c[name]}\")"

Network policies are the silent killer of sidecars. Default-deny egress is the right posture, but every sidecar that talks to the Kubernetes API needs an explicit toEntities: [kube-apiserver] egress rule. The symptoms (timeout, crash loop) don’t obviously point to network policy. They look like API server issues.

Go processes don’t return memory. The Go runtime uses MADV_FREE, which marks pages as available but doesn’t force the kernel to reclaim them. A long-running Go process that had a high-water mark will hold that RSS forever. The only fix is a restart. For Longhorn, this means periodic instance-manager restarts are worth considering if you see memory creep.

Check actualSize, not spec.size. Longhorn’s spec.size is the provisioned capacity. The status.actualSize includes snapshot chains and can far exceed the spec. Monitor this: it’s the real disk (and memory) cost.

Full Change Summary

All changes deployed via FluxCD GitOps: four commits to the home-ops repo, reconciled automatically. The whole audit and fix cycle took about an hour.