Right-Sizing a Memory-Starved Kubernetes Cluster

My homelab runs on a single Dell laptop. 15.6 GB of RAM, Talos Linux, Kubernetes. It hosts 14 application workloads alongside everything a cluster needs to function: kube-apiserver, Longhorn for storage, Cilium for networking, Prometheus for monitoring, Authentik for SSO. For months, it worked. Then I looked at the numbers.

Memory limits across all workloads summed to 98% of total node RAM. Not actual usage: limits. Every container was allocated a generous ceiling, and if enough of them spiked simultaneously, the OOM killer would start making choices I didn't want it to make.

This is the story of how I audited every workload, right-sized them to match reality, and brought that overcommit ratio down to a sustainable level, without removing a single application.

The Problem with Defaults

When I first deployed each application, I followed a common pattern: set memory limits to something that felt safe. 256Mi for a small app. 512Mi for a database. 1Gi for anything that seemed hungry. These numbers weren't based on observation. They were guesses dressed up as engineering.

On a cloud cluster with autoscaling, this barely matters. Over-provision and let the cloud bill absorb it. On a single laptop with 15.6 GB of physical RAM, every megabyte counts. The Kubernetes scheduler uses requests to decide where pods land, and the kubelet uses limits to decide when to kill them. Get either wrong, and you're either wasting capacity or inviting instability.

The distinction matters:

• Requests: what the scheduler reserves. If your requests sum to more than available memory, pods go Pending.
• Limits: the hard ceiling. If a container exceeds its limit, the OOM killer terminates it. No warning, no graceful shutdown.

On a single-node cluster, there's a subtlety most multi-node operators never think about: if total limits exceed total RAM, you're betting that not everything will spike at once. That bet has a name, overcommit, and mine was at 98%.

What You Can't Touch

Before tuning anything, I needed to understand what was non-negotiable. Not every consumer of memory is yours to configure.

On a Talos Linux cluster, several components sit outside your Kubernetes manifests entirely:

That's roughly 2.8 Gi of RAM consumed before a single application container starts. On a 15.6 GB node, that leaves about 12.8 Gi for everything else: your monitoring stack, your SSO provider, your actual apps. Knowing this number is the foundation of any honest capacity plan.

Measuring What's Actually Happening

The first step was gathering real data. Not what I thought workloads were using, but what they actually consumed under normal operation.

# Current memory usage by pod, sorted by consumption
kubectl top pods -A --sort-by=memory

# Resource requests and limits for all pods
kubectl get pods -A -o custom-columns=\
  NAMESPACE:.metadata.namespace,\
  NAME:.metadata.name,\
  MEM_REQ:.spec.containers[*].resources.requests.memory,\
  MEM_LIM:.spec.containers[*].resources.limits.memory \
  --sort-by='.metadata.namespace'

# Node-level summary
kubectl top nodes

I ran these commands several times over a few days to capture steady-state behavior, not just a single snapshot. Memory usage varies: Prometheus compacts TSDB blocks, Grafana loads dashboards, SSO handles login bursts. You want the typical operating range, not the minimum.

The Audit

With actual numbers in hand, the picture was striking. Some workloads were over-provisioned by an order of magnitude. Others were silently suffocating.

The Extreme Over-Provisions

The most egregious offender was Miniflux's PostgreSQL sidecar: 31Mi actual usage with a 512Mi limit. That's a 16x overcommit on a single container. The database was handling an RSS reader's workload: a few hundred feeds polled periodically. It didn't need half a gigabyte of headroom.

Other standouts:

• Navidrome: 57Mi actual, 512Mi limit (9x)
• Audiobookshelf: 82Mi actual, 512Mi limit (6x)
• Vaultwarden: 21Mi actual, 256Mi limit (12x)
• Speedtest Exporter: 23Mi actual, 256Mi limit (11x)

Each one individually is harmless: 256Mi here, 512Mi there. But on a single node, they compound. Fourteen workloads each with 2-3 containers, and suddenly those "safe" limits consume your entire node.

The Under-Provision

Grafana was the opposite problem. I had set its limit to 384Mi, but actual usage was 411Mi. It was being OOM-killed intermittently: restarting every few hours, losing in-flight dashboard queries, triggering pod restart alerts. I'd been chasing this as a "Grafana stability issue" when it was a resource limits issue the whole time.

This is the insidious thing about under-provisioned limits. The OOM kill happens silently from the application's perspective. Kubernetes restarts the pod, the readiness probe passes, everything looks healthy in the dashboard, until you notice the restart counter climbing.

# Check for OOM kills in pod events
kubectl get events -A --field-selector reason=OOMKilled

The JVM Tax

Stirling PDF is a Java application. Without explicit heap constraints, the JVM will happily claim as much memory as the container limit allows: it reads the cgroup limit and sizes its heap accordingly. With a 512Mi container limit, the JVM was reserving ~400Mi for heap alone, even though the actual working set was much smaller.

The fix is a single environment variable:

env:
  - name: JAVA_TOOL_OPTIONS
    value: "-Xmx128m"

This caps the JVM heap at 128Mi regardless of the container limit. Combined with reducing the container limit from 512Mi to 384Mi, Stirling PDF dropped from one of the heavier workloads to a manageable 278Mi actual usage. If you run any Java application in Kubernetes without setting -Xmx, you're giving the JVM a blank check drawn on your node's RAM.

The Python Baseline

Mealie, a recipe manager built on Python and FastAPI, was using 415Mi at idle. No active requests, no background processing: just the Python runtime, the ASGI server, and the application framework sitting in memory. This is the reality of Python web applications: the interpreter, loaded modules, and framework abstractions have a high fixed cost.

There's no trick to reduce it. You can't cap Python's memory like a JVM heap. You just have to account for it honestly. I set Mealie's limit to 448Mi: tight, but the workload is stable and doesn't spike significantly under load. It's the largest single-application memory consumer in the cluster.

The Framework

After the audit, I established a systematic approach rather than tuning ad hoc:

1. Measure actual usage over several days under normal load.
2. Set requests at or near actual usage. This tells the scheduler the truth about what the pod needs. Lying here (setting requests low to "fit more stuff") defeats the purpose: the scheduler makes bad decisions with bad data.
3. Set limits at 2-4x actual usage. This provides headroom for legitimate spikes (startup, compaction, burst traffic) without giving workloads a blank check. A container using 100Mi normally might spike to 200Mi during startup; a 2x limit covers that. Something like Prometheus that does periodic TSDB compaction might need 3-4x.
4. Never go below 64Mi. Below this threshold, even trivial operations can trigger OOM kills. It's not worth the savings.

This isn't the only valid approach. Some teams use vertical pod autoscaling. Others use percentile-based calculations from long-term metrics. For a homelab with stable workloads and no autoscaling, the 2-4x rule is simple, predictable, and sufficient.

Tuning Prometheus

Prometheus deserved its own section because it has the most tuning levers and the largest impact. It was consuming nearly 1 Gi with a 90-day retention window and 30-second scrape interval.

Three changes brought it under control:

prometheus:
  prometheusSpec:
    retention: 30d          # was 90d
    retentionSize: "5GB"
    scrapeInterval: 60s     # was 30s
    evaluationInterval: 60s # was 30s
    resources:
      requests:
        cpu: 200m
        memory: 512Mi
      limits:
        cpu: "1"
        memory: 768Mi       # was 1Gi

Retention 90d to 30d. In a homelab, I don't need three months of metrics history. Thirty days covers any reasonable troubleshooting window. Old TSDB blocks age out gradually after the change; don't expect immediate disk or memory savings.

Scrape interval 30s to 60s. Doubling the interval halves the number of samples ingested per minute. For a homelab monitoring stack, 60-second granularity is more than sufficient. You lose the ability to catch sub-minute spikes, but those are rarely actionable in this context anyway.

Memory limit 1Gi to 768Mi. With less data being ingested and retained, the working set shrank. Actual usage settled around 512Mi, giving comfortable headroom at 768Mi.

TSDB compaction is the primary memory spike event for Prometheus. It happens periodically as the write-ahead log is flushed to persistent blocks, and during this window, memory usage can temporarily double. The 2-4x framework accounts for this: 512Mi baseline with a 768Mi limit provides 50% headroom, enough for compaction without risking OOM.

The Before and After

Here's what a right-sized deployment looks like compared to the original, using Miniflux's PostgreSQL sidecar as the example, the most over-provisioned workload in the cluster:

Before

resources:
  requests:
    cpu: 100m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 512Mi

After

resources:
  requests:
    cpu: 100m
    memory: 64Mi
  limits:
    cpu: 500m
    memory: 128Mi

Actual usage: 31Mi. The new limit of 128Mi provides 4x headroom, generous for a PostgreSQL instance serving an RSS reader. The request of 64Mi tells the scheduler the truth: this container reliably uses about 30-60Mi.

Multiply this kind of correction across 14 workloads and the aggregate effect is dramatic.

The Full Picture

Here are the most significant changes, ranked by memory reclaimed:

Grafana is the only workload where the limit went up. That negative number is the cost of fixing an under-provision, but it stopped the OOM kills, which were costing more in disruption than the 128Mi costs in capacity.

Results

After applying all changes and letting workloads settle (allow 10-15 minutes after bulk restarts for memory to stabilize):

• Total memory overcommit: 98% down to 77%
• Actual memory usage: ~8.9 Gi / 15.6 Gi (59%)
• Grafana OOM kills: Eliminated
• Applications removed: Zero

A 21-percentage-point reduction in overcommit, achieved entirely by measuring reality and adjusting configuration to match. No hardware changes, no application removals, no architectural compromises.

The gap between 59% actual and 77% overcommit is intentional. That 18-point buffer is the collective headroom: room for Prometheus compaction, Authentik login bursts, Longhorn rebuilds. It's the difference between a cluster that runs and a cluster that runs reliably.

Lessons

Measure before you tune. Intuition about memory usage is almost always wrong. I would have guessed Miniflux's Postgres needed more than Vaultwarden. The opposite was true, and both were a fraction of what I'd allocated.

The OOM killer doesn't send a Slack notification. Under-provisioned limits manifest as mysterious restarts, not as clear errors. If a pod's restart counter is climbing, check its limits before debugging the application.

JVM applications need explicit heap caps. Without -Xmx, the JVM reads the cgroup limit and claims as much as it can. A 512Mi container limit becomes a 400Mi heap reservation, even if the application only needs 50Mi of live data.

Python has a high floor. FastAPI, Django, Flask: the runtime itself costs hundreds of megabytes. You can't tune it down. Budget for it honestly and don't waste time looking for leaks that aren't there.

Single-node clusters are unforgiving. On a multi-node cluster, the scheduler can spread overcommit across nodes. If one node OOMs, pods reschedule. On a single node, an OOM event means the workload dies and restarts in place, competing with the same neighbors that caused the problem. Getting limits right isn't optimization; it's survival.

Build it right, even when shortcuts are tempting. It would have been easier to slap 512Mi on everything and buy more RAM when things broke. But understanding the actual resource profile of every workload in your cluster is engineering. The shortcut is technical debt that compounds: every new deployment makes the guessing game worse, and eventually the OOM killer makes a choice you can't afford.

What's Next

Right-sizing bought breathing room, but 77% overcommit on a single node is still tight. The next step was adding hardware: a second node to distribute the load and enable storage redundancy.

In Part 2, I walk through joining an Intel NUC as a worker node to this single-node Talos cluster, configuring Longhorn for cross-node replication, and steering workloads with soft affinity: Adding a Worker Node to a Single-Node Talos Kubernetes Cluster.