When Your AI Agents Starve: Debugging a V8 Garbage Collector Death Spiral in Kubernetes

Seven AI agents went dark for over a day, and I didn't notice until I opened the dashboard and saw 1/2 Ready on the pod. The OpenClaw gateway (the process that routes messages to all seven agents, manages their Slack integration, and runs their cron jobs) had been crash-looping since sometime the previous night. Exit code 137, over and over, like clockwork.

This is the story of a two-phase outage, two different root causes hiding behind the same symptom, and the Kubernetes probe configuration that made it all worse.

The Setup

OpenClaw runs on my two-node Talos Linux homelab, managed via Flux GitOps. The pod has two containers: the main gateway (Node.js, port 18789) and a Mission Control sidecar (Express.js dashboard, port 3456). A 5 Gi Longhorn PVC at /home/node/.openclaw stores agent sessions, memory files, conversation history, and media: everything that makes the agents persistent across restarts.

The pod had been running for four days. During that time, the agents had been active: conversations via Slack and the web UI, cron-triggered morning briefings, cross-agent delegation. Normal operations. What I didn't account for was how much data that generates.

First Investigation

$ kubectl get pods -n openclaw
NAME                        READY   STATUS    RESTARTS
openclaw-859b844d6b-7phjb   1/2     Running   3 (11s ago)

One container running (Mission Control), one crash-looping (the gateway). kubectl describe showed the pattern:

Warning  Unhealthy  Startup probe failed: dial tcp: connection refused
Normal   Killing    Container openclaw failed startup probe, will be restarted

The startup probe was configured for 30 failures at 5-second intervals: a 150-second window. The gateway wasn't starting in time.

I checked what was happening inside:

$ kubectl top pod -n openclaw
NAME                        CPU(cores)   MEMORY(bytes)
openclaw-859b844d6b-7phjb   1010m        1123Mi

$ kubectl exec ... -- ps aux
node  14  99.4  3.3  openclaw-gateway

The gateway process was consuming 99.4% of its 1-core CPU limit and 1,123 Mi of memory. And producing zero logs. Not a single line to stdout. It hadn't even reached the point where it prints "listening on port 18789."

Root Cause #1: The GC Death Spiral

The NODE_OPTIONS were set to --max-old-space-size=1024: a 1 GB V8 heap limit. The gateway's RSS was at 1,079 Mi, meaning the heap was pressed right against the ceiling.

When V8's heap approaches its configured maximum, the garbage collector kicks into high gear. It runs more frequently, scans more aggressively, and tries harder to reclaim memory. But if the application genuinely needs most of that heap (because it's indexing 45 MB of agent data on startup), there's nothing to reclaim. The GC runs, finds nothing to free, returns control briefly, and the application allocates a few more bytes before the GC fires again.

This is a death spiral. The garbage collector consumes 99% of CPU time. Application code barely executes. The gateway never finishes initialization, never binds to port 18789, and the startup probe kills it after 150 seconds. Then it restarts and does it all over again.

The "why now?" was data growth. Four days of agent activity had accumulated 45 MB on the PVC: 21 MB of agent session files, 6.7 MB of memory indexes, 11 MB of media. The gateway indexes all of this on startup. At the original resource settings, there wasn't enough heap headroom to do it.

The First Fix

# Before → After
NODE_OPTIONS: "--max-old-space-size=1024"  →  "--max-old-space-size=1536"
cpu limit:    "1"                           →  "2"
memory limit: 1792Mi                        →  2304Mi
startup probe failureThreshold: 30          →  60  # 300s window

More heap, more CPU for the startup burst, more memory headroom, longer startup window. Applied the deployment, watched the rollout:

$ kubectl rollout status deployment/openclaw -n openclaw
deployment "openclaw" successfully rolled out

CPU settled from 1,425m during startup to 189m at steady state. Memory dropped from 1,560 Mi to 566 Mi. The GC thrash was gone.

Fixed.

Eight Minutes Later

Warning  Unhealthy  Liveness probe failed: context deadline exceeded
Normal   Killing    Container openclaw failed liveness probe, will be restarted

Different probe, different failure mode. The startup probe had passed: the gateway was listening, serving websocket connections, connected to Slack. It had been up for eight minutes, actively handling traffic. Then the liveness probe killed it.

The previous container's logs told the story:

02:35:50 [gateway] listening on ws://0.0.0.0:18789
02:35:50 [slack] socket mode connected
02:35:57 [agents/tool-images] Image resized: 2940x1912px 1.04MB → 100.3KB (-90.5%)
02:40:24 [ws] webchat connected
02:40:39 [agents/tool-images] Image resized: 2940x1912px 1.04MB → 100.3KB
02:41:38 [agents/tool-images] Image resized: 2940x1912px 1.04MB → 100.3KB
02:42:42 [ws] webchat connected
         — silence —
02:43:56 — killed by liveness probe —

The gateway was alive. It was serving traffic. But something made /healthz stop responding.

Root Cause #2: The Event Loop Blockade

The agents/tool-images subsystem was resizing a 2,940 x 1,912 pixel PNG screenshot (a session history image from Mission Control) three times in seven minutes. Each resize operation converts 1.04 MB of PNG into a 100 KB JPEG. This is CPU-intensive work, and in this version of OpenClaw, it happens synchronously on the main Node.js event loop.

Node.js is single-threaded. When the event loop is blocked by image processing, nothing else runs: not websocket messages, not Slack keepalives, and not HTTP health check responses. The /healthz endpoint is just another HTTP request waiting in the queue behind a multi-second image resize operation.

The liveness probe was configured with a 3-second timeout. Image processing takes 2-5 seconds. Three consecutive timeouts (the default failureThreshold: 3) and kubelet kills the container. Total window: 45 seconds of degraded responsiveness.

The container wasn't unhealthy. It was busy.

Here's the part that makes it worse: the container restarted and survived the second time. Why? Because the PVC data was already indexed from the first startup. The warm cache meant faster initialization, lighter GC pressure, and more event loop headroom. The first websocket response after restart took 1,750 ms, dangerously close to the 3-second timeout, but under it. Subsequent responses: 121 ms. Normal.

The second startup was lucky. Any coincidence of cron firing, image processing, and a health check could trigger the same kill.

The Second Fix

# Liveness probe — before → after
timeoutSeconds: 3    →  10
failureThreshold: 3  →  5   # kill window: 45s → 75s

# Readiness probe — before → after
timeoutSeconds: 3    →  10

The 10-second timeout accommodates event loop delays during image processing. The higher failure threshold means 75 seconds of sustained unresponsiveness before kubelet intervenes: long enough to ride out transient CPU spikes, short enough to catch genuine hangs.

After applying:

$ kubectl get pods -n openclaw
openclaw-bc9d694f6-kqcrn   2/2   Running   0   115s

$ kubectl top pod -n openclaw
openclaw-bc9d694f6-kqcrn   47m   469Mi

Stable. Zero restarts. 47 millicores at steady state.

What I Got Wrong

This was a two-phase failure, and I only fixed one phase at a time. The first fix (heap + CPU + startup window) addressed the GC thrash but didn't consider what happens after the startup probe passes. The gateway's HTTP server binds to the port early in initialization. That's when the startup probe succeeds. But background work continues for minutes: indexing sessions, building memory caches, processing queued images. During that post-startup warmup, the liveness probe is already active with production-grade expectations.

In retrospect, the liveness probe should have been part of the first fix. Both probes share the same underlying constraint: the gateway is a heavy Node.js application with a single-threaded event loop, and its responsiveness depends entirely on what else the event loop is doing.

Lessons Learned

1. Startup probe passing does not mean the application is ready. Kubernetes probes check if a port is accepting connections, not whether the application has finished warming up. OpenClaw's HTTP server binds in 40 seconds; its background initialization takes minutes. The gap between "listening" and "ready" is where liveness probes kill you.

2. Default failureThreshold (3) is aggressive for heavy applications. Three consecutive failures at a 15-second interval means 45 seconds of degraded performance triggers a container kill. For a stateless web server, that's reasonable. For a Node.js process doing synchronous image processing with active websocket sessions, it's a hair trigger.

3. Synchronous image processing is a hidden event-loop bomb. The agents/tool-images subsystem never showed up in resource monitoring because it's intermittent: it fires when agents process screenshots. But when it fires, it blocks the event loop for 2-5 seconds, long enough to miss a health check. This is an upstream concern; the fix for now is probe tolerance. The real fix is worker threads.

4. V8 heap ceiling + GC = death spiral, not graceful degradation. When V8 can't allocate past --max-old-space-size, it doesn't crash with an OOM error. It enters a GC loop that consumes all CPU while the application makes zero progress. This looks like a hang, not a memory problem. The only clue was kubectl top showing 99% CPU with zero log output.

5. Fix one probe failure mode, test the next. Startup and liveness are different failure surfaces. They share constraints (CPU, memory, event loop) but fire at different lifecycle phases with different thresholds. Fixing one doesn't protect the other. After any resource change, verify that all probes survive the worst-case scenario, not just the one that was failing.

Next Steps

Alerting. The agents were down for over a day before I noticed. An alert on kube_pod_container_status_restarts_total{namespace="openclaw"} > 2 in the existing kube-prometheus-stack would have caught this within minutes. That's the immediate next action.

Data growth monitoring. The PVC held 45 MB after four days. Agent activity scales linearly: more conversations, more sessions, more media. At some point the startup indexing will outgrow even the current 300-second probe window. I need a Grafana panel tracking PVC usage over time, with an alert at 80% of the thresholds.

Upstream improvements. If OpenClaw adds worker thread support for image processing, the event loop blocking disappears entirely. Worth watching their releases and changelogs.

The Bigger Picture

There's an irony in this outage that's hard to ignore. These seven agents are sophisticated enough to write Kubernetes manifests, analyze security configurations, draft governance documents, and orchestrate complex multi-step tasks across codebases. Henry, the orchestrator, recently redesigned his own Mission Control dashboard, unprompted, because he thought the UX could be better.

But none of them noticed they were dying.

The gateway crash-looped for over a day. The agents had no awareness of it: no self-monitoring, no heartbeat escalation, no "I haven't been able to reach my gateway in 6 hours, maybe I should alert Steven via an alternate channel." They're intelligent within their conversation context but blind to their own infrastructure.

That gap, between capability and self-awareness, is the next frontier for this platform. Not making the agents smarter at their tasks, but giving them basic operational consciousness: the ability to detect that their own container is under memory pressure, that probe timeouts are trending upward, that a restart is imminent. An agent that can say "my heap is at 92% and GC is running every 200 ms, I should reduce my in-memory session cache" would have prevented this outage entirely.

For now, that consciousness lives in Prometheus, Grafana, and a human checking dashboards. But someday, the agents should be their own first responders.

The garbage collector doesn't care how intelligent your agents are. It just needs more heap.