Subscribe

Operational Lessons from 48 Hours of AI Agent Debugging

Eight hard-won lessons from running seven AI agents on a two-node Talos Linux Kubernetes cluster.

Operational Lessons from 48 Hours of AI Agent Debugging

Running AI agents on Kubernetes sounds like a weekend project until you're three hours into a debugging session and the agent you deployed to help you code is itself stuck in CrashLoopBackOff. I spent 48 hours getting OpenClaw (a platform that runs 7 AI coding agents) stable on my 2-node Talos Linux homelab cluster. Here's what broke and why.

The Setup

  • Cluster: 2-node Talos Linux, Dell laptop (control plane, 15.6 GB RAM) + Intel NUC (worker, 32 GB RAM)
  • GitOps: Flux, reconciling every 10 minutes
  • Storage: Longhorn with 2 replicas across nodes
  • The app: OpenClaw, running 7 AI agents with a Mission Control sidecar for the web UI
  • The goal: Get agents working, persistent, and not bankrupting me

Every lesson below cost me real debugging time. Most of them aren't in any documentation.

1. Backup CronJobs + RWO PVCs = Multi-Attach Errors

Three backup CronJobs (OpenClaw, Ghost, and Life Hub) were all stuck in ContainerCreating. No errors in the pod spec, no image pull issues. Just... waiting.

Warning  FailedAttachVolume  multi-attach error for volume "pvc-abc123"
         Volume is already exclusively attached to one node
         and can't be attached to another

Root cause: Longhorn PVCs are RWO (ReadWriteOnce). The backup pods got scheduled on the Dell while the PVCs were attached to pods running on the NUC. Kubernetes can't mount the same RWO volume on two different nodes simultaneously.

Fix: Add node affinity to all backup CronJobs so they schedule on the same node as the workload that owns the PVC. Also change restartPolicy from OnFailure to Never: OnFailure causes an infinite CrashLoop when a backup fails for a transient reason (like the volume issue), because Kubernetes keeps retrying the failed pod on the same wrong node.

affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 80
        preference:
          matchExpressions:
            - key: homelab.naviauxlab.com/workload-tier
              operator: In
              values: ["primary"]
restartPolicy: Never

This one was obvious in hindsight. RWO means what it says.

2. Relative Workspace Paths Resolve Against CWD, Not the Data Directory

OpenClaw's workspace configuration used relative paths:

workspaces:
  - path: "workspaces/henry-calloway"

The OpenClaw binary's working directory is /app/. So it resolved to /app/workspaces/henry-calloway, which doesn't exist. The actual workspace files live at /home/node/.openclaw/workspaces/henry-calloway on the PVC.

The symptom was subtle: the Control Panel loaded fine, agents appeared configured, but every workspace file showed as MISSING. No error in the logs. The app was perfectly happy looking at an empty directory.

Fix: Use absolute paths.

workspaces:
  - path: "/home/node/.openclaw/workspaces/henry-calloway"

If you're mounting a PVC at a specific path and the app has a different CWD, relative paths will silently resolve to the wrong place. Always use absolute paths in container configs.

3. chmod 700 Fails on PVC Root

The init container ran chmod 700 on the PVC mount point to lock down permissions. Seemed reasonable.

chmod: changing permissions of '/home/node/.openclaw': Operation not permitted

The PVC root directory is created by Longhorn and owned by root:node. The init container runs as UID 1000 (non-root, as it should). A non-root user can't chmod a directory they don't own, even if they have group write access.

The thing is, fsGroup: 1000 in the pod's security context already handles this. Kubernetes applies the fsGroup to all files on the mounted volume, giving the container's group full access. The chmod was redundant. And broken.

Fix: Remove the chmod. Trust fsGroup.

4. OPENAI_API_KEY: REPLACE_ME Blocks the Event Loop

The SOPS-encrypted secret had a placeholder value:

OPENAI_API_KEY: REPLACE_ME

I figured it would fail gracefully: maybe log a warning, skip OpenAI features. Instead, OpenClaw tried to authenticate with OpenAI on every agent request, got a 401 back, and retried. The retry logic ran synchronously and blocked the Node.js event loop.

The cascade: blocked event loop means the HTTP server can't respond to anything, including the liveness probe. Kubelet sees the probe timeout, kills the pod, restarts it, and the cycle repeats. Classic CrashLoopBackOff, but the root cause is a bad API key, not resource exhaustion or a code bug.

Fix: Use a real key or remove the env var entirely. Don't leave placeholders in secrets that get injected into apps that will actually try to use them.

5. Sidecar Probes Need Matching Timeouts

OpenClaw's liveness probe was timing out during heavy agent work: the Node.js process was CPU-bound running LLM response parsing and couldn't serve the health endpoint fast enough. I widened the main container's probe timeout from 3 seconds to 10 seconds. Problem solved.

Except the Mission Control sidecar (a lightweight web UI running in the same pod) started failing its own liveness probe. Same root cause: both containers share the pod's CPU and IO budget. When OpenClaw is thrashing, Mission Control gets starved too. Its default 1-second timeout wasn't enough.

Fix: When you widen probe timeouts on one container in a pod, check every other container in that pod. They share resources, and they'll fail together under load.

# Both containers need generous timeouts
livenessProbe:
  httpGet:
    path: /health
    port: 3000
  timeoutSeconds: 10
  periodSeconds: 30
  failureThreshold: 5

6. Init Container Writes Don't Persist to Main Container Layer

I tried to create a symlink in the init container:

ln -sf /home/node/.openclaw/config /home/node/.config/openclaw

The symlink existed during init. I verified with an ls -la in the init container's command chain. But when the main container started, it was gone. The app couldn't find its config and fell back to defaults.

The issue: init containers and main containers share *volume mounts* but not *filesystem layers*. A symlink written to the container's ephemeral filesystem (not a mounted volume) exists only in that container's overlay layer. When the init container exits and the main container starts, it gets its own fresh overlay.

Fix: Don't use symlinks between the ephemeral filesystem and PVC paths. Instead, point the app at the PVC location directly using environment variables:

env:
  - name: XDG_CONFIG_HOME
    value: "/home/node/.openclaw"

If the app respects XDG_CONFIG_HOME (or has its own config path env var), use that. If it doesn't, the volume mount needs to be at the exact path the app expects.

7. Agents Forget Their Own Documentation

This one isn't a Kubernetes issue. It's an AI operations issue.

Henry, one of the OpenClaw agents, spent 30 minutes debugging a Mission Control sidecar restart. He went through four 10-minute update cycles: checking logs, theorizing about memory pressure, adjusting resource limits, restarting the pod. Each cycle, he'd wait for reconciliation, observe the same failure, and try something new.

The diagnosis he eventually arrived at was already documented in CLUSTER.md, a file in his own workspace that I had written specifically for this class of problem.

The fix was a single sentence: "Remember, you're running on a Talos Kubernetes cluster. There is information on this in your memory and files." He immediately pulled up CLUSTER.md, found the answer, and self-corrected.

Lesson: Documentation for AI agents is only useful if the agent reads it before it starts debugging. Agents have the same problem as junior engineers: they jump to troubleshooting before checking the runbook. The difference is you can't sit an agent down and teach habits. You have to structure the information so it's impossible to miss, or you have to interrupt and redirect.

Put the critical operational context in the system prompt, not in a file the agent might choose not to open.

8. Model Routing Is Billing Policy

OpenClaw supports fallback chains for LLM providers. If the primary model is rate-limited or down, it falls to the next one. Sounds like good resilience engineering.

Here's the problem: I had a subscription-based provider (Anthropic Max) as primary and OpenRouter with Opus 4.6 as the fallback. Subscription means flat monthly cost. OpenRouter means per-token billing, and Opus 4.6 is not cheap.

When the primary hit a transient rate limit during a burst of agent activity, all 7 agents fell through to OpenRouter simultaneously. A few minutes of fallback traffic generated a bill that made the subscription look like a rounding error.

Fix: Subscription-based providers should always be primary *and* secondary if possible. Pay-per-token providers should be last resort, with spend caps configured at the provider level. The fallback chain isn't just a reliability decision. It's a billing policy. Your worst-case spend is determined by whatever's at the bottom of that chain.

Think of it this way: having OpenRouter Opus as an uncapped fallback is like wiring a credit card with no limit to your AI agent's rate limit handler.

The Bigger Picture

Running AI agents on Kubernetes is a new operational domain. The failure modes combine the worst of both worlds: you get all the classic distributed systems problems (scheduling, storage, networking, resource contention) plus a new category of application-level issues that come from the agents themselves (retry storms from bad credentials, agents ignoring their own docs, cost explosions from fallback routing).

None of this is in any runbook yet. The agents can't Google the answer because nobody's written it down. So you build your own operational knowledge base, one incident at a time.

And then (here's the important part) you make sure your agents can actually read it.

*Tags: Kubernetes, AI Agents, OpenClaw, Talos Linux, Self-Hosting, Homelab, Lessons Learned*