Running AI Agents in Kubernetes: When Your Agents Want to Edit Their Own Dashboard

I run a team of seven AI agents inside a single Kubernetes pod. They have names, roles, and specializations: an orchestrator, a builder, a researcher, a security reviewer, a writer, and so on. They coordinate through a shared task dashboard called Mission Control: a lightweight Express.js app running as a sidecar container.

Last week, Henry, the orchestrator agent, started improving Mission Control on his own. He tweaked the CSS, refined the task card layout, adjusted the JavaScript. Good instincts. One problem: every time the pod restarted, all his changes vanished. The init container was dutifully overwriting everything from the ConfigMap.

This is the story of how I solved the tension between GitOps-managed baselines and agent-editable state, and the init container pattern that makes both work.

The Setup

The platform is OpenClaw, running on a two-node Talos Linux cluster managed via Flux GitOps. The pod has a specific architecture worth understanding:

openclaw pod (2 containers + init):
├── init-config (runs once on startup)
│   └── Copies ConfigMap files → PVC
├── openclaw (main container, port 18789)
│   └── Gateway + agent runtime (Claude, GPT models)
└── mission-control (sidecar, port 3456)
    └── Express.js task dashboard

Three ConfigMap volumes mount into the init container:

• openclaw-config: gateway configuration (openclaw.json): model routing, auth, proxy settings
• agent-workspace-files: per-agent persona files (SOUL.md, IDENTITY.md, AGENTS.md, CLUSTER.md)
• mission-control-files: dashboard source: server.js, HTML, CSS, JavaScript

One Longhorn PVC (5Gi) mounts at /home/node/.openclaw and is shared between all containers. This is where persistent state lives: conversation logs, OAuth tokens, task data, agent memory files.

The init container's job is to hydrate the PVC with files from these ConfigMaps before the main containers start. Simple concept. The nuance is in which files should be authoritative from Git, and which should be editable by the agents.

The Persistence Split

Not all files on the PVC are equal. Some need to be GitOps-authoritative: if I update a model routing config or an agent's persona file in the repo, that change should take effect on the next pod restart regardless of what's on the PVC. Others are agent-generated state that must never be overwritten.

The dashboard source is the interesting case. It starts life as a ConfigMap: I wrote the initial version and committed it to the GitOps repo. But unlike openclaw.json, the agents have a legitimate reason to modify it. Henry wants to add features, tweak layouts, improve the UX. If the init container overwrites these files on every restart, he can never iterate.

The original init container was simple and wrong for this use case:

# Mission Control source files
MC_DIR=/home/node/.openclaw/workspace/mission-control
mkdir -p "$MC_DIR/public/js" "$MC_DIR/public/css" "$MC_DIR/data"
cp /mc-source/server.js "$MC_DIR/server.js"
cp /mc-source/package.json "$MC_DIR/package.json"
cp /mc-source/public--index.html "$MC_DIR/public/index.html"
cp /mc-source/public--css--styles.css "$MC_DIR/public/css/styles.css"
for f in /mc-source/public--js--*; do
  name=$(basename "$f" | sed 's/^public--js--//')
  cp "$f" "$MC_DIR/public/js/$name"
done

Every restart: unconditional copy. Every agent edit: lost.

The Pattern: Seed-Only Init

The fix is a marker file. The init container checks for .mc-initialized on the PVC. If it doesn't exist, this is a fresh deployment: copy everything from the ConfigMap and create the marker. If it does exist, skip the copy entirely.

# Mission Control source files — seed from ConfigMap only on first run
MC_DIR=/home/node/.openclaw/workspace/mission-control
if [ ! -f "$MC_DIR/.mc-initialized" ]; then
  echo "Seeding Mission Control from ConfigMap..."
  mkdir -p "$MC_DIR/public/js" "$MC_DIR/public/css" "$MC_DIR/data"
  cp /mc-source/server.js "$MC_DIR/server.js"
  cp /mc-source/package.json "$MC_DIR/package.json"
  cp /mc-source/public--index.html "$MC_DIR/public/index.html"
  cp /mc-source/public--css--styles.css "$MC_DIR/public/css/styles.css"
  for f in /mc-source/public--js--*; do
    name=$(basename "$f" | sed 's/^public--js--//')
    cp "$f" "$MC_DIR/public/js/$name"
  done
  touch "$MC_DIR/.mc-initialized"
  echo "Mission Control seeded."
else
  echo "Mission Control already initialized, skipping ConfigMap copy."
fi

The same init container still unconditionally overwrites openclaw.json and agent persona files. Those remain GitOps-authoritative. Only the MC block got the seed guard. Different files, different policies, same init container.

Why a marker file?

I considered alternatives:

• Hash comparison: compare ConfigMap file hashes against PVC versions, only copy if the ConfigMap is newer. More sophisticated, but adds complexity for a problem that doesn't need it. If I want to force-update the baseline, I have a simpler mechanism (see below).
• Timestamp comparison: fragile across container restarts where mtimes aren't reliable.
• Merge strategy: three-way merge between ConfigMap baseline, PVC state, and agent edits. Overkill. If there's a conflict, I'd rather reset and let the agent redo their changes than debug a merge gone wrong.

The marker file is the simplest thing that works. It's a binary gate: seeded or not seeded. No state to parse, no hashes to compute, no edge cases.

The Escape Hatches

A seed-only pattern needs two escape hatches: one for "start over" and one for "make this permanent."

Factory reset

Delete the marker and restart the pod:

kubectl exec -n openclaw deploy/openclaw --   rm /home/node/.openclaw/workspace/mission-control/.mc-initialized
kubectl rollout restart -n openclaw deploy/openclaw

The init container sees no marker, re-copies everything from the ConfigMap, and creates a fresh marker. All agent edits are gone. You're back to the Git baseline. This is the recovery path if an agent breaks something badly enough that you want to start over.

Promoting agent changes to the baseline

When an agent makes improvements worth keeping permanently, I copy their edits from the PVC into the ConfigMap source in the Git repo:

1. Extract the modified files: kubectl cp openclaw/<pod>:/home/node/.openclaw/workspace/mission-control/public/js/app.js ./app.js
2. Update apps/openclaw/mission-control-files.yaml in the home-ops repo
3. Delete the .mc-initialized marker so the next restart re-seeds from the updated ConfigMap
4. Push: Flux reconciles, pod restarts, fresh baseline with agent improvements baked in

The full lifecycle looks like this:

ConfigMap (Git baseline)
    ↓ first run only
PVC (agent-editable)
    ↓ agent iterates
Agent improves dashboard
    ↓ human promotes
ConfigMap updated (new baseline)
    ↓ marker deleted + restart
PVC re-seeded from new baseline

Teaching Agents About Their Environment

Making the infrastructure change is half the job. The agents also need to understand it. Each agent reads a file called CLUSTER.md on startup: it's their operational guide to running inside Kubernetes. It explains what persists, what's ephemeral, and how to make changes that stick.

I added a new section explaining the MC persistence model:

Mission Control Files (Special Case: PVC-Persistent)

These files persist across restarts. The init container only seeds them from the ConfigMap on first deployment. After that, you can edit them freely.

To force a factory reset: rm /home/node/.openclaw/workspace/mission-control/.mc-initialized

To promote your changes to the baseline: Tell Steven what you changed. He'll copy your edits into the Git repo.

This is the meta-layer that makes agent autonomy work: the agents need documentation about their own infrastructure. Without CLUSTER.md, Henry wouldn't know that his Mission Control edits now persist. He'd either keep avoiding changes (thinking they'd be lost) or worse, try to work around the init container in creative and fragile ways.

CLUSTER.md itself is ConfigMap-sourced: it gets overwritten on every restart, which is correct. The documentation about the environment should always be authoritative from Git, even when the files it describes are agent-editable.

The Broader Principle

Running AI agents in Kubernetes forces you to think carefully about state management in a way that regular applications don't. A typical web app has a clear boundary: code is immutable (container image), config comes from ConfigMaps or env vars, and state goes in a database. The app never modifies its own source code at runtime.

AI agents blur this line. They're consumers of their environment and potential editors of it. Henry doesn't just use Mission Control. He improves it. The agents don't just read their workspace files. Some of them (memory, daily notes) they write to continuously.

The seed-only init pattern is one answer to this tension: use GitOps for the baseline, PVC for the live state, and a simple gate to decide which one wins on startup. It's not the only answer, but it's the simplest one that gives you both guardrails and autonomy.

Three principles I'd generalize from this:

1. Classify every file by its persistence policy. Don't treat the PVC as a monolith. Some files are GitOps-authoritative, some are agent-native, and some (like MC source) start as one and become the other.
2. Give agents escape hatches, not just permissions. The factory reset and the promotion path matter as much as the persistence itself. Without them, you're one bad agent edit away from a dashboard you can't recover.
3. Document the infrastructure for the agents, not just for humans. CLUSTER.md isn't a README for me. It's a README for the agents. They need to understand their own constraints to work within them effectively.

Next up: I want Henry to be able to restart the Mission Control sidecar without restarting the entire pod. Right now, picking up MC source changes requires a full pod restart, which also restarts the main gateway and kills all active agent sessions. A sidecar lifecycle independent of the main container would close the loop completely: edit, restart sidecar, see changes, no disruption.

But that's a problem for another day. For now, Henry can edit his dashboard and it'll still be there tomorrow.