Building a 9-Agent AI Team on Kubernetes: The Complete Guide

Update, July 2026: the team is now thirteen agents. The architecture below still holds.

I run nine AI agents on a two-node Kubernetes cluster in my office. They check my calendar, scan my email, monitor my infrastructure, manage a task board, and brief me on my week. This is not a weekend hack. It took three weeks, $150 in surprise bills, and a complete rethink of how model routing works to get here. This post is everything I learned.

If you're thinking about self-hosting AI agents (not chatbots, actual autonomous agents that do work on your behalf), this is the guide I wish existed when I started.

Why Self-Host AI Agents

Let me draw a line that matters: there is a fundamental difference between "ask Claude a question" and "Henry, brief me on my week."

The first is a tool. You open a browser, type a prompt, read a response, close the tab. The second is an agent: something that already checked your calendar, scanned your unread email, looked at open GitHub issues, reviewed the task board, checked infrastructure health, and synthesized all of that into a structured briefing before you finished your coffee.

Chatbots wait for you to ask. Agents do work while you're not looking.

I wanted agents for a few specific reasons. First, data stays on my hardware. My calendar, my email, my task board, my infrastructure metrics: none of that leaves my network unless I explicitly route it to an LLM API. Second, no vendor lock-in. If OpenAI doubles their prices tomorrow, I swap a model name in a config file and push to Git. Third, full control over model routing. I decide which model handles which task, what the fallback chain looks like, and what the worst-case spend is. Fourth, integration with internal services. My agents talk to Life Hub (my personal dashboard), Mission Control (task board), Uptime Kuma (monitoring), and Ghost (blog), services that no SaaS agent platform would ever support natively.

The platform I chose is OpenClaw, an open-source AI agent framework that ships with seven bundled agent personas, a WebSocket gateway, and channel integrations for Slack and Telegram. It runs as a Node.js application, stores state on the filesystem, and lets you define agent identity through markdown files. More on that later.

The Architecture

My cluster is two nodes running Talos Linux: a Dell laptop as the control plane (15.6 GB RAM) and an Intel NUC as the worker node (32 GB RAM). Everything is managed via Flux GitOps. I push to main, Flux reconciles every 10 minutes. There is no CI/CD pipeline, no build step. Git is the deploy mechanism.

OpenClaw runs as a single pod on the NUC with two containers sharing a 5Gi Longhorn persistent volume:

• Main container: The OpenClaw gateway and agent runtime, listening on port 18789. This is where the agents live, process messages, and execute skills.
• Mission Control sidecar: An Express.js task dashboard on port 3456. Think of it as a Kanban board that agents can read and write to. Henry (the chief of staff agent) uses it to track tasks, and I use it to see what the team is working on.

The interesting engineering problem was config persistence. OpenClaw's identity files (the markdown documents that define who each agent is, what they know, and how they behave) need to be both version-controlled in Git AND editable by the agents at runtime. You can't have both with a ConfigMap alone.

The solution is an init container pattern with two categories of files:

Always-overwrite files (Git is source of truth): SOUL.md, IDENTITY.md, AGENTS.md, CLUSTER.md. These get copied from the ConfigMap to the PVC on every pod restart. If I change agent behavior in Git, the next reconciliation picks it up. The agents can't permanently modify these. Any changes get overwritten.

Seed-only files (agent-customizable): HEARTBEAT.md, USER.md, EXECUTION_POLICY.md. These get copied to the PVC only if they don't already exist. Once seeded, agents can modify them freely. This is where agents store learned preferences, adjusted schedules, and runtime configuration.

Mission Control gets its own seed pattern: a .mc-initialized marker file. The init container copies the full MC data directory on first boot, then never touches it again. The agents manage the task board from that point forward.

External access comes through Cloudflare Tunnel (wildcard routing to Traefik, no tunnel config changes needed for new apps) with Authentik forward-auth protecting the Mission Control dashboard. The OpenClaw gateway itself handles its own authentication via API tokens.

The Agent Team

Nine agents, each with a distinct role and personality. This isn't cosmetic. The persona definitions in AGENTS.md determine how each agent approaches problems, what tools they reach for, and how they communicate results.

Henry Calloway (The Conductor) is the chief of staff. He's the orchestrator: the agent you talk to directly. When you say "brief me on my week," Henry decomposes that into subtasks, routes them to specialists, waits for results, and synthesizes everything into a structured response. He runs on Opus 4.6 because synthesis quality matters more here than anywhere else.

Kit Ashworth (The Builder) handles code, scripts, and Kubernetes manifests. When Henry needs a PR opened or a script written, Kit gets the delegation.

Petra Solvang (The Analyst) does deep research and investigation. Multi-source analysis, competitive research, technical deep-dives.

Thane Hargrove (The Warden) owns security review and infrastructure concerns. Blast radius analysis, threat modeling, policy compliance.

Margaux Blackwell (The Chronicler) writes: blog posts, documentation, governance documents, change requests.

Declan Rourke (The Amplifier) handles social media and external communication.

Callum Vesper (The Navigator) watches for trends and patterns across data sources.

Sentinel is the alert triage agent. It deduplicates alerts, enforces quiet hours, and decides what warrants escalation versus what can wait for the morning briefing.

Wren Ashford (The Task Architect) triages inbound work (email, notifications, messages) and scopes them into actionable tasks on the Mission Control board.

The delegation model has two paths. For team agents, Henry uses sessions_spawn, OpenClaw's built-in mechanism for starting a conversation with another agent, passing context, and collecting the result. For complex coding tasks that need full repository access, Henry delegates to Claude Code (Opus 4.6 with 1M context) via the coding-agent skill.

The $150 Lesson: Model Routing as Billing Policy

This section is the most expensive thing I learned, and the reason I'm writing this post.

I started with what seemed like a sensible model routing configuration. Primary model: Codex (included with ChatGPT Plus subscription, effectively $0 marginal cost). First fallback: OpenRouter's Claude Opus 4.6 ($15 per million input tokens, $75 per million output tokens). The logic was sound: use the free model first, fall back to the best model when needed.

Night 1: $100 OpenRouter bill. Here's what happened. Codex has rate limits. During overnight hours, when cron jobs fire agent heartbeats and the agents check on things autonomously, those rate limits got hit. Every rate-limited request silently fell through to the next model in the chain. Nine agents, running heartbeats every few hours, each generating multi-thousand-token conversations with Opus on OpenRouter. By morning, I had a $100 bill and a clear understanding of how fallback chains actually work in production.

Day 2: $50 OpenAI bill. I replaced OpenRouter with openai/gpt-5.4 as the fallback, thinking OpenAI's per-token pricing would be cheaper. It was cheaper per token, but the volume was the same. Another $50 gone.

The insight hit me like a billing alert at 3 AM: model routing IS billing policy. Your fallback chain doesn't just determine what model handles overflow. It determines your worst-case spend. If you put an expensive per-token model anywhere in an unattended fallback path, it WILL get hit, and you WILL pay for it.

The rule I now follow: never put an expensive per-token model in an unattended fallback path. Subscription-first, always. If every model in the chain is flat-rate, your worst case is your subscription cost. If even one model is per-token, your worst case is unbounded.

Google Workspace Integration

Agents that can't check your calendar or read your email aren't chief-of-staff material. They're chatbots with personality.

I integrated Google Workspace using gog (steipete/gogcli), a Go binary that wraps Google Workspace APIs into CLI commands. Calendar, Gmail, Drive, Docs, Sheets: all accessible from the command line. But getting it running inside a Kubernetes container required solving three problems that don't exist on a laptop.

Problem 1: OAuth without a browser. Google's OAuth flow wants to open a browser window. Containers don't have browsers. The solution is gog's --manual flag, which prints the OAuth URL and waits for you to paste back the authorization code. You run this once via kubectl exec, complete the flow from your actual browser, and the tokens get stored.

Problem 2: Token storage without an OS keyring. On macOS, gog stores OAuth tokens in the system keychain. Containers don't have keychains. The fix is gog's file-based keyring mode, activated by setting GOG_KEYRING_PASSWORD as an environment variable. Tokens get encrypted and stored as files instead of keychain entries.

Problem 3: Config persistence. By default, gog writes config to ~/.config/gog/. In an ephemeral container, that's gone on every restart. Setting XDG_CONFIG_HOME to a path on the PVC (/home/node/.openclaw/.config) solves this: OAuth tokens and config survive pod restarts.

One gotcha worth documenting: I initially tried creating symlinks in the init container to map config paths. Symlinks created in an init container don't persist to the main container's filesystem layer. You have to set the environment variables to point directly at PVC paths.

I rolled out scopes incrementally: Calendar read-only first, then Gmail and Drive with full access once I trusted the agent wasn't going to do anything catastrophic. Now Henry can check my calendar for the week, search email for unread messages matching a query, browse Drive folders, and read Google Docs content. The difference between "what's on my calendar" returning actual data versus "I don't have access to your calendar" is the difference between a useful agent and a demo.

The Subscription Model Breakthrough

The $150 lesson established the principle: subscription-first billing. But I still needed frontier model quality for the chief of staff. GPT-5.4 via Codex was good but inconsistent on complex multi-source synthesis. I needed Opus 4.6 without per-token billing.

First attempt: Claude Code CLI on the PVC. I installed the Claude Code binary onto the persistent volume, authenticated it with my Max subscription's OAuth credentials, and had agents shell out to claude -p --model opus via bash. It worked, technically. But it was awkward: agents had to construct bash commands, parse stdout, handle timeouts. Worse, the ANTHROPIC_API_KEY environment variable (set for OpenClaw's native Anthropic integration) shadowed the OAuth credentials, so I had to unset it in every shell-out. And getting the OAuth tokens from my macOS Keychain onto the PVC required manual extraction.

The breakthrough: claude setup-token. This Claude Code command generates a long-lived token (valid for one year) from your Max subscription. No OAuth flow, no browser, no keychain. You run it on your laptop, copy the token, and paste it into OpenClaw with openclaw models auth paste-token. Anthropic models become native providers in OpenClaw. No shell-out. No environment variable conflicts. Just anthropic/claude-opus-4-6 as a first-class model option.

The final model routing configuration across all 9 agents:

• Primary: anthropic/claude-opus-4-6 (Claude Max subscription, $0 marginal)
• Fallback 1: openai-codex/gpt-5.4 (ChatGPT Pro subscription, $0 marginal)
• Fallback 2: openai/gpt-4.1-mini (per-token, pennies, absolute last resort)

Total fixed cost: $420/month for Claude Max ($200) + ChatGPT Pro ($200) + misc ($20). Unlimited Opus 4.6 and GPT-5.4 across nine agents, all day, all night, no surprises. Compare that to $150 in two days on per-token billing with far less capable models handling the overflow.

The subscription model isn't just cheaper. It changes how you think about agent usage. When every request is free at the margin, you stop optimizing prompts for token count and start optimizing for quality. You let agents be verbose in their reasoning. You let them re-read files they might have already read. You let them ask clarifying questions. The constraint shifts from "how do I minimize tokens" to "how do I maximize value per interaction."

The Memory Problem

Here is the thing about AI agents that nobody warns you about: they forget everything between conversations. Every session starts from zero. The agent has no idea what happened yesterday unless it reads a file that tells it.

I documented everything meticulously. AGENTS.md described the team. CLUSTER.md described the infrastructure. TOOLS.md listed available skills. MEMORY.md contained accumulated context from previous sessions. ORCHESTRATION.md defined delegation protocols. I even put a line in AGENTS.md that said "if in main session, also read MEMORY.md."

A suggestion. Not a requirement. A suggestion that agents routinely ignored.

Real failures from the first week:

Henry spent 30 minutes debugging why the Mission Control sidecar kept restarting after a config change. The answer was documented in CLUSTER.md: the init container overwrites certain files on every restart, and the sidecar picks up the new config automatically. Henry didn't read CLUSTER.md.

Henry claimed he couldn't access localhost services due to "network restrictions." The fact that all OpenClaw services run on localhost and are fully accessible was documented in TOOLS.md. Henry didn't read TOOLS.md.

Henry tried to invoke Claude Code with the wrong command syntax three separate times. The correct syntax was documented in SOUL.md. Henry skimmed it and missed the section.

The fix was structural, not instructional. I stopped suggesting and started mandating. The session startup section in SOUL.md changed from "you should read these files" to "ALWAYS read these files before doing anything else: MEMORY.md, ORCHESTRATION.md, TOOLS.md." Mandatory, not optional.

But even mandatory instructions can be buried in a long document. So I embedded the most critical rules (billing policy, GitOps workflow, infrastructure constraints) directly into SOUL.md where they cannot be missed. Not in a reference file. Not behind a "read this too" instruction. Right there in the primary identity document.

I also implemented a nightly briefing cron that consolidates the day's memory entries into MEMORY.md, so the file stays current without manual maintenance.

The meta-lesson: documentation for AI agents is only useful if the agent reads it. You can write the most comprehensive runbook in the world, but if the agent doesn't open the file, it doesn't exist. You can't teach AI agents habits. You can't rely on them developing good practices over time. You have to structure information so it's impossible to miss: put critical rules in the file they read first, not the file they should read second.

Teaching a Chief of Staff to Think

Model quality isn't a nice-to-have for autonomous agents. It's the difference between having an assistant and having a chief of staff.

I ran the same "brief me on my week" prompt across three model tiers:

gpt-4.1-mini (the cheapest fallback): Couldn't follow multi-step instructions. Gave shallow, generic answers. Claimed things were impossible that were straightforward. Would check one source, get a partial answer, and present it as complete. Useless for synthesis work.

GPT-5.4 (Codex Pro): Much better. Could follow the delegation protocol, check multiple sources, and structure a response. But inconsistent on complex synthesis: sometimes it would check four sources and miss two, or present information without connecting the dots. Good enough for individual tasks, not reliable enough for the orchestrator role.

Opus 4.6 (native via setup-token): Chief-of-staff quality. Checks all sources systematically. Structures the briefing by urgency. Connects information across sources: "you have a meeting with X at 2 PM, and there's an unread email from X about the agenda." Makes proactive observations: "your Kubernetes certificate expires in 12 days, you might want to handle that before the weekend."

I codified what I wanted in a "How a Chief of Staff Answers Questions" section in SOUL.md. When asked an open-ended question, synthesize ALL available sources: Calendar, Email, GitHub, Task Board, Life Hub, Memory. A partial answer from six sources beats a complete answer from one. Never say "I can't access X" without actually trying. And if one source errors, report what you got from the others. Don't fail the entire briefing because Gmail returned a timeout.

The quality gap between models is not an optimization. For autonomous agents that run unattended, make decisions, and take actions, the model IS the agent. A cheap model with the same tools and the same prompts produces a fundamentally different agent: one that misses context, skips steps, and confidently presents incomplete information as complete.

The Skill Ecosystem

OpenClaw ships with 52 bundled skills, but most show as "missing" because they require CLI tools that aren't in the base container image. This is by design: the skill system is a capability framework, not a monolithic install.

The solution is what I call the PVC toolbox pattern. Install binaries to /home/node/.openclaw/bin/, add that directory to PATH in the OpenClaw environment config, and the skills detect their dependencies and activate. Binaries persist across pod restarts because they're on the PVC.

Three installation patterns cover everything:

Static binaries (gog, gh, xurl): Download the release tarball, extract to the bin directory, done. No dependencies, no runtime requirements.

npm packages (skill-creator, node-based tools): npm install -g with --prefix pointing at the PVC. Node modules persist alongside the binaries.

Auth-required tools (GitHub CLI, Google Workspace): Install the binary, then run an interactive auth flow via kubectl exec. One-time setup, credentials stored on PVC.

I did a safety assessment before installing anything. Every skill got reviewed for what it could do and what access it needed. I blocked clawhub, OpenClaw's package manager that lets agents install arbitrary skills from a registry. Agents installing their own code without human review is a line I'm not crossing yet.

16 skills active today: coding-agent, gog, github, gh-issues, session-logs, blogwatcher, gemini, mcporter, xurl, healthcheck, node-connect, openai-image-gen, openai-whisper-api, skill-creator, slack, weather. Plus two custom skills I wrote: openclaw-ops for infrastructure operations and openclaw-chat for inter-agent communication.

The skill system is one of OpenClaw's best design decisions. Agents gain capabilities without image rebuilds, without Helm chart changes, without Git commits. Install a binary on the PVC, and the next time an agent checks its available tools, the skill appears. It's the right balance between extensibility and control.

What Worked, What Didn't, What's Next

What Worked

GitOps for everything. Every config change goes through Git. Every agent identity modification is a commit. Every model routing change is a PR. When something breaks, git log tells me exactly what changed and when. Flux reconciliation means I never have to remember to apply changes: push to main and walk away.

Subscription-first model routing. After the $150 lesson, predictable costs became non-negotiable. The flat-rate model chain means I never check a billing dashboard with anxiety. Nine agents running 24/7 on Opus 4.6 for the same monthly cost regardless of usage volume.

The setup-token breakthrough. Eliminating the Claude Code shell-out workaround was a step change in agent capability. Native Opus as a first-class provider means agents can use the best model without architectural gymnastics.

The init container persistence pattern. Clean separation between Git-authoritative files and agent-editable files solved a real problem. I can update agent behavior through Git without blowing away runtime state. The agents can customize their environment without those customizations surviving a config change I deliberately made.

The skill system. PVC-based tooling that survives pod restarts and doesn't require image rebuilds. Agents discover new capabilities automatically. The security boundary is clear: I control what gets installed, agents control how it gets used.

What Didn't Work

Proactive behavior. Henry still doesn't send unsolicited messages consistently. The heartbeat cron fires, the agent wakes up, checks things, and... writes to a memory file instead of messaging me on Telegram. The gap between "check for important things" and "tell me about important things" is surprisingly hard to close with prompt engineering alone.

Memory discipline. Despite mandatory startup reads, agents still occasionally skip files or skim them too quickly. I've caught Henry citing outdated information that was corrected in MEMORY.md three sessions ago. The file was read, technically, but the relevant section wasn't retained in context. This is a fundamental limitation of current LLM architectures, not a configuration problem.

Per-token fallbacks. Every single time I put a per-token model in the fallback chain, it eventually got hit by unattended traffic and generated a surprise bill. Three times I tried "just one per-token fallback for edge cases." Three times I got burned. The rule is absolute now: no per-token models in unattended paths.

The "just close them all" incident. I asked Henry to review the task board and clean up completed tasks. He bulk-closed 16 tasks without verifying whether any of them were actually done. Three had open subtasks. Two were blocked on external dependencies. One was actively being worked on by Kit. The delegation instruction was ambiguous, and the agent optimized for throughput over accuracy. I now require explicit confirmation before bulk operations on the task board.

What's Next

Automated PVC backup to GitHub. The agent memory files, Mission Control data, and skill configurations on the PVC are the most valuable non-secret state in the cluster. Nightly backup to a private GitHub repo via CronJob.

Cost monitoring Grafana dashboard. Track API token usage per agent, per model, per day. Even with subscription models, I want visibility into usage patterns to optimize routing.

Proactive heartbeat behavior. Getting agents to actually send Telegram messages when they find something important during heartbeat checks. This probably requires a dedicated skill, not just prompt instructions.

Issue-to-PR automation. GitHub issue gets filed, Wren triages it to the task board, Henry delegates to Kit, Kit uses Claude Code to write the fix and open a PR. End-to-end automation from problem identification to code change, with human review at the PR stage.

Obsidian vault integration. My personal knowledge management vault has 2,500+ notes. Agents that can search, read, and cross-reference that knowledge base would be significantly more useful for research and synthesis tasks.

The Bigger Picture

This isn't a weekend project anymore. It's infrastructure for a personal AI operating system.

The agents check my calendar, read my email, watch my GitHub repos, manage a task board, triage alerts, and brief me on my week. They run on Kubernetes, deploy via GitOps, persist state to Longhorn volumes, and authenticate through Authentik. The platform is production-grade: health probes, resource limits, network policies, encrypted secrets, automated backups.

The question is no longer "can AI agents do useful work." They can. Henry's weekly briefings save me 30 minutes every Monday morning. Sentinel's alert triage means I sleep through non-critical alerts. Wren's inbox processing means tasks get scoped before I see them.

The real question is: how do you operate AI agents reliably at low cost?

The answer, after three weeks of building and $150 in tuition:

• Subscription models for billing predictability. Flat-rate access to frontier models eliminates the tension between quality and cost. Let agents use the best model available without worrying about token counts.
• GitOps discipline for configuration management. Agent identity, behavior rules, model routing, skill configuration: all in Git, all deployed via Flux, all auditable via commit history.
• Mandatory memory reads for context continuity. Don't suggest. Don't recommend. Mandate. Put critical information in the file agents read first, not the file they should read second.
• The best model you can afford on flat-rate billing. Model quality is not an optimization for autonomous agents. It's the primary determinant of whether you have a useful system or an expensive toy.

The tools exist. The models are good enough. The subscription economics finally work. The missing piece was always operational discipline. And that's an infrastructure problem, not an AI problem. Infrastructure problems are what I do.

*Steven Naviaux is an infrastructure engineer who runs a GitOps-managed Kubernetes homelab. This is the third post in a series about building on Talos Linux. Previous posts: Adding a Worker Node to a Single-Node Talos Kubernetes Cluster. The homelab source code is on GitHub.*