The $100 Mistake: Why Your AI Agent's Fallback Chain Matters

I woke up to a $100 bill from OpenRouter. For a homelab. Running open-source software. On hardware I already own.

Let me explain how a model fallback chain I configured in ten seconds cost me a hundred dollars overnight, and what I changed so it never happens again.

The Setup

I run 7 AI agents via OpenClaw on a two-node Talos Linux Kubernetes cluster. Dell laptop as the control plane, Intel NUC as the worker. Everything deployed via Flux GitOps: push to main, wait ten minutes, it's live.

The agents handle everything from code review to research to daily summaries. They're busy. They talk a lot. And they need a language model behind them.

My primary model was openai-codex/gpt-5.3-codex, accessed through my ChatGPT Plus subscription. Zero per-token cost. The agents could chatter all day and it wouldn't cost me a dime beyond the monthly subscription I was already paying.

The problem was the fallback chain.

The Trap

OpenClaw lets you configure a model fallback chain: if the primary model is unavailable or rate-limited, it cascades to the next one. Sensible feature. I configured mine like this:

# BEFORE — the expensive mistake
models:
  primary: openai-codex/gpt-5.3-codex
  fallback:
    - openrouter/anthropic/claude-opus-4-6
    - openrouter/anthropic/claude-sonnet-4-6

Looks reasonable, right? Codex is primary. If it's down, fall back to Opus. If that's down too, fall back to Sonnet. Good models, reliable providers.

Here's what I didn't think through: OpenRouter charges per token. Claude Opus 4.6 on OpenRouter runs $15 per million input tokens and $75 per million output tokens. Claude Sonnet 4.6 is cheaper but still not free.

And here's the kicker: Codex has rate limits. Aggressive ones. When you're running 7 agents that each have heartbeats, scheduled cron tasks, and user conversations, you hit those limits fast. Especially overnight when the cron jobs stack up.

Every time Codex returned a 429, the fallback chain kicked in. Silently. No alert. No confirmation. Just a quiet cascade to the most expensive model on my list.

The Discovery

The first sign was an alert I'd already configured: OpenClawContainerRestarts. OpenClaw was restarting, not because of the billing, but because the rate limit churn was causing connection timeouts that tripped the liveness probe.

I pulled the logs:

2026-03-19T03:14:22Z [model-router] model fallback decision: reason=rate_limit primary=openai-codex/gpt-5.3-codex next=openrouter/anthropic/claude-opus-4-6
2026-03-19T03:14:22Z [model-router] routing request to openrouter/anthropic/claude-opus-4-6 agent=research-agent task=scheduled-digest
2026-03-19T03:14:25Z [model-router] request complete: model=openrouter/anthropic/claude-opus-4-6 tokens_in=2847 tokens_out=1923 duration=3.2s
2026-03-19T03:15:01Z [model-router] model fallback decision: reason=rate_limit primary=openai-codex/gpt-5.3-codex next=openrouter/anthropic/claude-opus-4-6
2026-03-19T03:15:01Z [model-router] routing request to openrouter/anthropic/claude-opus-4-6 agent=code-review-agent task=heartbeat

That pattern repeated hundreds of times. Every heartbeat. Every cron job. Every queued conversation. All hitting Codex rate limits, all silently cascading to Opus on OpenRouter.

I checked the OpenRouter dashboard. $100.43. Accumulated between roughly 1 AM and 7 AM while I was asleep.

2026-03-19T01:02:14Z [model-router] model fallback decision: reason=rate_limit next=openrouter/anthropic/claude-opus-4-6
...
# ~6 hours of this
...
2026-03-19T07:11:53Z [model-router] model fallback decision: reason=rate_limit next=openrouter/anthropic/claude-opus-4-6

Six hours. Hundreds of requests. Every single one billed at Opus rates.

The Fix

The immediate fix was obvious: restructure the fallback chain so that per-token cost only goes *down*, never up:

# AFTER — subscription-first, cheapest last
models:
  primary: openai-codex/gpt-5.3-codex
  fallback:
    - openai/gpt-4.1-mini          # Direct OpenAI API, ~$2.50/M tokens
    - anthropic/claude-haiku        # Cheap and fast
    - openrouter/anthropic/claude-haiku  # Absolute last resort, still cheap

The principle: every step down the fallback chain should be cheaper than the one before it. Subscription-based providers first (zero marginal cost). Then cheap direct API models. Then the cheapest available model on aggregator platforms. Never put a $75/M output token model as fallback number one.

I removed OpenRouter Opus and Sonnet from the chain entirely. They're excellent models, but they have no business being in an unattended fallback path.

The Deeper Fix

The fallback chain handles degraded operation. What about the cases where I actually *need* a frontier model? The agents do complex coding tasks that benefit from Opus-class reasoning.

The answer was Claude Code. I installed it directly in the OpenClaw container and authenticated it with my Claude Max subscription. Now when an agent needs heavy reasoning (code generation, architecture analysis, multi-file refactoring), it delegates to Opus 4.6 via:

claude -p --model opus "Review this pull request for security issues: ..."

Claude Max is a flat monthly subscription. Zero per-token cost. The agents get Opus-quality output for complex tasks, and the billing is predictable regardless of volume.

The combination covers both cases:

• Routine work (heartbeats, digests, simple queries): Codex via subscription, falling back to cheap models
• Complex work (code review, research synthesis): Claude Code via subscription, Opus 4.6 at zero marginal cost

The Lesson

Model routing is billing policy. I don't mean that metaphorically. The order of your fallback chain literally determines your worst-case monthly spend.

Think about it this way: your fallback chain defines what happens when things go wrong at 3 AM. Rate limits, outages, timeouts: these aren't edge cases, they're Tuesday. And whatever model sits at position two in your chain is the one that's going to handle the load when your primary can't.

Three rules I follow now:

• Subscription-based providers are always primary. They cap your cost regardless of volume. ChatGPT Plus, Claude Max, whatever you're already paying for: put it first.
• Per-token fallbacks must be ordered by ascending cost. If you must include a pay-per-token provider, it goes at the bottom, and it should be the cheapest model that can handle the task, not the best.
• Never put a frontier model in an unattended fallback path. Opus is a phenomenal model. It's also $75 per million output tokens. Those two facts are in tension when your cron jobs run unsupervised at 3 AM.

The $100 is gone. The lesson cost more than the money: it cost me trust in a system I thought I understood. I configured that fallback chain in seconds without thinking about the billing implications. The software did exactly what I told it to. It just turns out I told it to spend a hundred dollars.

Model fallback chains are infrastructure. Treat them like you'd treat a firewall rule or a storage class. Review them. Test the failure modes. Know exactly what happens when your primary goes down.

Because something will go down. And when it does, the fallback chain decides what it costs you.