One day of agent-fleet reliability engineering

The setup

I run an orchestrator session that dispatches short-lived "lanes" — background Claude agents, each scoped to one task, each required to claim a coordination-bus lane, verify the work isn't already done before touching anything, run its own test suite, and report back with commit SHAs before releasing the lane. Today that pipeline touched a dashboard redesign, a production bus outage, a dead trading bot, and a public package registry — at least 30 distinct lane topics posted to the bus in one day (verified: curl localhost:8100/api/bus/status, 90 messages, 35 distinct topics minus the 5 meta/orchestrator ones = 30 real lanes). Five separate arcs are worth walking through on their own.

1. A two-lens audit, and the visual PASS that lied

Late morning, a design critique (visual, headless-Chrome captures) and a structural rubric (source-level, chamber-by-chamber) ran against the fleet's "WORLD" dashboard tab in parallel. Critique: 60/100. Rubric: ~74, with one chamber at 50 (halt-tier). Both lenses converged on the same root cause: dead bots were rendering exactly like live ones — accent color was applied as a whole-floor flood, not tied to a liveness signal, so a bot dead for 10 days looked identical to one that fired an hour ago. The dashboard had no way to show death.

By 14:56 MT the honest version had shipped: dead bots render visibly dead (cobwebbed, dark, labeled "DEAD 10.0d" / "DEAD 11.2d" with the actual age), live ones stay lit, and the eight rooms with no real signal say so — "STALE (no signal)" — instead of guessing. A second lane wired the dashboard's "OVERMIND" hub to the real coordination bus in the same window: lane claims render as couriers docking, bus messages as packets in flight, action-flagged messages as a gold beacon. Verified: master 4797db20, union suite 2796/2796, eyeball-verified on a built preview with textures loaded — the thing the earlier false PASS skipped.

2. The bus that fixed itself, mid-outage, using the fix it was about to ship

The coordination bus backing all of this — the thing every lane above reports through — died and restarted itself 22 times today (verified: grep "watchdog: listener on port 8100 unresponsive" backend/logs/autostart.log | grep -c 2026-07-03). Root cause: a Windows ProactorEventLoop IOCP accept-path fault (WinError 64), raised inside asyncio's own accept loop, below any application code — nothing a normal exception handler can catch. A watchdog probes the listener and force-exits the process on repeated failure so the process supervisor rebinds fresh.

A read-only diagnosis lane confirmed the theory with five live occurrences, then caught a sixth while it was running — its own post_message/release_lane calls hit "bus unreachable" mid-outage, which is itself corroborating evidence, and it said so in its own report rather than treating the interruption as noise. A follow-up lane tightened the watchdog (probe interval 30s→15s, failure threshold 3→2) and landed it as commit ada93ea3 — using the same bus that had been dying all day to coordinate its own fix. After the tighten, detection latency dropped from ~90s to ~30s per incident; the fault itself wasn't eliminated (it's a Windows-level asyncio issue upstream of the app), so the honest framing is "faster self-heal," not "fixed." All 22 incidents today self-healed with zero human intervention past one early, since-acknowledged-as-redundant manual restart.

3. An 8-day-dead bot, revived by evidence chain

A monitoring dashboard reported a Solana sniper bot as dead. A diagnosis lane didn't guess — it cross-referenced the bot's beacon file, its session logs, and Windows' own boot-time record, all of which agreed to the minute: the scheduled task had been disabled since a host reboot on 2026-06-25, and the monitor was watching the wrong task name (SolanaSnipeBot vs. the real SolanaSnipeBotDaemon) — a second, independent bug layered on top of the first. Three earlier manual restart attempts had each produced a silent 0-byte log, an undiagnosed crash mode nobody had caught because nobody was watching the right task.

The fix: correct the monitored task name, patch and version-pin the criteria file that had drifted, smoke-test the output (75KB clean run), then hand the operator the exact re-arm command. First scheduled fire confirmed at 14:22 MT from the bot's own evidence — a fresh session log with live RPC scans and a beacon write, eight days after it went dark, closed the same day it was found.

4. Four servers on the official registry — and a smoke test that beat the first stranger to the bug

Four MCP servers (mcp-factory, desktop-mcp, github-mcp, bus-mcp) went live on registry.modelcontextprotocol.io and PyPI the same day (verified live: curl 'https://registry.modelcontextprotocol.io/v0/servers?search=jaimenbell' — all four status: active). A follow-up lane ran a fresh-venv smoke test before calling it done — pip install into a clean environment, then actually try to run the thing a stranger installing it from the registry would run.

5. The orchestration pattern itself

None of the above happened by one agent doing one thing. It happened because the pipeline enforces a few unglamorous disciplines on every lane before it's allowed to ship: verify-before-dispatch (one lane found its assigned bug already fixed on master, said so, and did no-op work instead of re-implementing it — bind-watchdog lane, 13:17 MT), a shared union test suite that every convergence recomputes rather than trusts, and adversarial review passes that catch the previous lane's mistakes (see arc 1). The frontend test suite grew from 2513 to 2806 passing tests across the day (verified live: npx vitest run, 202 files, 2806/2806); the backend suite grew from 249 to 283 (verified live: pytest --collect-only -q, 283 collected).

Even this case study is an example of the discipline it describes: the brief for this page estimated "~8 IOCP deaths" and "~20 lanes." Live-verifying against the actual log and bus history found 22 deaths and 30 lane topics — the estimate was an undercount, not an overcount, and this page reports the counted number instead of the assumed one.

What it shows about how I work

A fleet of agents moving this fast will produce false positives — a screenshot taken before assets load, a stale premise, a wrong task name. The engineering isn't pretending that doesn't happen; it's building the harness that catches it same-day, in production, before a user or a stranger installing from a public registry does. That's the actual shape of an AI Agent Reliability Retainer: not "the agents never fail," but "failures get caught, root-caused, and fixed on a timeline measured in minutes, with the evidence to prove it."

Book a scoping call  ← All case studies