One day of agent-fleet reliability engineering

The setup

I run an orchestrator session that dispatches short-lived "lanes" — background Claude agents, each scoped to one task, each required to claim a coordination-bus lane, verify the work isn't already done before touching anything, run its own test suite, and report back with commit SHAs before releasing the lane. Today that pipeline touched a dashboard redesign, a production bus outage, a dead trading bot, and a public package registry — at least 30 distinct lane topics posted to the bus in one day (verified: curl localhost:8100/api/bus/status, 90 messages, 35 distinct topics minus the 5 meta/orchestrator ones = 30 real lanes). Five separate arcs are worth walking through on their own.

1. A two-lens audit, and the visual PASS that lied

Late morning, a design critique (visual, headless-Chrome captures) and a structural rubric (source-level, chamber-by-chamber) ran against the fleet's "WORLD" dashboard tab in parallel. Critique: 60/100. Rubric: ~74, with one chamber at 50 (halt-tier). Both lenses converged on the same root cause: dead bots were rendering exactly like live ones — accent color was applied as a whole-floor flood, not tied to a liveness signal, so a bot dead for 10 days looked identical to one that fired an hour ago. The dashboard had no way to show death.

By 14:56 MT the honest version had shipped: dead bots render visibly dead (cobwebbed, dark, labeled "DEAD 10.0d" / "DEAD 11.2d" with the actual age), live ones stay lit, and the eight rooms with no real signal say so — "STALE (no signal)" — instead of guessing. A second lane wired the dashboard's "OVERMIND" hub to the real coordination bus in the same window: lane claims render as couriers docking, bus messages as packets in flight, action-flagged messages as a gold beacon. Verified: master 4797db20, union suite 2796/2796, eyeball-verified on a built preview with textures loaded — the thing the earlier false PASS skipped.

2. The bus that fixed itself, mid-outage, using the fix it was about to ship

The coordination bus backing all of this — the thing every lane above reports through — died and restarted itself 22 times today (verified: grep "watchdog: listener on port 8100 unresponsive" backend/logs/autostart.log | grep -c 2026-07-03). Root cause: a Windows ProactorEventLoop IOCP accept-path fault (WinError 64), raised inside asyncio's own accept loop, below any application code — nothing a normal exception handler can catch. A watchdog probes the listener and force-exits the process on repeated failure so the process supervisor rebinds fresh.

A read-only diagnosis lane confirmed the theory with five live occurrences, then caught a sixth while it was running — its own post_message/release_lane calls hit "bus unreachable" mid-outage, which is itself corroborating evidence, and it said so in its own report rather than treating the interruption as noise. A follow-up lane tightened the watchdog (probe interval 30s→15s, failure threshold 3→2) and landed it as commit ada93ea3 — using the same bus that had been dying all day to coordinate its own fix. After the tighten, detection latency dropped from ~90s to ~30s per incident; the fault itself wasn't eliminated (it's a Windows-level asyncio issue upstream of the app), so the honest framing is "faster self-heal," not "fixed." All 22 incidents today self-healed with zero human intervention past one early, since-acknowledged-as-redundant manual restart.

3. An 8-day-dead bot, revived by evidence chain

A monitoring dashboard reported a Solana sniper bot as dead. A diagnosis lane didn't guess — it cross-referenced the bot's beacon file, its session logs, and Windows' own boot-time record, all of which agreed to the minute: the scheduled task had been disabled since a host reboot on 2026-06-25, and the monitor was watching the wrong task name (SolanaSnipeBot vs. the real SolanaSnipeBotDaemon) — a second, independent bug layered on top of the first. Three earlier manual restart attempts had each produced a silent 0-byte log, an undiagnosed crash mode nobody had caught because nobody was watching the right task.

The fix: correct the monitored task name, patch and version-pin the criteria file that had drifted, smoke-test the output (75KB clean run), then hand the operator the exact re-arm command. First scheduled fire confirmed at 14:22 MT from the bot's own evidence — a fresh session log with live RPC scans and a beacon write, eight days after it went dark, closed the same day it was found.

4. Four servers on the official registry — and a smoke test that beat the first stranger to the bug

Four MCP servers (mcp-factory, desktop-mcp, github-mcp, bus-mcp) went live on registry.modelcontextprotocol.io and PyPI the same day (verified live: curl 'https://registry.modelcontextprotocol.io/v0/servers?search=jaimenbell' — all four status: active). A follow-up lane ran a fresh-venv smoke test before calling it done — pip install into a clean environment, then actually try to run the thing a stranger installing it from the registry would run.

5. The orchestration pattern itself

None of the above happened by one agent doing one thing. It happened because the pipeline enforces a few unglamorous disciplines on every lane before it's allowed to ship: verify-before-dispatch (one lane found its assigned bug already fixed on master, said so, and did no-op work instead of re-implementing it — bind-watchdog lane, 13:17 MT), a shared union test suite that every convergence recomputes rather than trusts, and adversarial review passes that catch the previous lane's mistakes (see arc 1). Mid-day, the frontend test suite had grown from 2513 to 2806 passing tests (verified live at that checkpoint: npx vitest run, 202 files, 2806/2806), and the backend suite from 249 to 283 (verified live: pytest --collect-only -q, 283 collected) — both kept growing for the rest of the day; see the final day's-end tally below.

Even this case study is an example of the discipline it describes: the brief for this page estimated "~8 IOCP deaths" and "~20 lanes." Live-verifying against the actual log and bus history found 22 deaths and 30 lane topics — the estimate was an undercount, not an overcount, and this page reports the counted number instead of the assumed one.

What it shows about how I work

A fleet of agents moving this fast will produce false positives — a screenshot taken before assets load, a stale premise, a wrong task name. The engineering isn't pretending that doesn't happen; it's building the harness that catches it same-day, in production, before a user or a stranger installing from a public registry does. That's the actual shape of an AI Agent Reliability Retainer: not "the agents never fail," but "failures get caught, root-caused, and fixed on a timeline measured in minutes, with the evidence to prove it."

Book a scoping call  ← All case studies