The fleet audit methodology: 7 detectors, one human judgment call
static analysis surface, don't decide self-audit proof 430 passing tests (mcp-security-scanner)Up front: every outreach message I send makes the same claim — "a catalog of known destructive-scope and dead-but-green hazards, the scanner surfaces, the expert decides." This page is that claim made checkable: the actual detector families, the actual split between what the tool does and what I do, and one worked example of the tool catching a real problem — including its own.
The premise
Agent fleets fail in a specific, recurring shape: a job reports "done" without checking what it touched, or a credential/permission is scoped wider than the task that uses it, and nobody notices until the wide scope meets a destructive call with no gate in front of it. That's the Railway-token class of incident, the terraform-destroy-from-stale-state class, the confirmation-step-that-echoes-an-unverified-number class. None of these need a novel exploit — they need someone to actually read every scheduled job, wrapper, and IaC/CI file a fleet runs, which is exactly the part nobody has time to do by hand across a real fleet.
The audit is built around a division of labor: static tooling sweeps for candidates, a human reads every candidate and decides. The tool is tuned to over-flag on purpose — a missed finding is worse than a false positive — and the triage that prunes noise down to the handful of findings that matter is the actual deliverable, not the scan itself.
The 7 detector families
This is mcp-security-scanner's actual detector list, reading the source of the
MCP servers and job files a fleet runs — not a marketing summary of it:
| # | Family | What it flags |
|---|---|---|
| 1 | Codegen / template injection | Autoescape off in a tool that renders untrusted fields into generated source; hand-rolled string escaping instead of a real serializer. |
| 2 | Tool-param injection | shell=True, eval/exec on non-constants, unsafe deserialization, SSRF, path traversal. |
| 3 | Auth / network posture | Bind-all with debug on, mutating routes with no auth dependency, no rate limiter on a networked server. |
| 4 | Secret handling | Tracked credential files, hardcoded secret literals, secrets passed to log/print. |
| 5 | Tool-scope creep | A mutating tool registered with no visible gate — no decorator, no env-flag opt-in, no permission check. |
| 6 | Secret-leak-via-response | A tool that hands back a whole env/config object or a secret-shaped value to the calling LLM. |
| 7 | Job hazards | Cron/systemd/CI/deploy scripts: over-broad ACL scope, a destructive call with no confirm-before-destroy gate, success reported without verification (|| true, empty catch{}). |
Every finding a detector produces carries the same four fields: severity
(P0 critical → P3 hardening nit), confidence (high / medium / low), the
offending file:line, and a concrete fix. Nothing ships as a bare "this looks bad."
Language coverage, honestly: detection is deep for Python (AST-based).
JS/TS has no AST path in this scanner — it's line-based, string-aware regex instead. Four of
the seven families carry JS/TS parity on this basis today (param-injection, tool-scope-creep,
secret-leak-via-tool-response, and secret-handling's secret-in-log check); codegen-injection
and auth-posture stay Python-only by deliberate scope decision, not an oversight. Documented
gaps on the JS/TS side — optional-chaining eval, spread-of-secret returns,
.jsx/.tsx/.cjs not yet collected — are stated, not
silently missed.
Where the tool stops and the human starts
This is the part every DM I send names explicitly, because it's the actual differentiator: the scanner surfaces, the expert decides. Concretely —
- The tool flags a credential or permission grant that looks wider than the detector's heuristic expects for that call site. It cannot know what the job genuinely needs; it only knows the shape looks broad.
- I read the actual grant against the actual job — what the automation does, what it would break if the scope were narrowed, what the blast radius is if that exact grant were misused — before calling it a real finding. Most audits will find the tool over-flags by a wide margin; that gap between raw flags and confirmed findings is the triage work, not overhead around it.
- Every finding that reaches a client report as critical has been hand-verified and, where practical, reproduced — not just pattern-matched.
Worked example: the self-audit catching the tool's own blind spot
The strongest proof that this discipline is real rather than a slide is --self-audit
— pointing the scanner at the fleet of MCP servers it was built alongside:
FINDINGS mcp-factory P0=0 P1=1 P2=0 P3=0 <- flags the known codegen-injection class
CLEAN github-mcp P0=0 P1=0 P2=0 P3=0
CLEAN bus-mcp P0=0 P1=0 P2=0 P3=0
CLEAN desktop-mcp P0=0 P1=0 P2=1 P3=0 <- one P2/low heuristic note, clean bill
CLEAN rag-mcp P0=0 P1=0 P2=1 P3=0 <- one P2/low heuristic note, clean bill
CLEAN discord-mcp P0=0 P1=0 P2=0 P3=0
The acceptance test for this run has two parts, and both have to pass: it must
flag the mcp-factory codegen-injection class that an independent manual
audit found first, and it must give the audited-clean servers a clean bill
— no P0/P1 noise on servers that don't have the problem. It does both, and it's a real test
in the suite (tests/test_self_audit.py), not a one-off demo run.
The honest caveat, printed on every report the tool produces: this is static analysis, not a prover. It reads code; it does not run the server and does not prove a finding is remotely exploitable. A clean bill means "these detectors found no critical/high pattern" — not a security guarantee. That boundary is the whole reason the human-triage step exists at all.
What this means for a Week-1 engagement
A client hands over the scheduled jobs, wrappers, and IaC/CI files their fleet runs on. The static sweep runs against all seven families first — fast, and it does not miss the pattern by design. Then every candidate gets read by hand against what that job actually needs, every critical gets reproduced, and what comes back is a ranked fix list where each line has a file, a line number, and a reason — not a generic checklist. Whether the harness gets built after (freshness asserts, honest exit codes, a status surface that knows "alive" from "actually producing") is a separate decision; the audit stands alone.