The fleet audit methodology: 7 detectors, one human judgment call
static analysis surface, don't decide self-audit: blind spot found 781 passing tests (mcp-security-scanner)Up front: every outreach message I send makes the same claim — "a catalog of known destructive-scope and dead-but-green hazards, the scanner surfaces, the expert decides." This page is that claim made checkable: the actual detector families, the actual split between what the tool does and what I do, and one worked example of the tool catching a real problem — including its own.
The premise
Agent fleets fail in a specific, recurring shape: a job reports "done" without checking what it touched, or a credential/permission is scoped wider than the task that uses it, and nobody notices until the wide scope meets a destructive call with no gate in front of it. That's the Railway-token class of incident, the terraform-destroy-from-stale-state class, the confirmation-step-that-echoes-an-unverified-number class. None of these need a novel exploit — they need someone to actually read every scheduled job, wrapper, and IaC/CI file a fleet runs, which is exactly the part nobody has time to do by hand across a real fleet.
The audit is built around a division of labor: static tooling sweeps for candidates, a human reads every candidate and decides. The tool is tuned to over-flag on purpose — a missed finding is worse than a false positive — and the triage that prunes noise down to the handful of findings that matter is the actual deliverable, not the scan itself.
The 7 detector families
This is mcp-security-scanner's actual detector list, reading the source of the
MCP servers and job files a fleet runs — not a marketing summary of it:
| # | Family | What it flags |
|---|---|---|
| 1 | Codegen / template injection | Autoescape off in a tool that renders untrusted fields into generated source; hand-rolled string escaping instead of a real serializer. |
| 2 | Tool-param injection | shell=True, eval/exec on non-constants, unsafe deserialization, SSRF, path traversal. |
| 3 | Auth / network posture | Bind-all with debug on, mutating routes with no auth dependency, no rate limiter on a networked server. |
| 4 | Secret handling | Tracked credential files, hardcoded secret literals, secrets passed to log/print. |
| 5 | Tool-scope creep | A mutating tool registered with no visible gate — no decorator, no env-flag opt-in, no permission check. |
| 6 | Secret-leak-via-response | A tool that hands back a whole env/config object or a secret-shaped value to the calling LLM. |
| 7 | Job hazards | Cron/systemd/CI/deploy scripts: over-broad ACL scope, a destructive call with no confirm-before-destroy gate, success reported without verification (|| true, empty catch{}). |
Every finding a detector produces carries the same four fields: severity
(P0 critical → P3 hardening nit), confidence (high / medium / low), the
offending file:line, and a concrete fix. Nothing ships as a bare "this looks bad."
Language coverage, honestly: detection is deep for Python (AST-based).
JS/TS has no AST path in this scanner — it's line-based, string-aware regex instead. Four of
the seven families carry JS/TS parity on this basis today (param-injection, tool-scope-creep,
secret-leak-via-tool-response, and secret-handling's secret-in-log check); codegen-injection
and auth-posture stay Python-only by deliberate scope decision, not an oversight. Documented
gaps on the JS/TS side — optional-chaining eval, spread-of-secret returns,
.jsx/.tsx/.cjs not yet collected — are stated, not
silently missed.
Where the tool stops and the human starts
This is the part every DM I send names explicitly, because it's the actual differentiator: the scanner surfaces, the expert decides. Concretely —
- The tool flags a credential or permission grant that looks wider than the detector's heuristic expects for that call site. It cannot know what the job genuinely needs; it only knows the shape looks broad.
- I read the actual grant against the actual job — what the automation does, what it would break if the scope were narrowed, what the blast radius is if that exact grant were misused — before calling it a real finding. Most audits will find the tool over-flags by a wide margin; that gap between raw flags and confirmed findings is the triage work, not overhead around it.
- Every finding that reaches a client report as critical has been hand-verified and, where practical, reproduced — not just pattern-matched.
Worked example: the self-audit catching the tool's own blind spot
--self-audit points the scanner at the fleet of MCP servers it was built
alongside, locked by a real test in the suite (tests/test_self_audit.py) and
reproducible yourself rather than a one-off demo run: eight of my own MCP servers scanned, P0=0 fleet-wide, six returned nothing, two flagged — github-mcp (2x P1/LOW, Hardcoded Bearer-prefixed token in source (fake-marker, test-path), honestly reported) and discord-mcp (1x P1/LOW, Hardcoded secret assigned to 'TEST_TOKEN' (fake-marker, non-credential-shape, mock-name, test-path), honestly reported). No other server carries a P0 or P1.
Read that as a regression gate, not a security result. Measured blind against five third-party MCP servers on 2026-07-29, this scanner returned 0 true positives across 58 findings and recall 0/69 (0/7 restricted to the three classes it implements). Recall fixes the next day raised it to two findings that survive a hand audit — one of real-world consequence — and pooled recall has not been re-scored since, so no current number is claimed either way. A server that returns nothing here has produced silence, not a clean bill. I pulled the paid audit tier the day that first measurement landed. This table shows the self-audit runs in CI and that the report format is real; it is not evidence that any server listed is secure.
Live sample run (2026-08-19 — regenerated by scripts/generate_self_audit_table.py, not hand-typed; re-run it and commit whenever the fleet changes. Per-server P2/ungraded counts still drift
run-to-run as detectors evolve and are deliberately not part of the headline
above, only the P0/P1 tally is):
NO P0/P1 mcp-factory
FLAGGED github-mcp P1=2
NO P0/P1 bus-mcp
NO P0/P1 desktop-mcp
NO P0/P1 rag-mcp
FLAGGED discord-mcp P1=1
NO P0/P1 rails-mcp
NO P0/P1 vllm-ops-mcpThe honest caveat, printed on every report the tool produces: this is static analysis, not a prover. It reads code; it does not run the server and does not prove a finding is remotely exploitable. A clean bill means "these detectors found no critical/high pattern" — not a security guarantee. That boundary is the whole reason the human-triage step exists at all.
One of these detectors later produced a CVE, and that does not change the recall number above. Outside the frozen five-server corpus, the codegen-injection detector flagged a real code injection defect in an AWS MCP server, now CVE-2026-85654 at CVSS 7.8. The write-up is I found a CVE with a scanner that misses almost everything, and it leads with the 0 of 69, because one credited coordination is one credited coordination, not a detection track record.
What this means for a Week-1 engagement
A client hands over the scheduled jobs, wrappers, and IaC/CI files their fleet runs on. The static sweep runs against all seven families first — fast, and it does not miss the pattern by design. Then every candidate gets read by hand against what that job actually needs, every critical gets reproduced, and what comes back is a ranked fix list where each line has a file, a line number, and a reason — not a generic checklist. Whether the harness gets built after (freshness asserts, honest exit codes, a status surface that knows "alive" from "actually producing") is a separate decision; the audit stands alone.
If you want to run a similar checklist against your own server, there is a free 10-minute self-score on the products page. It asks for an email and nothing else, and it is the same lens as the paid audit, one rung down.