An invisible position, a same-day fix, and the hardening that follows it

// reliability incident · root cause + fix · own fleet, paper-only

reconciler bug silent-failure class same-day hardening 17 new tests, own fleet's own suite

Up front, because honesty is the whole point of this page: this is my own paper-trading fleet, not a client engagement — no client name, no P&L figure, nothing here is a live-money claim. What it shows is exactly the failure class the audit exists to catch, on infrastructure I built and had to debug myself, with the actual commits as proof.

The incident

A reconciler in a paper-trading day-trader bot compares broker-held positions against the bot's own local position records every cycle, and flags anything that doesn't match as status="orphan" for manual review. A symbol-matching bug (already fixed in an earlier commit, but the live process was still running the pre-fix build) made the reconciler fail to match a genuinely-held options position against the broker's record — so it flagged a real, live position as orphaned.

That flag turned out to be worse than a false alarm: get_open_positions() filters strictly on status == "open", and every exit path in the system — day_trader.check_exits and the end-of-day sweep — reads from that same filtered list. An orphaned position isn't flagged loudly and left alone; it becomes invisible to every mechanism that would ever close it. No alert fired. No exit path caught it. The only thing that actually recovered the position that night was a manual file edit.

The fix — same day, then hardened past "fixed"

The immediate fix (a prior commit) corrected the symbol-matching bug so new reconciles stop producing false orphans. But a point fix doesn't answer the harder question: what happens the next time something — this bug's cousin, a broker API hiccup, anything — flags a real position wrongly? The hardening pass that followed the same evening answered that directly, with four concrete specs against broker/reconciler.py, day_trader.py, position_manager.py, and eod_sweep.py:

Orphan self-heal. Every reconcile cycle now re-verifies each pre-existing orphan-flagged position against the broker. If the broker still holds the lot, the position self-heals back to status="open" (adopting the broker's quantity as ground truth) and writes an orphan_recovered journal event. A wrong orphan flag is no longer a dead end — it's a quarantine state with a way back out.
Aging + escalation. If the broker genuinely doesn't hold the position, the quarantine ages (orphan_cycles) and escalates to a loud ERROR after a fixed number of cycles — a real orphan can no longer sit silent indefinitely either.
Loud visibility. The daily stats log and the end-of-day sweep now count and WARNING/ERROR-log any non-open, non-closed position — before the sweep's own "nothing to do" early return, which is exactly the code path that let the original incident stay silent.
A standing invariant. Every broker-held lot must now map to exactly one live-exit-eligible record: zero matches raises an UNTRACKED warning, more than one raises a loud ERROR. Recovered orphans count as tracked, so the fix doesn't trade one false alarm for another.

Evidence: proof this isn't just a description

17 new tests shipped with the hardening — tests/test_orphan_lifecycle.py and tests/test_exit_visibility.py — covering the self-heal path, the escalation path, the invariant check, and the sweep's visibility change.
Merged via a 2-vote review tagged MERGE-SAFE, not a solo self-approval.
The bot's full suite is 1,002 tests, live-run in its own environment as part of verifying this write-up — not a number carried over from an old note.

Why this is the actual pitch

The failure class here — an automation quietly loses the ability to act on the thing it's supposed to be watching, with no alert — is precisely detector family #7 (job hazards: success-reported-without-verification) in the audit's methodology, and it's the same shape as "a job reports done without checking what it touched." I didn't just write the detector for this pattern in the abstract; I lived it on my own infrastructure, root-caused it, and shipped the same-day hardening with tests to prove it holds. That's the standard the audit is built to bring to someone else's fleet — a ranked, file:line finding, not a vague warning, and a fix that's actually verified rather than assumed.

Book a scoping call ← All case studies