The Scoreboard
Holm + BH-FDR two-gate discipline same gate the paper-only fleet holds itself to 47 rows tracked
The rule, up front: nothing gets marketed as accurate before this
scoreboard says so, in public. The category this page sits in — signal
services, whale-tracking feeds, alt-data alerts — mostly sells confidence without
receipts. This is the receipts, published before anything is for sale.
Today that mostly means UNVALIDATED and INSUFFICIENT_DATA
rows below — that is the honest state of a young signal-validation campaign, and
publishing it plainly (rather than hiding it behind marketing copy) is the actual
product. One more honesty note: the whole fleet these signals come from trades
paper money only, nothing live — see the
validation-harness case study for that
context.
How a row gets promoted
A signal starts UNVALIDATED (weight zero, logged only). It is promoted to
VALIDATED only after it survives a pre-registered backtest against a minimum
out-of-sample window, Holm-corrected within its own test run and
Benjamini-Hochberg FDR-corrected across the whole campaign (q=0.10) — the same two-gate
discipline applied to every strategy before it is trusted with (paper) position sizing.
A signal
that backtests as anti-predictive is marked REMOVED (quarantined at weight
zero) and requires a root-cause analysis before it can re-enter. A signal with a real but
structurally small edge (e.g. compressed funding-rate arb spreads) is marked
THIN-EDGE — capped, never allowed to solo-drive a decision, and that cap is
not revisited by more data because it's a structural ceiling, not a sample-size problem.
Validated rows still carry a decay clock: 90 days without a re-test demotes them back to
UNVALIDATED.
41
UNVALIDATED
2
VALIDATED
2
REMOVED (anti-edge)
1
PENDING REPRO
1
THIN-EDGE (capped)
Parsed live from the vault/Reports/Signal Validation Scoreboard.md (47 rows) at generation time 2026-07-28T23:55:15.819Z. 1 row live-reverified against current data during this run (see "live-reverified" tag below).
| Signal | Status | N | OOS / wave progress | Notes |
|---|---|---|---|---|
whale-tracker.confluencewhale-tracker |
UNVALIDATED — PENDING REPRO live-reverified |
127 | OOS BLOCKED — 44 calendar days of ticker-scoped signal history (2026-04-07→2026-05-21) vs 90-day minimum; embargo (14d) consumes the entire 30% OOS window | Only validated return-predictor fleet-wide; 70.5% win at 10d, p=0.0004 per 2026-05-20 Whale-Tracker Signal Efficacy Backtest; **Wave 0-B.1 re-test (2026-05-21) reproduces under the corrected harness: 72.1% win at 10d, binomial p=5.08e-5, n=86 deduped — Class-1 reproducibility achieved**; decay-retest due ≥90d post-2026-05-20; mega-cap-momentum edge concentration is an honest-flag risk Override evidence: research/whale_flow_check.py, re-run 2026-07-04 against whale-tracker/data/signal_log.jsonl (1218 rows) during scoreboard generation. Reverses the vault scoreboard's VALIDATED classification (70.5% win-rate @10d, p=0.0004, 2026-05-20 methodology) — a Spearman re-test on the same signal returned NO-EDGE (p=0.1247), an unresolved methodology mismatch per the vault's own Learning/Glossary ("Scoreboard and VALIDATED promotion") and the 2026-07-03 Data Feed Products design doc. Publishing the more cautious label per the "never price ahead of the honesty label" rule. |
cross-platform-arb.tte_filtercross-platform-arb |
VALIDATED | — | — | Gate-layer (not committee-eligible); post-mortem driven — 35/36 trades timed out flat before filter per 2026-05-15 horizon-filter analysis |
solana-sniper.safety_bundlesolana-sniper-bot |
VALIDATED | — | — | Gate-layer (not committee-eligible); defensive rug-prevention (mint revoked, freeze revoked, LP locked/burned, top-10 concentration, no bundle/insider, Jupiter sell-route honeypot check); not return-predictive |
whale-tracker.insider_clusterwhale-tracker |
REMOVED (anti-edge) | — | — | Spearman ρ = −0.35, p = 0.007; top picks −19% at 10d; excluded from day-trader scheduler (`scheduler.py:51`) and whale-tracker writer (commit 7106d58); **still consumed by options-bot score blob** — unclosed gap per audit |
kronos.edge_pctday-trader |
REMOVED (anti-edge) | — | — | 2026-05-20 NO-GO; anti-edge magnitude grows with horizon; re-entry only after upstream RCA per 2026-05-20 Kronos Anti-Edge Root Cause; day-trader is SHADOW-ONLY for the entire campaign |
funding-rate-arb.funding_ratefunding-rate-arb |
THIN-EDGE (capped) | — | parallel-track — scope decision pending, not a backtest | Industry funding APY compressed to ~5% in 2025; structural ceiling, not a backtest failure; scope decision (accept low-weight or sunset) due by end of Wave 3 |
whale-tracker.congressionalwhale-tracker |
UNVALIDATED | — | — | Sub-tag of whale score blob; no per-tag backtest; consumed via score_bonus in options-bot and day-trader universe builder |
whale-tracker.institutionalwhale-tracker |
UNVALIDATED | — | — | Sub-tag; same gap as congressional — no isolated backtest |
whale-tracker.options_flowwhale-tracker |
UNVALIDATED | — | — | Sub-tag; same gap; options-flow activity as a signal, not a gate |
whale-tracker.kronos_edge_pctwhale-tracker |
UNVALIDATED | — | — | Kronos-edge% sub-tag stored on whale signals; distinct from raw `kronos.edge_pct`; likely follows NO-GO verdict per roadmap since upstream Kronos is ANTI-PREDICTIVE |
prediction-trader.electionsprediction_trader |
UNVALIDATED | — | — | Highest-weight prediction_trader scanner (weight = 0.30); polls → Kalshi election markets |
options-bot.connors_rsioptions-bot |
UNVALIDATED | — | — | Primary mean-reversion signal; Connors RSI (3,2,100) oversold/overbought; +3 score contribution |
options-bot.rsi_14options-bot |
UNVALIDATED | — | — | Secondary mean-reversion confirmation; RSI(14); +2 score |
options-bot.macd_crossoptions-bot |
UNVALIDATED | — | — | Bull/bear MACD cross; +2 score; direction-only signal |
options-bot.bb_squeeze_breakoutoptions-bot |
UNVALIDATED | — | — | Bollinger Band squeeze (+3) and breakout (+4) signals |
options-bot.volume_spikeoptions-bot |
UNVALIDATED | — | — | Volume vs 20d SMA; +2 score |
prediction-trader.ai_techprediction_trader |
UNVALIDATED | — | — | AI benchmark rank → Kalshi AI/tech markets; weight = 0.25 |
options-bot.iv_rankoptions-bot |
UNVALIDATED | — | — | Real ATM IV rank (or 20d HV fallback); HIGH/LOW labels; score input |
options-bot.vix_rankoptions-bot |
UNVALIDATED | — | — | 52-week VIX percentile; position-sizing scalar and halt thresholds |
options-bot.voi_sweepoptions-bot |
UNVALIDATED | — | — | Call/put V/OI directional sweep; +/−2 score; also feeds the cross-bot sweep bus to prediction_trader (sweep bus validated separately in Wave 7) |
options-bot.put_call_iv_skewoptions-bot |
UNVALIDATED | — | — | ATM put/call IV skew; +/−1 score |
options-bot.earnings_0_7d_bonusoptions-bot |
UNVALIDATED | — | — | +1 score bonus if earnings 0–7d out; the 0–2d hard skip is a rule and is excluded from backtest scope |
prediction-trader.weatherprediction_trader |
UNVALIDATED | — | — | Poisson model → Kalshi weather markets; weight = 0.20 |
grid-trading.adxgrid-trading-bot |
UNVALIDATED | — | — | Low-ADX gate for range-bound regime; validated against realised grid PnL, not SPY excess return |
grid-trading.bb_widthgrid-trading-bot |
UNVALIDATED | — | — | Bollinger Band width; grid suitability scoring input |
grid-trading.grid_scoregrid-trading-bot |
UNVALIDATED | — | — | Composite suitability score (ADX + BB + ATR); gates which pairs get launched; metric is realised fills + grid PnL, not directional excess return |
prediction-trader.econprediction_trader |
UNVALIDATED | — | — | FRED/BLS macro releases → Kalshi econ markets; weight = 0.15 |
solana-sniper.m1_momentumsolana-sniper-bot |
UNVALIDATED | — | — | GeckoTerminal m1 candle accel/decel; +/−8 score delta + Tier-1 gate |
solana-sniper.buy_sell_ratiosolana-sniper-bot |
UNVALIDATED | — | — | m5 buy/sell txn ratio; +/−10 / +/−5 score delta |
solana-sniper.bonding_curve_positionsolana-sniper-bot |
UNVALIDATED | — | — | Pump.fun bonding-curve 40–60% sweet-spot bonus |
solana-sniper.creator_wallet_age_tiersolana-sniper-bot |
UNVALIDATED | — | — | Tiered age penalty (weight 10); reasonable defensive heuristic but unvalidated for return prediction |
solana-sniper.price_dump_penaltysolana-sniper-bot |
UNVALIDATED | — | — | Penalties on price_change_m5 / price_change_h1 dump patterns |
solana-sniper.volume_spike_ratio_h1_h24solana-sniper-bot |
UNVALIDATED | — | — | h1 vs h24/24 volume spike ratio; currently informational sizing input |
prediction-trader.mirofishprediction_trader |
UNVALIDATED | — | — | External Monte Carlo / simulation engine; weight = 0.20; requires live external MiroFish service |
cross-platform-arb.polymarket_leads_kalshicross-platform-arb |
UNVALIDATED | — | — | Core arb thesis: Polymarket is the more-liquid leading venue; entirely thesis-level; requires paired Polymarket/Kalshi tick archive from Wave 0-A |
cross-platform-arb.manifold_agreement_boostcross-platform-arb |
UNVALIDATED | — | — | +0.05 confidence boost when Manifold direction agrees with Polymarket |
cross-platform-arb.polymarket_volume_confidence_tierscross-platform-arb |
UNVALIDATED | — | — | Volume-tiered confidence weight (0.30 → 1.00); calibration vs actual hit-rate unknown |
cross-platform-arb.fuzzy_match_confidencecross-platform-arb |
UNVALIDATED | — | — | 0.50 confidence weight for fuzzy Kalshi event-match score; calibration unknown |
prediction-trader.news_sentimentprediction_trader |
UNVALIDATED | — | — | Lowest-weight prediction_trader scanner (weight = 0.10); sentiment score → Kalshi market direction |
news-bot.market_regime_fngnews-bot |
UNVALIDATED | 2 | INSUFFICIENT-DATA on 2026-05-20 backtest (N = 2 defensive days) | INSUFFICIENT-DATA on 2026-05-20 backtest (N = 2 defensive days); requires Wave 0-D archive (~4+ weeks); consumed as hard-pause gate by grid-trading, options, funding-arb, day-trader |
news-bot.alerts_highnews-bot |
UNVALIDATED | — | — | LIKELY-REDUNDANT hint from 2026-05-20 News-Gate Efficacy Backtest; Wave 7 tests whether this collapses into news_veto with no independent lift |
news-bot.news_vetonews-bot |
UNVALIDATED | — | — | Ticker-level veto gate; consumed by grid-trading, options, funding-arb, day-trader; never backtested as a predictor; validation linked to the Wave 7 alerts_high collapse study |
options-bot.wheel_csp_iv_rankoptions-bot |
UNVALIDATED | — | — | Wheel/CSP entry path via IV-rank scan; deferred from Wave 3 — insufficient entry-event history at Wave 3 fire time |
options-bot.cross_bot_sweep_busoptions-bot |
UNVALIDATED | — | — | +/−0.05 conviction nudge from options-bot to prediction_trader; can be stripped pre-emptively if Waves 2–3 both return NO-GO on voi_sweep + put_call_iv_skew |
sage-bot.core_analystssage-bot |
UNVALIDATED | — | — | 4 core LLM analysts (technical / fundamentals / sentiment / news_sentiment); gated on ≥90 calendar days AND ≥100 decisions per analyst from Wave 0-C clock; clock_start_date = 2026-05-21 (Component C deployed 2026-05-21; 39 per-analyst rows live) |
sage-bot.full_rostersage-bot |
UNVALIDATED | — | 90-day clock | 11+ named-investor LLM analysts (Buffett, Munger, Burry, etc.); same 90-day clock; per-analyst N may stay thin for trigger-only analysts — those remain clocked past Wave 8 |
day-trader.deerflow_scorerday-trader |
UNVALIDATED | — | — | DeerFlow LLM composite scorer atop Kronos; day-trader is SHADOW-ONLY for the entire campaign; independent validation deferred to Wave 9 rebuild or follow-on campaign; not independently testable while Kronos (its primary input) remains ANTI-PREDICTIVE |
What this page deliberately does NOT do
- It does not round an
UNVALIDATEDrow up to "promising" in the copy. If the table says unvalidated, the header text above says unvalidated too. - It does not hide the one row (
whale-tracker.confluence) whose earlier VALIDATED verdict was walked back after a methodology re-check found the result didn't reproduce under a second, independent statistical test. That reversal is the scoreboard working as intended, not a failure to hide. - It does not backdate a graduation. Every VALIDATED row here still carries its 90-day decay clock and re-test requirement.