Daggy vs. the Market
We’re testing our AI forecast agents against the hardest possible benchmark: a real-time, liquid market. Before each game, we post our predictions along with those from Kalshi’s prediction market and score them as they settle. Why we’re doing this.
Week by Week
| Week | Method | Daggy W-L | Kalshi W-L | Daggy Brier | Kalshi Brier |
|---|---|---|---|---|---|
| · so far |  Supervisor | 1-0 | 1-0 | 0.212 | 0.160 |
| Supervisor | 7-7 | 9-5 | 0.267 | 0.267 | |
| Supervisor | 8-7 | 9-6 | 0.240 | 0.239 | |
| Supervisor | 11-5 | 9-7 | 0.214 | 0.223 | |
| Supervisor | 9-7 | 8-8 | 0.243 | 0.258 | |
 Extremized | 11-5 | 11-5 | 0.248 | 0.231 | |
 Mean | 12-4 | 12-4 | 0.216 | 0.211 | |
| Season | Our call | 59-35 | 59-35 | 0.237 | 0.237 |
Week 7
1 of 14 final
Agent Breakdown
Models against Kalshi, over time
Click a provider to compare its engine-on line (solid) with its engine-off line (dashed). You can also click a line or its name.
Each line is a running total: game by game, it adds how much lower (up) or higher (down) that agent’s Brier was than Kalshi’s on the same game, so a rising line is beating the market. For example, Moonshot · engine on ends at +.78 over 94 games: +.008 per game, the gap between its season Brier and Kalshi’s in the table below. Click any Brier there to see the game-by-game sum. Totals run all season per provider; the week-3 rule marks the model upgrade.
Season and week by week
Engine on means the agent ran with the Daggy stats engine; engine off is the same model without it. Each cell shows Brier (lower is better) over W–L; a forecast of exactly 50% counts as picking the home team. Kalshi’s row is the market to beat, and the best Brier in each column is bold. Click a column to sort, a provider to highlight it on the chart, or a Brier to see its games.
| Market-redacted | Market-aware | |||||||
|---|---|---|---|---|---|---|---|---|
| market-redacted | market-redacted | market-redacted | market-redacted | market-redacted | market-aware | market-aware | ||
| Kalshi | 59–35 | 1–0 | 9–5 | 9–6 | 9–7 | 8–8 | 11–5 | 12–4 |
| Average of 4 | ||||||||
| Engine on | 59–35 | 1–0 | 7–7 | 8–7 | 11–5 | 9–7 | 11–5 | 12–4 |
| Engine off | 60–34 | 1–0 | 7–7 | 9–6 | 11–5 | 9–7 | 11–5 | 12–4 |
| Moonshot | ||||||||
| Engine on | 58–36 | 1–0 | 6–8 | 8–7 | 11–5 | 10–6 | 10–6 | 12–4 |
| Engine off | 59–35 | 1–0 | 8–6 | 8–7 | 11–5 | 9–7 | 10–6 | 12–4 |
| OpenAI | ||||||||
| Engine on | 58–36 | 1–0 | 8–6 | 8–7 | 9–7 | 10–6 | 11–5 | 11–5 |
| Engine off | 60–34 | 1–0 | 7–7 | 9–6 | 11–5 | 9–7 | 11–5 | 12–4 |
| Engine on | 60–34 | 1–0 | 7–7 | 8–7 | 11–5 | 10–6 | 11–5 | 12–4 |
| Engine off | 58–36 | 1–0 | 7–7 | 9–6 | 9–7 | 9–7 | 11–5 | 12–4 |
| Anthropic | ||||||||
| Engine on | 57–37 | 0–1 | 7–7 | 8–7 | 10–6 | 9–7 | 11–5 | 12–4 |
| Engine off | 56–38 | 1–0 | 6–8 | 9–6 | 9–7 | 8–8 | 11–5 | 12–4 |
How This Whole Racket Works
Why We're Running This Game
Superforecasters typically turn qualitative signals into probabilities via judgment. In a football game, these signals could be player-coach friction. But probabilistic modeling and machine learning generally beat qualitative judgment, especially when we have a well-defined quantitative target. This page documents our journey to build an AI forecasting system that mixes the judgment of frontier LLMs with Daggy’s statistical capabilities.
Every forecast is posted before kickoff and scored in public, so if the approach doesn’t work, it shows here first.
The Two Players
Daggy is not one model. It's 8 agents forecasting every game — 4 of them wired to the Daggy statistical engine, the rest working from what they can reason out — combined into a single pick per game, before kickoff.
Kalshi isn't a rival forecaster. It's the live price real money is trading at, on a market where anyone can bet on the outcome. When we say “Daggy vs. Kalshi,” we mean model vs. market — not two people guessing.
The Score
In addition to win-loss, we use the Brier score, which judges based on confidence:
Aggregation Methods
We have 8 agents predict each game, but what's left is to aggregate all 8 predictions to one forecast. We use the following methods, but commit to one before each week.
- Mean: A simple average of each agent's prediction
- Extremized: Agents often overly hedge their bets toward 50/50. Extremizing pushes the predictions toward zero-one by using Platt scaling
- Supervisor: An AI agent looks at all 8 predictions, thinks them through and then comes up with a final forecast.
For information and entertainment only. Not betting, trading, or investment advice.