Calibration — the agent measures itself
Once a market on Polymarket or Kalshi resolves, the resolver back-fills the outcome onto every receipt the agent emitted for it. That gives us ground truth for every prediction. We then compute a Brier score (mean squared error between predicted probability and actual outcome — lower is better, perfect forecaster scores 0) plus a 10-bucket reliability curve.
A trivial “50% on everything” forecaster scores ~0.25. A good prediction-market analyst typically lands between 0.10 and 0.18.
Brier score
0.2461
across 1180 resolved receipts
High-conf Brier
0.2475
confidence ≥ 0.7
Low-conf Brier
0.2443
confidence < 0.7
Resolved markets
55
out of 1180 receipts
Reliability — predicted vs actual
dots near y=x = well calibrated · dot size = receipts in bucket
Brier over time — is the agent learning?
lower = sharper · dashed 0.25 = coin-flip baseline
Bucket breakdown
| Bucket | n | mean predicted | mean actual | drift |
|---|---|---|---|---|
| 0.10-0.20 | 4 | 15.8% | 25.0% | +9.3 pp |
| 0.20-0.30 | 13 | 25.8% | 7.7% | -18.1 pp |
| 0.30-0.40 | 58 | 35.5% | 19.0% | -16.5 pp |
| 0.40-0.50 | 204 | 46.1% | 26.5% | -19.6 pp |
| 0.50-0.60 | 462 | 54.7% | 36.8% | -17.9 pp |
| 0.60-0.70 | 285 | 63.9% | 60.0% | -3.9 pp |
| 0.70-0.80 | 71 | 73.7% | 70.4% | -3.2 pp |
| 0.80-0.90 | 46 | 83.5% | 60.9% | -22.7 pp |
| 0.90-1.00 | 27 | 94.2% | 55.6% | -38.7 pp |
Note: a market is treated as resolved YES iff Polymarket Gamma reports closed=true and the YES close price is within 5% of 1.0; resolved NO iff the YES close price is within 5% of 0.0. Ambiguous markets (close price near 0.5) are not counted.