Brier score

The Brier score measures the accuracy of probabilistic predictions: for each prediction, take the squared difference between the stated probability and the outcome (1 if it happened, 0 if not), then average across all predictions. Zero is perfect; lower is better.

Say you predict "this migration will apply cleanly" at 90% confidence. If it does, that prediction contributes (0.9 − 1)² = 0.01 to your score. If it fails, it contributes (0.9 − 0)² = 0.81 — a heavy penalty for being confidently wrong. Averaged over many predictions, the score rewards two things at once: being right, and knowing how sure to be.

Reference points

A perfect forecaster scores 0. Always saying 50% — pure fence-sitting — scores exactly 0.25 regardless of outcomes, which makes 0.25 the "no information" baseline. Scoring above 0.25 means your confidence is actively anti-correlated with reality: you'd do better flipping a coin. Skilled forecasters on hard real-world questions typically land in the 0.1–0.2 range; on easier factual questions, well-calibrated people get closer to 0.1.

Why squared error, not just accuracy

Plain accuracy ignores confidence entirely — someone who says 51% and someone who says 99% get the same credit for a correct call. The Brier score's squared penalty makes overconfidence expensive in exactly the way real decisions are: acting on a wrong 99% claim costs far more than hedging on a wrong 55% one. This property makes it a proper scoring rule — the score is minimized by honestly reporting your true belief, so there is no way to game it except by actually being calibrated.

Why it matters in AI work

Brier scoring turns judgment into a measurable, trainable quantity. When you log a prediction before a consequential AI run and reconcile it afterwards, each reconciliation writes one Brier component; the running mean is your calibration in a single honest number. It's the flagship metric of the Miatz Tuukul practice — and the reason its credential can't be earned by memorization: you cannot braindump your way into a good Brier score computed from months of your own real decisions.