Ask one expert for a prediction and you get one opinion, with all its blind spots. Ask twenty and average their answers, and something odd tends to happen: the average is often more accurate than most of the people who contributed to it, and sometimes better than any of them. Forecasters call this forecast combination; machine-learning engineers call it an ensemble. Both rely on the same idea, and it explains why so many sports models are really several models working together.
A famous first example
In 1907 the scientist Francis Galton wrote in Nature about a weight-judging competition at a livestock show in Plymouth. Visitors paid to guess the weight of an ox once it had been slaughtered and dressed. Galton collected 787 usable tickets and found that the middle estimate was 1,207 lb, while the ox actually weighed 1,198 lb: an error of under 1%. Few individual visitors were that close.
The point is not that crowds are magic. Each guess combined some real knowledge with some personal error. Because those errors pointed in different directions, many of them cancelled out when the guesses were combined.
Why combining works
Every forecast can be thought of as two parts: the signal it shares with the truth, and its own error. When forecasts are combined:
- the signal is kept, because every good forecast contains some of it;
- the errors partly cancel, as long as they are not all the same.
The second condition is the important one. Averaging ten copies of the same model gives you the same model. Averaging ten models built on different data, assumptions or methods can give you something better than each of them.
Example: Invented numbers to show the arithmetic. Three models give the home team these chances of winning: 52%, 44% and 49%. Their average is (52 + 44 + 49) ÷ 3 = 48.3%. If one model overrates home teams and another underrates them, the average sits between their errors, and over many matches it is usually closer to the truth than the most extreme model.
The research behind it
In 1969, J. M. Bates and C. W. J. Granger published "The Combination of Forecasts", usually treated as the starting point of the field. They showed that a combination of two forecasts could have a lower error than either one, and proposed ways to weight each forecast by how well it had performed.
Twenty years later, Robert Clemen reviewed the literature that had grown out of that paper. His conclusions are still quoted: combining forecasts can improve accuracy substantially, and simple methods, such as a plain average, often work about as well as complicated weighting schemes. Clever weights estimated from a short history can chase noise, while an equal-weight average has nothing to overfit.
Ensembles in machine learning
Machine learning uses the same principle in several standard ways:
| Method | How the models differ | Example |
|---|---|---|
| Bagging | Each model is trained on a different random sample of the data | Random forests, built from many decision trees |
| Boosting | Each new model focuses on the mistakes of the previous ones | Gradient-boosted trees |
| Stacking | A final model learns how to combine the predictions of several others | A model that blends an Elo rating, a goals model and market odds |
In football forecasting, a common recipe is to combine a team-strength rating, a goals-based model and information that neither captures well, such as injuries.
The betting market as a consensus
Betting odds are themselves a kind of consensus. They reflect the bookmakers' own models and the money placed by thousands of bettors. That is why they are such a strong benchmark, and why research on "beating the bookies" has often looked at the average of many bookmakers' odds as the best available estimate of the true probabilities. We look at that research in Can AI Beat the Bookmakers?.
When combining stops helping
Watch out: A consensus is only as good as the independence of its parts. If every forecaster reads the same tipster, uses the same data feed or copies the market, their errors stop cancelling and start adding up.
- Correlated errors. Models built from the same data with similar methods tend to make the same mistakes.
- Adding bad forecasts. A combination of good and very poor forecasts can be worse than the good ones alone. Combining helps most when each part is reasonably good on its own.
- Overfitted weights. Giving one model 80% of the weight because it did best last month usually rewards luck.
- Herding. In public prediction contests, people who see the current consensus tend to move towards it, which makes the crowd less diverse and less wise.
How to use the idea
If you build or follow predictions, a few practical rules follow from all this:
- Prefer several reasonable, different approaches to one complicated one.
- Start with an equal-weight average before trying anything cleverer.
- Judge the combination on matches it has not seen, with a proper scoring rule such as the Brier score.
To see what one of the simplest ingredients of such a consensus looks like, try the Elo rating calculator, and read How AI Predicts Football Matches for the other common building blocks.
- Combining forecasts works because their errors partly cancel while their signal adds up.
- It only works when the forecasts are reasonably good and genuinely different.
- A plain average is a strong, hard-to-beat starting point.
Further reading
- Revisiting Francis Galton's forecasting competition — Kenneth F. Wallis, arXiv, 2014: the 1907 data and its transcription errors
- The combination of forecasts — Bates and Granger, Operational Research Quarterly 20(4), 1969
- Combining forecasts: a review and annotated bibliography — Robert T. Clemen, International Journal of Forecasting, 1989
The AI Sports team writes about data, models and technology in sport.