← Research

When the model says another pick was better, how much of that shows up?

Published October 5, 2026 · 90,408 draft states from 12,000 ranked games

How it was measured

Only the pick that was made has a result, so “what if you had picked X” is never observed. What can be tested is whether the model's differences between picks show up across many states where different players made different picks.

• For each real draft state, every legal candidate is scored by a model trained only on earlier games. The average over the candidates is the board's value; the pick's own value is its score minus that average.
• The game's result is regressed on the board's value and on the pick's own value. The second coefficient is the share: 1 means the model's gaps between picks show up in full, 0 means they carry nothing.
• Every game was played after the model that scored it was built, and the player histories used were frozen before the day began.
• Errors are clustered by game, because the states of one game share one result.

The share, in three populations

The champions-only figure is two measurements of the same model on different days, pooled. The first, on its own, said 0.675 ± 0.118; the second said 0.969 ± 0.118. They differ by less than twice their combined error, so neither alone is the number.

Until 5 October 2026 the site used 0.29 for the player-known reading. That number came from a player's own pool, the last row, and was being applied to every candidate. Measured where it is applied, the share is 0.83.

Is it one straight line?

If small gaps were noise and only large ones real, a single share would mislead. Sorting the champions-only states into ten equal groups by the pick's claimed value, what showed up follows the claim across the range, and a curvature term is indistinguishable from zero.

The ranges are wide: each group's result is uncertain by more than a point. The picture supports a line through zero, not any particular bend.

Does scaling help the forecast?

On the same states, three forecasts of who wins: the board alone, the board plus the model's full gap for the pick made, and the board plus the gap at the share the site had been using. Knowing the pick beats the board alone on AUC, log loss and Brier score. Scaling the gap down did not beat the full gap on these games.

What this changes on the site

Arena reports and game reviews print a cost in observed outcomes: the model's gap times the share, with a 95% range from the share's own uncertainty. An alternative is called likely better only when the low end of that range still clears half a point.

Rankings, grades and ratings read the order of the picks, which a common share cannot change. The draft arena and the game review use the same numbers.

What it does not show

• It is observational. Players pick champions for reasons the model only partly sees, so a share near 1 is necessary for the advice to be right, not proof that a different pick would have won.
• Four days, one region, one model. The site now serves a later model trained on all three regions, which has not been measured this way.
• The data has no observed pick order, so each side's order is assigned the way training assigns it.
• The player-known figure knows the picking player only; the review's “everyone known” reading knows all ten.
• Inside a player's own pool, picks valued up to about two points above the board showed almost nothing; the effect sat in the tails.