A Better Projection Made Worse Lineups
On November 10 I tested a change to my daily fantasy football lineups and rejected it the same night. It made the player projections more accurate. On the backtest I'd been measuring everything against, it made the lineups worse.
What the project does
Each week the model projects how many fantasy points every player on a slate will score. A slate is the set of NFL games a contest covers. Mine are the Sunday main slates, the games kicking off between 1:00 and 4:25 Eastern. Every player has a salary, and an optimizer picks the lineup that fits under the salary cap with the best projection.
I score a lineup with a number I call Beat: the share of real entries in a contest that my lineup outscored, with ties counting half. The backtest rebuilds the lineup for 145 real past contests across 55 Sundays from 2021 to 2026, using only what was known before kickoff, and averages Beat across them.
The idea
Expert analysts publish a consensus ranking of players every week. The bet was that a late-week ranking reflects news my model hasn't priced in, like practice reports. The change took the part of each player's rank that his salary and my own projection don't already explain, and moved his projection up or down by it. The average move was half a point. The biggest was 10.5.
Before the lineup test, I registered it. That's a commit with the exact recipe and two ways it could fail. If any held-out season's projections got less accurate, it failed. If it missed my ship bar on both versions of the backtest, it failed. The registration went in at 9:54 that night and the result at 10:41.
The projections got better
Accuracy here is the average squared miss per player: projected points minus actual points, squared. The blend's weights were fit only on earlier seasons and tested on later ones. It cut that average by 0.42 per player, with a likely range of 0.22 to 0.64. It helped in every held-out season from 2022 through 2026, and on 33 of 43 held-out slates.
About half the gain was at quarterback. For running backs and wide receivers the likely range crossed zero, so I can't claim a gain there.
The first failure condition didn't fire. By the usual standard, that's a better model.
The lineups didn't
The lineup test runs the backtest once on the plain projections. Then it runs 20 more times with every player's projection nudged by random noise of about 1%. Each of those runs is a seed. Both versions get the same nudges, so the comparison is paired, and a gain that only shows up on one lucky draw gets caught. My ship bar wants the 20-seed average up by at least a full point of Beat, up on at least 15 of the 20 seeds, and no season that flips the result negative when it's left out.
On the main backtest, Beat fell from 65.72 to 62.55 averaged over the seeds. That's down 3.17, and it was up on 1 of 20 seeds. The plain run fell from 69.69 to 62.11.
There's a second version of the backtest, and I owe it a paragraph. The main one runs with a feature off that the live system has on. That feature projects team defenses from the betting line, and I had no pre-kickoff lines saved for 2021 to 2025. Three days earlier I'd found that switching it on, using closing lines, drops the baseline itself. On that version the blend was a wash: 63.19 to 63.49 over the seeds, up on 11 of 20. The plain run went from 62.76 to 61.49.
Neither cleared the bar. The second failure condition fired, and that's where I rejected it.
Why better rows made a worse lineup
The blend changed the lineup on 53 of 55 Sundays in the plain run, and on about 53 in every seed. In a changed lineup, four or five of the nine players were different on average.
So I looked at who came in and who went out. Across the 20 seeds on the main backtest there were 4,862 player swaps. The players swapped in were projected for 2,745 more points than the players they replaced. They scored 410 fewer. They got 0.874 of their projection, against 0.916 for the players they pushed out. On the second version it was 0.872 against 0.883.
That's what an optimizer does. It hunts for the players whose projection is highest for their salary. A blend that's better on average still puts some players too high, and those are the ones it grabs. The accuracy gain was spread across thousands of player rows. The lineup is nine of them, picked for looking best.
A second change tested the same night went the same way. It fixed how salary feeds into the projections. The rows got more accurate, and Beat fell 2.21 over the seeds on the main backtest, up on 3 of 20. Its swapped-in players got 0.92 of their projection, against 1.01 for the ones swapped out.
How much of this is the baseline
I don't think all of the drop belongs to the blend, and it'd be dishonest to say it does. The main lineup method was chosen by trying many rules on these same 145 contests. A method picked that way sits on a peak in this data, and most changes slide off it. My notes from that week say two earlier sets of changes to it lost about 2.5 to 3 points of seed average each. The audit called it the selected-peak effect.
So here's what I'm sure of. On the backtest tuned to these contests, the blend lost about three points. On the one closer to live, it gained nothing. On neither did a more accurate projection turn into a better lineup.
What I do now
The next day I decided the consensus ranks stay out of the real picks. They now build a separate weekly lineup, labeled a shadow lineup and shown beside the picks. It went live November 13 and has no verdict yet. New ideas get measured on both versions of the backtest.
I still track projection accuracy. I don't count it as evidence a lineup will improve until the lineup test says so.
-- Justin Higgins. Software Engineer, Midwest. Made the projections more accurate, then checked who the optimizer actually picked.
Companion piece: The Winner Was Array Order - a seven-model ranking that was mostly an artifact of the eval.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com