PennantBench: LLMs beat a projection system at predicting MLB player value, but not the front offices

I gave frontier models the stat lines MLB clubs had before the 2026 season and scored their forecasts against what actually happened. Sonnet 5.5 and Opus 5.5 beat the Marcels, a standard projection system, by about 8%. All of that edge came from predicting playing time, and none of it from projecting talent. Then I replayed 811 real keep-or-release decisions, and nothing beat the clubs.

These are the results so far from PennantBench, a project I’ve been working on to see whether LLMs can forecast players and make roster decisions better than the usual baselines.

The setup

Every policy in PennantBench sees only what was public on a real decision date, and it’s scored against the real 2026 season.

I started with a simulated league, but the early pilots gave small, noisy differences in wins across four simulated worlds. Real outcomes avoid that. The 2026 season happened once, the same way for every policy, so when two forecasts differ, the difference isn’t game noise.

The models get information through 2025 and are scored on 2026. That season ended after their training data, and when I probed them for 2026 knowledge, they didn’t have any. The players are the 983 who had at least 100 plate appearances or 100 batters faced in 2025, and the models see them as anonymized stat lines.

Here’s what that looks like. Each request has 40 players, and the model answers with one tool call. This is one player from one request, trimmed:

System: You are an MLB front-office analyst. It is the 2025-26
offseason: you know results only through the end of the 2025
regular season. […]

User: Forecast the 2026 season for each anonymized MLB hitter
below. […] Each forecast is scored by absolute error, so give
your median expectation and account for injury, role, aging and
roster risk. […]

H003164 | age 26
  2023: PA 44, […] wOBA 0.356
  2024: PA 259, […] wOBA 0.355
  2025: PA 452, AB 388, H 110, […] HR 19, BB 43, […] wOBA 0.367

Sonnet 5.5 (medium), submit_hitter_forecasts:
{"id": "H003164", "pa": 480, "woba": 0.348}

The Marcels projected 452 plate appearances and a .358 wOBA for this player. Sonnet’s two samples both said 480 PA, at .348 and .347. The player actually got 717 PA at .330. Here Sonnet beat the Marcels on both, by 28 PA and by 10 points of wOBA. Across all 983 players, only the playing-time half of that held up.

The baseline is the Marcels, a projection system that takes a weighted average of a player’s last three seasons, regresses it toward the league average and adjusts for age. I also built an “easy hybrid” that uses Marcel rates plus last season’s playing time. Everything is scored in rar-v1, a simple runs-above-replacement number built from wOBA for hitters and FIP for pitchers.¹ It isn’t WAR, and I’ll come back to what it leaves out.

I froze each study’s protocol before any model saw a prompt. Where I worked something out afterwards, I’ll say so. Every comparison is paired: the same players, scored against the same season, so each interval is on the difference itself, from 2,000 bootstrap resamples of players (or of decisions, in Finding 4). The whole thing cost about $13 of API spend, plus some subscription quota for the DeepSeek runs.

Finding 1: LLMs beat the Marcels, all through playing time

Sonnet 5.5 and Opus 5.5 cut player-value error by about 8% against the Marcels and about 4% against the easy hybrid. The result held up when I reran it on fresh prompts.

Negative means less error than the baseline. The Marcels’ error is 9.77 runs per player. The Sonnet and Opus rows come from the replication run, and the Luna, DeepSeek and Haiku rows from the first run. The 95% intervals come from resampling players.

Splitting the error into rates and playing time shows where the gain comes from. No model beat the Marcels at projecting rates (wOBA for hitters, FIP for pitchers), over the full season or in August and September. The frontier models were better at guessing who would actually be on the field: they cut playing-time error by about 32 plate appearances per hitter and 4 to 5 innings per pitcher.

The gain also levels off. Sonnet at high effort² was only −0.04 runs better than Sonnet at medium [−0.09, 0.00], and Opus was −0.03 better [−0.15, +0.09]. My guess is that three seasons of stat lines are the limit here, and a bigger model can’t get more out of them.

I don’t think it’s memory either. Sonnet at medium effort could name only 5% of a random sample of players from their stat lines, and it gained as much as Opus, which could name every star I tested.

Smaller models didn’t help. Haiku was worse than the Marcels, and DeepSeek and Luna only matched the easy hybrid.

Finding 2: names make forecasts worse

Next I gave Sonnet 5.5 more to work with, and in a separate test told it who each player was. I planned both comparisons before any model output.

The extra context helped a lot. Adding injured-list history, Triple-A stat lines, 40-man roster status³ and position cut error by −0.40 runs per player [−0.56, −0.24], which moves the edge over the Marcels from about 8% to about 12% (−1.14 runs per player). Adding names to those same records made the forecasts worse, by +0.28 runs [+0.07, +0.49].

I worked out where that came from after the fact, so treat this part as a hypothesis. Names made the model more optimistic about well-known players:

For the top 100 hitters, the named model predicted 608 plate appearances and a .350 wOBA. The anonymous model predicted 592 and .346. They actually finished at 564 and .336.

It shows up player by player too. Of the 134 hitters the Marcels projected for at least 500 plate appearances, the name raised the forecast’s value for 96. These are the five it raised the most, each shown as plate appearances and wOBA:

The name made four of the five forecasts worse, by 7.7 to 8.9 runs each. The exception is James Wood, who outhit even the named forecast. Each figure is the average of the model’s two samples, and each prompt showed a player either anonymously or by name.

The name pulls the forecast toward the player’s reputation and away from the evidence in the stat line. So if you’re using an LLM to evaluate players, give it de-identified records with as much information as you can. I’ve only tested this on Sonnet 5.5 at medium effort, though, and the reputation explanation still needs to be replicated.

Finding 3: the context gain was mostly the clubs’ own roster moves

Most of the gain from extra context came from one field, and that field already records what front offices decided over the winter.

I ran DeepSeek V4.1 Flash at low effort with three information sets: stat lines only, what was known on November 1, 2025, and what was known on February 15, 2026. The February context beat the easy hybrid by −0.57 runs [−0.83, −0.33]. The November context didn’t: −0.10 [−0.32, +0.11].

When I broke that down afterwards, 86% of the gap between the two came from the 159 players who weren’t on any 40-man roster on February 15. Their error fell from 5.63 runs to 3.15. For the other 824 players, it moved by only −0.08.

February roster status reflects non-tenders, releases and signings. For those 159 players, the model was mostly reading an answer key the front offices had already written.

I only did this breakdown for DeepSeek. The extra context in Finding 2 used the same roster field, so part of Sonnet’s gain probably came from the same place, but I haven’t measured that.

Finding 4: nothing beat the front offices on keep-or-release decisions

Then I checked whether better forecasts lead to better decisions. I replayed 811 real club options and arbitration tenders across three offseasons and compared each rule’s keep-or-release calls with what the clubs actually did.

No rule measurably beat the clubs. Every interval but one spans zero, and the best point estimates are under 2% of total surplus. The exception is keeping everyone, which would have cost $207M, so the clubs’ cuts carried real information.

The replay also couldn’t have caught a small edge. The clubs’ decisions earned $3,446M of surplus in total, and the intervals are about ±$150M wide, so a rule would have had to beat the clubs by more than 4% of everything they earned before this replay could tell it apart from luck. More offseasons will narrow the intervals.

A plain regression also out-forecast the LLM. stat-v1 is a ridge regression on the November 1 information, and it beat the Marcels by −0.87 runs per player and the easy hybrid by −0.52. DeepSeek, given the same information, did worse than it by +0.36 [+0.03, +0.73]. The closest any rule came to the clubs was stat-v1 on the 90 club options, at +$50M [−2, +112].

The clubs also had medicals, Statcast and scouting reports that none of these rules saw, and as Finding 3 showed, the LLMs’ best context was partly the clubs’ own decisions.

I left out DeepSeek’s 2024 results because it knew how 2024 turned out. Sonnet and Opus were only scored on the 2026 decisions, as a descriptive comparison: −$3M and −$21M against the clubs, both with wide intervals.

Caveats

People who know baseball analytics will look for these.

• Findings 1 to 3 rest on one outcome season, 2026. Finding 4 pools three offseasons.
• rar-v1 is a rough metric. It ignores defense, position, baserunning, park effects and reliever leverage, and it runs about 20% generous on total wins above replacement.
• Model coverage is narrow. Only Sonnet 5.5 at medium effort was tested with extra context and names, and DeepSeek ran at low effort.
• I didn’t plan the reputation explanation in Finding 2 or the 40-man breakdown in Finding 3 in advance.
• The clubs had more information, and goals beyond one season’s surplus: roster fit, payroll, years of control and trade value.
• This isn’t a calibrated benchmark, a publication-ready result or a ranking of LLMs.

Where this leaves things

LLMs help with forecasting, mostly because they’re better at guessing who will play. I haven’t found evidence that they make better decisions than the people who make them for a living, and some of what looked like the models’ judgment was the clubs’ judgment showing up in the data.

Next, I want to replicate all of this on the 2027 season, check whether an LLM adds anything on top of the stat-v1 regression, and run Sonnet and Opus on the full three-offseason replay.

If you have a forecasting rule or a baseline you think would do better than stat-v1, or than the clubs, send it to me. I’d like to run outside policies against the same 811 decisions in the 2027 round.