The Cheaper AI Model Cost More in 32% of Matchups. Only 11% of Companies Can Predict the Bill. | Edition 345
Edition 345 — Researchers ran 8 top AI models through 12 tasks. The list price turned out to be a poor guide to the real bill.
The AI model with the lower price tag turned out to cost more in nearly one of every three head-to-head comparisons.
That is the finding of a study by researchers at Stanford, UC Berkeley, Carnegie Mellon and Microsoft Research, who ran eight leading AI models through 12 kinds of work and compared what each one was supposed to cost with what it actually cost. On Monday, The Wall Street Journal paired that study with a survey finding that only 11% of companies can predict their AI spending within 10%.
If you pay for AI by usage, or your company does, the price list is not telling you what you will spend.
What the researchers found
The team tested eight “reasoning” models, the kind that work through a problem step by step before answering. They used 12 tasks covering competition math, science questions, writing code, and multi-step agent work, where the AI takes several actions in a row.
Then they lined the models up in pairs. In 32% of those pairings, the model with the lower listed price ran up the higher total cost. In the most extreme case the gap was 28 times.
One example from the paper, using two of the models it tested: Google’s Gemini 3 Flash is listed at a price 80% lower than OpenAI’s GPT-5.4. Across all the tasks in the study, it actually cost 38% more.
“Price alone should not be used to infer which model is actually cheaper,” said Lingjiao Chen, the paper’s lead author.
Why the cheap one can cost more
AI companies bill by the token, which is roughly a piece of a word. You pay for the tokens you send, the tokens in the answer, and, with reasoning models, the tokens the model uses to think before it answers. You usually never see that thinking. You do pay for it.
The study traced the gap mainly to two things: how much hidden thinking each model does, and, on multi-step agent tasks, how many back-and-forth steps it takes. On the very same question, one model might use 900% more thinking tokens than another, or take 10 times as many steps to finish. A model that charges a fifth as much per token but burns seven times the tokens is the expensive one.
Why nobody can budget for it
It gets harder. The researchers ran the same question through the same model more than once and saw the thinking tokens vary by as much as 9.7 times from one run to the next. They call that a noise floor: some of the cost simply cannot be predicted in advance, no matter how good your spreadsheet is.
That lines up with what companies report. In a survey of 396 companies taken in April and May by Mavvrik and Benchmarkit, only 11% said they could forecast AI spending within 10%, down from 15% in the 2025 survey. Mavvrik reports that 62% had an unexpected AI cost change a business decision in the past year. Among the fallout it lists: 40% escalated to the board, 33% put in an emergency spending freeze, and 25% delayed or cancelled an AI project.
One caution on that survey: Mavvrik sells software for tracking AI costs, so it has a reason to highlight the problem. The research paper is independent of it and points the same way.
What it means for you
On a flat monthly plan? You’re mostly fine. If you pay a set monthly price for ChatGPT, Claude or Gemini and haven’t switched on extra-usage credits, none of this changes your bill. This is about pay-per-use: company accounts billed by usage, AI agents, coding tools that charge by the token, and the optional top-up credits those subscriptions now sell.
Compare the total, not the price list. Take 20 real tasks from your own work, run them on two models, and look at what each run cost and whether the work was right. That one afternoon tells you more than any pricing page.
Budget a range, not a number. If the same job can cost several times more on a different day, plan for the high end and treat the low end as a bonus.
Put a ceiling on it. If your provider offers spending limits or alerts, turn them on before you scale anything up.
Don’t write off the premium model. Sometimes the pricier one finishes with less thinking or fewer steps and comes out cheaper.
For two years the pitch has been that AI keeps getting cheaper, and for the same level of capability, per token it has. But nobody buys tokens. You buy finished work, and the price of finished work depends on how hard the model has to think to get there. The good news is that this is easy to check for yourself, and most people haven’t yet.