When Will More Thinking Help a Reasoning Model? Ask the Cheap Eval First
Reasoning models often score higher when you let them think longer, and evaluating them at long thinking budgets is expensive. Our lab measured what a cheap, capped eval can honestly tell you about the expensive one, before you pay for it.
Quick summary: We ran three open reasoning models on four math and science benchmarks across eight thinking budgets (caps on how many tokens a model may spend reasoning), froze curve-based forecasts of the high-budget scores before measuring them, then checked the forecasts. As a forecast, the cheap eval fails: the frozen curves missed by up to 24 accuracy points. But one extra check inside the cheap run, whether the model actually used up its thinking allowance, turns it into a reliable purchasing signal. Across all 36 measured combinations of model, benchmark, and thinking budget (one combination is a cell), the resulting buy rule recommended paying for a bigger budget 18 times. Every buy was right: zero over-promises across the 18 buy calls, including on a model lineage held out until after the rule was frozen. The rule still misses small gains; 9 of its 36 calls declined budget that would have helped a little. The scope is narrow: models at 14B parameters and smaller, math and science benchmarks, and 24 of the 36 cells are the cells the rule was built on, so treat the perfect record as promising, not proven.
The eval you actually want to run is the expensive one
A thinking budget is the number of tokens (the word-pieces a model reads and writes) a reasoning model is allowed to spend working through a problem before it answers. Give the model more room and it often scores higher. So the eval you really want (an eval is a benchmark test of a model's accuracy) is the one at a big budget, 32,768 thinking tokens, because that is closer to how you would deploy the model. We tested three open models this way: Qwen3-8B, Qwen3-14B, and DeepSeek-R1-Distill-Qwen-14B (R1-Distill below), built on Qwen's and DeepSeek's model releases.
That eval is painful to buy. The single most expensive run in our campaign, one model at a 32,768-token budget across all four benchmarks, took about 19 A100 GPU-hours (hours on one A100 graphics card, the unit compute is billed in). The entire low-budget probe for Qwen3-8B, four benchmarks at five budgets from 0 to 4,096 tokens, cost 6.76 GPU-hours. For each model, the whole probe grid costs less than any single high-budget run above 8,192 tokens, and about 45 percent of the 8B model's 32,768-token run.
The whole 20-cell probe grid (4 benchmarks x 5 budgets up to 4k) costs less than any single high-budget run above 8k for the same model.
A100 GPU-hours per model. Asking the question (the full low-budget probe grid) costs slightly more than a single 8k run, and roughly 21 to 25% of the three high-budget runs combined. In thinking tokens the probe is 35 to 39% of those runs across the three models, which missed our own 20% cheapness bar, set in advance; the probe is cheap, not free.
You have the cheap probe results in hand. Should you pay for the big-budget run? The probe already answers that: the model reveals in the cheap run whether it is still using the budget it has. That one free measurement is the buy signal this post tests.
The cheap curve cannot see what happens at 32k
The obvious move is extrapolation. Fit a curve through your probe scores at 512 to 4,096 tokens, extend it out to 32,768, and read off the predicted score. We did exactly that. We froze the curves in advance so we could not fool ourselves, and the predictions are consistently wrong. They are wrong in two opposite ways, and which way depends on the benchmark and the model.
On hard competition math (AIME), all three models are budget-hungry. At the 4,096-token probe cap their realized thinking tokens, the tokens the model actually spent thinking as opposed to the cap you set, run at essentially the whole allowance, around 3,800 to 4,080 of it. When you raise the cap they keep thinking; the two Qwen3 models plateau naturally around 12,000 to 17,000 tokens, and R1-Distill earlier, near 10,700. The extra thinking pays: measured scores at high budgets beat the frozen curve by 9 to 24 accuracy points on every Qwen3 AIME cell at 16k and 32k. The worst miss was a model scoring 0.723 where the curve predicted 0.485.
On easier math (MATH-500) and science questions (GPQA-Diamond), the Qwen3 models are budget-indifferent. They simply stop thinking, at roughly 3,500 to 6,600 tokens, no matter how much room you give them. The bigger budget buys almost nothing. The curve keeps rising anyway and over-predicts by up to 10 points. R1-Distill is budget-indifferent on MATH-500 too, stalling near 3,000 tokens, but not on GPQA-Diamond: at 8,192 tokens it was still hitting the cap on 30 percent of its responses. That cell comes back below; it is the one call the buy signal's saturation check got wrong.
On AIME the stars land above the frozen curve at every budget: the probe forecast undersold the model, by 9 to 24 accuracy points on the Qwen3 16k and 32k cells. The curve was a floor, not a score. Filled circles are the four cheap probe cells; the curve was frozen from them before any high-budget cell ran. Each star is a measured expensive eval plotted at the tokens the model actually used; the dashed link connects it to the open triangle, where the frozen curve predicted that cap. The gap between star and triangle is the forecast miss, and it goes in opposite directions on AIME versus GPQA and MATH-500.
Both failures come from the same blind spot. The curve is fit on accuracy against the realized thinking tokens of the probe cells, but only over the probe range. It cannot see whether the model will keep consuming budget beyond that range. That second quantity is the one that matters. The probe already measured it for free.
We froze every prediction before buying the answer
The numbers above are only meaningful because we locked the experiment down in advance.
Our eval software enforces a hard thinking cap: generation is cut at exactly the budget, and a response that hits the cap mid-thought is truncated (forced to stop thinking and answer with whatever it has). A cap-fidelity control metric confirmed the cap behaved exactly as specified on every cell we measured. The benchmarks are AIME-2024 and AIME-2025, MATH-500, and GPQA-Diamond, run with 4 to 16 seeds per benchmark (a seed is the random-number starting point, so repeated runs sample different responses).
For each model and benchmark we fit curves on the probe cells at 512 through 4,096 tokens, plotting accuracy against the mean realized thinking tokens of each cell rather than its cap. We compared four candidate curve shapes and picked one per benchmark by held-out validation. Then the preregistration step: we wrote down every prediction at 8,192, 16,384, and 32,768 tokens, with 90 percent prediction intervals, and froze them with a SHA-256 hash (a cryptographic fingerprint of the file) before that model's high-budget runs were queued. Preregistration just means committing to your predictions in a tamper-evident way before seeing the answers, so a curve cannot be quietly refit after the fact.
The saturation check (call it the screen) is one line of arithmetic on data the probe already collected: a model-benchmark pair is flagged budget-indifferent if its mean realized thinking at the 4,096-token probe is under 90 percent of the cap. No extra generation, no extra cost.
Finally, we froze the buy rule itself about three hours before the first R1-Distill high-budget run finished. Every R1-Distill high-budget cell is therefore a genuine holdout: the rule never saw that model lineage's expensive results.
The buy signal was never wrong in the expensive direction
The rule is simple. Buy the bigger budget only if the frozen curve forecasts a gain of at least delta over your probe-top (4,096-token) score, and the saturation check classifies the pair budget-hungry. We tested delta thresholds of 2 and 5 accuracy points. A buy counts as correct when the measured gain at the target budget is at least delta; a don't-buy counts as correct when the measured gain falls below delta.
Across the 36 measured cells the rule got 27 decisions right at the 2-point delta and 32 at the 5-point delta: 75 and 88.9 percent (Wilson 95 percent intervals, a standard confidence range for a success count, of 58.9 to 86.3 percent and 74.7 to 95.6 percent). The rule said buy 18 times at the 2-point threshold and 16 times at the 5-point threshold, and every single buy paid off. Every error it made was an under-promise: it told you to keep your money when you would in fact have gained a little, 2.3 to 5.2 points at the 2-point delta.
The saturation check is what does the work. A naive rule using the curve alone hits the same 75 percent accuracy at the 2-point delta, but all 9 of its errors are over-promises, the expensive kind where you pay for a big run that does not deliver. The saturation check converts costly errors into cheap ones.
The check has one documented failure, also in the cheap direction. R1-Distill on GPQA-Diamond used about 82 percent of its probe allowance, so the screen classified the pair budget-indifferent. The cell was in fact still truncation-bound at 8,192 tokens (30 percent of responses hit the cap) and gained 2.3 points there, just over the 2-point bar. That don't-buy is one of the 9 under-promises. The lesson generalizes: whether a pair is budget-hungry depends on the model and the budget, not the benchmark alone, so measure it rather than assume it. Treat the 90 percent threshold itself with care too. On our grid every flagged pair sat at or below 86.4 percent of the cap and every passing pair at or above 92.8 percent, so the line sits in an empty gap and is untested inside that margin.
(said buy, budget was wasted)
(said skip, model still gained)
BUY iff (thinking at 4k probe >= 90% of cap) AND (forecast gain >= 2 points)
One cheap check decides every call in this grid: did the model actually use its 4k thinking allowance? All six AIME cells sit above the 90% line and the rule says buy; every one of those purchases paid off. The six GPQA and MATH-500 pairs sit below it and the rule says keep your money. One of those calls was wrong: R1-Distill GPQA used only 82% of the probe cap, yet at an 8k budget (a cell not shown on the card) it was still hitting the cap on 30% of responses and gained 2.3 points. That is the screen's known failure mode, and the error is in the cheap direction. Across all 36 measured cells the rule made 0 over-promises and 9 under-promises: when it was wrong, the cost was missed upside (largest +5.2 points, R1-Distill MATH-500 at 32k), never wasted spend. Cards show the highest measured budget per pair (32k for every pair).
The holdout is complete, and all 6 R1-Distill buy decisions were correct. The standout was AIME-2024 at 16,384 tokens: the curve frozen two days earlier predicted 0.6296 and the model measured 0.6292, within 0.0005. On AIME-2025 at the same budget the model beat its forecast by 6 points. The final cell of the campaign then delivered the record's one honest blemish. On AIME-2024 at 32,768 tokens the frozen curve forecast 0.7353 and the model measured 0.6750: the forecast overshot the score by 6.0 points, because R1-Distill's natural thinking saturates near 10,700 tokens, short of what the curve assumed. The buy call was still right by a wide margin; the model gained 27.1 points over its probe top against a forecast gain of 33.1. So the honest pattern across the 18 buy cells in our grid: the measured score matched the frozen forecast (to within 0.3 points) or beat it on 17 of the 18 buys, and fell 6.0 points short once, on a cell where the purchase still paid for itself many times over.
The sample is still small. Eighteen buys with zero errors leaves a 95 percent upper bound of about 17 percent on the true over-promise rate, by the rule of three (the textbook upper bound when you have seen zero failures in n tries). Twelve of those 18 buys are Qwen3 cells the rule was built on. The six holdout buys alone bound the rate only at 50 percent; six decisions cannot bound much yet.
Point prediction failed, and that failure is why the buy signal holds
We held the forecasts to preregistered quality gates, and they failed them. On budget-hungry cells, only 50 percent of frozen point predictions landed within twice the measurement noise of the truth. The frozen 90 percent intervals covered the measured score in 19 of the 36 cells, 52.8 percent, far below their target. For score prediction the forecasts are unusable.
The misses have structure, though. Of the 18 cells the screen classed budget-hungry, 15 measured above their frozen prediction. The three exceptions are all holdout cells: two landed essentially on the curve (within 0.05 and 0.3 points), and one, R1-Distill AIME-2024 at 32,768 tokens, fell 6.0 points below it. So on cells where the model wanted more budget, the curve was always low on the Qwen3 grid, and on the holdout it overshot the score once, by 6 points, on a cell where the decision still cleared the 2-point bar by 25 points of measured gain. Combine that mostly one-sided miss with the saturation check, which filters out the cells where the curve reliably overshoots, and 17 of the 18 surviving buy forecasts were met or beaten; the one that overshot missed the score, not the decision. That structured failure, plus the wide margin on its single exception, is the evidence that makes the purchasing signal trustworthy.
Where the signal has not been tested
The claims stop at the edges of the grid. All three models are 14B parameters or smaller; nothing here is evidence about frontier-scale models. The four benchmarks are math and science with checkable answers; no code, agentic, or open-ended tasks. The budget knob is a hard token cap, and the results may not transfer to models that control thinking length through prompting or training. AIME has only 30 items per year, so a single item swings a cell by 3.3 points. The companion paper reports the slightly smaller pre-holdout grid, so a few of its percentages differ from this post's by a point or two.
The rule also has two costs. First, it was constructed on the 24 Qwen3 cells, so those cells are in-sample; only the R1-Distill cells test it out of sample, and there are 6 buy decisions there. Second, caution has a price. At the 5-point delta the rule declined two cells that went on to gain 15.0 and 17.1 points, the largest upside it left on the table; at the 2-point delta the largest missed gain was 5.2 points. Those two big misses were near-threshold calls: the frozen forecasts predicted gains of 4.2 and 4.5 points, just under the 5-point bar, and because forecasts on budget-hungry cells are biased low, near-threshold declines are where the rule leaves the most value. The rule never wastes your money, but it will sometimes leave real gains unclaimed. The probe itself is not free either. Across the three models it cost 35 to 39 percent of the thinking tokens of the three high-budget runs, and for the 8B model 21 percent of their GPU-hours. The token fraction missed our own preregistered 20 percent bar.
What you can take to your own evals
- Log realized thinking tokens, not just the budget you set. Whether the model uses its allowance is the single most informative number in a capped eval, and it is free.
- Before paying for a high-budget run, check the probe: if the model left thinking room unused at 4k, a bigger budget bought at most 5.2 points in our grid and usually nothing.
- Treat an extrapolated scaling curve as approximately a floor on budget-hungry cells, never as a forecast. Ours missed by up to 24 points, essentially always low; the one meaningful overshoot in 18 buys was 6 points, on a cell that still gained 27.
- Freeze predictions before running the expensive cells. "Zero over-promises" is only a checkable claim because the curves were hashed and the rule frozen before the answers existed.
The full paper (PDF linked above) has the complete grid, the preregistration artifacts, and the statistical treatment. Code and per-response records are available on request.