When can an AI improve its own math by grading itself?
There is a cheap way to try to make a language model better at math: let it grade itself instead of using an answer key. It changes some models and does almost nothing to others. Our lab found that what decides this is whether the model writes down a clean final answer at all, and that fixing even that is not enough on its own. Understanding why is the useful part.
Why you would care
Training a model with reinforcement learning normally needs an answer key. Reinforcement learning, or RL, means the model tries something, gets a reward when it does well, and shifts its behavior toward whatever earned the reward. For math, the usual reward checks each answer against the correct one.
A newer and cheaper idea skips the answer key. The model writes several answers to the same question, sees which answer comes up most often, and rewards the tries that match that most common answer. This is called verifier-free RL, because there is no external verifier holding the answer key. It is attractive because answer keys are expensive to collect. Prior reports describe it working on some base models, like the Qwen family, and doing little on others, like Llama and OLMo. A base model is the raw pretrained model, before the extra training that turns it into a chat assistant.
If you know in advance which models this trick will change, and whether the change is real, you save wasted training runs. So we tried to find the rule.
What decides it is the answering rate
The common assumption is that the strong model is simply better at math. We measured three base models on GSM8K, a standard set of grade-school math word problems, and the pattern points somewhere else.
The variable that lines up with everything is the parseable-answering rate: how often the model writes its final answer in the requested format, a number wrapped in a box, the way a textbook prints the answer at the end. A grader can only read the answer if it is in that box. Call it the answering rate for short. Here is what the three models look like before any training, measured on 200 test problems:
| Base model | Answering rate | Accuracy (pass@1) |
|---|---|---|
| Qwen2.5-3B | 0.95 | 0.80 |
| OLMo-2-7B | 0.48 | 0.43 |
| Llama-3.2-3B | 0.01 | 0.00 |
Pass@1 is the fraction of problems the model gets right on its single most-likely answer. Qwen writes a clean answer almost every time and scores well. Llama almost never writes one. It scores near zero.
That zero is not mostly about math ability. It is mostly about formatting. Here is the evidence. Give the model eight tries at each problem instead of one, and count it correct if any of the eight is right (this looser score is called pass@8). Under pass@8, Llama solves 0.14 of the problems. So it can reach the right answer. It usually just does not write one down in the box.
A closed gate makes every reward useless
Take Llama, the model that almost never writes an answer, and train it three different ways: with the self-grading reward, with a perfect answer key, and with a completely random reward. All three leave its accuracy exactly where it started, at 0.000. Even accounting for the small test set, the true effect is at most about one or two problems in a hundred.
A perfect answer key changes accuracy no more than a random reward does. When the model almost never produces a readable answer, there is nothing for any reward to grade. The reward has no effect on training. For this model, the answering rate acts as a gate. When it is closed, no reward gets through.
Writes a clean answer almost every time, and scores well. The gate is wide open.
Across the three base models, how often a model writes a readable final answer tracks how often it gets one right. When that answering rate is near zero, no reward can train the model.
Only one of our three models arrived with the gate this far closed, so this is one clean observation, not a law across many models. It is also not the training method quietly failing to run. The same method moves this same model's answering rate a great deal once the gate is open, which is what the next section shows.
A short format lesson opens the gate
If the answering rate is the gate, we should be able to open it and watch what happens. So we gave Llama a short lesson in format only, with no math taught. We fine-tuned it on a generic template: a few lines of plain reasoning that end by writing a final number in the box. Fine-tuning means training the model a little more on a small set of examples. The template used training questions the model never sees at test time, so it teaches the shape of an answer without leaking any test answers. We call this a format prime.
The prime works. Llama's answering rate jumps from near zero to 0.535. It goes from almost never answering to answering about half the time, with real numbers. Its accuracy even rises, from 0.00 to 0.05. So the answering rate can be changed by training. It is not a fixed trait of the model.
The same format lesson hurts the models that were already good. Qwen's accuracy drops from 0.80 to 0.27, and OLMo's from 0.47 to 0.12. (These prime runs measure each model in a separate pass from the table above, on the same 200 problems, so the untrained numbers differ by a point or two.) The plain template overwrites the careful step-by-step reasoning those models already had. So a habit, writing an answer, and a skill, working out the answer, are two separate things. You can teach the first without the second. On a strong model, teaching only the first does real damage.
Opening the gate is not enough
With the gate open, the self-grading reward still does not help Llama. It pushes the answering rate from 0.535 to 0.93. But accuracy does not follow. It stays flat, moving from 0.05 to 0.01, a change too small to read as real. The model learns to always commit an answer, and to commit the same one it already prefers. On a model that cannot actually do the math, that preferred answer is usually wrong. So self-grading makes the model more confident without making it more correct.
This is Llama after the format prime opened its gate. Click the button to run verifier-free RL on top and watch which bar moves.
Verifier-free RL rewards agreement with the model's own most common answer, not correctness. On a model that cannot do the math, it raises confidence without raising accuracy.
Verifier-free RL optimizes agreement, not correctness. It rewards the model for matching its own most common answer. Where that common answer tends to be right, which needs real ability, agreement and accuracy improve together. Where it tends to be wrong, the model just gets more sure of a wrong answer.
The two models with more ability move up after the format prime, by 0.02 on OLMo and 0.04 on Qwen. Those numbers are recoveries from the damage the prime did, not gains over the untrained model. Qwen went 0.80 at the start, down to 0.27 after the prime, back up to 0.31 after RL, still far below where it began. Across the whole study, verifier-free RL never produced a clean net gain over any untrained model. What tracks a model's math ability is only how much of the prime's damage the RL step claws back, and even that is a small move from a single random seed (one training run from one random starting point).
Where the finding stops
The clean, load-bearing results here do not depend on a lucky run. A perfect answer key did exactly nothing on a model that will not answer. An answering rate that climbed past 0.9 while accuracy stayed near zero is a large, plain effect. Neither is a close call.
The softer parts are held back on purpose. Every number comes from a single random seed on 200 problems, so the small recovery moves are directional, not proven. The claim that math ability governs how RL behaves rests on three models, so it is a pattern across three models, not a proven law. We studied one dataset (GSM8K), one training method (a standard reinforcement-learning method called GRPO), and one self-grading reward (majority vote, which rewards the answer the model produces most often), with the answer-key and random rewards as controls. A gentler prime that raises the answering rate without harming a strong model would let us test the ability question more cleanly, and we did not build one here.
Takeaways
- Verifier-free RL for math needs two separate things from the base model: a habit of writing a clean final answer, and the ability to get that answer right. Miss either one and the trick does nothing useful. In our tests it never cleanly beat the untrained model on any of the three.
- When the answering rate is near zero, no reward helps, not even a perfect answer key. There is nothing to grade. Check the answering rate before you spend a training run.
- Self-grading rewards optimize agreement, not correctness. A rising answering rate can look like progress while accuracy stays flat. Watch accuracy, not how often the model commits an answer.
- Answering habit and reasoning ability are separable. Teaching format is cheap and can pull a model off the floor, but the same lesson can wreck a model that already reasons well.
Built on Alibaba's Qwen2.5, Meta's Llama 3.2, and Ai2's OLMo 2 base models, evaluated on GSM8K. The full paper can be downloaded below. Code, configurations, and the exact evaluation recipe are available from the authors on request.