Reusing a training-difficulty score across runs: when it holds up and when it doesn't
Picture two researchers training the same kind of model on the same dataset. Each starts from a different random seed, a starting point for the model's random numbers. Two runs with different seeds are two honest, independent attempts at the same job. Before either run finishes, each one can rank every example in the dataset from easy to hard. If the two rankings roughly agree, "hard" is a real, stable property of the data. If they don't agree, "hard" was an artifact of one particular training run, and reusing that ranking anywhere else is a mistake. We tested this directly. The ranking method you use decides the answer.
Why you'd care about this
Some training setups spend extra compute on the examples a model finds hardest, on the theory that a model learns more from a hard example than an easy one. To do that, you need a difficulty score for every example in your dataset. That score can be computed two ways. You can compute it once, from a separate "scout" run, then reuse it unchanged for every run after that. Or you can recompute it live, inside the run it's steering. That costs more compute and engineering effort, but it tracks the model's own current state.
Which one you should do depends on a single question: is per-example difficulty a stable, transferable property of the data, or is it mostly noise from one particular training run? If it's stable, reusing an old score is fine and saves you real compute. If it's noise, reusing it is actively misleading the run you feed it into. A recent paper on one popular type of difficulty score reported the second answer. The score barely agreed with itself across two independent training runs: a correlation of about 0.01. We'll use this 0-to-1 correlation scale throughout this post. A 1.0 means two rankings agree perfectly; a 0 means no relationship at all. A 0.01 is close enough to zero to mean the score isn't capturing anything real in the data itself. We wanted to know if that holds for difficulty scores in general, or just for the one type of score that paper tested.
We built a mechanism that pays extra attention to hard examples
We built a small compute-allocation mechanism. We call it an auction because every example "bids" with its own difficulty score, and the highest bidders win extra training. Every training example gets a difficulty score. The 10% highest-scoring examples (the hardest, by whatever scoring method we're testing) get two optimizer steps that epoch, an optimizer step being one weight update to the model. Every other example gets one step, the normal amount. That 10% cutoff and the 2x step count were fixed for the whole study; we didn't try other settings, so we can't say whether a different cutoff or a bigger step bonus would change the results below.
We ran this on two image classifiers: ResNet-18 and a compact vision transformer we built specifically for small images (32-by-32 pixels, not the standard 224-pixel input a vision transformer usually expects), which we call DeiT-Small. Both trained from scratch, with no pretrained starting weights, on CIFAR-10 and CIFAR-100, two standard small-image benchmarks.
A group of training examples arrives.
The scores above are a stand-in for illustration. The top 10% by score get 2 optimizer steps that epoch; the remaining 90% get 1. We fixed this ratio for the whole study.
We tested two kinds of difficulty score. A forgetting score counts how many times a model's prediction on a given example flips from correct to incorrect as training goes on. An example whose prediction keeps flipping back and forth gets a high forgetting score. A loss score is an exponential moving average, a rolling average that weights recent steps more heavily, of how wrong the model's prediction is on that example at each epoch. Both are established methods from the research literature. We tested how each one behaves when reused across runs.
For each score, we compared a static version (scored once by a separate scout run and frozen) against an online version (recomputed live, inside the run it steers). The baseline was a uniform-compute control: no scoring at all, every example gets one step. For the forgetting score, we also tried a fourth variant, an ensembled static score averaged over three separate scout runs instead of one, to see whether averaging away scout-to-scout noise closes any gap to the online version. If reusing a stale score is actually harmful, the static arm should train worse than the online arm.
For the forgetting score, we couldn't detect the auction doing much
We ran the forgetting-score auction across both datasets and both architectures, three separate training runs (seeds) per combination. Every version landed close to the uniform-compute control's final accuracy: static, online, and the ensembled static variant, all within 1.4 percentage points of control. For comparison, just rerunning the control itself with a different random seed moves its own accuracy by 0.2 to 2.1 percentage points. The auction's effect, whatever it was, was smaller than the noise you'd see from changing nothing but the random seed. We could not detect the auction doing anything meaningful for the forgetting score, static or online, at this scale. With only three seeds per condition and no formal statistical test, that is an absence of a detected effect, not proof the true effect is exactly zero.
That result makes the static-versus-online question hard to answer directly for the forgetting score. If a mechanism doesn't move the outcome either way, comparing two versions of feeding it a score can't tell you much. So we went and measured the underlying question directly, without routing it through the auction at all.
Loss scores hold up across runs much better than forgetting scores do
We ran a scout pass and a second, fully independent pass: same setup, different seed, no auction at all. We compared their scores directly, using the same 0-to-1 Spearman correlation scale from above. We measured it at four points during training: 25%, 50%, 75%, and 100% of the way through. That is a direct measurement of the question that matters: does this score, computed by one run, predict what a completely different run will compute for the same examples?
Each gauge is the Spearman correlation (0 to 1, how well two independently trained runs agree on which examples are hardest) between a frozen scout score and a separate run's own score, on CIFAR-10, at this point in training. The loss score's two runs agree closely from early in training onward. The forgetting score's two runs agree much less, especially early on. It only partly catches up by the end.
By the end of training, the loss score's two independent runs agree at a correlation of 0.79 on CIFAR-10 and 0.92 on CIFAR-100. The forgetting score's two runs agree far less: 0.58 on CIFAR-10 and 0.47 on CIFAR-100. Both numbers sit well above the near-zero result the earlier paper reported for a forgetting-style score. Forgetting scores are not pure noise here. They do, however, track "which particular training run happened" much more than the loss score does. The gap between the two score types holds at every checkpoint we measured, not just at the end. It is largest early in training: at the first checkpoint, the loss score's agreement is already roughly 3 times higher than the forgetting score's (2.9x on CIFAR-10, 3.3x on CIFAR-100). That early-training ratio is dividing by a small number, the forgetting score's own correlation is only 0.20 to 0.22 at that point, so it moves around more than the steadier 1.3x to 2x gap at the end of training.
If you're deciding whether to reuse a difficulty score or pay to recompute it live, this is the finding that answers it: the type of score you picked matters more than whether you reuse it. A loss score computed once holds up well enough that reusing it is a reasonable bet. A forgetting score computed once is noisier across runs and a shakier bet, even though it is nowhere near as unstable as the near-zero result from prior work.
What didn't work (the useful part)
We also ran the loss score through the same auction, on CIFAR-10 only, as a check on whether a more stable score behaves differently inside the mechanism. On DeiT-Small, the static loss-scored auction dropped accuracy by about 6 percentage points relative to control, negative on all three seeds. The online version dropped less, about 2.7 percentage points on average and negative on two of the three seeds. ResNet-18, under the identical scorer and threshold, stayed within about 1 percentage point of control on both versions, but that ResNet-18 comparison only has two seeds, too few to know its own natural spread, so it isn't on the same evidentiary footing as DeiT-Small's three.
We added the third DeiT-Small seed after the first two already showed the drop, a decision we hadn't planned in advance. That makes the effect look more unanimous than a pre-planned three-seed run would have.
The static and online loss scores agreed closely with each other on DeiT-Small, a correlation of 0.74 to 0.79, similar to the overall loss-score result above. That rules out one explanation for the drop: the two scoring passes disagreeing with each other. It doesn't tell us what did cause it. We don't have a full explanation yet. We're flagging it as an open question, not a resolved one. DeiT-Small trained from scratch was already the shakiest setup in our whole study: its own baseline accuracy on CIFAR-10 sits around 0.53 to 0.56, against ResNet-18's 0.93. Be cautious about generalizing this specific result; we're not weighting it as heavily as the correlation finding above.
Where this doesn't (yet) apply
Everything here ran at CIFAR scale, which is small and cheap by modern standards, with two scorer types and two architectures. We didn't test other difficulty-scoring methods, like ones based on a model's internal gradients or its learned representations of each example. We also didn't test whether any of this holds at the scale of a real production training run. The headline correlation result, the loss score beating the forgetting score across training, was measured on ResNet-18 only. We left DeiT-Small out of that specific comparison because its from-scratch training is the shakiest setup in the whole study. We didn't want that fragility to distort the trajectory. That means we haven't shown the loss-versus-forgetting gap holds on DeiT-Small itself. The correlation result is also whole-dataset agreement. It doesn't directly measure agreement specifically among the top 10% hardest examples, which is the slice the auction mechanism actually acts on. Every number here comes from 2 to 3 training runs per condition, with no formal statistical test behind the comparisons. We're reporting a clear, consistent pattern, not a statistically certified one.
Takeaways
- The type of difficulty score matters more than whether you recompute it. A loss score's ranking of "which examples are hard" holds up well across independent training runs. A forgetting score's ranking holds up too, just noticeably less. Check which kind of score you're using before deciding a one-time scoring pass is good enough.
- A near-zero cross-run correlation isn't a universal property of difficulty scores. It shows up for one score type here, but the same measurement on a different score type lands far higher. Don't assume a finding about one scoring method generalizes to every scoring method.
- A mechanism that doesn't move the outcome metric can't tell you much about its own design choices. We couldn't learn whether static or online scoring was better for the forgetting score, because the auction itself barely changed accuracy either way. Measure the underlying question directly if your outcome metric is too flat to discriminate.
- A stable score isn't automatically a safe score. The loss score's two runs agreed closely on DeiT-Small. The auction still hurt accuracy there. Score stability rules out one failure mode (the score disagreeing with itself), not every failure mode.