The Copy Test: How Much of a Foundation Model's Skill Is Already in the Obvious Answer?

Foundation models are trained on enormous amounts of data and then asked to predict things they've never seen. A stock predictor, trained on years of market history, tells you tomorrow's price. But when you judge it, accuracy alone means nothing. You have to ask how much better it is than the silliest possible answer, and for a stock, "tomorrow's price is the same as today's" is already pretty accurate. If the model barely beats that, what exactly did the training buy?

We turned that question into a test. Take any model, and run two rivals against it on the same task: a do-nothing guess, and the silliest answer: copy the thing you just saw. Now measure the two gaps. If the model scores ten points better than doing nothing, and the silly answer scores seven points better than doing nothing, then seven-tenths of what the model gained was free. Across four standard forecasting benchmarks, that fraction averaged 72%. On one of them, two models built by different teams scored so close to "copy the last value" that we couldn't statistically tell them apart. Then we ran the same test in a completely unrelated field, labeling immune cells from their gene activity, where the silly answer is to reuse the label of the most similar cell. It beat the specialized genomics model outright.

This isn't a takedown. On three of the four forecasting benchmarks the models do beat the silly answer, and now we can say by how much. That's the point: the copy test doesn't tell you a model is bad, it tells you which part of its score it actually earned.

Why you'd care

A foundation model is a large model pretrained on a lot of data and then pointed at a new task zero-shot (used directly, with no task-specific training). This works in language. The same idea is now sold for forecasting (Chronos, TimesFM) and for single-cell genomics (scGPT). The pitch is that pretraining taught the model something general, so it can handle your new task in context.

There is a cheaper explanation for a lot of that apparent skill. Maybe the model is mostly finding the closest thing it has seen and repeating it. In forecasting people call this "context parroting." In single-cell work, researchers say "one PCA still rules them all." Both hint that a trivial copy might match the foundation model. As far as we can tell, no one had checked whether that is a general property of these models or a set of unrelated quirks. If it is general, then a benchmark number that looks like intelligence might just be retrieval. You would want a way to tell the difference before trusting the model.

How much of the skill is just copying

The test puts three predictors on the exact same held-out task and the same score.

1. A floor: a genuinely no-skill baseline that does not copy. For forecasting it predicts the average of the recent past, the way you might guess a day from the season's average (this is called climatology). For cell annotation it guesses the most common cell type. This is the "do nothing" score.
2. A copy baseline: retrieve the nearest example in a cheap representation and copy its answer. For forecasting, copy the last value, or the value from exactly one cycle ago (the "last season," for example the same hour yesterday). For single-cell, each cell is described by thousands of gene-activity numbers; we use PCA to boil that long list down to a handful of summary numbers, then find the most similar already-labeled cells and take their majority label (a nearest-neighbor vote).
3. The foundation model itself, zero-shot.

From these three scores we compute one number, the Retrieval-Explained Fraction (REF): the share of the model's above-floor skill that the copy already gets. If the model scores 10 points above the floor and the copy scores 7 above the floor, REF is 0.7. So 70% of what the model added over doing nothing is matched by copying. REF near 1 means the model is mostly retrieving. REF above 1 means the copy beats the model outright.

There is one honesty point built into REF. We summarize a small set of cheap copies, fixed in advance so we could not cherry-pick, by its strongest member. So REF is a best-case rather than a typical number: at most this much of the skill is retrievable. We report it with confidence intervals (the range the true value very likely falls in). When that interval includes 1, we cannot tell the model apart from a pure copy.

A cheap copy gets most of the forecasting skill

We audited Chronos-Bolt on four standard forecasting datasets (the ETT benchmark, electrical-transformer sensor readings over time), and ran a second, independent model family, TimesFM, on two of them (ETTm1 and ETTh2) as a cross-family check. The diagram shows the REF for each case, with its 95% confidence interval, against the line where copying explains everything.

the model's gain over doing nothingthe cheap copy's gain, as a share of it: model adds skill (CI below 1) tied with copying (CI spans 1) copy beats the model (CI above 1)

Each pair of bars is one model on one dataset. The grey bar is everything the model gained over the do-nothing floor; the coloured bar is how much of that gain the strongest cheap copy already got (the Retrieval-Explained Fraction). The grey bar is 1.0 for every case by definition, since it is the yardstick each copy is measured against. Whiskers are 95% confidence intervals on the copy. On ETTm1 both model families are statistically tied with copying the last value; on the other forecasting series the model adds real skill. The single-cell case is on macro-F1, where the copy edges the model out; on plain accuracy the copy and scGPT are tied.

Averaged over the four series, the strongest copy already gets about 72% of Chronos's above-floor skill. On one series, ETTm1, the model is statistically indistinguishable from copying the last value: the REF confidence interval includes 1 for both Chronos (0.99) and TimesFM (1.00). Two model families, built by different teams, both reduce to "repeat the last value" on that series. Model size did not help where we checked it. On ETTh1, Chronos-tiny and Chronos-base land at essentially the same REF (0.71 versus 0.70), so scaling the model up does not reduce the retrieval fraction.

On the other three series the model does add real skill over copying. The confidence interval sits below 1, roughly 30% to 43% of its above-floor performance. Which trivial rule the model imitates also changes by series (last value on some, last season on others), which is why the copy has to be a small family and not a single rule.

The same test in single-cell genomics

None of this is specific to time-series. We ran the identical three-predictor audit on a completely different problem: labeling immune cells by type from their gene expression (the pbmc3k dataset), with scGPT as the foundation model and the same PCA-plus-nearest-neighbor copy. In the chart below, both bars start at the do-nothing floor, so each bar's length is what that predictor actually added. Toggle between macro-F1 (an accuracy score averaged evenly over cell types, so rare types count as much as common ones) and plain accuracy.

Labelling immune cells by type (pbmc3k), scored by macro-F1, which counts rare cell types as much as common ones. Both bars start at the do-nothing floor (guess the most common cell type, which scores 0.076) so each bar's length is what that predictor added.

scGPT (the foundation model)

+0.737

the cheap copy

+0.829

The copy gained 0.829, the model gained 0.737. The copy therefore got 112% of what the model got: more than all of it, and the gap is big enough to be real (95% CI [1.04, 1.22]).

On macro-F1 the cheap copy is significantly ahead of scGPT (gap 0.09, CI [0.03, 0.15] excludes 0). The cheap copy here finds the most similar cells that someone has already labelled and reuses their label. Switch the metric to see that it leads on the class-balanced score and ties on plain accuracy.

The floor (guess the most common cell type) scores 0.076 on macro-F1. scGPT scores about 0.81, and the plain PCA copy scores 0.905. On macro-F1 the PCA copy is significantly ahead of scGPT (a gap of 0.09, with a confidence interval of [0.03, 0.15] that excludes zero). On plain accuracy the two are tied (0.940 versus 0.936, too close to call). So a plain PCA copy is at least as good as scGPT on this dataset, better on the class-balanced score and tied on accuracy. The copy-baseline pattern from forecasting shows up in genomics too.

What we're careful about

• REF is an upper bound, not the typical copy. We pick the strongest copy in the family, so REF says "at least this much of the skill is retrievable," not "the average trivial method does this well." We report the range across the family and the confidence interval, so the claim stays honest.
• The forecasting series are the easy case for copying. All four ETT series are highly persistent, where the recent past predicts the near future well. A low-seasonality series (like exchange rates or traffic), where copying should fail and a good model should pull ahead, is the obvious next test. We did not run it. Do not read "72% is copying" as a universal rate.
• Single-cell is one dataset and one model. One annotation task (pbmc3k), one foundation model (scGPT), one donor. scGPT's macro-F1 also moved from run to run (about 0.81 to 0.85), though the PCA copy stayed above it in each. A batched, multi-donor dataset and a second genomics model would test how far this goes.
• Everything is a single run (seed 0). The confidence intervals resample the evaluation windows and cells, not training seeds, so they do not capture run-to-run variation except where we note it above.

Takeaways

• Score-level skill is not the same as learning. A benchmark number above a floor can be mostly retrieval. Measure how much a cheap copy gets before crediting the model.
• The copy phenomenon transfers across fields. The same trivial-copy result holds in forecasting and in single-cell genomics. That points to a property of the zero-shot paradigm rather than a time-series quirk.
• Bigger did not fix it. Scaling the forecasting model up left the retrieval fraction unchanged.
• Run the audit on your own model. The REF harness is a fixed evaluator plus a swappable predictor: point it at a model and task and it returns the floor, copy, and model scores plus the REF, so you can check learning versus retrieving for yourself. The code and configs are available on request.