The crystal-structure advantage in materials AI is real, not leakage
Machine-learning models that predict a crystal's properties come in two kinds: ones that read only the chemical formula, and ones that also read the 3D arrangement of atoms. On the standard benchmark, the structure-reading models win. A common worry is that they win by cheating, seeing near-identical crystals in both training and testing. We measured that directly. The cheating is not there, and we can show exactly where the real advantage comes from.
Two kinds of materials models, and a nagging suspicion
Say you want to predict a property of a crystal, like how stiff it is or its band gap (the energy jump that decides whether it conducts electricity). You can build two kinds of model.
The first is a composition-only model. It reads just the chemical formula, which elements are present and in what ratio, and predicts the property. The second is a structure model, usually a graph neural network (a model that treats the crystal as atoms connected by bonds and passes information along those bonds). It reads the full 3D atomic arrangement.
On Matbench, the standard benchmark for this task, the structure models win, often by a lot. That result gets a common reaction: maybe the win is leakage. Leakage is when near-identical examples land in both the training set and the test set, so the model can score well by memorizing rather than by learning something general. It has been documented in other fields; in one drug-discovery benchmark, a model with no learned parameters at all, one that just copies the answer from the most similar training example, matched the state of the art. Materials benchmarks are known to contain duplicate crystals, and people deduplicate to guard against it. So the suspicion is reasonable. Nobody had actually measured whether it explains the structure advantage. We did.
First finding: the leakage is not there
The formula-collision rate is misleading. If you group crystals by formula, a huge fraction share a formula with another crystal in the data, up to 98% on one task. That number looks like the benchmark is riddled with copies.
But sharing a formula is not the same as being a near-identical crystal. Many crystals have the same formula and genuinely different structures. This is called polymorphism (diamond and graphite are both pure carbon). What matters for leakage is whether the actual 3D structures are near-duplicates across train and test. We checked that directly with a structure-matching tool (pymatgen's StructureMatcher, which compares crystals up to small distortions), and swept it from strict to loose to be sure.
Toggle between the two numbers below.
Up to 98% of crystals share a chemical formula with another. This looks like leakage, but it is genuine polymorphism (same formula, different structure), not near-identical crystals.
The formula-collision rate collapses to almost nothing once you ask the real question. On perovskites it is exactly 0% at strict, default, and loose matching. On the elastic-property tasks it tops out at 2.8% even under deliberately over-loose matching. We did not run the exact all-pairs sweep to completion on the two biggest tasks: on the band-gap task it came back near zero over the first three-quarters of its pairs, and the very largest task we did not finish, so we do not claim a completed count for those two. Every task we measured to completion came back near zero. A de-duplicated split is essentially the same as the random split. Near-duplicate leakage cannot be what makes the structure models win.
Second finding: where composition looks bad, the benchmark is the reason
If leakage is not inflating the structure advantage, why does composition look so weak on some tasks? On perovskites the composition model's error is 0.581 eV/atom, about eight times worse than the structure model.
The answer is that polymorphism, the same thing that produced those formula collisions, hands composition models an impossible task. When one formula appears with several different structures that each have a different target value, a composition model sees identical inputs with different correct answers. It cannot tell them apart. There is a hard floor on how low its error can go, and we can compute that floor exactly, no training needed. On perovskites the floor is 0.379 eV/atom, which is 67% of the values' mean absolute deviation (a measure of their spread). Two-thirds of what looks like "composition is terrible here" is a property of how the benchmark is built, not a failure of the model.
The structure model, which can see the different arrangements, is not bound by that floor. Its perovskite error (0.071) sits far below it. This is the honest version of the perovskite story: structure wins big there mostly because composition is handed an unsolvable degeneracy, and structure resolves it.
The real question: how much does structure actually help, per property?
With leakage ruled out and the floor accounted for, we can ask the clean question. We trained a structure model (a from-scratch CGCNN, a crystal graph neural network we wrote in plain PyTorch) and compared it to the composition model on all five tasks. We test rigorously: same five-fold split with a fixed seed for every model, and the composition numbers reproduce the published benchmark references.
Pick a task below.
The biggest advantage. Composition cannot get below 0.379 no matter what (the floor). The structure model falls well below it.
The structure model beats composition on all five tasks, so the advantage is real everywhere. But its size depends heavily on the property, from a gap of +0.510 on perovskites down to +0.012 on shear modulus, about a 40-fold range. Structure matters enormously for some properties and barely for others. The blanket claim "structure models are better" is true but hides most of the story.
Third finding: the advantage grows on unfamiliar crystals
A memorization story makes a specific prediction. If the structure model wins by recognizing crystals close to ones it trained on, its advantage should be largest for test crystals near the training set and should shrink for crystals far away. We checked. For each test crystal we measured how far it is from the nearest training crystal, then split the test set into four bands from nearest to farthest and looked at the structure advantage in each band.
Press play.
The structure advantage per crystal, split by how far the crystal is from anything in training. For bulk modulus it grows with distance. If the advantage were memorized duplicates, it would shrink far from training, not grow. Band gap is the exception: structural features stop helping the farther out you go. (Each row is scaled to its own range; perovskites gaps are far larger than the others.)
For bulk modulus the advantage grows the farther a crystal is from training data. That is the opposite of memorization. It is what genuine generalization looks like: the structure model helps most exactly where a composition model is most lost. We saw the same direction with a harder out-of-distribution split, where we hold out whole clusters of similar crystals; the structure advantage stays positive on the structural tasks (perovskites +0.295, formation energy +0.063) and survives the distribution shift.
What did not work: band gap, and a dependency wall
Band gap is the honest exception. When we appended structural features to the composition model, they did not help, they hurt (a gap of -0.027). Band gap is driven mostly by electronic structure that our geometric features do not capture, and the extra features just added noise. Only the end-to-end graph network recovered a small gain (+0.034). Band gap is the one task where bolting structural features onto the composition model made the predictions worse instead of better. For band gap, structure barely helps.
We also hit a wall worth naming. The standard structure-GNN library (dgl) would not install cleanly against the GPU we had, and three attempts to fix the dependency chain failed. Rather than keep fighting it, we wrote a compact CGCNN in plain PyTorch with no dgl at all. It trained in about two minutes on a single GPU and reproduced the expected accuracy. That workaround is why this result exists at all.
Where this does and does not apply
These are five Matbench regression tasks with DFT-computed labels (density functional theory, a simulation rather than a lab measurement). DFT band gaps carry a known systematic error, so the band-gap conclusions are about a proxy, not experiment. Our CGCNN is a from-scratch model, not tuned to leaderboard state of the art, and our composition baseline (Magpie features with a random forest) is weaker than the strongest composition models; a stronger composition model would shrink every gap we report. The numbers are single point estimates without error bars, so read the smallest gaps as directional. The distance analysis uses the feature-based structure model rather than the graph network, because per-crystal predictions were available there. We list these limits in full in the paper.
Takeaways
- The structure-model advantage on Matbench is not near-duplicate leakage. Structural duplicates are essentially absent, and the advantage grows on crystals far from training data.
- The high "same formula" collision rate is genuine polymorphism, not leakage. It creates a hard error floor for composition models, which explains where composition looks weak.
- The structure advantage is real but strongly property-specific, from huge on perovskites to tiny on elastic moduli, and smallest of all on band gap.
- When a benchmark makes one model look bad, check whether the benchmark's construction, not the model, is responsible before drawing a conclusion.
The leakage-checked splits and the analysis are available from the authors on request.