Is skipping zeros free on a wafer-scale chip?
Our lab measured sparse decode math one cycle at a time on a Cerebras wafer-scale simulator. Skipping zeros turns out to be nearly free, but only if you write the kernel to walk the non-zeros, and only on a single core. Spread the same work across a thousand cores and the bottleneck moves off the math and onto the cores coordinating with each other.
When a language model writes a word, most of the work is one specific operation repeated over and over: a matrix-vector product, or GEMV (multiply a list of numbers by a big grid of weights). On a normal GPU this step is memory-bound: the chip finishes the math long before it finishes hauling the weights in from its main memory (HBM). So it sits idle, waiting for data. In roofline terms (a chart that compares how much math a step does against how many bytes it moves, to show which of the two is the limit), this step does very little math per byte. Its arithmetic intensity is about 0.5 math operations per byte in the precision we ran, and up to about 1.0 in lower precision. A common GPU like an A100 needs roughly 153 (in those same units) before its math units stay busy. So the chip is starved of data, not short of compute.
Two ideas fall out of that one fact. First: if the weights already lived on the chip instead of in far-away HBM, the starvation would go away. Second: if most of those weights are zero (a sparse model), and the chip can skip a zero cheaply, then the number of cycles should drop right along with the fraction of zeros, and it should not matter how the zeros are arranged.
The Cerebras Wafer-Scale Engine (WSE) is built for the first idea. It is one giant chip made of hundreds of thousands of small cores called processing elements (PEs), and each PE keeps its slice of the weights in its own on-board memory (about 48 KB of SRAM per core). No HBM trip, no starvation. That makes it the right place to ask the second question, which as far as we can tell has not been answered with real cycle counts: is skipping zeros actually cheap here?
So we measured it. We ran small sparse GEMV kernels on the Cerebras SDK fabric simulator, which reports the exact number of cycles a real WSE would take, and we checked every result against a plain NumPy computation so we know the kernels are correct, not just fast. The whole study is cheap to reproduce: about 1.8 machine-hours on the simulator, end to end.
Skipping zeros the obvious way does almost nothing
The natural first kernel is what we call approach A: loop over every position in the weight grid, and when a weight is zero, skip the multiply. It sounds like it should save all the work on the zeros, but it does not, because the loop still visits every position. Skipping the multiply saves the multiply, but you still pay for the loop step and the "is this a zero?" branch on every single element.
In our fit the cycles come out to about 112,876 + 8.0 x (number of non-zeros). That fixed part, 112,876, is essentially the whole dense cost. So even at 90% sparsity, approach A takes about 116,000 cycles, which is actually a hair more than just doing all the work densely (about 112,900). Measured against its own full-density point it looks like a 1.25 times win, but against the fastest dense kernel it is no win at all. The slope of cycles against sparsity, how steeply cycles fall as you remove weights, where 1.0 would mean perfectly proportional, is just 0.225. At 90% sparsity that is a real speedup of essentially nothing, against about 3.6 times for a GPU on the same sparsity.
Walking only the non-zeros makes it nearly free
The fix is approach B: store the weights in a compressed format (compressed sparse column, or CSC, which keeps only the non-zeros and their positions) and write the inner loop to run exactly once per non-zero. The zeros are never visited at all.
Now the cycles track the non-zeros almost perfectly: 4,083 + 39.0 x (number of non-zeros). The per-non-zero cost here (39 cycles) is actually higher than approach A's (8 cycles), but the fixed overhead nearly vanishes (4,083 instead of 112,876), and that is what makes the line fall. The slope is 0.975 out of an ideal 1.0. At 90% sparsity approach B runs about 8 times faster than the same compressed kernel does on a fully dense grid. That 8 times is measured against approach B's own dense point, which is the fair way to line it up against a GPU (whose best on the same unstructured sparsity is about 3.6 times, measured the same self-relative way). Against the single fastest dense kernel we have, which is cheaper than B's own dense point, approach B at 90% is about 5.6 times faster. Either way the savings start immediately and stay close to linear, while the GPU curve barely moves until the model is very sparse.
The slider below shows both kernels at once. Drag it and watch them diverge: approach A stays flat, approach B falls with the zeros.
At 75% sparsity, approach B runs 3.8x faster than it does on a dense grid. Approach A barely moves. Above about 32% sparsity, walking the non-zeros beats doing all the work.
Drag through the five measured points. Approach B (compressed) falls almost linearly with the zeros. Approach A (skip-in-a-loop) stays near the dense cost because it still visits every position.
Walking the non-zeros is not free, and the reason matters if you go build this. It costs more per non-zero (39 cycles) than a plain dense multiply does per element (about 27.5 cycles), because you have to look up where each non-zero lives. So approach B only wins once the model is more than about 32% sparse. Below that, the bookkeeping costs more than it saves. Most pruned models are far more than 32% sparse, so this rarely bites, but the crossover is real.
Structured sparsity buys nothing on one core
There is a popular trick on GPUs called structured sparsity, usually 2:4 (in every group of four weights, exactly two are zero). GPUs like it because the regular pattern is easy for their hardware to accelerate. The problem is that forcing a model into that rigid pattern hurts its accuracy: in the literature, strict 2:4 pruning dropped one hard reasoning score from 75.6 to 6.7.
On a single core of the wafer that trade is pointless, because the pattern makes no difference to the cycle count. We compared unstructured sparsity, 2:4, and a third, more balanced pattern (constant fan-in, where every output gets the same number of non-zero inputs), all at the same 50% density. The per-non-zero cost was identical to within 0.02% (41.00 versus 40.99 cycles per non-zero). Our kernel walks the non-zeros and simply cannot tell how they are arranged. On one core, then, structure buys no speed, so there is no reason to accept its accuracy loss. The one place structure might still earn its keep is across many cores, where it could balance the work evenly, which is a case we did not test (more on that next).
Spreading the work across a thousand cores appears to stall on communication
Everything above lives on a single PE at a small size. Real models are far too big for one core, so the interesting question is what happens when you tile the weights across a mesh: a square grid of PEs that each hold a piece and cooperate. Each PE does its slice of the multiply, and then the partial answers have to be combined, which means the cores broadcast and reduce numbers across the grid (a collective).
We ran a fixed problem across meshes of 64, 256, and 1024 PEs and measured how the cycles fell. This is called strong scaling: same problem, more cores, hopefully proportionally faster. It was not proportional. Going from 64 to 1024 cores (16 times as many) made it only about 1.9 times faster, not 16. Expressed as a scaling exponent (how close to proportional the speedup is, where 1.0 is perfect), we measured about a quarter, 0.23 to 0.30 across the sizes we ran.
16x more cores gives only 1.91x fewer cycles, not the ideal 16x. The tick on each bar marks where perfect scaling would land.
Strong scaling (same problem, more cores). Measured cycles fall far short of the ideal 1/cores line because the cost of cores broadcasting and reducing across the grid grows with the grid.
The likely reason is that as you add cores, each core's share of the math shrinks, but the cost of all those cores broadcasting and reducing across the grid grows with the size of the grid. Past a very early point, the chip is no longer waiting on math. It seems to be waiting on cores talking to each other. We want to be careful about that word "seems": our timers measure the whole pipeline end to end, not the communication in isolation. So "communication-bound" is the reading most consistent with the numbers, not a separately measured fact. What the data show for certain is that the scaling is far from ideal.
We also hit a hard wall from the same 48 KB memory that made single-core sparsity work. At the largest size we tried, each core's tile of weights (64 by 64 floats, 16 KB) plus its communication buffers no longer fit in the core, and the program refused to compile. We did not separately measure how much of that was the buffers, so a leaner kernel might push the wall a little further. The direction is the point, though. The same small on-core memory that makes weights free to reach also forces a big model to spread across more cores, and that is where the communication cost takes over.
Where this holds and where it does not
These are cycle-accurate simulator measurements, not wall-clock time on a physical wafer. Our inner loop is plain and not hand-optimized. So every claim here is about slopes and ratios, not raw speed against a real GPU. The single-core sparsity results are at a small size (a 64 by 64 grid), so we report them as trends, not as guarantees that hold unchanged at full scale. The pattern-independence result was measured directly at 50% density; we infer higher densities from the same cost model rather than measuring each one. And the wafer's roofline advantage, the claim that this operation can run from on-chip compute instead of starving, is an argument from how the chip is built. We did not measure the wafer's own crossover point.
Takeaways
- On a wafer-scale chip, unstructured sparsity is close to free, but the kernel has to walk the non-zeros (approach B), not skip them inside a full loop (approach A). The chip makes the good kernel possible; it does not fix the naive one, which stays about as slow as dense.
- On a single core, structured sparsity like 2:4 buys no cycles, so its accuracy loss is not worth paying there. Across many cores it might still help balance the work, which we did not test.
- The single-core win does not automatically become a whole-chip win. Once you tile across many cores, the collective communication, not the math, looks like the bottleneck, and small per-core memory forces exactly the wide layouts where that bottleneck bites.
The paper (with the full method, tables, and honest limitations) can be downloaded below. The kernels, the input generator, the GPU roofline scripts, and the raw cycle counts are available on request.