Two Kinds of Unreadable: what obfuscating code actually costs a language model
GLOSSOPETRAE, a self-published research project, claimed that making code unreadable, by swapping its syntax for alien glyphs, made a model solve hard problems better. We tried to reproduce it across five models and did not find the benefit on any of them. The one thing we did find is worth knowing: a model reads obfuscated code about as well as plain code, it just cannot write its answer back in the strange form. And we could not test the model the claim was made on, because today's Claude Opus refuses the obfuscated input outright.
Quick summary. GLOSSOPETRAE, a self-published project, reported that rendering code in unreadable glyphs improved a frontier model's score on hard programming problems (57% to 93% on Claude Opus), on the theory that unfamiliar code blocks pattern-matching and forces real reasoning. Three findings. First, we could not run the test on Opus itself: its current version refuses the obfuscated input on every problem we tried. Second, on the models we could test, the benefit never appeared. Third, obfuscation does not help, and naively it even looks like it hurts, but that apparent harm is mostly an artifact of how the answer is scored. We split a question the usual setup runs together: can a model read obfuscated code (let it answer in ordinary Python), versus write the answer back in glyphs. Reading glyph-obfuscated syntax is cheap, the drop is small and not statistically distinguishable from zero; being forced to write the glyphs is what collapses the score, 80 to 95% of the penalty. One honest twist: obfuscating the names in the code, rather than the syntax, is a different thing, and does impose a real comprehension cost of roughly 9 to 16 points.
The claim we set out to explain
The idea is appealing: if code looks unfamiliar, a model cannot lean on a memorized or superficially-cued answer, so it has to compute the result. The original report showed a large jump on hard coding problems and gave no mechanism. We set out to supply one, expecting either to confirm the effect and explain it, or to find it was fragile. Reported reasoning gains often shrink once the evaluation is tightened, so a careful replication was the right first step.
What we did
We reimplemented the original glyph transform for Python, a reversible mapping verified to round-trip exactly on hundreds of programs, and ran the same underlying problems through it. We tested two open models (Qwen2.5-Coder-7B, Llama-3.1-8B) and three frontier models (Claude Sonnet, GPT-5, Claude Opus) on 132 held-out problems from HumanEval, MBPP, and LiveCodeBench, scored by unit tests.
The design choice that mattered was splitting reading from writing:
- Reading: show the model the glyph-encoded problem, but let it answer in ordinary Python.
- Writing: show the same glyph problem and require the answer in glyphs too, which is the original protocol. The glyph answer is decoded before the tests run.
The first surprise: the original model will not look at it
Before any of the analysis below, one thing happened immediately. The original result was on Claude Opus, so we tried it there first. The current Opus refused the obfuscated input on all 12 problems we sampled, while solving the readable version on all 12. The refusal is triggered by the unusual input itself, not by anything harmful in the task. So the specific claim is not just unreplicated, it is currently untestable on the model that made it: the guardrails now reject the exact kind of input the method depends on. That is a small n, so we treat it as an observation rather than a measured rate, but it is a preview of a larger reproducibility problem for any research that feeds models deliberately strange inputs. Everything below is on the four models that do engage.
Reading is cheap. Writing is the tax.
Use the buttons to switch the answer format. When each model only has to read the glyphs, it lands near its readable baseline; when it has to write them back too, the score falls off a cliff.
Allowed to read the glyphs but answer in plain Python, every model lands right near its readable baseline. Comprehension was never the problem.
Here are the same numbers with 95% intervals from a paired bootstrap over problems.
| Model | Read cost (readable minus reading) | Write cost (reading minus writing) |
|---|---|---|
| Qwen2.5-Coder-7B | +0.015 [-0.05, +0.08] | +0.174 [+0.11, +0.25] |
| Llama-3.1-8B | +0.030 [-0.03, +0.09] | +0.121 [+0.06, +0.18] |
| Sonnet | +0.038 [-0.03, +0.11] | +0.652 [+0.57, +0.74] |
The read-cost intervals all include zero: at this sample size we cannot resolve a comprehension drop from reading glyph-obfuscated syntax. The write-cost intervals all exclude zero and are large. Generation accounts for 80 to 95% of the total penalty. The model understood the problem the whole time. It could not hand the answer back in the unfamiliar format. GPT-5 behaves the same on reading (glyph-in 0.86 versus readable 0.84 at the full sample) but uses provider-default sampling, so we report it separately rather than pooling it.
We stop short of saying reading is "provably free." The intervals are wide (upper bounds near 8 to 11 points), so the honest statement is "no resolved difference," not "identical."
The twist: two kinds of unreadable
Everything above scrambles the syntax and keeps the variable names readable, which is what the original tool actually does. Scrambling the names is a different matter. Reading code whose identifiers are replaced with glyphs costs Qwen +15.9 points ([+9.8, +22.7]) and Llama +10.6 ([+4.5, +17.4]); replacing names with misleading ones costs +10.6 ([+3.8, +17.4]) and +9.1 ([+3.0, +15.9]). All four intervals exclude zero. So opacity of the names a model reasons with is a genuine comprehension cost, and it is separate from the syntactic surface, which is not. "Unreadable" is not one thing.
Use the buttons to compare the two kinds of obfuscation. Each bar is the accuracy a model loses just from reading, with its 95% interval. Scrambling syntax leaves every interval crossing zero (no resolved cost); scrambling names pushes them all to the right of it (a real cost).
Scrambling the syntax barely moves reading: the point estimates are small and every 95% interval crosses zero, so no comprehension cost is resolved. The whole penalty was in writing, not reading.
The benefit did not appear, and one honest disclosure
No model solved obfuscated problems better than plain ones; the pre-registered "does obfuscation help" gate fails. Our study also pre-registered a secondary hypothesis, that obfuscation helps most when the names are misleading. Computed the naive way, on the glyph-output cells, that interaction reaches significance (pooled +0.057, interval excluding zero), though at 132 problems it is short of the 300 per cell the pre-registration called for. We disclose this because it was our registered primary, but it is computed on exactly the collapsed glyph-output cells that the decomposition above shows are an output-channel artifact: a floored cell that started lower falls less far, which reads as a positive interaction without any reasoning being suppressed. Recomputed on the clean reading-only cells, it is small and inconsistent across models. So we do not read it as evidence of shortcut suppression.
What we are careful about
- Conceptual replication. The source ships no fixed problem set, so we reimplemented its transform rather than rerunning its pipeline. Our glyph vocabulary differs, and our problems are public benchmarks.
- We could not run the source's model. Current Opus refuses the input, so we cannot confirm or refute the original number on it.
- No equivalence claim. At 132 problems the read-cost intervals are wide; "no resolved difference" is not "identical."
- Output channel, not proven generation. A correct answer with a malformed glyph surface fails the decode gate and scores zero. We did not separately re-score to split "malformed but correct" from "genuinely wrong," so we credit the syntactic penalty to the output channel, not to model generation specifically.
Takeaways
- Measure comprehension apart from generation. If you study transformed or obfuscated inputs, let the model answer in the untransformed form to test understanding, and only require the transformed output when writing it is the thing you care about. Otherwise an output-format effect masquerades as a change in reasoning.
- "Unreadable" splits in two. Obfuscating syntax is an output-channel cost; obfuscating names is a real comprehension cost. Neither improved reasoning here.
- Guardrails are becoming a reproducibility variable. The model behind the original claim now refuses the input, which increasingly bounds what these studies can even measure. Code and data are available on request.