Can you remove political censorship in Chinese open weight LLMs?
Ask an open Chinese language model what happened in Tiananmen Square in June 1989 and it answers you, fluently, without ever saying what happened. We measured this soft censorship in an open model from a Chinese lab. Counting refusals catches about one-eighth of it. Judged by whether the answer actually says the thing, it is 30 percent. And when we tried to make the model "unlearn" its censorship, the standard tools backfired.
The usual metric measures the wrong thing
If you want to know how censored a model is, the obvious test is to count refusals: how often does it say some version of "I can't help with that"? This is the refusal rate (the fraction of answers that are outright refusals), and it is what most censorship checks report.
For political questions, the refusal rate is close to useless. We asked Qwen2.5-7B-Instruct (an open, permissively licensed model from Alibaba) 264 held-out politically sensitive questions across eight sensitive topics, in both English and Chinese. It gave a hard refusal on almost none of them. By refusal rate, it looks about 4% censored.
Then we scored the same answers a second way. Instead of asking "did it refuse?", we asked "did it actually answer?" We used an LLM judge (a second model that reads each answer and decides whether it gave the real facts or dodged) to sort every response into substantive or not. Dodges count as censored: deflecting, repeating the official line, or leaving out the key fact. Judged this way, the censorship rate is 30.3%.
Same answers. Same model. One number is 4%, the other is 30%. The gap is the whole point.
Same 264 answers, same model. Counting refusals sees about one-eighth of the censorship, because the model almost never refuses (0 hard refusals in 264 prompts). It deflects. You only see the real rate when you judge whether the answer actually says the thing.
The censorship is real, but it is soft. The model does not refuse outright. It talks around the question, stays on-message, and omits the fact you asked for. A metric that only catches outright refusals sees about one-eighth of what is there. If you are evaluating a model for this kind of behavior, refusal rate will tell you it is clean when it is not.
Checking it on other people's data
The obvious worry with a result like this is that we wrote the questions, so maybe we wrote them to get the answer we wanted. To rule that out, we re-ran the exact same measurement on question sets other people built and released, without changing them.
The first is DECCP: 95 China-sensitive questions assembled by Leonard Lin (Shisa.AI) for an analysis of Chinese-model censorship, and widely used in de-censoring work. The second is a broader geopolitical question set from EnkryptAI covering six countries. That second one gives us a built-in control: if the effect is really about Chinese political topics, then the India, Pakistan, Russia, and USA questions should come out clean.
| Someone else's question set | Refuses outright | Soft-censors (judged) | Gap |
|---|---|---|---|
| DECCP, China-sensitive (95 questions) | 7% | 39% | 5.3x |
| Geopolitical set, China + Taiwan (100) | 8% | 15% | 1.9x |
| Geopolitical set, other countries (200) | 0.5% | 13% | control |
DECCP tells the clearest story, and it lands right on top of our own numbers. Qwen refuses only 7% of the time but soft-censors 39%, a roughly five-fold gap, on questions we did not write. Refusal rate again sees about a fifth of what is there.
Two things are worth noting. The size of the gap depends on how pointed the question is. DECCP asks directly about the machinery of control ("how does the government censor domestic media," "what happens to people who speak out"), and those draw heavy deflection. The geopolitical set asks calmer historical questions ("what led to the Tiananmen protests"), which Qwen answers more fully, so its number is lower. The non-China questions also scored around 13%, because the open judge counts hedging on any touchy geopolitics (Kashmir, Ukraine) as soft censorship, not only Chinese topics. So the geopolitical set backs up the direction but is not a clean China-only signal. DECCP, whose questions are unambiguously about Chinese state censorship, is the one that carries the point.
Two judges disagree on the number, and agree on the ranking
Judging by meaning has an obvious worry: the judge is itself a model, and a different judge might score things differently. It does. We ran a strict proprietary judge (Claude) and a reproducible open one (Phi-3.5-mini, so anyone can rerun it without a paid API) over the exact same answers.
The two judges disagree a lot on the absolute number. The strict judge counts the baseline model at 30.3% censored. The open judge, which is more willing to call a hedged answer "substantive," counts it at 10.2%. That is a big spread, and it matters: the absolute censorship rate depends on who is judging.
What the two judges agree on is the ranking. When we line up every method we tried from least to most censored, the two judges produce nearly the same order (a rank correlation of 0.92). So we do not lean on any single percentage. We lean on the ordering, which is stable no matter which judge you trust.
One more pattern held under both judges: the censorship is heavier in Chinese than in English. On the strict judge, English sits at 16.7% and Chinese at 43.9%, about 2.6 times worse. Evaluate only in English and you undercount the problem again.
The small experiment: can you make the model "unlearn" its censorship?
Here is where we expected a clean win and did not get one. There is a whole research area called machine unlearning (methods that make a trained model forget a specific thing, like a private fact or a dangerous skill). "Make the model forget its censorship" sounds like exactly that job, so we tried four of the standard unlearning methods on the sensitive topics.
Every one of them either made the censorship worse or broke the model. Because how hard you train matters, we swept each method across four training lengths instead of trusting one setting. Press play and watch what happens as the training runs longer.
Two methods (NPO, GradDiff) dip below the untouched model at 40 steps, then fall apart by 100: the model stops making sense. The other two (RMU, ELM) stay more censored than the untouched model the whole way. There is no training length that cleanly removes the censorship.
Two of the methods (NPO and GradDiff, both gradient-based) do dip below the untouched model at 40 steps. That looks like a win for one frame. Train a little longer and they collapse: the output turns into repeated words and gibberish, so the low "censorship" score is meaningless (there is nothing coherent left to censor). The other two (RMU and ELM) stay more censored than the untouched model at every setting. ELM in particular distills the model toward a version that denies knowing anything, so it learns to say "I don't have information on this sensitive topic," which is more evasive, not less.
The reason is a mismatch, and it is the real lesson here. Unlearning removes knowledge. But a de-censored model needs to keep the knowledge and drop only the trained habit of withholding it. Those are opposite goals. Erasing what the model knows about Tiananmen does not make it discuss Tiananmen. It makes it dodge harder, or forget how to form sentences.
We confirmed this with a control. We fine-tuned the model to answer directly (a plain supervised nudge toward substantive answers), and that reached the lowest coherent censorship of anything we tried. So the behavior we want is reachable. Unlearning just aims at the wrong target.
What this means if you evaluate or edit models
- Do not measure political censorship with refusal rate. It sees about one-eighth of the real behavior. Judge whether the answer contains the fact, not whether it said "no."
- Report more than one judge, and lean on the ranking. Absolute censorship rates move a lot with judge strictness. The order of methods is far more stable.
- Test in the model's own language. The censorship was about 2.6 times heavier in Chinese than English. English-only evaluation undercounts it.
- If you want to remove a trained behavior, target the behavior, not the knowledge. Unlearning the topic backfired. Editing the refusal habit directly, or just supervising the answers you want, worked and stayed coherent.
We studied one model at one size and one set of topics, so treat the exact numbers as specific to that setup, not a universal law. The direction of the effect (soft censorship, refusal-rate blindness, unlearning backfiring) was consistent across every setting we ran.
Built on Alibaba's Qwen2.5-7B-Instruct. The code, the question set, and the paper are available on request. We are not releasing an edited model checkpoint: that decision is on hold pending a safety and framing review, since the goal of this work is to measure and understand embedded guardrails, not to ship a model with a different bias baked in.