[Observation] Punctuation loss after extended generation in uncensored QWEN / DeepSeek models — absent in Gemma 4 variants

Summary

After testing multiple uncensored / abliterated releases of Chinese-origin open-source models, I observed a consistent issue: punctuation marks gradually disappear after the model generates a certain number of tokens. This issue is reproducible across several model families but is notably absent in uncensored Gemma 4 variants, even when published by the same author (who likely used the same uncensoring tooling).

Affected Models (punctuation loss observed)

Model	Variant
QWEN 3.6	uncensored / abliterated
QWEN 3.8	uncensored / abliterated
QWEN 3.8 NEXT	uncensored / abliterated
DeepSeek V4 FLASH	uncensored / abliterated

Unaffected Models (no punctuation loss)

Model	Variant
Gemma 4 12B	uncensored / abliterated
Gemma 4 26B	uncensored / abliterated
Gemma 4 31B	uncensored / abliterated

Key Observation

• The punctuation loss does not occur from the very first token. The model starts generating normally, and punctuation gradually fades out after a certain generation length.
• This pattern suggests a state drift or distribution shift during long-sequence generation, rather than a fundamental tokenizer defect or a system-prompt issue.
• The same publisher released uncensored versions of both QWEN and Gemma 4 models, and the discrepancy persists. This implies the same uncensoring tool/pipeline was likely used, yet the outcome differs significantly across model families.

Hypotheses for the Root Cause

Based on the observed pattern, the following hypotheses are proposed for discussion:

1. Tokenizer differences × data preprocessing mismatch (most likely)

• QWEN / DeepSeek use BPE-based tokenizers (QWEN uses a tiktoken-based vocabulary of ~152K; DeepSeek uses its own BPE). In these tokenizers, Chinese punctuation marks (e.g., 。, ，, ；) are often tightly coupled with preceding text, encoded as merged tokens or split context-dependently.
• Gemma 4 uses SentencePiece (~256K vocabulary), where punctuation marks are typically encoded as independent, high-frequency, positionally stable tokens.

If the uncensoring pipeline includes any text preprocessing (cleaning, newline normalization, Unicode normalization) that is not tokenizer-aware, it could subtly corrupt the punctuation token distribution for BPE-based models while leaving SentencePiece-based models unaffected.

Over long generations, the accumulated distribution shift causes punctuation token probabilities to be progressively suppressed until they effectively vanish.

This explains why the same tool produces different outcomes: the tool is identical, but the tokenizer it operates on is not.

2. Coupling between safety alignment and format control

Safety alignment (RLHF / DPO / refusal-vector removal) does not merely add a "refuse to answer" switch. During alignment training, models simultaneously learn to:

• Follow instruction formatting
• Maintain structural integrity (punctuation, paragraphs, lists)
• Sustain consistency over long outputs

QWEN and DeepSeek may encode "format control" and "safety compliance" in overlapping representational subspaces. Removing the refusal direction could inadvertently degrade format-maintenance capability, manifesting as gradual punctuation loss in long outputs.

Gemma 4's alignment pipeline (Google's) may encode format control and safety refusal in more decoupled subspaces, so uncensoring only removes the refusal direction without harming formatting.

3. Length distribution bias in uncensoring fine-tune data

Uncensoring fine-tune datasets (e.g., WildChat, Dolphin, various "uncensored" conversation corpora) are typically dominated by short dialogue turns, rarely exceeding a few hundred tokens per response.

• After fine-tuning on such data, the model's "punctuation rhythm" for long-form generation is diluted.
• QWEN / DeepSeek's original Chinese pre-training corpora contain extensive long-form text with stable punctuation patterns, but a sufficiently aggressive fine-tune can partially overwrite this.
• Gemma 4's pre-training is more multilingual and its punctuation generation patterns may be inherently more resilient to fine-tuning perturbation.

The symptom — punctuation intact at first, then fading — is consistent with this: short outputs are still dominated by pre-trained punctuation habits; longer outputs gradually shift toward the fine-tuned distribution.

4. Architectural robustness to activation steering

If the uncensoring method is abliteration / activation steering (removing a refusal direction vector) rather than fine-tuning:

• QWEN uses GQA (Grouped Query Attention); DeepSeek V4 uses MLA (Multi-head Latent Attention). Both produce more compact high-dimensional representations. Removing a direction vector in such spaces has a higher probability of collateral damage to adjacent capabilities (e.g., punctuation generation).
• Gemma 4 uses more standard MHA, which offers greater representational redundancy. The same vector-removal operation is less likely to disrupt neighboring functions.

Likelihood Assessment

Hypothesis	Likelihood	Key Evidence
Tokenizer mismatch + data preprocessing	★★★★★	Same tool, different outcome; BPE vs SentencePiece
Safety-alignment / format-control coupling	★★★★☆	"After N tokens" pattern implies format-maintenance degradation
Fine-tune data length bias	★★★☆☆	Short conversation data cannot sustain long-form punctuation rhythm
Architectural robustness to steering	★★★☆☆	GQA/MLA vs MHA representational compactness

Most Probable Explanation

A combination of Hypothesis 1 and 2: the uncensoring pipeline's data preprocessing subtly corrupts punctuation encoding for BPE tokenizers, while QWEN/DeepSeek's tighter coupling between safety alignment and format control means the uncensoring operation inadvertently weakens long-form punctuation generation. Gemma 4 escapes due to its more robust SentencePiece encoding and more decoupled alignment.

Questions for Discussion

1. Can the publisher confirm whether the uncensoring method was fine-tuning or activation steering / abliteration?
2. Has anyone else reproduced this punctuation-loss behavior on QWEN / DeepSeek uncensored variants?
3. Is there a way to inspect the logit distribution of punctuation tokens during generation to confirm the progressive suppression hypothesis?
4. Could a tokenizer-aware data preprocessing step in the uncensoring pipeline mitigate this issue?

Environment

• Observation based on local inference; specific backend (llama.cpp / vLLM / TGI) may vary.
• Generation parameters kept consistent across all model comparisons.

---

This is an observation report and discussion thread, not a bug report against any specific repository. The goal is to understand the root cause and potentially improve uncensoring pipelines for BPE-tokenizer-based models.