EncodingVisualizer: data-* attributes built from token text are not HTML-escaped (XSS gap left by #1937)

Summary

EncodingVisualizer.consecutive_chars_to_html (bindings/python/py_src/tokenizers/tools/visualizer.py) builds data-* HTML attributes directly from raw token text without escaping, so a token whose text contains a " can break out of the attribute and inject arbitrary HTML/JS into the visualizer's output. This is a leftover gap in #1937, which fixed HTML-escaping for the visible span text but not for the data-stoken / data-stok attribute values built from the same untrusted token strings.

Where

Two spots in consecutive_chars_to_html:

# special-token branch, unquoted attribute value
return f'<span class="special-token" data-stoken={stoken}></span>'

# regular branch, built from encoding.tokens[...], not escaped
data_items["stok"] = encoding.tokens[first.token_ix]
...
for key, val in data_items.items():
    data += f' data-{key}="{val}"'
span_text = html.escape(span_text)
return f"<span {css} {data} >{span_text}</span>"

html.escape is only applied to span_text, the visible text. data, which is built from the same token strings, is inserted into the HTML output as-is.

Reproduction

from tokenizers import Tokenizer
from tokenizers.models import WordLevel
from tokenizers.pre_tokenizers import WhitespaceSplit
from tokenizers.tools import EncodingVisualizer

payload = '"><script>alert(document.domain)</script>'
adversarial_token = f"unk{payload}"

vocab = {"[UNK]": 0, "hello": 1, adversarial_token: 2}
tokenizer = Tokenizer(WordLevel(vocab, unk_token="[UNK]"))
tokenizer.pre_tokenizer = WhitespaceSplit()

text = f"hello {adversarial_token}"
visualizer = EncodingVisualizer(tokenizer, default_to_notebook=False)
html_output = visualizer(text)
print(html_output)

I ran this against the current main branch source of visualizer.py (imported directly, so it exercises the actual unmodified upstream logic) with the real installed tokenizers Rust core. Output includes:

<span class="token odd-token special-token"  data-stok="unk"><script>alert(document.domain)</script>" >unk&quot;&gt;&lt;script&gt;alert(document.domain)&lt;/script&gt;</span>

The data-stok="..." attribute terminates early at the embedded ", the following > closes the <span> tag early, and <script>alert(document.domain)</script> becomes a live, executable tag in the DOM. Note the visible text at the end of the span is correctly escaped (<script>) — only the data-stok attribute path is unescaped, confirming this is specifically the gap left by #1937 rather than a regression of it.

The token only needs to match EncodingVisualizer.unk_token_regex (case-insensitive search for "unk", matching common words like "unknown", "chunk", "drunk") to be routed into this code path via the data_items["stok"] = ... branch; the data-stoken branch (for actual special tokens) is reachable even more directly, from any special token whose text isn't escaped.

Impact

Any consumer that renders EncodingVisualizer output as HTML (Jupyter/JupyterLab, an exported HTML report, a web dashboard embedding this output) executes attacker-controlled script if the token text is influenced by an untrusted source, for example a vocabulary loaded from a downloaded tokenizer file, or user input that ends up as raw token text. This is the same class of issue #1937 was written to close, just via a different attribute.

Suggested fix

Escape all data-* attribute values the same way span_text already is, e.g.:

for key, val in data_items.items():
    data += f' data-{key}="{html.escape(str(val))}"'

and for the special-token branch:

return f'<span class="special-token" data-stoken="{html.escape(stoken)}"></span>'

(also quoting the attribute, since it's currently emitted unquoted).

Environment

Reproduced against tokenizers 0.22.2 (installed via pip) using the current main branch source of visualizer.py. Python 3.12.10, Windows.