#4111·prism

Use the token overlay helper for the remaining structure-merging code

Author: LeaVerouCreated Sep 15, 2026Updated Sep 17, 2026

"Merge one tokenized structure with another over the same text" is reimplemented or worked around in many places. The $inner selectors PR adds the primitives (splitTokenStream() and insertTokens() in src/util/token-stream.js) and uses them for meta-languages. These are the remaining instances found by scanning the repo, roughly in order of payoff:

DOM / string level today, should be token level

  • keep-markup (src/plugins/keep-markup/keep-markup.js:62-168): records text offsets of existing elements, then re-wraps them after highlighting with Range.extractContents(), which cuts through token spans. → wrap ranges on after-tokenize with splitTokenStream. Its node-identity test (tests/plugins/keep-markup/test.js) must keep passing. normalize-whitespace's special branch for keep-markup (normalize-whitespace.js:213-223) goes away with it.
  • command-line (src/plugins/command-line/command-line.js:60-131): stashes output lines and strips continuation prefixes on env.code, re-inserts on the highlighted HTML string split by \n (breaks any multi-line token). → the same shape as diff: select the command lines.
  • markdown code blocks after autoload (src/languages/markdown.js:298-337): re-highlights via element.innerHTML = Prism.highlight(element.textContent).
  • inline-color (src/plugins/inline-color/inline-color.js:92): recovers text by stripping tags from HTML.

Per-line / per-node grammar workarounds

  • xml-doc /// lines (src/languages/xml-doc.js:11-26), javadoclike / jsdoc * prefixes (javadoclike.js:5-15, jsdoc.js, javascript.js:93-98): every pattern is /m and anchored per line because the prefixes are never removed, so nothing can match across lines. → select the prefix-stripped lines, tokenize as one document (exactly what diff does now).
  • markdown blockquotes (markdown.js:49-52): > prefix per line, the remainder is not re-parsed as markdown.
  • shell-session (src/languages/shell-session.js:58-71): each command highlighted as bash separately; output is one opaque token.

Per-string overlays that miss matches across token boundaries

  • tokenizeStrings() (src/shared/tokenize-strings.js) and its users show-invisibles, autolinker, data-uri-highlight: a URL or data URI split by a token boundary is never matched. → :text-style selection of all strings at any depth, highlighted as one whole.

Same algorithm twice

  • jsx and xquery walkTokens (src/languages/jsx.js:24-109, src/languages/xquery.js:8-105): concatenate selected nodes into plain-text tokens. → select + an empty inner language.

Ordering hazards that exist only because these overlays don't compose: command-line.js:82-84, line-highlight.js:348-370.