Use the token overlay helper for the remaining structure-merging code
Author: LeaVerouCreated Sep 15, 2026Updated Sep 17, 2026
"Merge one tokenized structure with another over the same text" is reimplemented or worked around in many places. The $inner selectors PR adds the primitives (splitTokenStream() and insertTokens() in src/util/token-stream.js) and uses them for meta-languages. These are the remaining instances found by scanning the repo, roughly in order of payoff:
DOM / string level today, should be token level
- keep-markup (
src/plugins/keep-markup/keep-markup.js:62-168): records text offsets of existing elements, then re-wraps them after highlighting withRange.extractContents(), which cuts through token spans. → wrap ranges onafter-tokenizewithsplitTokenStream. Its node-identity test (tests/plugins/keep-markup/test.js) must keep passing. normalize-whitespace's special branch for keep-markup (normalize-whitespace.js:213-223) goes away with it. - command-line (
src/plugins/command-line/command-line.js:60-131): stashes output lines and strips continuation prefixes onenv.code, re-inserts on the highlighted HTML string split by\n(breaks any multi-line token). → the same shape asdiff: select the command lines. - markdown code blocks after autoload (
src/languages/markdown.js:298-337): re-highlights viaelement.innerHTML = Prism.highlight(element.textContent). - inline-color (
src/plugins/inline-color/inline-color.js:92): recovers text by stripping tags from HTML.
Per-line / per-node grammar workarounds
- xml-doc
///lines (src/languages/xml-doc.js:11-26), javadoclike / jsdoc*prefixes (javadoclike.js:5-15,jsdoc.js,javascript.js:93-98): every pattern is/mand anchored per line because the prefixes are never removed, so nothing can match across lines. → select the prefix-stripped lines, tokenize as one document (exactly whatdiffdoes now). - markdown blockquotes (
markdown.js:49-52):>prefix per line, the remainder is not re-parsed as markdown. - shell-session (
src/languages/shell-session.js:58-71): each command highlighted as bash separately; output is one opaque token.
Per-string overlays that miss matches across token boundaries
tokenizeStrings()(src/shared/tokenize-strings.js) and its users show-invisibles, autolinker, data-uri-highlight: a URL or data URI split by a token boundary is never matched. →:text-style selection of all strings at any depth, highlighted as one whole.
Same algorithm twice
- jsx and xquery
walkTokens(src/languages/jsx.js:24-109,src/languages/xquery.js:8-105): concatenate selected nodes intoplain-texttokens. → select + an empty inner language.
Ordering hazards that exist only because these overlays don't compose: command-line.js:82-84, line-highlight.js:348-370.
Source: PrismJS/prism