Root string starting with U+FEFF is emitted unquoted and loses its first character on decode
Summary
When the root value is a string whose first character is U+FEFF, the encoder emits it unquoted. The decoder then treats that leading U+FEFF as a byte-order mark and strips it (SPEC §12), so decode(encode(x)) no longer equals x. In the worst case the whole value disappears.
Verified on main (f151a5d) by running the source directly, and on the published 4.1.1 package.
Reproduction
import { encode, decode } from '@toon-format/toon'
decode(encode('8')) // 8 (number, expected the string "8")
decode(encode('abc')) // "abc" (expected "abc")
decode(encode('')) // {} (expected "")Encoded output for each is the raw string, e.g. encode('8') returns "8" with no quotes.
Only the root primitive position is affected. Keys starting with U+FEFF are already quoted by encodeKey (non-ASCII fails the unquoted-key pattern), and inline array values, tabular cells, list items, and object field values never start the document:
decode(encode({ 'k': 1 })) // { 'k': 1 } ok
decode(encode(['x'])) // ['x'] ok
decode(encode({ k: 'x' })) // { k: 'x' } okWhy this is a bug
- SPEC §12: "A U+FEFF anywhere else is content. Encoders MUST NOT emit one." A document that begins with U+FEFF has, by the spec's own definition, been emitted with a BOM.
- SPEC §2:
decode(encode(x))must equalxunder JSON-model equality. - The encoder already knows about this trap:
RawStringinpackages/toon/src/encode/raw-string.tsrejects raw strings that would read as a comment line after a leading BOM is removed. The normal string path inisSafeUnquoted(packages/toon/src/shared/validation.ts) has no equivalent check.
Suggested fix
Two options, either is a one-liner:
- In
encodeJsonValue(packages/toon/src/encode/encoders.ts), when the root value is a string that starts with U+FEFF, force the quoted form. This keeps the change scoped to the only position where it matters. - Add
value.startsWith(BYTE_ORDER_MARK)toisSafeUnquoted. Simpler, but it also quotes such strings in positions where it isn't needed.
Option 1 seems closer to the spirit of "quote only when required" (§7.2). Happy to open a PR for whichever you prefer.
It may also be worth a note in §7.2 of the spec, since the current quoting rules do not mention U+FEFF and any implementation that follows them literally will have the same gap. The Rust port has the identical bug and a fix has been requested there.
Source: toon-format/toon