#339·toon

Root string starting with U+FEFF is emitted unquoted and loses its first character on decode

Author: shreyasbhat0Created Sep 15, 2026Updated Sep 15, 2026

Summary

When the root value is a string whose first character is U+FEFF, the encoder emits it unquoted. The decoder then treats that leading U+FEFF as a byte-order mark and strips it (SPEC §12), so decode(encode(x)) no longer equals x. In the worst case the whole value disappears.

Verified on main (f151a5d) by running the source directly, and on the published 4.1.1 package.

Reproduction

typescript
import { encode, decode } from '@toon-format/toon'

decode(encode('8'))    // 8        (number, expected the string "8")
decode(encode('abc'))  // "abc"    (expected "abc")
decode(encode(''))     // {}       (expected "")

Encoded output for each is the raw string, e.g. encode('8') returns "8" with no quotes.

Only the root primitive position is affected. Keys starting with U+FEFF are already quoted by encodeKey (non-ASCII fails the unquoted-key pattern), and inline array values, tabular cells, list items, and object field values never start the document:

typescript
decode(encode({ 'k': 1 }))  // { 'k': 1 }  ok
decode(encode(['x']))       // ['x']       ok
decode(encode({ k: 'x' }))  // { k: 'x' }  ok

Why this is a bug

  • SPEC §12: "A U+FEFF anywhere else is content. Encoders MUST NOT emit one." A document that begins with U+FEFF has, by the spec's own definition, been emitted with a BOM.
  • SPEC §2: decode(encode(x)) must equal x under JSON-model equality.
  • The encoder already knows about this trap: RawString in packages/toon/src/encode/raw-string.ts rejects raw strings that would read as a comment line after a leading BOM is removed. The normal string path in isSafeUnquoted (packages/toon/src/shared/validation.ts) has no equivalent check.

Suggested fix

Two options, either is a one-liner:

  1. In encodeJsonValue (packages/toon/src/encode/encoders.ts), when the root value is a string that starts with U+FEFF, force the quoted form. This keeps the change scoped to the only position where it matters.
  2. Add value.startsWith(BYTE_ORDER_MARK) to isSafeUnquoted. Simpler, but it also quotes such strings in positions where it isn't needed.

Option 1 seems closer to the spirit of "quote only when required" (§7.2). Happy to open a PR for whichever you prefer.

It may also be worth a note in §7.2 of the spec, since the current quoting rules do not mention U+FEFF and any implementation that follows them literally will have the same gap. The Rust port has the identical bug and a fix has been requested there.