Error recovery regression in 0.27.0: A single malformed nested object collapses the whole document into an `ERROR` root

AI Policy

• I have read the AI Policy and this issue complies with it.

Problem

In 0.27.0, nested malformed input (e.g., JSON {"a": {"b" "c"}}) parses to an ERROR against the root document. In 0.26.10, the same input gives document > object > pair > object > ERROR; that is, confined to the inner object. It looks like nesting is the trigger: Flat {"b" "c"} keeps a document root (though 0.27.0 loses the object node), as does [1, 2 3, 4].

Secondary to this, recovery resumes lexing at a shifted quote boundary, so }, , and : get absorbed into string tokens and every byte range thereafter is misaligned.

A bisect shows 869638f6 as the first bad commit, which itself is a fix; so it probably shouldn't be idly reverted. I've confirmed this with git revert --no-commit 869638f6 against 0.27.0, which restores the same 0.26.12 sexpr.

AI disclosure: I used Claude Code to bisect this and build the reproduction; I verified the bisect result and the revert myself.

Steps to reproduce

With Rust:

[dependencies]
tree-sitter = "=0.27.0"    # or "=0.26.12"
tree-sitter-json = "0.24.8"

use tree_sitter::Parser;

fn main() {
    let src = r#"{"a":{"b" "c"}}"#;
    let mut parser = Parser::new();
    parser.set_language(&tree_sitter_json::LANGUAGE.into()).unwrap();
    let tree = parser.parse(src, None).unwrap();
    println!("root: {}", tree.root_node().kind());
    println!("sexp: {}", tree.root_node().to_sexp());
}

Expected behavior

As an error_corpus test case, the 0.26.12 behaviour was:

==================================
Malformed object nested in a pair
==================================

{"a":{"b" "c"}}

---

(document
  (object
    (pair
      key: (string (string_content))
      value: (object
        (ERROR (string (string_content)) (string (string_content)))))))

That is, the error is confined to the inner object:

root: document
sexp: (document (object (pair key: (string (string_content)) value: (object (ERROR (string (string_content)) (string (string_content)))))))

In 0.27.0, the entire document becomes an ERROR:

root: ERROR
sexp: (ERROR (string (string_content)) (ERROR (string (string_content)) (UNEXPECTED 'c')) (string_content))

Knock-on effect: Token boundaries shift

Beyond the tree shape, recovery in 0.27.0 resumes lexing at a shifted quote boundary, so the structural } and , get absorbed into a string token. For {"one":{"bar" "baz"},"two":"bar"}:

ERROR [0..33] "{\"one\":{\"bar\" \"baz\"},\"two\":\"bar\"}"
  { [0..1] "{"
  string [1..6] "\"one\""
  : [6..7] ":"
  { [7..8] "{"
  ERROR [8..31] "\"bar\" \"baz\"},\"two\":\"bar"
    string [8..13] "\"bar\""
    " [14..15] "\""
    ERROR [15..18] "baz"
    string [18..22] "\"},\""        <-- `}` and `,` swallowed into a string
      string_content [19..21] "},"
    ERROR [22..25] "two"
    string [25..28] "\":\""         <-- `:` swallowed into a string
    ERROR [28..31] "bar"
  " [31..32] "\""
  string_content [32..33] "}"

Every token from byte 14 onwards is misaligned against the source.

Tree-sitter version (tree-sitter --version)

tree-sitter 0.27.0

Operating system/version

Linux 6.18.34 x86_64