lib: lexical conflicts can change keyword tokenization

AI Policy

• I have read the AI Policy and this issue complies with it.

Problem

tree-sitter can tokenize the same higher-precedence keyword differently depending on whether it is successfully promoted into the keyword lexer. If an otherwise unrelated lexical conflict prevents that promotion, the keyword can suddenly split a longer 'word' match that would otherwise be consumed as a single identifier.

Repro case and reasoning provided in PR. PR may be incorrect and/or incomplete.

Opened to discuss issue presented in #5614. This appears to be similar to #5607.

Steps to reproduce

Provided in #5614

Expected behavior

Provided in #5614

Tree-sitter version (tree-sitter --version)

tree-sitter 0.27.0

Operating system/version

Ubuntu Linux 24.04