add `rewind` callback to `TSLexer`

AI Policy

• I have read the AI Policy and this issue complies with it.

Problem

grammars that use whitespace to indicate structure often use the canonical INDENT/DEDENT tokens, but sometimes those are not sufficient. if all tokens are pure zero-width peeking ahead, then the author can probably figure out code to deterministically react to each lookahead as advance marches linearly through the input. and since everything is zero-width the rules get messy to consume what was peeked. returning positive-width tokens would require sprinkling mark_begin (yay #2985) and mark_end into the confusing code. if you're really unlucky (like I was) two of the tokens will want to put the marks in incompatible places, so marking is completely impossible.

Expected behavior

these snippets are naive sketches, probably buggy, I haven't had time to dig into the code deeply yet...

add a field to:

struct TSLexer {
  void (*rewind)(TSLexer *);

the start_position is only on the stack, near top of ts_parser__lex. wouldn't hurt to leave it there, but it will be needed inside struct Lexer, and it looks like it could just be moved. add to struct Lexer and change references in ts_parser__lex to use the struct instead of the stack:

struct Lexer {
  Length start_position;

add a new function ts_lexer__rewind and initialize it into the .data.rewind field:

static void ts_lexer__rewind(TSLexer *_self) {
  Lexer *self = (Lexer *)_self;
  ts_lexer_reset(self, self->start_position);
  ts_lexer_start(self);
}

and I think that is all it would take. famous last words, I know. sorry I haven't found time to put together a PR, but maybe that's for the best if this idea doesn't pass muster.