parser: extra statements left in AST after unambiguous reparsing `await`
When parsing in "unambiguous" mode, a top level await can be parsed initially as an identifier, then when some module syntax is found later, the statement containing await is re-parsed, treating await as a keyword.
The newly re-parsed statement is inserted back into the AST, replacing the original statement.
#26619 also fixed up the token stream, splicing in the new tokens.
The problem
The decision whether to treat await as an identifier or keyword can have knock-on effects on parsing the remainder of the file.
One example is:
await /a(); b(); c(); d(); e(); f()/g;
export {}
Initially the first line is parsed as 6 statements. The file is parsed as equivalent to:
await / a();
b();
c();
d();
e();
f() / g;
export {}
When re-parsed with await as a keyword, the first line is parsed as a single statement await /regex/g.
That statement replaces the 1st statement only, and the remaining statements stay in place. Resulting AST prints as:
await /a(); b(); c(); d(); e(); f()/g;
b();
c();
d();
e();
f() / g;
export {};
Scope of the bug
The above is just one example. It's possible that this problem manifests in other ways. Mis-interpreting await is a lexer-level mistake, which can potentially produce a completely different token stream, and therefore a completely different AST.
I'm not clear whether:
- division/regex ambiguity is required to produce this situation.
- the opposite is true - code is originally parsed as more than 1 statement, and on re-parse shrinks to less statements.
- just splicing out more statements is sufficient to fix all cases.
What Babel does
Babel gets it right - not sure how.
Possible solution
If it's excessively complicated to fully fix and the cases where await gets misinterpreted are vanishingly rare, it might be simpler to just detect when it's happened, bail out of parsing (create a fatal error), and then re-parse the whole file again from scratch as module source type.
That might be preferable to going to more and more extreme lengths to handle truly bizarre cases.
Source: oxc-project/oxc