MontyRepl exhausts the u16 name id space after ~65k inputs: each <python-input-N> filename is interned permanently
Summary
Every MontyRepl::feed_run, feed_start and call_function mints a new <python-input-N> script name and interns it as an ordinary string (parse.rs, parse_with_interner: interner.intern(filename); run.rs, new_repl_function_call). Interned strings are never released, and name-bearing opcodes carry string ids as u16 operands. So after roughly 65,000 inputs, the table is full. From then on, any snippet that needs a new name fails to compile with:
SyntaxError: module has too many distinct names; the bytecode format supports up to 65536 interned stringsThe error points at whatever line happened to need a new id. The code is fine; the session is just old.
A host that drives a game loop through the REPL, calling update() and draw() as two snippets per frame, reaches the limit after about 32,500 frames: roughly nine minutes at 60 Hz.
Reproduction
Against monty = "0.0.23" (also reproduces on main at 7134ec1e):
use monty::MontyRepl;
use monty_types::{CompileOptions, PrintWriter, ResourceTracker};
fn main() {
let mut repl = MontyRepl::new("<main>", ResourceTracker::default(), CompileOptions::default());
repl.feed_run("x = 0\ndef update():\n global x\n x += 1\n", vec![], PrintWriter::Disabled).unwrap();
// A game loop: the same one-line snippet, over and over.
for _ in 0..70_000 {
repl.feed_run("update()", vec![], PrintWriter::Disabled).unwrap();
}
// Any snippet that needs a new name now fails to compile.
repl.feed_run("def brand_new_function():\n return 1\nbrand_new_function()\n", vec![], PrintWriter::Disabled)
.unwrap();
}Traceback (most recent call last):
File "<python-input-70001>", line 3
brand_new_function()
~~~~~~~~~~~~~~~~~~
SyntaxError: module has too many distinct names; the bytecode format supports up to 65536 interned stringsReplacing the loop body with repl.call_function("update", vec![], PrintWriter::Disabled) fails at the same count. call_function also mints and interns a <python-input-N> name.
What fails and what doesn't
The limit is only checked when a string id is emitted as an operand. Globals are addressed by namespace slot, so some snippets still compile in a full session. Measured on a session filled with 66,000 snippets of 0:
| Snippet | Result |
|---|---|
update() (existing function) |
OK |
x + 1 (existing global) |
OK |
brand_new_global = 1 |
OK |
def brand_new_fn(): ... (no call) |
OK |
brand_new_fn() |
SyntaxError |
p.brand_new_attr = 1 |
SyntaxError |
[].append(1) (first use of a method name) |
SyntaxError |
kw(brand_new_kw=1) |
SyntaxError |
import math |
SyntaxError |
call_function("update") |
OK |
Whether a snippet fails depends on whether it needs a new name, not on its size, which makes this hard to diagnose from outside.
Cause
Each input adds exactly one entry to the string table (its filename). Everything else deduplicates, and heap entries stay flat. A dump of an idle session grows by about 49 bytes per input: the filename string, plus the same name as a key in MontyRepl::sources.
PR #808 noted that the intern table grows on every feed through the unique filename, and fixed the resulting slowdown. The growth itself remained. #855 has since added a separate filename-only id range (SOURCE_ID_BASE) for exec() / eval() snippets, which never touches the u16 space; REPL inputs don't use it yet.
Environment
monty0.0.23 from crates.io; also reproduced onmain(7134ec1e)- Rust 1.95; macOS aarch64 (also seen on Linux x86_64 and
wasm32-unknown-unknown)
I have a fix ready and will open a PR referencing this issue.
Source: pydantic/monty