An apostrophe in a CLI value silently swallows every following key=value argument

Bug

An apostrophe inside a value silently swallows every key=value pair that follows it.

split_top_level (lm_eval/_cli/utils.py) treats any ' or " as opening a quoted span. A value such as until=Bob's therefore leaves the parser inside a quote for the rest of the string, so all subsequent commas are treated as quoted and the remaining pairs are folded into that one value.

There is no error and no warning — the run just proceeds with different generation settings than the ones on the command line, which makes the reported numbers wrong and unreproducible.

This affects all seven flags that use MergeDictAction: --model_args, --gen_kwargs, --metadata, --wandb_args, --wandb_config_args, --trackio_args and --hf_hub_log_args.

Steps to reproduce

import argparse
from lm_eval._cli.run import Run

parser = argparse.ArgumentParser()
Run.create(parser.add_subparsers())

args = parser.parse_args(
    ["run", "--gen_kwargs", "until=Bob's,temperature=0,max_gen_toks=256"]
)
print(args.gen_kwargs)

args = parser.parse_args(
    ["run", "--model_args", "pretrained=gpt2,system_prompt=Answer what's asked,dtype=float16"]
)
print(args.model_args)

Actual:

{'until': "Bob's,temperature=0,max_gen_toks=256"}
{'pretrained': 'gpt2', 'system_prompt': "Answer what's asked,dtype=float16"}

Expected:

{'until': "Bob's", 'temperature': 0, 'max_gen_toks': 256}
{'pretrained': 'gpt2', 'system_prompt': "Answer what's asked", 'dtype': 'float16'}

Equivalently, from the shell:

lm-eval run --model hf --tasks gsm8k \
  --model_args pretrained=gpt2 \
  --gen_kwargs "until=Bob's,temperature=0,max_gen_toks=256"

temperature=0 and max_gen_toks=256 never reach the model.

There is no traceback: the failure is silent.

A related case, same root cause: an unterminated quote (--model_args 'a="x,b=1') consumes the rest of the string instead of being treated as a literal character.

Version

lm_eval 0.4.14.dev0, at commit d6de8164. Python 3.12.12.

I have a fix ready: a quote opens a span only when it begins a token (start of string or right after a separator), so an in-word apostrophe stays literal; it must also have a partner later in the string, so unterminated quotes degrade to literals. Values that are genuinely quoted (a='x,y', desc=say "hi") and bracket/brace protection are unchanged. Happy to open the PR.