Python API guide passes unsupported num_fewshot to evaluate()
Problem
The low-level Python API example passes num_fewshot to lm_eval.evaluate(), but that function has no such parameter. Copying the documented example fails before evaluation begins.
Documented call
results = lm_eval.evaluate(
lm=lm,
task_dict=task_dict,
num_fewshot=5,
limit=100,
)Reproduction
import lm_eval
lm_eval.evaluate(
lm=None,
task_dict={"tasks": {}},
num_fewshot=5,
)Observed:
TypeError: evaluate() got an unexpected keyword argument 'num_fewshot'Cause
simple_evaluate() is the high-level entry point that accepts num_fewshot, applies the override to loaded task objects, seeds their few-shot samplers, and then calls evaluate().
evaluate() is the lower-level execution function. It accepts an initialized model and preconfigured task dictionary after those setup decisions have already been made, so its signature does not contain num_fewshot.
The example mixes the high-level configuration argument into the low-level API.
It also builds the task dictionary with the deprecated get_task_dict() helper, while the current evaluate() documentation and type expect the dictionary returned by TaskManager.load().
Impact
Users following the Python API guide receive an immediate TypeError. Removing the argument without understanding the API boundary can also leave them unsure how few-shot configuration should be supplied.
Expected documentation
The example should:
- build tasks with
TaskManager.load(); - pass only supported arguments to
evaluate(); - explain that low-level callers configure task objects before evaluation;
- direct users who want a global
num_fewshotoverride tosimple_evaluate().
No runtime API change is needed.
Source: EleutherAI/lm-evaluation-harness