Issue with `bigbench_gender_inclusive_sentences_german_multiple_choice`
Author: ayulockinCreated Feb 26, 2024Updated Sep 18, 2026
While running this task I get the following error stack trace:
Traceback (most recent call last):
File "/opt/conda/envs/fr-de-lb/bin/lm_eval", line 8, in <module>
sys.exit(cli_evaluate())
File "/home/ayushthakur/lm-eval/llm-leaderboard-fr-de/lm-evaluation-harness/lm_eval/__main__.py", line 288, in cli_evaluate
results = evaluator.simple_evaluate(
File "/home/ayushthakur/lm-eval/llm-leaderboard-fr-de/lm-evaluation-harness/lm_eval/utils.py", line 288, in _wrapper
return fn(*args, **kwargs)
File "/home/ayushthakur/lm-eval/llm-leaderboard-fr-de/lm-evaluation-harness/lm_eval/evaluator.py", line 154, in simple_evaluate
task_dict = get_task_dict(tasks, task_manager)
File "/home/ayushthakur/lm-eval/llm-leaderboard-fr-de/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 412, in get_task_dict
task_name_from_string_dict = task_manager.load_task_or_group(string_task_name_list)
File "/home/ayushthakur/lm-eval/llm-leaderboard-fr-de/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 253, in load_task_or_group
collections.ChainMap(
File "/home/ayushthakur/lm-eval/llm-leaderboard-fr-de/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 164, in _load_individual_task_or_group
return load_task(task_config, task=name_or_config, group=parent_name)
File "/home/ayushthakur/lm-eval/llm-leaderboard-fr-de/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 153, in load_task
task_object = ConfigurableTask(config=config)
File "/home/ayushthakur/lm-eval/llm-leaderboard-fr-de/lm-evaluation-harness/lm_eval/api/task.py", line 731, in __init__
test_target = self.doc_to_target(test_doc)
File "/home/ayushthakur/lm-eval/llm-leaderboard-fr-de/lm-evaluation-harness/lm_eval/api/task.py", line 962, in doc_to_target
target_string = utils.apply_template(doc_to_target, doc)
File "/home/ayushthakur/lm-eval/llm-leaderboard-fr-de/lm-evaluation-harness/lm_eval/utils.py", line 426, in apply_template
return rtemplate.render(**doc)
File "/opt/conda/envs/fr-de-lb/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
self.environment.handle_exception()
File "/opt/conda/envs/fr-de-lb/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
raise rewrite_traceback_stack(source=source)
File "<template>", line 1, in top-level template code
ValueError: 'Potsdam ist eine kreisfreie Stadt und mit gut 180.000 Einwohner*innen die bevölkerungsreichste Stadt und Hauptstadt des Landes Brandenburg.' is not in listI am using the following command:
lm_eval --model hf --model_args pretrained=microsoft/phi-2,trust_remote_code=True --tasks bigbench_gender_inclusive_sentences_german_multiple_choice --device cuda:0 --batch_size 1 --output_path output/phi-2-mmlu-arc --limit 2 --wandb_args project=lm-eval-harness-integration --log_samplescc: @haileyschoelkopf
Source: EleutherAI/lm-evaluation-harness