MELA group omits Icelandic and lists Arabic twice
Problem
The mela group contains mela_ar twice and does not include the existing mela_is task. The task README repeats the same identifier typo by labeling mela_ar as Icelandic.
Reproduction
Parsing the group configuration produces:
configured: [mela_en, mela_zh, mela_it, mela_ru, mela_de, mela_fr, mela_es, mela_ja, mela_ar, mela_ar]
duplicates: {mela_ar: 2}
unique language tasks: 9The task directory defines 10 language tasks, including mela_is.
Impact
Running the documented mela group silently evaluates 9 languages instead of the benchmark's 10. Group children are keyed by task name, so the second Arabic entry replaces the first without an error. Icelandic does not contribute to the group's aggregate MCC.
Expected behavior
Each of the 10 defined MELA language tasks should appear exactly once in the group, and the README should identify Icelandic as mela_is.
Proposed resolution
Replace the duplicate final mela_ar entry with mela_is, correct the README entry, and add a configuration regression test that compares the group members with the defined MELA task files.
Source: EleutherAI/lm-evaluation-harness