MELA group omits Icelandic and lists Arabic twice

Author: sdivyanshu90Created Sep 14, 2026Updated Sep 14, 2026

Problem

The mela group contains mela_ar twice and does not include the existing mela_is task. The task README repeats the same identifier typo by labeling mela_ar as Icelandic.

Reproduction

Parsing the group configuration produces:

configured: [mela_en, mela_zh, mela_it, mela_ru, mela_de, mela_fr, mela_es, mela_ja, mela_ar, mela_ar]
duplicates: {mela_ar: 2}
unique language tasks: 9

The task directory defines 10 language tasks, including mela_is.

Impact

Running the documented mela group silently evaluates 9 languages instead of the benchmark's 10. Group children are keyed by task name, so the second Arabic entry replaces the first without an error. Icelandic does not contribute to the group's aggregate MCC.

Expected behavior

Each of the 10 defined MELA language tasks should appear exactly once in the group, and the README should identify Icelandic as mela_is.

Proposed resolution

Replace the duplicate final mela_ar entry with mela_is, correct the README entry, and add a configuration regression test that compares the group members with the defined MELA task files.

Source: EleutherAI/lm-evaluation-harness