Metadata n-shot override is ignored in collected results

Problem

Task metadata can declare num_fewshot to override the n-shot value printed in result tables. This is useful when a task embeds demonstrations directly in its prompt while keeping the runtime num_fewshot setting at zero.

The current result collector ignores that metadata and always reports the runtime configuration value.

Reproduction

Create a task with:

num_fewshot: 0
metadata:
  version: 1
  num_fewshot: 5

Then collect its result:

acc = {
    "hardcoded_fewshot": {
        "task": task,
        "raw_metrics": defaultdict(list),
        "logged_samples": [],
    }
}

result = _collect_results(acc, bootstrap_iters=0)
print(result.num_fewshot["hardcoded_fewshot"])

Observed:

0

Expected:

5

Why the two values differ

For tasks with demonstrations hard-coded into doc_to_text, runtime num_fewshot must remain zero so the harness does not sample and prepend additional examples. The metadata value describes how many demonstrations are already present and is intended only for the printed and serialized n-shot result field.

Without the override, a prompt that is actually five-shot is published as zero-shot even though inference itself ran correctly.

Cause

The current result collector assigns:

result.num_fewshot[task_name] = task_config.get("num_fewshot", 0)

It never examines task_config["metadata"]["num_fewshot"].

The override was introduced by #1379 to resolve #1360. Before the evaluator refactor, result collection used the metadata value when runtime num_fewshot was zero. The documentation still promises this behavior, but the refactored collector dropped it.

Impact

The prompt and metric values are unchanged, but result provenance is wrong. Hard-coded few-shot tasks can be mislabeled as zero-shot in console tables and serialized output, making runs harder to reproduce and compare correctly.

Three shipped task configurations still contain metadata.num_fewshot, currently with value zero. Custom tasks using the documented nonzero override expose the visible regression.

Proposed resolution

Restore the original display semantics in result collection:

• when runtime num_fewshot is zero, use metadata.num_fewshot if present;
• otherwise retain the runtime value;
• do not change prompt construction or few-shot sampling.

Add regression coverage for a nonzero metadata override and the ordinary configured-value path.