#3208·deepeval

HumanEval benchmark validates n/k with a bare assert and accepts invalid values

Author: Asthenia0412Created Aug 31, 2026Updated Sep 14, 2026

Description

The HumanEval benchmark never validates its two numeric parameters, and the one check it does have uses a bare assert:

python
class HumanEval(DeepEvalBaseBenchmark):
    def __init__(self, tasks=None, n: int = 200, verbose_mode=False, **kwargs):
        ...
        self.n = n

    def evaluate(self, model, *args, k: int = 1, **kwargs):
        import pandas as pd
        with capture_benchmark_run("HumanEval", len(self.tasks)):
            assert self.n >= k   # <-- bare assert
            ...

Problems:

  1. assert self.n >= k is stripped under python -O/-OO, so the n < k configuration silently proceeds and produces a meaningless run. It also raises AssertionError instead of the ValueError callers expect.
  2. n is never validated at construction: HumanEval(n=0) or HumanEval(n=-1) are accepted. n=0 makes evaluate generate zero samples per task, which flows into Scorer.pass_at_k as a degenerate n=0.
  3. k is never validated: evaluate(..., k=0) or a negative k is accepted, producing a meaningless pass@k.

Expected behavior

  • HumanEval(n=...) raises ValueError unless n is a positive integer.
  • evaluate(..., k=...) raises ValueError unless k is a positive integer, and replaces the bare assert self.n >= k with an explicit ValueError.
  • Valid configurations (n >= 1, k >= 1, k <= n) behave exactly as today.

This is input validation only: it never changes results for valid inputs, and it converts a -O-stripped assert and silently-wrong degenerate runs into actionable errors at the source.