HumanEval benchmark validates n/k with a bare assert and accepts invalid values
Author: Asthenia0412Created Aug 31, 2026Updated Sep 14, 2026
Description
The HumanEval benchmark never validates its two numeric parameters, and the one check it does have uses a bare assert:
class HumanEval(DeepEvalBaseBenchmark):
def __init__(self, tasks=None, n: int = 200, verbose_mode=False, **kwargs):
...
self.n = n
def evaluate(self, model, *args, k: int = 1, **kwargs):
import pandas as pd
with capture_benchmark_run("HumanEval", len(self.tasks)):
assert self.n >= k # <-- bare assert
...Problems:
assert self.n >= kis stripped underpython -O/-OO, so then < kconfiguration silently proceeds and produces a meaningless run. It also raisesAssertionErrorinstead of theValueErrorcallers expect.nis never validated at construction:HumanEval(n=0)orHumanEval(n=-1)are accepted.n=0makesevaluategenerate zero samples per task, which flows intoScorer.pass_at_kas a degeneraten=0.kis never validated:evaluate(..., k=0)or a negativekis accepted, producing a meaningless pass@k.
Expected behavior
HumanEval(n=...)raisesValueErrorunlessnis a positive integer.evaluate(..., k=...)raisesValueErrorunlesskis a positive integer, and replaces the bareassert self.n >= kwith an explicitValueError.- Valid configurations (
n >= 1,k >= 1,k <= n) behave exactly as today.
This is input validation only: it never changes results for valid inputs, and it converts a -O-stripped assert and silently-wrong degenerate runs into actionable errors at the source.
Source: confident-ai/deepeval