evaluate_print and the time-series tests do not catch a non-discriminating detector
evaluate_print reports ROC and precision@n with no chance level, so a number that looks reasonable can be sitting on the floor and nothing in the output says so.
At 5% contamination a random-scoring detector prints roughly:
dummy ROC:0.5244, precision @ rank n:0.120.12 reads as a low but real number. The floor is 0.05. A constant-scoring detector prints precision @ rank n:0.0 and emits a silent sklearn UndefinedMetricWarning.
The test does not catch it because it asserts nothing:
https://github.com/yzhao062/pyod/blob/master/pyod/test/test_data.py#L470
It passes y_train * 0.1, which is the labels scaled, so the input is a perfect oracle by construction.
The seven time-series test files have the same shape. Each binds self.y_train, self.y_test in setUp and then never references them again:
https://github.com/yzhao062/pyod/blob/master/pyod/test/test_ts_spectral_residual.py#L20
The assertions are on length, non-negativity and NaN. A detector returning np.ones(n) passes all of them, at lift 1.00 against the fixture's own prevalence. I checked five of the seven that way.
You already wrote the guard for this, in one place:
https://github.com/yzhao062/pyod/blob/master/pyod/test/test_deepsvdd.py#L65
The comment there is exactly the point, that "a ROC floor can still be met by floating-point noise in the ordering". It runs for one detector out of ninety-four test files.
Proposed, and I am happy to open the PR:
detection_lift()inpyod/utils/utility.py, exported. Observed precision@n over the contamination floor.evaluate_printprints the chance level alongside precision@n, so the floor travels with the number.- Real assertions in
test_evaluate_print. - One shared helper the seven time-series tests call, asserting the scores separate the labels those fixtures already carry.
Under 200 lines, no API break, defaults unchanged. It goes in utils rather than test because pyod/test/ is excluded from the wheel, so users of the library get the floor too.
Background on why the floor moves with prevalence and why a threshold-independent number alone does not settle it: https://doi.org/10.1109/ACCESS.2026.3705430
Source: yzhao062/pyod