#736·pyod

evaluate_print and the time-series tests do not catch a non-discriminating detector

Author: mateenali66Created Sep 4, 2026Updated Sep 4, 2026

evaluate_print reports ROC and precision@n with no chance level, so a number that looks reasonable can be sitting on the floor and nothing in the output says so.

At 5% contamination a random-scoring detector prints roughly:

dummy ROC:0.5244, precision @ rank n:0.12

0.12 reads as a low but real number. The floor is 0.05. A constant-scoring detector prints precision @ rank n:0.0 and emits a silent sklearn UndefinedMetricWarning.

The test does not catch it because it asserts nothing:

https://github.com/yzhao062/pyod/blob/master/pyod/test/test_data.py#L470

It passes y_train * 0.1, which is the labels scaled, so the input is a perfect oracle by construction.

The seven time-series test files have the same shape. Each binds self.y_train, self.y_test in setUp and then never references them again:

https://github.com/yzhao062/pyod/blob/master/pyod/test/test_ts_spectral_residual.py#L20

The assertions are on length, non-negativity and NaN. A detector returning np.ones(n) passes all of them, at lift 1.00 against the fixture's own prevalence. I checked five of the seven that way.

You already wrote the guard for this, in one place:

https://github.com/yzhao062/pyod/blob/master/pyod/test/test_deepsvdd.py#L65

The comment there is exactly the point, that "a ROC floor can still be met by floating-point noise in the ordering". It runs for one detector out of ninety-four test files.

Proposed, and I am happy to open the PR:

  1. detection_lift() in pyod/utils/utility.py, exported. Observed precision@n over the contamination floor.
  2. evaluate_print prints the chance level alongside precision@n, so the floor travels with the number.
  3. Real assertions in test_evaluate_print.
  4. One shared helper the seven time-series tests call, asserting the scores separate the labels those fixtures already carry.

Under 200 lines, no API break, defaults unchanged. It goes in utils rather than test because pyod/test/ is excluded from the wheel, so users of the library get the floor too.

Background on why the floor moves with prevalence and why a threshold-independent number alone does not settle it: https://doi.org/10.1109/ACCESS.2026.3705430