[Task]: ML test suite passes with a detector that never fires and one that always fires

Problem / root cause

Two degenerate versions of the anomaly detector pass the entire ML test suite. ml_dimension_predict, the function that decides whether a dimension is anomalous, is never exercised by a test.

I built current master with -DENABLE_ML=On and ran netdata -W mltest. Baseline is 606 assertions, 0 failed. I then made one-line changes to ml_dimension_predict in src/ml/ml.cc and rebuilt.

Mutation 1, the detector can never report an anomaly:

        if (anomaly_score < (100 * Cfg.dimension_anomaly_score_threshold)) {
            ...
            return false;
        }

        sum += 0;   // was: sum += 1

ML tests: 606 run, 0 failed
ML TESTS PASSED

Mutation 2, every dimension is always anomalous (100% false-positive rate):

        if (false) {   // was: if (anomaly_score < (100 * Cfg.dimension_anomaly_score_threshold)) {

ML tests: 606 run, 0 failed
ML TESTS PASSED

Root cause, three parts:

1. ml_dimension_predict is never called by the suite. grep -n ml_dimension_predict src/ml/ml-unittest.cc returns one hit, inside a comment on line 635. The threshold comparison, the sum accumulation and the suppression logic are untested.

2. The scoring formula is tested, the decision is not. test_kmeans_scoring covers ml_kmeans_anomaly_score well enough that mutating it directly does get caught. Both mutations above leave that function untouched and change only the decision built on top of it.

3. The one detection-shaped assertion is not tied to the decision threshold. In test_full_pipeline:

ML_TEST_ASSERT_DOUBLE_EQ(normal_score, 0.0, 1e-6, "best training sample should score ~0");
...
ML_TEST_ASSERT(anomaly_score > normal_score, "anomalous data should score higher than normal");

normal_score is the minimum score across training samples, asserted to be about 0. So the bar is that an anomalous sample scores above roughly zero, while ml_dimension_predict only declares an anomaly at 100 * dimension_anomaly_score_threshold, which is 99.0 at the default 0.99. A detector whose anomalous scores top out at 0.5 satisfies the assertion and detects nothing in production.

The rest of the suite is thorough on the machinery: feature preprocessing, kmeans training and serialization, circular buffer equivalence, queue and model-publish behaviour, config parameter combinations. The gap is specifically the detection outcome.

Clean end state

src/ml/ml-unittest.cc covers the detection decision, not only the machinery that feeds it. A change that breaks detection while leaving the scoring formula intact fails the suite. Specifically, ml_dimension_predict is driven end to end, and at least one assertion is expressed in terms of dimension_anomaly_score_threshold so the score scale stays pinned to the decision it drives.

Acceptance criteria

• netdata -W mltest fails when sum += 1 in ml_dimension_predict is changed to sum += 0
• netdata -W mltest fails when the threshold comparison in ml_dimension_predict is made unconditional
• A test trains on a known-normal series, feeds a known anomaly through ml_dimension_predict, and asserts the return is true, and false for a normal sample
• A test asserts the decision changes when dimension_anomaly_score_threshold changes, with input held constant
• A test asserts that scores over a mixed normal and anomalous window are not all identical, and that anomalous samples rank above normal ones rather than above a single best-case sample

Category

test debt

Scope boundaries

In scope: additions to src/ml/ml-unittest.cc, and wiring -W mltest into CI if that is wanted.

Out of scope: any change to detector behaviour, to ml_kmeans_anomaly_score, to the default threshold, or to the training path. This is about what the tests assert, not about how the detector works. The two mutations above were reverted after measuring.

Validation

netdata -W mltest on an unmodified tree still reports 0 failed, and reports failures under each of the two mutations. I can supply the mutation patches so the result can be confirmed independently.

Risks / compatibility

Test-only, so no behaviour, API, schema, packaging or migration impact. The one risk worth naming is a flaky assertion: a detection test that depends on kmeans initialisation could be nondeterministic. Any test added here should use a fixed training set and assert on the decision rather than on an exact score value.

Related context

-W mltest does not appear in any workflow under .github/workflows/, in packaging scripts, or in an add_test registration, so the suite appears to run only when invoked by hand. Happy to be corrected if it runs somewhere I did not find.

I am willing to implement the additions above if the direction looks right. I wanted to check the framing before sending tests nobody asked for. Build used: master on macOS 15 arm64, Apple clang, cmake 4.4.0, -DENABLE_ML=On with the prometheus remote-write and mongodb exporters disabled.