`dimension anomaly score threshold` values above 1.0 (80 % of the documented 0.01–5.00 range) silently disable ML anomaly detection
Commit: master @ 1b3efaf3beef (2026-09-07)
What I observed
src/ml/ml_kmeans.cc:100-101caps the per-sample anomaly score at 100:return (anomaly_score > 100.0) ? 100.0 : anomaly_score;src/ml/ml.cc:1158only flags a dimension when the score clears the threshold:if (anomaly_score < (100 * Cfg.dimension_anomaly_score_threshold)) { ... return false; }src/ml/ml_config.cc:141,232readsdimension anomaly score threshold(default0.99) and stores it unchanged;src/ml/ml-configuration.md:230documents the valid range as0.01–5.00.
For any configured value > 1.00, 100 * threshold exceeds the hard cap of 100, so the comparison is always true and ml_dimension_predict can never return an anomaly. There is no log line, warning, or config error.
Why it matters
An operator who raises the threshold to reduce noise (1.5 or 2.0 are within the documented range and read as "stricter") turns the feature off instead of tightening it, while the config still shows ML enabled. That is a silent false-negative mode in a monitoring product.
Reproduction
Set dimension anomaly score threshold = 1.5 under [ml], feed any dimension a training window followed by an obviously out-of-distribution value, and observe that ml_dimension_is_anomalous never returns true. With 0.99 the same input is flagged.
Suggested fix
One of:
- clamp the accepted range to
(0, 1]and updateml-configuration.md; or - keep the wider range but scale the comparison so 5.00 means something (e.g. compare against a percentile of the training-window score distribution rather than the capped absolute score); or
- at minimum, log an error at startup when
threshold > 1.0stating that detection cannot fire.
Related (separate issue if preferred)
dimension anomaly rate suppression window is parsed as seconds (inicfg_get_duration_seconds, default 900) but ml.cc:1142 increments the counter once per collected sample, so the real window is 900 × update_every seconds, not the documented 15 minutes, for any chart with update_every != 1.
Additional verification
Re-reading confirms exactly one clamp (ml_config.cc:186, [0.01, 5.00]) and one comparison (ml.cc:1158) with no transform in between; threshold 1.0 still fires, anything above cannot. Severity note: this needs a non-default setting (the 0.99 default works), so it is a silent-misconfiguration defect rather than a default-broken one. On the related suppression note: a further read found both suppression counters reset on every successful retrain (ml.cc:915 → 860-861), and with train every defaulting to 10800 s this makes SILENCED unreachable for any chart with update_every > 12 s, independent of the units mismatch.
Found while running a read-only audit of threshold handling across several open-source projects; happy to send a PR for option 1 or 3.
Source: netdata/netdata