How to Overcome MannWhitney Judge Shortcomings

Hey Spinnaker folks,

TL;DR: MannWhitney doesn't work too well for us. We're looking for any guidance as to whether we should be writing our own judge, should tweak our metrics, or do something else that we haven't thought of yet.

We're experiencing a problem related to that in spinnaker/kayenta#364 and are looking for guidance as to how to combat it. I'll include unit tests below with actual values if anyone wants to test but here are some charts to illustrate our issue with the current MannWhitney test.

Disclaimer: We're aware that MannWhitney is intended for distributions the shape of which is equal (and the distribution itself is shifted from the baseline) and the baseline/canary charts below are clealy not shifted versions of each other. However, that's what real data sometimes looks like for us.

Case 1: Increase in Metric Variability

Result is PASS but it should be FAIL

  test("Mann-Whitney Classifier Test: Increase in Metric Variability") {
    val experimentData = Array(
      10.0,  10.1, 10.2,  10.3, 10.4,  10.5, 10.6,  17.0, 17.1,  17.2, 17.3,  17.4, 17.5,  17.6, 17.7
    )
    val controlData = Array(
       1.0, 100.0,  4.0, 101.0,  5.0, 102.0,  6.0, 103.0,  7.0, 104.0,  8.0, 105.0,  9.0, 106.0, 17.0
    )
    val experimentMetric = Metric("pass-metric", experimentData, "canary")
    val controlMetric = Metric("pass-metric", controlData, "baseline")
    val classifier = new MannWhitneyClassifier(tolerance = 0.10, confLevel = 0.95)
    val result = classifier.classify(controlMetric, experimentMetric, MetricDirection.Either)

    assert(result.classification == Pass) // Logically, it shouldn't pass!
  }

Our reasoning is: if an engineer saw this as a result of monitoring a deployment, they'd most certainly stop the deployment and investigate what is happening. This issue stems from how MannWhitney works. However: How can we guard against this type of false negative?

Case 2: Initial Fluctuation That Heals over Time

Result is FAIL but it should be PASS

  test("Mann-Whitney Classifier Test: Initial Fluctuation that Heals over Time") {
    val experimentData = Array(
      10.99, 10.991, 10.992, 10.993, 10.994, 10.995, 10.996, 11, 11.01, 11.02, 11.021, 11.023, 11.024, 11.025, 11.026
    )
    val controlData = Array(
      2.1, 15.05, 2.2, 15.01, 2.3, 10.982, 11.011, 11, 10.989, 10.988, 10.987, 10.986, 10.985, 10.984, 10.983
    )
    val experimentMetric = Metric("pass-metric", experimentData, "canary")
    val controlMetric = Metric("pass-metric", controlData, "baseline")
    val classifier = new MannWhitneyClassifier(tolerance = 0.10, confLevel = 0.95)
    val result = classifier.classify(controlMetric, experimentMetric, MetricDirection.Either)

    assert(result.classification == High) // Logically, this should pass!
  }

Similar reasoning as with Case 1, if an engineer was to monitor this, they'd likely decide that there was an initial glitch at boot-time and still let the deployment continue. (Due to the nature of our deployments, this fluctuation does happen quite often.) If I understand spinnaker/kayenta#364 correctly, then if the MannWhitney test fails, there's another level of judgement determined by a effect size parameter. Where is this effectSize parameter documented (i.e. how does it work)?