Every A/B testing tutorial ends the same way: run the test, wait for significance, ship the winner.
Then you run a real test and variant B converts 12% better on newsletter signups, brings in 4% less revenue per visitor, and bounce is flat.
Nothing is significant except the signups.
Ship it?
I spent an embarrassing amount of time on this question while building an A/B engine, and most of what I read online didn't help, because most of it assumes one metric.
This post is what I ended up with.
It's not novel — the statistics are decades old — but I couldn't find it written down in one place with working code, so here it is.
Why the p-value doesn't answer the question you're asking Two problems, and the second one is the bad one.
Multiple comparisons.
Three metrics at α = 0.05 means roughly a 14% chance of at least one false positive if nothing is actually different.
Bonferroni fixes this, but now you need α = 0.017 per metric and your test needs to run three times as long.
On a site doing 300 conversions a month that's not a fix, it's a refusal.
The p-value is answering a different question.
It tells you the probability of your data assuming no difference exists.
What you actually want to know is: if I ship B, how much do I expect to lose if I'm wrong?
Those are not the same question and no amount of Bonferroni turns one into the other.
There's also the peeking problem — everyone checks the dashboard daily and stops when it goes green, which quietly inflates the false positive rate well past whatever α you wrote down.
I'll come back to that, because Bayesian methods do not magically solve it, whatever you may have read.
Posterior first, decision second For a conversion rate, the Beta-Binomial conjugate pair gives you the posterior in one line.
With a uniform prior, after conversions out of visitors: That's it.
No closed-form comparison between two Betas that's worth implementing, so sample.
PHP has no Beta sampler in core, and no Gamma sampler either, so you build one.
Marsaglia–Tsang, which needs a normal sampler underneath: Now the single-metric comparison: Two numbers instead of one. is the intuitive one everybody quotes. is the one that should drive the decision: it's the average amount of conversion rate you give up, across the whole posterior, in the worlds where B is actually worse.
If that number is 0.0004 and you genuinely don't care about four hundredths of a percentage point, ship B and stop thinking about it, even at 88% probability.
Pick that threshold before the test.
Write it in the test config.
It's the smallest effect you'd bother shipping for, and it forces a conversation about what the test is actually for.
Revenue per visitor is not a Beta This is where I initially got it wrong.
I treated revenue per visitor as a conversion-like quantity and got posteriors that were far too confident.
Revenue per visitor is zero-inflated (most visitors buy nothing) and heavy-tailed (one enterprise order distorts everything).
A Beta is wrong, a Normal is wrong, a log-normal is closer but still assumes away the zeros.
I use a bootstrap instead.
Resample the observed per-visitor revenue values with replacement, times, take the mean of each resample.
That's your posterior-ish distribution, and it inherits whatever ugly shape your real data has without you having to name it: It's O(draws × n) and it will hurt on large samples.
Precompute a cumulative array and sample indices in blocks, or bootstrap on a fixed random subsample of 5,000 visitors per arm — the extra Monte Carlo noise is small compared to the sampling noise you're already living with.
Combining metrics without lying to yourself Now the actual question.
Three metrics, three posteriors, one decision.
The standard answer is an OEC — Overall Evaluation Criterion, from Kohavi's work at Microsoft.
One weighted composite, agreed in advance, that the test is scored against.
The two things people get wrong:
1.
Weight relative uplift, not raw values.
A conversion rate lives in