Image: images.ctfassets.net · rights & removal

Why Spotify Is Not Using Bayesian A/B Testing

Reporting by Spotify EngineeringRead the original at engineering.atspotify.com

Executive Summary

The debate surrounding the use of Bayesian A/B testing centers on claims that the framework is inherently superior to frequentist statistics, often citing flexibility and the automatic handling of complexities like multiple testing. The article argues that this appeal masks an oversimplification where specific configurations of Bayesian methods can reproduce outcomes similar to poor frequentist practices without providing genuine new guarantees. The authors posit that true advantages in Bayesian experimentation emerge only when explicitly defining goals through decision theory—such as minimizing cost functions—which leads to specific stopping rules like Bayes-factor stopping. Furthermore, the utility of Bayesian methods is heavily dependent on maintaining a high-quality historical corpus; poorly managed priors and pooled data can degrade the accuracy of estimates, suggesting that implementation complexity often outweighs the theoretical gains unless rigorous prior maintenance is guaranteed.

Facts Only

* Many commercial experimentation platforms have added Bayesian modes.
* Bayesian A/B testing is a family of configurations defined by stopping rules, priors, and likelihoods.
* A flat-prior stopping on a posterior-probability threshold reproduces the false positive rate of frequentist peeking.
* Bayes factor stopping controls the false positive rate under peeking.
* A calibrated empirical Bayes prior can shrink effect estimates to counter winner’s curse bias and bound the false discovery rate.
* Decision-theoretic approaches derive stopping rules that minimize a chosen cost function.
* For specific cost functions, the optimal stopping rule is often Bayes-factor stopping.
* Under a flat prior and the two-group normal model, Bayesian and frequentist procedures produce numerically identical outputs for mean estimates and probability comparisons.
* The mixture Sequential Probability Ratio Test is exactly the Bayes factor stopping under the same prior.
* Spotify historically used frequentist statistics for production experiments.
* Empirical Bayes requires a large, representative corpus to avoid poor pooling across metrics or programs.

Full Take

The narrative presents a tension between the perceived intuitive elegance of Bayesian inference and the necessary rigor required for reliable experimentation. The core pattern observed is that mathematical equivalence exists between frequentist and Bayesian concepts under specific, simple assumptions (like flat priors), suggesting the choice of framework often reflects differing vocabulary rather than ontological differences in guarantees. The manipulation occurs when complexity—specifically maintaining high-quality empirical Bayes priors across heterogeneous data streams—is obscured as inherent simplicity. The argument forces a critical look at where the "flexibility" is actually applied: whether it leads to genuine, traceable improvements or merely obfuscates failures related to prior selection and data coherence. This suggests that the true bottleneck in experimentation is not the choice between frameworks but the maintenance of epistemic integrity within any chosen configuration.
STEP 1 — DETECT SOURCE TYPE: SKEPTICAL MODE
Patterns detected: Ambiguity, Distortion

From the original · Spotify Engineering

In recent years, amid growing interest in Bayesian interference for A/B testing, many commercial experimentation platforms have added a Bayesian mode, including GrowthBook, LaunchDarkly, PostHog, Amplitude Experiment, Optimizely, VWO, Statsig, and Eppo. The main argument is that Bayesian A/B testing is modern, flexible, and easier to interpret than frequentist statistics.

Read the full story at engineering.atspotify.com

Sentinel — Human

Confidence

The article functions as a deep, structured critique of common claims surrounding Bayesian vs. frequentist A/B testing, grounded in detailed statistical configurations, suggesting a high degree of human analytical synthesis.

Signals Detected

Sentence length variance exhibits natural variation; complex structure is used to build layered arguments rather than uniform rhythm.

Argument flows logically from observation (tools) to interrogation (claims) to specific configuration analysis, demonstrating a focused, sustained thesis.

The structure directly maps out a complex analytical process using numbered steps and direct dissection of common claims, suggesting an authorial framework.

References to specific statistical concepts (Bayes factor stopping, Empirical Bayes prior) are used contextually rather than as standalone assertions, requiring domain expertise integration.

Human Indicators

The text exhibits a sustained, nuanced argument that resists simple dichotomies, focusing on the *conditions* under which claims hold (e.g., 'only true for people using an informative prior').

There is an acknowledgment of failure modes and practical implementation challenges (e.g., pooling metrics, inconsistent prior estimation), indicating experience with the practical difficulties of applied statistics.