Traditional A/B testsing rests on a fundamentally flawed premise. Most of the time, version A will be better for some subgroups, and version B will be better for others. Choosing either A or B is inherentlyinferior to choosing a targeted mix of A and B.
Michael Kaminsky locallyoptimistic.com
The quote above is from a post by Michael Kaminsky “Against A/B tests“. I’m still not fully convinced by Michael’s thesis but it is very interesting and thought-provoking.
It is like preparing dinner for a large crowd.
You decide on the menu which suits the majority.
You don’t cater to many different subgroups because you don’t have the resources to do that and because the ROI isn’t worth it.
It’s a nice idea but fails the math, developing and maintain endless branches is extremely costly.
LikeLike