Against A/B tests

Traditional A/B testsing rests on a fundamentally flawed premise. Most of the time, version A will be better for some subgroups, and version B will be better for others. Choosing either A or B is inherentlyinferior to choosing a targeted mix of A and B.

Michael Kaminsky

The quote above is from a post by Michael Kaminsky “Against A/B tests“. I’m still not fully convinced by Michael’s thesis but it is very interesting and thought-provoking. 

By Boris Gorelik

Machine learning, data science and visualization | CTO @

1 comment

  1. It is like preparing dinner for a large crowd.
    You decide on the menu which suits the majority.
    You don’t cater to many different subgroups because you don’t have the resources to do that and because the ROI isn’t worth it.
    It’s a nice idea but fails the math, developing and maintain endless branches is extremely costly.


Leave a comment

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: