If you’re testing a horse against a car, you don’t need an A/B test.

Sean Taylor (Ex-Lyft and Facebook data scientist and author of the prophet forecasting package) has started a newsletter. In his first edition he ponders the value added by analysing data.

The single biggest source of value from data often comes from A/B tests. However, if an A/B test works well you don't need statistics to tell if it worked. Therefore what's the point in analysing data!? If all the analysis gets you is incremental change, is that good enough? Can we not do better than that?

Maybe data tells you why things have gone wrong? This is the field of causal analysis. Or maybe data can give us idea about what to try in the first place? This is the role of nascent generative models like GPT-3 and DALLE.

An interesting, if slightly rambling start from Taylor.