Big data is currently a top geek fashion. But more data does not directly imply better understanding. Avinash Kaushik offers great wisdom on the mistakes of data puking. Collecting data and making tables and diagrams is no substitute for figuring out what to do.
Another top geek fashion is A/B testing. Kohavi et al. (2012) summarize the pitfalls with a proverb: “the difference between theory and practice is greater in practice than in theory.” The allure of controlled experiments is that you can directly test outcomes. In practice you need to address two big problems:
- Actually understanding the test. Kohavi et al. (2012) point to the importance of A/A testing. If you can’t understand and control the outcomes of A/A testing, don’t waste your time doing A/B testing. A test implicitly encompasses judgments about sample relevance and time-horizon relevance. A test also necessarily implies some anticipated change in behavior. Have you considered possibilities for changes in behavior apart from and unrelated to your proposed test change?
- Actually identifying a feasible and meaningful outcome measure. Optimizing what is measurable may not increase economic value. Figuring out the best feasible measure of the economic value relevant to a specific enterprise often isn’t easy. Kohavi et al. (2012) provide some insightful examples of trade-offs between short-run value and long-run value in the context of testing search advertising.
In short, big data and controlled experiments are tools for thinking, not substitutes for thinking.
* * * * *
Related post:
Reference:
Kohavi, Ron, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker, Ya Xu. 2012. “Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained.” To appear in KDD 2012 Aug 12-16, 2012, Beijing China.