knowing when to re-optimize your website

Google recently made website optimization freely available to everyone. Perhaps this truly is a key element in Google’s plan for world domination. But in any case, it’s good news for those times when you want to choose data analysis over whimsy and hope.

A danger of free, fast, and easy testing might be that its users will become complacent about the background structure of testing and optimization. Statistical tests always depend on background assumptions about data-generating processes. For example, when the number of users of a website increases by orders of magnitude, the attributes of its users (age, sex, internet experience, etc.) may change. What had been an optimal content design may no longer be optimal.

Economists have long pondered problems of behavioral estimates. To get a sense of the issues, suppose that you observe that over the past ten years both the price of widgets and aggregate sales of widgets rose strongly. That doesn’t mean that raising the widget price will increase widget sales. More likely the opposite is true. Increasing demand for widgets may account for the correlation between higher prices and greater sales.

The problems for human behavioral estimates are far worse than those of omitted variable biases in other physical systems. Humans are good at collecting and sharing information, and humans often change their behavior in response to new information, new rules, and new incentives. In economics, the Lucas critique is a famous recognition of this reality for parameter estimation.

Among empirical economists, a commonly expressed goal is to estimate “structural parameters.” Structural parameters are interpretable parameters relatively insensitive to plausible changes in the environment and rules of the game.[1] When you optimize with respect to such parameters, your optimization is more likely to be valid over time. Moreover, since you can interpret the parameters, you can anticipate changes in the environment that are likely to invalidate your optimization.

Economists have produced a huge, complex literature on structural estimation. But probably all you really need to know from it for website optimization are a few points:

• Website optimization is not forever. Changes in circumstances (New Year’s Day promotions won’t work year-round), changes in user characteristics, changes in the experience of users with other websites, and changes in your business reputation may change the optimal configuration of your website.
• Seek some understanding of what works for you. Testing can help you discover what works. But sound interpretation and knowledge remain valuable. Understanding why a particular configuration works can give you insights into when it might need to be changed (or re-tested). In addition, understanding what works might provide you with more general insights into your users.
• Expect your users to learn and respond to what you do. If an aspect of your optimization involves baiting and screwing your users, they will learn about it and adapt to it. Your optimal screw of your users can turn to screw you.

Note:

[1] To economists, parameters describing preference functions (demand) and cost functions (supply) are structural parameters. That economists consider parameters describing preferences and technology to be relatively stable may seem laughable to persons engaged in viral marketing and rapid web service development.