The good folks on the Microsoft Experimentation Platform team have published a paper which gives a great introduction to how and why one can go about using controlled experiments (i.e. A/B testing) to improve the usability of a website. The paper is titled Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO and will be published as part of the Thirteenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. The paper begins

In the 1700s, a British ship’s captain observed the lack of scurvy among sailors serving on the naval ships of Mediterranean countries, where citrus fruit was part of their rations. He then gave half his crew limes (the Treatment group) while the other half (the Control group) continued with their regular diet. Despite much grumbling among the crew in the Treatment group, the experiment was a success, showing that consuming limes prevented scurvy. While the captain did not realize that scurvy is a consequence of vitamin C deficiency, and that limes are rich in vitamin C, the intervention worked. British sailors eventually were compelled to consume citrus fruit regularly, a practice that gave rise to the still-popular label limeys.

Some 300 years later, Greg Linden at Amazon created a prototype to show personalized recommendations based on items in the shopping cart (2). You add an item, recommendations show up; add another item, different recommendations show up. Linden notes that while the prototype looked promising, ―a marketing senior vice-president was dead set against it, claiming it will distract people from checking out. Greg was ―forbidden to work on this any further. Nonetheless, Greg ran a controlled experiment, and the ―feature won by such a wide margin that not having it live was costing Amazon a noticeable chunk of change. With new urgency, shopping cart recommendations launched. Since then, multiple sites have copied cart recommendations.

The authors of this paper were involved in many experiments at Amazon, Microsoft, Dupont, and NASA. The culture of experimentation at Amazon, where data trumps intuition (3), and a system that made running experiments easy, allowed Amazon to innovate quickly and effectively. At Microsoft, there are multiple systems for running controlled experiments. We describe several architectures in this paper with their advantages and disadvantages. A unifying theme is that controlled experiments have great return-on-investment (ROI) and that building the appropriate infrastructure can accelerate innovation.

I learned quite a bit from reading the paper although I did somewhat skip over some of the parts that involved math. It's pretty interesting when you realize how huge the impact of changing the layout of a page or moving links can be on the bottom line of a Web company. Were talking millions of dollars for the most popular sites. That's pretty crazy.

Anyway, Ronny Kohavi from the team mentioned that they will be giving a talk related to the paper at eBay research labs tomorrow at 11AM. The talk will be in Building 0 (Toys) in room 0158F. The address is 2145 Hamilton Avenue, San Jose, CA. If you are in the silicon valley area, this might be a nice bring your own lunch event to attend.