The anatomy of a web analytics decision

…or the scientific approach to actionable web analytics

Web analytics masters like Eric Peterson, Avinash Kaushik, Anil Batra and many others have been advocating since always for actionable web analytics. They have dedicated blogs and entire books on how not to be a reporting monkey but rather be an web analytics ninja.

The road to web analytics ninjaness ain’t easy. Whenever you think you get a big WOW from your web analytics data don’t just run to the IT department and tell them everything they did is plain wrong. Take a deep breath and try to answer some questions before.

How reliable is the source of your discovery?

This one gave me shame moments for a couple of times so far. On my dashboard I have a big widget for the conversion rate. One day, entering the web analytics solution it showed a huge drop. I got everybody in the company fired up, just to realize that nothing happened to the conversion rate. It was just a bug in our website that affected the way we measured a couple of our metrics.

Now, whenever I get WOW or OMG moments, the first thing I do is to do a WASP debugging of the pages are guilty for my WOW moment. I make sure everything is still measured correctly.

Have the claims been verified by other departments?

In the above scenario, when I got everybody fired up with my conversion rate drop, just before entering the CEO and giving the awful news,  the sales department told me that according to their reports, for the same day sales were up and not down. The marketing department told me that no special marketing campaigns are on so the traffic should be the same… All of them suggested me to calm down, drink some tea and go double check.

Since then, whenever I analyze revenue reports in my analytics application, I make sure to double check it with the other departments. If they confirm, now it’s the moment to get excited.

Is this how websites work?

This question actually helped me quite some times so far. Here is an example: A report is showing that a page has 100% bounce rate however people visiting that page bought a product in the same session. How is that possible? No, this is not how websites work. If a user bounces than he can’t convert: he is not on your website anymore. So there must be something wrong with this report

Actually, it isn’t. What is wrong is the way some people look at it. Bounce rate is calculated only for landing pages. Let’s say that page has been seen by 1000 visitors out of which 5 coming from search engines directly to the page and the others coming to it from other pages in the website. If all the 5 visitors coming from Google bounce, than we have 100% bounce rate. However, any of the other 995 users might of have bought something.

If things just don’t make sense, try to figure them out. If they don’t make sense to you, they won’t to your management as well.

What else could be explained with the data you get?

This is a classic one. When you see that the time spent on your website increases it doesn’t necessary means that users decided to look into more products of your website. It can also mean they don’t find what they are looking for. The same goes for increases or decreases of the pages/visit rate.

Whenever you are ready to take a decision based on the data, question yourself  if the same data can’t explain for other behaviors of your users as well.

Where does the other evidence point to?

The metrics you get for your website visitors are interconnected. When one metric is affected bet on it that other metrics will be affected as well. What does it mean when pages/visit increases but time spent on website doesn’t? What about the other way around? What if the drop in bounce rate has a very small impact on time on website? Does it mean people really stopped bouncing or might there be another explanation?

Whenever you want to act based on the web analytics data you have, make sure that any other metric you take to analyze, fits in your theory.

Are you taking it personal?

This is a tough one. As mastering the web analytics ninja way of life, your intuition becomes stronger as well; sometimes too eager to adopt new things. If during your experience the conversion rate increased only by working on the funnel, it doesn’t mean that’s the only way it can be increased.

Always question your personal beliefs and share them with seniors. Don’t take their word for it (they have their own personal beliefs) but rather use their arguments to see if your belief can still stand up.

Conclusions

Be scientific about it. Web analytics should not be treated as a walk in the park but again, it’s not rocket science for MIT graduates as well. Be skeptical.

What other questions/methods are you using to validate your web analytics actions?

This article idea came out while going through the Skeptical Magazine, which I recommend to any geek out there.