Cool Resources for A/B Testing Newbies

Article written by Gabriel
Community manager at PadiCode
Google+ profile

Did you like science in school? Nevermind, you’re gonna love this!

Each business is like a big science experiment. Stop rubbing your eyes, I did say science experiment. Sometimes the entrepreneur knows exactly what’s needed for a successful experiment, but most of the time the rules aren’t so clear. So she makes a bunch of other, smaller, experiments to find out what’s going wrong and what improvements are necessary.

Which brings us to A/B tests. When you’re in charge of some website, or newsletter, or what have you, you make a plan. Because if you don’t (have a plan), a penguin commits suicide. It’s a fact.

So, you know what you want to do, and you expect certain results. If, for some obscure reason, the results you’re expecting aren’t happening, you start asking yourself what the problem might be. You make a supposition, make a change, and then you test that change. Half of your visitors will see the original version, half will see the changed one, and you’ll see if the change is also an improvement (in terms of results, of course). Afterwards, all your visitors will only see the new and improved version. And that was an A/B test.

Now that we cleared that up, let’s put our goggles on, and get ready to learn some scientific marketing. Just make sure you don’t stay too close, there might be sparkles.

Learn From Examples.

I love Anne Holand’s Which Test Won, it is perhaps the most useful resource out there, when it comes to split tests. This is a website curated by a team of journalists, and it contains over 200 case studies of web, email and direct mail tests from Europe and USA.

Each week you can learn more about A/B testing from the newly featured test: the two versions are presented side by side and you are invited to vote on the version you think got the best results.

After you vote, you get to see if you guessed correctly. So, which layout do you think was best? Each time one of you chose Version B, a puppy died. Just thought you should know.

Besides the weekly test you can see, free of charge, who won awards for the best tests on the web. This is a good way for you to find out what a successful test looks like (if it won, it also had great results). However, complete access to all case studies is offered for a fee.

If you don’t feel like clicking your way through WhichTestWon, looking for great tests (there is a lot of clicking going on), you might want to consider looking at one of the awarded companies. They love to show off their accomplishments, as they should, and the awarded tests are presented in a single article, pictures and all.

It is the case of the Visual Website Optimizer’s post on the 9 award winning tests that used their software. Or, even better, the series of articles on A/B testing from ion interactive. In their fourth article, they feature a test which won silver on WhichTestWon. It turns out the headline version did better than the product oriented version. Who would’ve known?

HubSpot has another really useful article presenting 3 real A/B testing experiments conducted by their marketing team: landing pages, calls to action and email marketing.

The experiments are quite well presented, but what I like best are the conclusions after each case and the advice based on those conclusions:

1. Optimize landing pages that don’t convert at a high rate.
2. Start your optimization process with an offer test.
3. Use the knowledge you’ve gained to improve existing processes.

Learn Through Reading.

Pictures are great, when you want to learn something, but words… words are fantastic! And I think you might need some enlightening articles to help you fully understand A/B testing. A couple of my favourite resources are the Marketing Experiments website and the Unbounce blog, let’s see what they have to say.

1. Create the right environment for testing. Some companies simply do not consider testing. Either because they think they already know all the answers, or they simply don’t believe testing is necessary. Well, that’s all just silliness. Even if you read everything there is to read on the subject and see everything that worked for others, it still doesn’t mean someone else’s solutions are the right thing for you.

“… only through testing and optimization will you find out what works best for your company and, more importantly, your customers. And doing this on a regular basis is the quickest path to maximizing your email deliverability and performance.” Brad Bortone in “Email Marketing Optimization: How to create a testing environment to improve your email results”

2. Why you need a split test. Make sure you’re asking the right questions (like what is going wrong and what you want to accomplish by testing). Only then you will know which way to go and what plans to make (remember the penguin?). If you don’t ask those questions, your business experiment will look pretty much like this:

“Alice: Would you tell me, please, which way I ought to go from here?
The Cheshire Cat: That depends a good deal on where you want to get to.
Alice: I don’t much care where.
The Cheshire Cat: Then it doesn’t much matter which way you go.
Alice: …so long as I get somewhere.
The Cheshire Cat: Oh, you’re sure to do that, if only you walk long enough.” Lewis Carrol, quoted by Daniel Burstein in “Marketing Optimization: You can’t find the true answer without the right question”

3. How and what you should test. You have many options, and some of them might not be obvious to you now. You can start with a short list of things that need improving; but before you do that, take a look at some of my favourite kinds of tests, they might give you some ideas:

“6. The 6ft Test. Print a screenshot of your page and pin it to the wall. Then stand 6-feet back and stare at it. What stands out to you and your colleagues? It should be the headline and CTA. If it’s not, revisit. (Change, rinse, repeat). Avoid biased responses by reaching beyond the design team for test subjects. If you’re super ballsy – go ask strangers on the street and video it to create an extra inbound content source that you can stick on YouTube.”

7. Make use of Twitter search. Search using your “companyname” in a Twitter search (TIP: don’t use # or @ in the search as that limits the results – many people will just use your name without these prefixes), you can quickly find out what anyone and everyone is saying about you on Twitter. The good, the bad and the ugly!

18. Ask your users to describe your page in their own words. You’ll be surprised what you can learn by hearing your product or service described in your users voice. Use this to alter your communication style to something that more closely aligns with your visitors communication style.” Angela Stringfellow in “29 Ways to A/B Test Pages Based on Real User Insight”

4. Let’s talk results. When you say results, companies usually think of ROI (return on investment), percentages of people buying, number of visitors, etcetera. They are important, of course, but try to remember that behind all those numbers are people, with likes and dislikes. If you focus on what they want, instead of the numbers, you’ll be better off in the long run.

“… more valuable than any quick fix is the reason behind customer behavior.

So while most marketers can certainly slap together two treatments and see which one gets a higher conversion rate, the marketer that asks “why” more customers responded to one treatment over the other is gleaning the maximum customer insight.

Once you’ve attained enough customer insight, you can begin to predict your customer’s behavior. And when you can predict your customer’s behavior, it takes less and less time and resources to get the results you want (or most likely didn’t think you could achieve).

This is precisely why the goal of a test is not to get a lift, but rather to get a learning.” Paul Cheney in “Optimization 101: How to get results from A/B testing”

Learn By Doing.

I hope that, now that you’ve learned some things about A/B testing, you’re not just going to say: “Oh, that’s nice!”, and go on about your usual business. I hope you’ll do something about it. Leaving a comment is great, but A/B testing something is way better.

Because while reading about A/B testing and seeing what others did is extremely important, nothing can replace your own experience. After all the learning and practicing, this is the best resource you will have. So find out what needs to change in your website / email / mail and start experimenting with A/B tests.

If you don’t… let’s see… we’ve had penguins… we’ve had puppies… cats! If don’t go right now and A/B test something, a little cat will suddenly die, and it will be all your fault. You’ve been warned!

Did you find this article helpful? Are there any other cool resources we missed, but should  definitely  be included here? Please tell us what you think in the comments below.

Cool Resources for A/B Testing Newbies by

1.100.000+ email leads collected with

PadiAct helps you get more email leads. More and better.

  • Geoff Clarke

    This is a great article! I’m looking forward to spending more time looking at the ‘WhichTestWon’ website to learn more. I’m particularly fascinated by the relationship between strong keyword phrases and great creative on the web. I’ve reached the conclusion that it’s the keywords that create the organic traffic, but the creative that closes the sale – you need both to be successful!
    Thanks for sharing this article.

  • Pingback: Web Analytics - 5 things that separate men from boys | PadiCode