How to Check the Significance of an A/B Split Test

A/B Split TestingSo you’re doing some split testing right? I’m sure you are (or should be! I mean it!). Anyway, are you sure that the results you’re having are significant, statistically significant?

Here’s what I mean

Suppose you have two different banners. No matter if you’re a web designer or an internet marketer, either way, you want to know which one is better. The easiest thing to do is to perform some A/B split testing.

After the test is over you find out that 600 impressions of banner A resulted in 50 clicks. However, 600 impressions of banner B resulted in 70 clicks. Great! So banner B is the better one you may think. The only problem is that these results are not statistically significant. This means that the results should not really be considered when deciding which banner is the better one. It seems like banner B is better but the number of impressions and clicks is so small that the results might be completely accidental. How to find out whether or not the results after a split test are significant?

There’s a solution

You just have to get yourself a cool, easy to use mathematical tool.

Introducing the statistical significance checker:

Statistical Significance Checker Screen

It’s really easy to use. Just input the number of trials and the number of actions for both variants of your A/B split test and press the “calculate” button.

Statistical Significance Checker calculate button

This tool will tell you whether or not your results are significant.

Here’s an example:

Statistical Significance Checker example 1

And another, significant one:

Statistical Significance Checker example 2

As you can see you will also get a level of certainty associated with your test. (A hint: everything above 95% is a great result.)

Now the best part. The tool is free. Just use the link below to download it. Have fun! 🙂

Actually, quite recently I made the tool available online. Here it is: LINK.

Download Statistical Significance Checker Here

(Quick note. An internet connection is required for this tool to work.
The tool runs on Windows and needs a thingy called .NET Framework.)

Here’s a list of articles you may also enjoy:

  • A/B testing – Wikipedia, the free encyclopedia – A/B testing or bucket testing is a method of marketing testing by which a baseline control sample is compared to a variety of single-variable test samples in order to improve response rates…
  • The Ultimate Guide To A/B Testing – /B testing isn’t a buzz term. A lot of savvy marketers and designs are using it right now to gain insight into visitor behavior and to increase conversion rate…