diegopoza

System Engineer, geek, foodie, technology lover, speaker.

Sr Manager, Developer Advocacy

Diego Poza

How A/B Testing can help you taking decisions that make impact



Even large successful companies like Facebook understand and [utilize the power of A/B testing](https://fbinsights.files.wordpress.com/2016/07/facebookiq_placement-optimization.pdf) to ensure they stay ahead. **A/B testing** is a powerful tool you can use to drive more conversions from your site. 

The main goal of an A/B test is to test two or more variants of a page to determine which option works best. Starting your tests high-level before moving to test smaller details is the best way to maximize these conversions. 

It's important to make sure you choose a page that has [high traffic but low conversions](http://unbounce.com/a-b-testing/data-based-a-b-testing-process/) so that you're maximizing returns. If conversions are high but traffic low, then you need to find a way to drive more traffic which isn't something you can A/B test for. Low conversions and low traffic also won't benefit from an A/B test because without any traffic to gather data from, your A/B test won't have results.

<include src="TweetQuote" quoteText="Choose a page that has high traffic but low conversions, so that you're maximizing returns."/>

![Conversion-Traffic graphic](https://images.ctfassets.net/23aumh6u8s0i/Qr8MVveoPHIFnyydvD7No/acebffd65d8b957da32d073863a6c4a9/conversion-traffic-graph)

Source: [Unbounce](http://webinar.unbounce.com/analytics-for-conversions/?utm_source=Webinar&utm_medium=Blog-Post&utm_content=conversion-rate-optimization&utm_campaign=Hiten-Shah&_ga=1.218726042.2055948509.1470424864)

A/B testing one change at a time allows you to fully determine the effect of that variation. This way, you can combine the effects of multiple changes over time to optimize for the best page possible. But most people take this to mean they should A/B test minor changes to their site. Here we are going to show you why that thinking is wrong, and how big A/B tests allow you to get great data while making great changes to your site.

## The Problems With Starting Small

When deciding how to conduct A/B tests, Facebook knows to start big picture to have the biggest impact before even thinking about the smaller details. 

As we've mentioned, one of the most important [A/B testing rules](https://blog.kissmetrics.com/know-about-ab-testing/) is to test only one thing at a time. This way you'll make sure it's something that counts, rather than trying out multiple little things. 

The other problem with testing multiple small things like colors, small text, or call-to-action is that you can't be entirely sure what is really making a difference if at all. You're likely not making much of an impact. Conversion rates may be changed by a percent of a percent at most, if there is even enough traffic to make a statistically significant comparison.

![A/B Testing small changes](https://images.ctfassets.net/23aumh6u8s0i/5DKY6JTYpdFXnmnifrvPzQ/e8649f3fe04e8c14cc460f79befee649/small-changes)

For example, if we made the above changes — slight color and font resize — to the Auth0 homepage, it likely wouldn't make a difference. In fact, most people probably wouldn't even notice that we made a change. It certainly wouldn't effect the number of conversions.

When looking to A/B test, it's better to make big changes first to have the most significant effect. Small changes are good for optimizing further down the road, but those first need to be guided by larger-scale tests.

## Go Big, Then Go Small

Starting big will ensure that your A/B test will “[create results, not statistics](https://blog.kissmetrics.com/ab-testing-results/)” by focusing on the message you're trying to present, rather than the words you use to do so.

There are countless examples of impressive A/B testing case studies available online, but even in this compilation of “[10 kickass A/B testing case studies](https://vwo.com/blog/10-kickass-ab-testing-case-studies/),” the ones that made the biggest changes saw the biggest improvements. Testing the things that have the [biggest impact](https://blog.kissmetrics.com/ab-testing-introduction/) — such as headlines, graphics related to sales or product description — will present the biggest returns.

![Radical vs Incremental changes](https://images.ctfassets.net/23aumh6u8s0i/274U4RTIBWcC6dBDe6iW1h/2aaff2dce6c932fddb32eeb9adae6a06/radical-vs-incremental-changes)

Companies should also not be afraid to make radical changes to test a completely new idea. As shown in a study done by [conversion optimization platform Experiment Engine](https://www.experimentengine.com/blog/2016/08/02/what-exactly-are-companies-ab-testing/), radical changes are more likely to show impact. With more general idea testing, if the change doesn't work, it is easy to just not launch rather than having to re-do your entire page for a small change that proved unsuccessful. 

### How This Works For Any Industry

In any industry, starting [high-level](https://knowledge.hubspot.com/landing-page-user-guide-v2/how-to-ab-test-landing-pages) and testing the entire landing page can open many doors for things to A/B test further down the line. Making big changes can reveal [surprising and maybe even unintuitive results](http://unbounce.com/a-b-testing/surprising-a-b-test-results/) that you might've never expected. 

For web app automation service [Zapier](https://zapier.com/app/explore), completely changing their landing page gave them the boost they needed. Though they [received criticism](http://www.usertesting.com/blog/2015/06/09/homepage/) for having a simplistic landing page, this choice was actually a result of a surprising A/B test.

![Zapier Comparison](https://images.ctfassets.net/23aumh6u8s0i/45Yb6jXlGr0niIbVK2gg2N/5dc52e011faa3a6a278d2b30f474eb37/zapier-comparison)

They had compared the two above landing pages, thinking that the more informative page would lead to more conversions. Instead, they discovered that the simplistic page ended up leading to statistically significant more converted signups and active users than the other.

![Zapier Homepage Test Statistics](https://images.ctfassets.net/23aumh6u8s0i/6Tuu214p10GtTqUFjuUdJ/9e0028cfd2ff96553c5dd97388af18ab/zapier-homepage-test-stats)

Zapier isn't the only company to make big changes to their web page to find out what appeals most to users.

Health-related eCommerce store [Body Ecology](http://bodyecology.com/) added product categories to their homepage instead of a drop-down menu, [increasing revenue by 56.43%](https://vwo.com/blog/10-kickass-ab-testing-case-studies/).

![Body Ecology A/B testing in the home page](https://images.ctfassets.net/23aumh6u8s0i/3t6R6GvjUDFsA79ySbKCKJ/ce439588667b2a6f4dc49e8f70671d66/body-ecology)

Business training company [Dale Carnegie](http://www.dalecarnegie.com/) redesigned their main lead generation page to more clearly display the benefits of their course in a 2-column design. This change led to a [17.1% increase in downloads](https://blog.optimizely.com/2016/03/02/form-redesign-increase-leads/).

![Dale Carnegie A/B testing](https://images.ctfassets.net/23aumh6u8s0i/1tBodW8DxRQWGzhYrmmZzU/aa3d004ff198a3b81db4797523fafb3b/dale-carnegie)

[Spreadshirt](https://www.spreadshirt.com/), where people sell their T-shirt ideas online, [increased conversions by 606%](http://blog.wishpond.com/post/98235786280/50-a-b-split-test-conversion-optimization-case-studies) by simplifying the “how to” section of their website.

![Spreadshirt A/B Testing](https://images.ctfassets.net/23aumh6u8s0i/3Xd2Af2qYb3y8mPc5Td627/4b8bc0ca51c6af4ca53c70c8425a267b/spreadshirt)

By improving their value proposition and emphasizing a free new town, The Sims 3 [increased visitor registration by 128%](https://blog.kissmetrics.com/100-conversion-optimization-case-studies/).

![The Sims 3 A/B Testing](https://images.ctfassets.net/23aumh6u8s0i/3h8MrFLvXcaINrYQTmNdCO/87ee22bafd6b2a98d32021a7dbbb02ae/the-sims)

By making entirely new pages and A/B testing more general ideas, these companies were able to see a more immediate difference that they could then start to iterate and optimize. 

## How To Run An A/B Test 

There is a scientific process you can follow when [running A/B test](https://www.optimizely.com/ab-testing/) experiments:

1. **Conduct background research**: Use data from analytics to provide insight on what to test.
2. **Determine your goal:** Know what you're working towards and never test without a goal in mind.
3. **Construct a hypothesis:** Identify and prioritize reasons on why you think this variation will be more effective.
4. **Prepare your experiment:** Make the changes! Using a software platform can make creating these variations much easier.
5. **Run your experiment:** Run the A/B test. Make sure your visitors are split evenly and interactions with your site are measured accurately.
6. **Analyze your results:** Use the data collected to see if there was a statistically significant difference and determine how to move forward.

This process, similar to the experimental framework from [A/B Testing and Personalization Platform Optimizely](https://www.optimizely.com/), is very general and can be applied to any A/B testing situation.

![The A/B Testing process](https://images.ctfassets.net/23aumh6u8s0i/5hM22tci6A2Vr93fpdJWYx/c5bcd5937da3622aeb6df27450c884f4/ab-testing-process)

Following this process will make sure you are testing with a purpose and will learn something from the results, regardless of success. 

## A/B Test Everything — But Start With The Big Things

Starting your A/B tests high-level will give you the biggest returns quickly. By incorporating each successful change into the next A/B test, you will compound on these benefits to your conversions even more.

And the best part? If the change doesn't give the result you want, you can simply not launch the new page. Even if you get bad results, by testing, at least you know what doesn't work.

<include src="TweetQuote" quoteText="If the change doesn't give the result you want, you can simply not launch the new page."/>

The main takeaway message is that you are free to test anything on your website, so it's best to start big. You may learn that something you never expected is what appeals most to your customers.


"ab-testing"