Why A/B Testing is Suspect – Read This Before Signing Up
If you’ve been in the market for digital marketing services, you’ve probably come across a few A/B Testing (aka: Conversion Optimization) sales pitches. These two dynamite-drop-in terms seems to be all the rage for those offering internet marketing services lately; mainly by those who have no clue what they are doing.
Why is this? Because A/B Testing is:
- Easy to explain and understand
- Makes you sound like an expert regardless if you are or not
Let me give you a quick rundown about A/B Testing to help you decide if it is right for your business or not.
What is A/B Testing
A/B Testing is when you compare one version of a webpage to another*. More specifically, how Version A converts** website visitors compared to Version B. You keep the winner, and, as a result, optimize your conversions.
The difference between Version A and Version B may or may not vary much. Variances could be anything from rephrasing a headline to changing the color of a button to a completely different page design. All depends on what you want to test.
Why A/B Testing is Easy to Sell
It is this easy… “By changing a single word on your website we increased your conversions by 20% in just 1 week!”
Version A
Free Evaluation
Sales 20%
Version B
Free Consultation
Sales 0%
You get it. I get it. We all get it.
You’d have to be a genius to pull something like this off, right?! And, of course, you’ll be willing to shell out more cash because they just made you 20% more!
I see this almost exact same pitch on Linkedin and Facebook posts all the time. I’ll bet people buy it. You should do your homework before doing so… here’s why.
Why A/B Testing is Suspect
Here is a example as to why A/B Testing may have little to do with conversions and/or sales.
| Date | Visitors | Calls | Sales | Revenue |
|---|---|---|---|---|
| May 23 | 958 | 143 | 40 | $12,272 |
| May 30 | 1,048 | 138 | 34 | $9,256 |
| Jun 6 | 868 | 144 | 26 | $7,356 |
| Note: This is actual client data from 2017 | ||||
Few comments:
- All dates are Tuesdays, for a week-to-week comparison.
- In this case, the website’s goal is to drive calls that lead to sales.
- Jun 6 had the fewest visitors, the most calls, the lowest revenue
- Notice the ~$5,000 difference in revenue in just two weeks time
So what changed A/B Testing wise? Nothing.
In this case, the issue was consumer behavior; one of several unpredictable variables that can throw a serious wrench into your A/B testing data. You could end up making change after change that did nothing but waste your time, energy, and money.
Does A/B Testing Ever Work
Yes it can work… but we only recommend trying A/B testing for high traffic / transaction sites.
If you’re a small business and/or do not have many visitors or transactions on your site, you should focus your conversion-optimization efforts elsewhere – and/or – try to increase your website’s overall traffic.
And certainly do not fall for a sales pitch like the one above.
*The term A/B testing may also be used to compare different ad sets, mobile applications, etc. For the purposes of this article, we are only refering to webpage A/B testing.
**A conversion is when a website visitor executes the goal of the web page. This could be anything from filling out a contact form, signing up for a newsletter, downloading an app, or making a purchase on an ecommerce site.