”Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day…” – Jeff Bezos, CEO at Amazon
This is what Jeff Bezos had to say when asked about how much A/B testing his company does. Just in 2011, Amazon was already doing over 7000 tests annually and that number only increased over the years. Amazon isn’t the only company aware of the importance of testing constantly and basing their growth on data they get from tests. Companies like Google, Facebook, Microsoft, Netflix and Booking.com all understand the value of doing thousands of experiments per day, per week, per month.
They all have a testing strategy that prioritizes quantity over quality. Their goal is to run as many tests as their website and resources allow them, because they understand that whenever they are not testing, they’re wasting a chance to earn more money. They don’t spend a lot of time on hypotheses, research and analysis, but jump right in and immediately start testing.
Although companies like Google and Facebook are massive, small and mid-sized business can also benefit from a testing strategy that favors quantity over quality. Even if you do less than 30 tests per year, a testing strategy that pushes you to do 2 tests per month rather than 1 test every 2 months, gives you a much bigger chance to grow your business.
And here’s why.
Most of the ”A/B Testing Best Practices” and ”A/B Testing Common Mistakes” articles will tell you that you have to do a lot of research and analyze user behavior before you start a test. You’ll also be advised to never start a test without a hypothesis. You should take all that advice with a grain of salt. More often than not, those hypotheses are wrong and won’t lead you to finding any winning variations.
According to Bart Schutz, an expert in consumer psychology at Online Dialogue and the man behind Wheel of Persuasion, only 25% of tests are winners and the other 75% are wrong. This is just another way of saying that 75% of your assumptions are wrong. The only way for your company to grow is if you test as often as you can and rely on facts and findings from those tests, rather than your own test ideas. Companies like Google, Facebook and Microsoft know this and run tens of thousands of A/B experiments per year.
The reason why you can’t trust your own assumptions? You can’t trust your conscious mind. Essentially, your subconscious wants something completely different from what your thoughts are telling you. Consumer psychology explains that conscious control and self-control are very often just an illusion. The conscious part of the brain is the weakest part and it’s your subconscious that runs the show. This is why, very often, the most successful and profitable websites have been designed by our subconscious impulses. What we think we want is not actually what we want.
Let’s look at Amazon’s homepage.
If the users were asked to design the homepage, it wouldn’t look like this. It would probably be a lot sleeker, smoother, with a lot less happening on the page. But it turns out that numerous Call-To-Action buttons and pop-ups actually appeal to us.
If you asked your users in a survey what they thought about pop-ups, they’d tell you that they hate them and that they’ll buy less of whatever you’re offering because they’re so annoying. Well, turns out that pop-ups are extremely efficient when it comes to generating revenue and increasing conversion rates. You might hate the way it looks, but websites like Amazon’s current website drive more sales and traffic because they are appealing to the subconscious part of the brain. Your consciousness can only process a few things at a time, but your hidden mind is constantly firing at all cylinders, processing millions of operations every second.
Another company that is aware of this and has been working with consumer psychologists to optimize their website is Booking.com. They test absolutely everything on their pages. It’s no surprise that at any given moment they have around 1000 A/B tests running. This is what their website looks like after all that experimentation.
In order to figure out how to appeal to your user’s subconscious, you have to run as many tests as your test bandwidth will allow you. Your assumptions and hypotheses will only be a reflection of your conscious mind, and the only way to get to the truth about user behavior is by testing continuously.
Look at what some of the best and most profitable companies in the world do:
- Google runs an average of 100.000 tests per year;
- Netflix does around 1200 experiments annually;
- Whenever you visit Facebook there’s a good chance you’re involved in 10 A/B tests;
- Even back in 2011 Amazon was already running 7000 A/B tests annually.
They understand the value of experimenting and learning about user behavior, but do not assume they can predict what their customers want. These companies are basing their growth on hard, cold evidence and it’s been working for them so far.
The catch here is that, in order to run as many tests as your company resources and test bandwidth allow you, your testing process needs to be lean and agile. To test efficiently, you need to have the capabilities and resources, and your test turnaround time has to be short. You can’t spend weeks planning a test, designing and preparing it.
If you get bogged down with doing research, creating models, analyzing user behavior and perfecting the design- you’re wasting time and money. The only time when you should be doing this is if you’re already running at full speed and testing all the time. If you’re just starting out and have not used up all your testing bandwidth, don’t care about any of this and start testing right now. An experiment done is much more valuable than a perfect experiment.
Rather than agonizing over a hypothesis, go for the rapid and risky idea. Assume quickly what could help improve your website and test that. The higher the risk you take and the more you change on your website, the greater the chance you’re going to see an increase in your income. If you want your changes to make a big impact on your growth and income, you have to test disruptive and bold ideas. Make rapid and risky assumptions and don’t spend days planning your tests. The idea here is that you move fast and adapt faster, and that you keep your testing flow as lean and agile as possible.
Any moment when you are not testing is a wasted opportunity to earn money. You should always aim to use up all your test bandwidth.
If you’re running 2999 tests annually, but know that your testing bandwidth and capabilities allow you to run 3000 tests- you missed out on one and that’s a wasted opportunity to test something that could grow your company. You want to think about how fast you’re moving, how many tests you’re running and how many you could’ve ran. By not getting bogged down in research and hypotheses, you can look at the data your tests are giving you and quickly adapt, finding new elements you want to test.
You want to exploit your bandwidth and never have any regret because you missed out on testing something that could have a major impact on your company’s growth.
Consumer psychology has known for a while that our conscious brain doesn’t really control anything and that it’s the subconscious that determines when we’ll decide to click on that CTA button. Luckily, experimenting with your website through A/B tests allows you to hone in on what your users want and optimize your website.
If you have the testing capacity, the Evidence Based Growth model, psychology and best practices of most successful companies prove that the worst thing you can do for your business is not test at maximum capacity. Even a bad test has value and will bring you closer to more revenue and higher CRO.