Do you have an idea for your business that you think might be great? Do you need to convince yourself, your boss, or other stakeholders whether or not to proceed with an idea?
A/B testing is a very useful tool for calculating a business case for anything you are considering changing or implementing afresh on your website.
The aim of A/B testing should always be to either prove or disprove a business idea, but at the same time, you don’t want to waste a whole lot of effort on validating bad ideas. What makes a bad idea? From our perspective, it’s something that doesn’t give you any sort of significant uplift. It’s those minor changes that don’t produce anything of statistical significance, or that takes away from focusing on bigger issues.
We wouldn’t include a major test that failed as a “bad idea.” In these cases, you can genuinely learn something valuable from the exercise, but you really can’t from those minor changes.
Calculating a business case really begins with good hypotheses and following A/B testing best practices. You need to ask the right questions before testing so that you’ll understand what might produce the most meaningful insights.
Know your baseline
We’ve talked about the need to gather some baseline data previously. You have to define a beginning point in order to understand where you’re aiming to go. A/B testing is a science – you want to base business decisions on hard numbers rather than feelings. Knowing where you began helps you to justify the business case or not.
What goes into your baseline data? Here are a few points to consider:
- The composition of your customers or target audience.
- Definition of KPIs for your business. You should include target metrics here so that you’ve defined what “good” and “bad” looks like.
- Know what is happening with your website currently. How are you doing against website goals and business objectives?
- Know what your customers think. It may be helpful to gather feedback from key segments.
Know what to prioritize
It’s hard to make a strong business case for something that’s insignificant in the scheme of things or has a high cost to implement. You can help this along by understanding what should be prioritized.
On a website, top priority should always go to the most highly-trafficked pages. Once you’ve identified those, look for the pages with the most potential for improvement. Look to your analytics data for information like landing pages with high bounce rates.
You also need to take a holistic view of your website. If shopping cart abandonment is a problem, driving more traffic to the cart probably won’t help. Optimizing the checkout process may be a logical step.
Once you know what your issues are and the relative KPIs they impact, you can devise hypotheses for testing. Of course, you’re looking for anything that has the potential to provide a significant uplift, but when it comes to justifying a business case, cost probably matters too. Will the company have to invest a significant amount into making the change if it proves to be a winner? Prioritize for high-value, low-cost tests first, then look at other tests which may cost more to implement, but potentially are of high value.Prioritize high-value, low-cost elements for A/B testing first Click To Tweet
Understand the factors that impact your results
If you really want to make a strong case for a change, you need to understand the factors that can impact your results. Sometimes you may get a false positive or a false negative. It’s important to understand a couple of important terms:
- Validity. In A/B testing, validity looks at whether there are factors, besides your sample size, that are negatively impacting your data.
- Statistical significance. This is the degree of confidence to which your test results are considered reliable, rather than up to chance.
Statistical significance begins with knowing your baseline data. You also need to have an appropriate amount of traffic to be able to conduct a significant test (see our article on what to do if you’re too small to A/B test here). The standard accepted level of “confidence” tends to be 95%.
Validity can be impacted by a whole host of potential factors. For example, perhaps your test result appears significant, but there is something else going on affecting validity. What might make a difference? Here are some examples:
- Seasonality. This is a common impact. Maybe you sell products that people seek more at certain times of the year. Perhaps it’s the holiday season and this skews results. To avoid this, gathering data over a six month impact period can help.
- The length the test is run. A common mistake is to call tests too soon, claiming statistical significance. Sometimes that’s due to some kind of “noise” that impacts validity. Results that are just blips will usually even out over time if you keep running the test. Don’t pull them too soon!
- The “novelty effect.” This is a phenomenon where repeat traffic to your website is drawn to a change you’ve made because it’s something they’re not used to seeing. One way to combat this is to segment traffic and drive only new traffic to the variation.
- Random occurrences. There is any number of things that can occur at random that impact your test results. The weather, internet outages, press mentions, downed servers or a bug on your website can all impact validity. The treatment would again be to run tests for longer to help “even out” those impacts.
An Intercom article talks about looking beyond statistical significance to “statistical power,” or the likelihood that you will recognize a winner. There are three questions they ask to gain enough statistical power and run a test correctly:
- How much do you think the change will increase the associated key performance indicator (KPI)?
- Given this desired effect, how long will you need to run the test to get accurate results?
- Is it worth the wait?
These are definitely worth considering. If we take the example they use of a website redesign versus a different color scheme, development time is longer for the redesign than the color scheme. However, testing on the complete redesign could be done in two days to see significant results, compared to 49 days for the color scheme change.
Know what you gained from the test
A key part of A/B testing is interpreting the data that you get. One way to look at it is that there are very few tests that are actually “losers.” From our perspective, a losing test is one that was not set up well and the results can’t be relied upon. A test that was well put-together but didn’t come out a winner is still a valuable learning experience (or potential money-saver for your company).
A business case looks at how the company profits, so here are some ways to determine that from testing:
- You learn valuable information about your customers. If a test didn’t work out, you may have learned something you can use in further campaigns or testing. There may be a business case for another type of change, or not to change at all.
- Look at any “harvested revenue.” This is revenue directly gained from the test variation.
- Look at results within segments. Something that may appear not to work overall might be a winner within certain customer segments.
- Look at any shifts in the metrics that represent your KPIs.
Implement and monitor
Once you have a “winner” it’s important to implement it and then keep monitoring it. Run new tests on top of the implementation and see what can be done to optimize it further.
When you’ve prioritized the changes with the biggest uplifts, you can look to the smaller things that will create some uplift. Besides that, you can always keep a backlog of tests to try re-running. Sometimes you’ll be surprised to find that something that previously didn’t work provides you with great results when adjusted and re-tested. Maybe the change you implemented which you’re now monitoring has some kind of impact on those old tests.
As testing experts will tell you, no matter how well your landing page or website is doing, it could always be doing better.
The bottom line of calculating a business case with A/B testing is that you’ve got to have reliable data. This means you need to;
- Gather baseline data
- Prioritize key pages
- Set tests up well for statistical significance, statistical power, and validation.
- Interpret your results correctly.
By having a robust process in place, you’ll have accurate results to present a business case that will result in the likelihood of being approved to test further and provide more cases later on.