skip to Main Content
5 Reasons Why You Shouldn’t A/B Test In-House

5 Reasons Why You Shouldn’t A/B Test In-House

So you’re looking to optimize your ecommerce website. You’ve done a bit of research and you know that you should A/B test to discover the most effective elements for your website.

In fact, if you’re willing to spend a bit of time looking around, you can find all sorts of articles, tutorials and how-tos, delving into the world of A/B testing. Many businesses start out by trying to do it themselves in-house, but the question is, should you?

Perhaps. But there are a lot of contingencies you need in place to do so effectively. For many businesses, this simply isn’t practical and therefore, testing in-house is a bad idea. Here are some reasons why you might want to avoid taking on A/B testing in-house:

Get our quick guide here for choosing an external A/B testing company

#1. You need to invest in A/B test expertise

You can read up on A/B testing, take courses or hire team members who are experienced testers. In any case, A/B testing effectively requires expertise among the people doing the testing.

In fact, A/B testing when done well involves multiple segments within your business. The most successful internal testing teams really are just that – teams! For example, you might involve UX designers, graphic designers, test designers and of course, people who are skilled analysts of the tests.

Not only do all of these people need to understand principles and setup of A/B testing, but they may need to decipher complex software tools to help them do it. They must have knowledge of statistical principles and an overall understanding of what makes something a valid or good test.

There are many pitfalls associated with A/B testing which are more commonly fallen into by people who try to set up tests without having proper expertise. For example, you can expect many tests will not yield results, but an expert tends to have a better “win” rate on tests. Why? Because they understand the pitfalls in the first place and design tests to try to avoid them. They understand aspects such as how the timing or duration of the test can impact the results, or how to generate a good hypothesis.

This is where you need to look at what gets the best “bang for your buck” in your business. Is it worth you hiring internally for A/B testing expertise? Or, would that money be better spent elsewhere? You have the option of hiring an external team that is already set up and has the expertise for A/B testing, so it may be false economizing to pour investment into internal testing.

A/B test

#2. You need the right systems

A/B testing cannot be conducted without the right infrastructure in place. You will need a good internal testing system in place, and most of these cost a significant amount of money. Some large companies build their own system, particularly when they’re big enough to support a constant internal testing program (think Netflix or Facebook). However, doing this is the most costly outlay of all.

When we look more closely at some of the SaaS-based systems which you can subscribe to, they range in price, but they also tend to involve a large learning curve for users. Testing is never as easy as “plug and play,” not if you want to test things that will actually have a significant impact. This means not only do you invest in the system, but you’re then spending a lot of time on getting yourself or team members trained to use it.

Again, it’s worth weighing up the costs here. Can you justify the expense of an internal system, or would it be more prudent to leverage an external company with existing infrastructure?

You need to invest in the right systems if you’re going to A/B test internally Click To Tweet

#3. Testing properly takes a lot of time

One of the key mistakes made by companies when they run an A/B test is trying to rush it. Everyone wants results and they want them yesterday, so they think that a shorter timeline will do for testing.

This is not the case. By halting the test early, you can miss out on genuine results. We’ll commonly see something like a company getting excited because, after three days, the test seems positive. It’s tempting to pull the plug here and call it good, but what if that short period represents a blip? Testing for longer helps to even out any potential blips and give a fairer result. This helps you to avoid basing your actions on a test result that may not be accurate.

Another key point here is that tests fail. Sometimes they’re an outright failure, but other times they require tweaking and re-testing. Or, you may need to come up with alternative tests to see some results. All of this adds to the time required to A/B test effectively.

The question is, do you have time for all of this in your business? If you’re going to commit to A/B testing internally, then it should be done properly. There’s no sense in forging ahead if you’re going to want to shorten tests or give up when a test fails.

#4. You don’t want to spread resources too thinly

Let’s say you don’t have the budget to hire testing experts and expand your internal team. As an alternative, some companies will add A/B testing to the job roles of current team members.

Most often, the bulk of the workload will fall to developers who you already have in-house. They will now need to spend time on ideation, planning and implementation of tests. They might need to build new programs or even a from-scratch solution.

In smaller teams, this can easily end up with resources being spread far too thinly. You might need your developers to be working on ensuring that any bugs are fixed on your app or website for example, but they’re too busy working on testing.

Before making any decision to test in-house, assess the priorities and workload of your team members. Can they really take on this extra task, or will something else have to give?

A/B test

#5. You could endanger user experience

Your website users are the bread and butter of your business. In fact, the heart of A/B testing is about catering to user preference, so that they are more likely to take the actions you would like them to. So, what if user experience were to go horribly wrong?

During A/B testing, there can be glitches or incidents that impact load speeds, or even the overall usability of the aspects of your website. The ultimate horror can even be your website going down.

Internal teams would have to be working around the clock to ensure that websites remain usable and that there are no bugs introduced by testing. Just a small reduction in load speed could see you lose customers, as can features that no longer work on your website.

If you don’t have an experienced or dedicated internal team, this can be problematic. In the event of anything going wrong, you need your website to be operational again as quickly as possible.

The difference with hiring an external team is that they have the experience and the contingencies in place for such events. They’ve probably handled many issues before and know what safeguards to set up in the first place.

Download our quick guide to choosing an A/B testing company here

Final thoughts

Of course, you could opt to A/B test internally. If you have the resources, you can invest in infrastructure, team members and the time needed to set up testing effectively.

The major downside for businesses, especially if you don’t have a large team or huge resources, is that A/B testing does take a considerable investment of time and money. Do you really have the capacity to support proper testing?

If not, then hiring an external agency is a great option. You can leverage their experience and resources, leaving your team to get on with their core roles. Which will you choose?

Back To Top

Testing.Agency has a new name: Team Croco.

With this new brand, we focus even more on CO-operating on your CRO strategy. (Get it => CRO-CO )

Of course you can expect the same speed and quality as before, but we also commit to support you in the rest of your testing program: planning & prioritising, design, development, implementation, reporting.

Our goal is for you to run a consistent A/B testing program with minimal effort.

Want to know more?

Request a free demo and consultation!