Have you ever thought about questions of morals or ethics when it comes to A/B testing?
Ethical A/B testing is an issue that has hit the spotlight in some prominent cases. The bottom line is, whenever we are conducting experiments, they involve real people, and as such, it’s important to protect them and adhere to ethical principles.
We know an ecommerce business doesn’t set out to do anything questionable, it’s all about testing to deliver an optimal experience to users. Your hope is that by doing so, you achieve better results with business goals.
For most people, the idea that something may not be ethically right doesn’t sit well, so what exactly does this entail with A/B testing?
How do ethics fit into A/B testing?
As you scroll your Facebook timeline, you come across a post that captures your attention. In fact, the headline downright pushes your buttons! You give the post an “angry” reaction, perhaps you even voice your displeasure in a comment.
If you’re on social media, you are potentially part of any number of A/B tests. Sometimes they’re more innocuous (“if we rearrange the elements of the newsfeed, how will users react?”), but other times they are designed to elicit an emotional response (“if I write a headline this way, will the post get more engagement?”). Or worse, “can I manipulate how people feel?”
A/B testing itself is not inherently unethical; however, some uses of it are. A famous example was an experiment conducted by a Facebook researcher. Published in 2014, the paper outlined how 689,003 Facebook users had been randomly selected for a study about emotional contagion. They hypothesized that emotional states of users would spread through the network, depending on the tone of the content they were most exposed to.
The researcher ran two experiments in parallel; one where users had reduced-positive exposure to posts, and one where they had reduced-negative exposure. The results were that those who saw less positive posts had a larger percentage of negative words in their own status updates, while the opposite result was true for those who saw less negative posts. Emotional contagion was at work.
While the emotional impact on users was not considered to be great, this experiment stirred up a lot of controversy around the ethics of testing in this way. Facebook saw it as another newsfeed A/B test, much like they conduct regularly, while industry watchdogs considered it to be a gross manipulation of human subjects.
Social media experimentation remains a hot topic across the world. In the case of Facebook, the implications of data use and manipulation of the platform have gone before the U.S. Congress. It’s somewhat easy to see where things might go into murky territory on social media, but are there some clear lines we can draw with A/B testing as a whole?Ethical A/B testing means respecting the human test subject and their needs Click To Tweet
Where is the ethical line with A/B testing?
This is not an easy question to answer, but here are some points to consider that are regularly debated among researchers:
- Informed consent. This is at the basis of ethical academic research but is less often part of A/B testing for businesses. The general consensus tends to be “as long as you’re not doing anything illegal, you can do what you like without informed consent on your own website.”You could have something in your website terms and conditions, but how many people pay attention to those? Another form of informed consent is when you explain to people that you’re testing a new version of your website and allow users to toggle between versions. This is more for a huge change though, not for small tests such as a change to page layout. Another idea is to include a line on your website or in your newsletter along the lines of; “You may notice some changes on our website – we are testing to improve your experience.” This way if they’re still using the website, you could infer that they have consented.In practice, most companies don’t get informed consent for regular A/B tests – the rationalization tends to be around the outcomes of those tests. There’s also a real risk of bias in test results where people know they’re part of an experiment.
- Potential outcomes or impacts on the human subject. Some things are obvious when it comes to drawing ethical lines. As Ed Felton discusses, if you were manipulating information, leading to inciting violence or to negatively impacting a subject’s mental state, this would clearly be unethical.What if you are conducting A/B tests on your ecommerce website around generating urgency among visitors? If you test out countdown timers that aren’t really true (you’re going to give them the deal anyway) or stock counters that always show “only one left!” is this unethical?This is an argument that is very subjective to the beholder. Some people would argue “but everyone knows that companies use tactics like that,” while others would say you’re lying, therefore it is unethical. Did the user come to harm when they decided to make the purchase? Probably not – as you can tell, this is a can of worms!If we compare the sort of A/B tests you do on your website with the test conducted by Facebook, there is a fairly clear point of difference. You can argue that while tests on your ecommerce site are designed to make user experiences better, the Facebook test was purely psychological. Is this a good ethical line?
- Data protection. This is a huge deal related to A/B testing. Ethical principles attached to testing on human subjects, such as those laid out by the Nuremberg Code place an expectation that subjects agree to hold data about them.In most cases of A/B testing, you’re not looking at people on an individual basis, although it may be possible for you to dig down and do so. What you do with their personal data matters. This includes protecting it from breaches. The Facebook-Cambridge Analytica scandal is a good example, where illegally obtained data was allegedly used to influence political outcomes.
What should you consider for ethical A/B testing?
If you’re online, you’re bound to be part of any number of A/B tests without ever knowing it. In fact, in defense of their own A/B testing scandal, OKCupid’s response was; “We experiment on users. Everyone does.”
It’s true that A/B testing is everywhere and most of the time, no one raises an eyebrow over it. Did altering a page layout change your life? Unlikely, however, it may have changed the fortunes of the company conducting the test.
So, assuming that you don’t want your own business to be involved in the next A/B testing ethical scandal, what should you consider?
Are there any real risks to the human subjects of the experiment? Could it cause them harm in any way, whether physically or psychologically? For example, let’s say you sell health products and experiment with the copy that you use, implying that a product will cure a serious condition. This is not only unethical (and illegal) but could cause significant harm.
What are the potential benefits, both for your company and for the user? Is the user going to get a better overall experience out of the results of the test?
Choice and privacy
This is where that “informed consent” comes in. It’s fair to say that an experiment such as the one Facebook conducted, where there is a risk of harm to the user should involve informed consent.
A Hackernoon article also provides three suggestions for communicating your use of live testing:
- Clearly specifying an A/B testing section in a Terms of Service.
- Publish an ethical statement and guiding principles of A/B testing.
- Visually indicating an A/B test is being performed with opt-out.
If there are any vulnerable populations involved (children, economically disadvantaged people etc.), many suggest that additional safeguards be placed around experimentation; for example, checking in with test subjects to check that the user is safe or well. This pertains more to any test where there may be some risk.
Additionally, the privacy of the user and the protection of their data should be a priority.
Where does that leave us when it comes to ethical A/B testing? As far as testing goes, businesses operate within a different context to academia, and so they should. While strict constraints are necessary for academic testing, businesses operating in the free market don’t have the same level of rigor applied.
Perhaps it’s fair to break it down to a few simple questions:
- Does this test have the potential to cause any harm?
- What is being done with the data gathered?
- Should users have a reasonable expectation of being informed?
In the end, A/B testing is performed on people. If we respect the human element, we should be able to devise good tests that fall well within ethical boundaries.