Usability testing – the solution to bad design. Or is that so? It’s hard to deny that usability testing is good for evaluating how effective an interactive user experience on a website is, i.e. whether it is easy to learn and use, being effective for users to achieve their goals of using the website, and being an overall engaging and pleasant experience for users. But yet, there is just one thing that people sometimes fail to realize. Usability testing like any other science experiment, can be affected by external factors that contributes to biased results.
Yes, usability testing is biased.
No matter how “real” you emulate the environment for the usability testing of a website, it is still an artificial, controlled environment. It is at best a simulation of real life. Let’s get a better understanding of how reactivity bias can underlie usability testing:
Reactivity bias
Reactivity occurs when participants alter their feedback because of the awareness that they are being observed for what they do. It can also be present in situations where participants alter their feedback or behaviour to conform to the expectations of the experimenter/observer.
There are a few forms of reactivity bias and it is largely grouped under two categories – the observer effect and the experimenter effect.
The observer effect
“You and all those people online and in the next room are watching my every keystroke, I’m going to be more vigilant and determined than I ever would be to complete those tasks—I’ll even read the help text.”
Img src: http://www.widerfunnel.com/wp-content/uploads/2013/02/Hawthorne-verbalizeYSTT.gif
The tendency to exhibit a slightly different behavior when one is being observed is called the Hawthorne Effect, where participants become self-conscious of the presence of the experimenter.
Social desirability is also a form of reactivity bias whereby participants have the tendency to answer questions or give feedback in a more positive manner than it actually is. This is especially so when they know they are being observed and will usually want to be viewed favourably by others. This may end up with them over-exaggerating the positives of a web design. It might be that they do not want to seem like they do not know how to use the website, or they simply just tell you what they think you want to hear. Whatever the reason, it makes it hard to distinguish between participants who are responding factually, and those who simply distort their feedback in a positive manner.
Img src: http://www.usabilitycounts.com/2013/08/20/five-things-ive-learned-from-usability-testing/
You then have another situation where participants observe and evaluate beforehand what they think is demanded from them for the outcomes of the usability test. This being called “Demand characteristics” refers to participants forming an interpretation of the experiment’s purpose and its desired outcome, which may result in them subconsciously changing their behavior to fit that interpretation. You then end up with your participant giving overly keen answers on how great the website works, instead of how it actually meet or doesn’t meet users’ needs.
The experimenter effect
“In the late 19th century a horse known as Clever Hans astonished audiences with his ability to stamp responses to arithmetic questions. He did this with startling accuracy. His infamy grew until an investigation determined that Hans was, in fact, only able to respond to subtle cues in the body language of his trainer.”
This phenomenon is a significant example of how the experimenter can largely affect the reliability of a usability test – both in the potential flaws of the test design, and also the bias the experimenter may unknowingly bring into the test.
Experimenter effect occurs when the experimenter unconsciously communicates his/her expectations to the participants. It could also be a case of confirmation bias where the experimenter interprets results incorrectly due to the tendency to prioritise responses that conform to his/her hypothesis, and overlook others. This is all a result of experimenters conducting the usability test with an idea of how they expect or desire the feedback to be. For example, your website may have a particular function which you are very keen in introducing to your users. This may lead you to interpret the usability of the function to be much better that it actually is, even if very few respondents actually expressed it to be so.
So, now what?
And so, the result of all this? With all these biases, it’s likely that you can get skewed usability test results, with more people saying that your website design is great, than the actual website views you get.
It may be hard to eliminate such biases, but we can always take measures to reduce them. You can start by emulating as best the environment users are normally in when they need to use your website. Also, as an experimenter, take careful note of your own expectations and beliefs, and be aware overall about how your decisions can frame the test design.
With that said, usability testing can generate useful results that are of most value if handled well. But it should not always be taken as a rule of thumb or yardstick for great website usability. There is never a foolproof test. If we are not careful in understanding the context that the participants and we ourselves are in, it’s highly possible that the results from the usability tests will be a flop – one in which it becomes a test merely for confirming our own beliefs.
References:
Alla Kholmatova (2014). Collaborative User Testing: Less Bias, Better Research. Retrieved from A List Apart: http://alistapart.com/article/collaborative-user-testing-less-bias-better-research
Robert Hoekman JR (2009). The Myth of Usability Testing. Retrieved from A List Apart: http://alistapart.com/article/the-myth-of-usability-testing
Rob Kerr (2012). Better Experimental Design for Better User Testing. Retrieved from UX Booth: http://www.uxbooth.com/articles/better-experimental-design-for-better-user-testing/.
Jeff Sauro (2012). 9 Biases In Usability Testing. Retrieved from MeasuringU: http://www.measuringu.com/blog/ut-bias.php
Vicky Brown (2014). The active role of participants and facilitators. Retrieved from Nomensa: http://www.nomensa.com/blog/2014/active-role-participants-and-facilitators
Patrick Neeman (2013). Five Things I’ve Learned From Usability Testing. Retrieved from Usabilitycounts: http://www.usabilitycounts.com/2013/08/20/five-things-ive-learned-from-usability-testing/
CS2024/COM224 
You must be logged in to post a comment.