Here are some common biases you fall for during User Testing & practical tips to overcome these before you begin the process:

1. Think of the Method – The Medium Bias

The method you use to test your designs has a direct influence on the end result. Each medium carries its own positives & biases. Conducting remote studies will allow you to test a lot of participants (as compared to in-person sessions), but it also tends to fetch participants who are tech-savvy. But does this method match your product’s target audience? Are you building a product for people who are digital natives? Or is it being designed for first-time web users? Depending on the answers to these questions, you must choose the most appropriate method for user testing (and yes, budget & timelines are crucial variants too).

2. Think whom you’re Testing on – The Sampling Bias

Now that your medium is defined, who are your participants in the study? Their geographics, demographics? Are they a set type, i.e. all of one gender, similar age & income bracket? If the answer is YES, you know you need to build some diversity into your participant sample (or else risk sampling bias!). When you user test, you want diverse viewpoints to represent your real, diverse end-users (unless the product is for a super niche audience). One way to do this is demographic diversity: people from different socio-economic backgrounds, age groups, levels of education, etc. Another way to do it is domain diversity, i.e. test some people who are very comfortable with that domain, as well as people who know nothing about it.

For example, while user testing a food delivery app, a person who orders food everyday will have a different review of your app, versus someone who barely orders.

3. Think how you Interact with the Participant – The Moderator Bias

This is for the moderator who is conducting the test while it is in progress. Leading questions are bias-generators. A participant’s response to “Tell me what you like about this design” will differ from “How do you find this design?”. The former is more likely to get you positive feedback on your design, which is great to hear, but the latter prevents you from unduly influencing your participants, and hence maintains the accuracy of the responses you get from users. Another way moderators influence participants is through – Time Pressure. If you quickly rush through questions or tasks, you aren’t giving your participants time to think and verbalise what they feel regarding the product. Plan timelines for yourself (and the participants) to do justice to the process.

4. Think of your Result – The Confirmation Bias

If you’ve come to the interpretation stage, we know you’ve been through the grind. It is important at this stage to look at your results with no preconceptions (and we mean ANY!). If you only amplify results that confirm an existing belief, while ignoring facts that contradict it, you are committing Confirmation Bias. During user testing, you may be surprised to find that your prized feature is not being used, or something you were sure was broken, turned out to actually be useful to users. All in all, don’t let your love for the product get in the way of deriving real insights from the testing.

Takeaway:

Remember, the purpose of User Testing is to find out things we never knew, or to update our beliefs about the product, not to just confirm them! Recognize the possibility of biases & work around them.