When people think about user research, their image is often of our research lab. This is a room with a computer where real users complete normal tasks using our prototype website or software. Members of the product team observe the session live from another room, to learn from the user’s behaviour.
This lab-based ‘usability’ testing is part of what user research is, but not all of it.
So, I’d like to explore the attraction to usability testing in a lab, but also explain how we decide to use a particular research method.
Why do teams want usability testing?
Usability testing can be an enticing offer for product teams. It’s convenient as the users come to you, and they're all booked in for one day, which makes it easy to observe.
Team members get exposure to real people who use Parliament's services, which ‘feels’ like user research should feel. It’s also really interesting to watch, and observing sessions can sometimes feel like a break from ‘real’ work. However, none of those reasons are the right reason to go ahead with lab usability testing.
The most important thing to consider is what the team needs to know to inform their decisions.
On closer inspection, lab usability testing is often not the most efficient or appropriate way of answering questions. For example, this kind of testing takes a long time to organise, removes participants from their natural context, and is expensive to run. Sometimes you can get better quality answers through other means.
How do we decide which method to use?
To pick an appropriate method, we’ll start each round of research with a team session defining research questions. This helps product teams decide what information they need to discover to inform their decisions.
It’s very important that the team agree what they want to learn. Otherwise it’s not possible to have a sensible discussion about how we want to answer those questions. We’ll then match the research questions with our understanding of potential methods to pick the right one.
There’s a huge range of potential methods out there. The one we decide to use depends on a complex set of factors. We make sure we’re answering the team’s questions, but the final decision is shaped by other factors too. Factors like time, budget, staff availability, and how representative the prototype is of the final product or service.
Different methods at different stages
Broadly, there are some common trends about which methods are appropriate in each development phase. For discovery, at the start of a project, the questions may include:
- who are our users
- why do our users do x
- how do our users do x
Some methods for answering these questions include observing users, interviews, and looking at analytics and data sources, such as call logs and complaints.
For alpha, when exploring different things that could be built to meet user needs, the team’s questions include:
- does this help our users do x?
- is this a good way of doing x?
To answer these, it’s often appropriate to look at prototypes. This can be in the lab, but could also be less formal methods, such as guerrilla research, paper prototypes, or other ‘quick’ research.
And later on in beta, when the team knows what it’s building, they may be asking:
- we made this and we think it does x. Does it?
- can people understand and use the thing we’ve built?
Usability research is often the best way of tackling this, and again this can be in the lab. However, we also use unmoderated remote tools like ‘What Users Do’ to get answers quickly (assuming the audience is appropriate).
We can get insight from analytics and use techniques like A/B testing. These help to validate and improve the team’s decisions once we start getting real users using the service during this phase.
These methods may come up more commonly in certain phases, but they're not hard and fast rules. The context and setup of each team will need a careful selection of appropriate methods.
Making the right research choices
As we've seen, lab usability testing isn’t the only method of running research. Our researchers work with their product teams to help them make the right choices about how to run research. This is all based on what the team needs to know at that time. Research isn't just in the lab.