We’re lucky that at PDS there’s a lot of enthusiasm for research, and a huge demand to learn more about the users of our products and services. Just running research isn’t enough though. It’s important to make sure that it’s reliable, has an impact, and that product teams are ready to react to the insights learned from it.
I’ve made many mistakes as a researcher, and I learned through those mistakes that some issues a team faces can’t be overcome by enthusiasm alone.
In this post, I want to explore some of the challenges teams face that prevent useful research from happening. I also want to share some of the techniques we’ve used as researchers to help overcome them.
Why we run research
Our researchers run user research to help product teams learn about their users, and give the team confidence in their decision making. But that confidence, when not backed by rigorous and reliable research, is dangerous. It may lead to misplaced confidence in poor decisions which can be worse for the team than doing no research at all.
This is a particular challenge for user researchers. Product teams don’t have the depth of experience in research that professional user researchers do. They're also unlikely to notice that research isn’t having an impact. This means they won't raise concerns about the quality or the impact of research. Because of this, user researchers need to be vigilant and look out for doing research that looks good, but isn’t improving the quality of the final product.
To avoid this, I have some questions that I think are important to consider as a team before running a round of research. They can help make sure the team are in a place where the research is useful.
1. Does the team have things they want to know?
The point of user research is to learn things. If the team has nothing they need to know, they don’t need research. Instead, they should focus on thinking about the problem they’re trying to solve, and identify what obstacles they have to just getting on with it.
It’s part of a researcher’s role to help teams uncover and capture what they need to know to progress. But if a team is scheduling testing because “testing is what we do”, and have no real research questions, there’s an issue.
To overcome this, we’ve explored having a dedicated session to uncover research questions with the team, agree on them, and verify that they’re valuable things to learn before scheduling testing. This can make the team more confident that the research is directly relevant to what they need to know.
2. Does the team know what they’ll learn already?
If the team know what they’re going to learn from research before it happens, research isn’t appropriate. It implies that the problem the team are facing isn’t a lack of knowledge, it’s that something else is preventing progress.
Evidence from research does help make a convincing case to persuade people and get things moving but it’s also a very expensive way of doing that. A delivery manager can identify and overcome blockers, helping the team progress without spending the thousands of pounds that a round of research can cost.
Research insight can also be useful when the issues are obvious, without the need to test. Techniques like usability reviews, or unmoderated research can do this quickly, and cheaply.
3. Is the team working to a deadline created by testing?
There’s often a temptation in teams to schedule research as a deadline to get some work done. This is a bad idea. It’s not using research for its intended purpose to learn. Testing therefore becomes a project management tool.
This can also occur when testing is scheduled before the team have confidence that the ‘thing’ they want to test will be ready. This is often due to poor scheduling, when teams haven’t been realistic about the time they need to deliver.
When the schedule is unrealistic, the whole team is under a lot of pressure to prepare at the last minute. This can lead to delivering things that don’t represent the team's intended product, making research pointless.
Scheduling needs to be balanced. It needs to avoid a long delay between ‘we have a thing that needs insight’ and research starting. Especially as finding real participants takes time too.
There’s no quick fix to this unfortunately. As a team matures, they can start to anticipate how long it needs for appropriate iteration, and testing can fit in with the team’s deadlines. Being realistic about how long designing and building something takes is important to avoid wasting research sessions.
4. Is the researcher able to run rigorous research?
Doing good work takes time. It’s often tempting to compromise on ‘doing things properly’ to get a quick answer.
User research is an applied science. This means that researchers approach problems with a scientific methodology and design appropriate experiments to find reliable answers to the questions the team have. When doing this, researchers always have to compromise but it’s important that the researcher is still confident with the findings.
It takes expertise to identify where compromises are appropriate, to avoid affecting the results. Our researchers have a huge amount of academic and practical experience that inform their decisions. So, when teams want to make shortcuts that their researcher isn’t comfortable with, that’s usually a bad sign and will lead to unreliable results. If the results won't be reliable, there’s no point in running the research.
Taking a guess could save everyone’s time and effort. A much more sensible approach, however, is to educate a team to recognise that the researcher’s advice is coming from a sensible place and find a way of working together to get actionable insights in an appropriate timeframe.
5. Do the team commit to understanding what was learned?
“We’ve run some research” isn’t enough if no-one knows what was learned from it. Often the findings from research can be nuanced, and so researchers use a variety of methods to make sure that the team understand it.
We hope to achieve this by doing more than writing reports. We encourage teams to be involved in the research and analysis process. The object of this is to raise the team’s understanding of their users, and what occurred in the sessions.
For this to work, it takes time and commitment from the team. Commitment to turning up to research activities and paying attention, while recognising where the researcher brings objectivity and value into interpreting what happened.
This commitment can be hard for team members when it’s not a main part of their job. Especially when they have many other responsibilities to juggle.
There’s no quick fix to this for teams. Patience, making sure the whole team’s aware of why research is being run in this way, and making sure they understand the benefits of their involvement, are all parts of helping the team develop a collaborative way of working. This leads to a deeper understanding of the insights from research.
6. Is the team ready to react?
Doing good research is all well and good but it can have no impact if the team don’t have the time to react to what they learned. Technical constraints can also prevent teams doing anything about the problems they discover from the research.
To help make sure the research is effective, it’s important that the team discuss what they’re going to do with the findings after testing, and plan appropriately to be able to react to them.
When starting research, it’s important to work closely with the delivery and product manager to do this. This makes sure that there's time to discuss the research results, and is an important part of running good research. This is something that we’ve been trying to improve at PDS.
Getting insights that we can use
Useful insights are ones that are timely, reliable, and relevant to what the team need to know. Failing any of these means the value of research is greatly reduced.
None of this is easy though. It takes time and you have to work closely with other disciplines to overcome. This is one of the challenges our research team will be improving over the next year.
Read more about the work the user research team are doing.