Continuous discovery has a structural weakness. You talk to a small group of people, often repeatedly, usually about a product you already have opinions about. That is close to ideal conditions for confirmation bias.
The practitioners I interviewed for my MSc thesis were candid about this, and most had built specific habits to counter it. Here they are, along with how teams decide whether to trust a finding.
Where the bias comes from
Three sources came up repeatedly.
The people closest to the product. Product people often have fixed ideas about what the product should be, and hear confirmation of those ideas in ambiguous feedback.
The same small group. Using a small, repeated pool of users is convenient, but it narrows what you can learn and gradually turns your participants into insiders.
Untrained interviewers. In organisations without researchers, designers often run the sessions. Without training, questions get leading and quality suffers, which was raised directly as a risk to the integrity of the findings.
Seven techniques that help
1. Deliberately detach from what you know
One participant described their approach as acting as if they had never seen the product before. It sounds simple, but going into a session having consciously set aside what you already believe changes the questions you ask.
2. Ask about past actions, not opinions
This is the single most useful technique in the list. Instead of asking what someone would do, or what they think of an idea, ask what they actually did last time. Get them thinking out loud about a real past experience and excavate the story from there.
Opinions and predictions are cheap and unreliable. Behaviour already happened, so it cannot be shaped by how you phrased the question.
3. Set the metric before the session
Participants stressed having very clear metrics attached to assumption testing: deciding in advance what would need to be true for something to succeed, and phrasing questions around that in an unbiased way.
The reason this works is that it makes cherry-picking visible. If you decide afterwards what counts as success, you will always find evidence for what you hoped.
4. Triangulate
Do not let one vivid interview become a finding. Check what you heard against other sources: analytics, support tickets, sales conversations, app store reviews. Using multiple data sources was named specifically as a way to counteract confirmation bias.
5. Rotate and diversify who you talk to
Regularly evaluate and adjust your user group so you keep getting fresh perspectives. This is the same argument as the cooling period in recruitment: a rotating group protects the quality of what you learn.
6. Use someone less attached
Researchers with no stake in a particular outcome give a more neutral read. Where that is not possible, get other people to review your research plans and scripts before you run them, and discuss findings as a team afterwards so no single interpretation goes unchallenged.
For major projects, some teams ran company-wide reviews, bringing in designers, product teams and key stakeholders to stress-test what had been found.
7. Handle feedback openly
Encourage people to say what they dislike, and treat criticism of things you are close to as useful rather than threatening. As one participant put it, one of the big ways to overcome bias is simply getting feedback from others.
How to know whether to trust a finding
Techniques reduce bias, but at some point you have to decide how much confidence a finding deserves. I put this question to a twelve-person product team in a workshop, and their answers are a practical checklist.
The clear top answer, with four votes, was trends and data being consistent across the research. Not one strong interview, but the same thing appearing repeatedly.
Then, each with two votes:
- Well-structured data outputs, so the finding can be examined rather than taken on trust.
- A representative group, rather than whoever was easiest to book.
- Validation from multiple sources, which is triangulation again.
- A/B testing, where the question can be settled with behaviour at scale.
Two single-vote answers are worth more attention than their score suggests:
- Give each finding an explicit confidence score, assigned by the research team based on the quantity and quality of the data behind it. This is a genuinely good practice and rarely done. It stops all findings being presented as equally solid.
- Be transparent about participation and findings, the good, the bad and the ugly. Reporting what did not work, and where your sample was thin, is what makes the rest of your reporting credible.
The uncomfortable part
Some bias is unavoidable. Cognitive biases happen in people’s heads, and a small repeated group is always going to be a limitation of continuous discovery rather than a solved problem.
What the good teams did was not eliminate bias. They made it visible: they wrote down assumptions in advance, checked findings against other data, involved people who disagreed with them, and were honest about how confident they actually were.
That is a lower bar than objectivity, and a much more useful one.
Further reading
- Teresa Torres, Product Talk. producttalk.org
- My MSc research. Eight expert interviews and a twelve-person workshop. Read the case study
Next, read 11 challenges of continuous product discovery or how to turn research insights into action.
Based on my MSc thesis on continuous product discovery. The research is written up in the case study.
