Usability Testing & Research Synthesis

Usability Testing and Research Synthesis

Nothing reveals the problems in a design faster than watching real people try to use it. In a usability test, representative users attempt realistic tasks while a facilitator observes where they hesitate, misunderstand, or fail. Five sessions with a prototype can surface the issues a team has been arguing about for weeks — and several nobody had imagined. The hard part is not running sessions; it is recruiting the right people, writing tasks that do not give away the answer, staying neutral, and then synthesizing dozens of observations into a short list of prioritized, evidence-backed changes.

This page covers planning, running, measuring, and synthesizing usability research so it actually changes what gets built.

TL;DR

Decide what you need: qualitative tests find problems (small samples); quantitative tests measure performance (larger samples).
Recruit people who match your users — not colleagues — with a screener, fair incentives, and assistive technology users where relevant.
Write scenario-based tasks with clear success criteria and no hints in the wording.
Moderate neutrally: ask participants to think aloud, answer questions with questions, and never rescue too early.
Measure task success, time, errors, and satisfaction (SEQ, SUS) alongside observations.
Synthesize with affinity mapping, rate severity, and turn findings into prioritized recommendations that teams can act on.

Quick Example

A task written as a realistic scenario, with success defined before the session:

Scoring the System Usability Scale questionnaire after each session:

A synthesized finding, ready to share:

Core Concepts

Types of Usability Tests

How Many Participants

For qualitative testing, around five participants per distinct user group typically reveal most major issues in a design iteration; it is better to test five people three times across iterations than fifteen people once. For quantitative measures — success rates, time on task, SUS benchmarks — use larger samples (often 20 or more per group) and report confidence intervals.

Recruiting

Write a screener that selects for behaviors and context (how often they do the task, which tools they use), not just demographics. Include people with disabilities and assistive technology users where the product serves them. Offer fair incentives, get informed consent for recordings, and protect participant data. Internal colleagues know too much to be valid participants.

Writing Tasks

Good tasks are realistic scenarios with a goal, not instructions naming the interface ("Click Upgrade"). Avoid words that appear in the UI, define success criteria in advance, order tasks naturally, and keep sessions to about 45–60 minutes.

Moderation

Ask participants to think aloud, reassure them that the design — not they — is being tested, and stay neutral. When asked "Should I click this?", reply "What would you expect to happen?" Let people struggle long enough to learn something, then move on with a planned prompt. Note behavior (what they did) separately from opinions (what they said); behavior is more reliable.

Measures

Synthesis

Debrief right after each session while memories are fresh: top observations, surprises, quotes.
Capture observations as atomic notes — one observation, one participant, one task — with timestamps to recordings.
Affinity map: cluster notes into themes, then into problems with clear descriptions.
Rate severity using frequency (how many participants), impact (blocked versus slowed), and persistence (one-time or repeated).
Write findings with evidence, a representative quote or clip, and a recommendation.
Prioritize with the team by severity and effort, and track which findings were fixed.

Sharing and Using Research

Short readouts with prioritized findings, clips, and next steps work better than long reports.
Invite engineers and product managers to observe sessions; watching users is more persuasive than any summary.
Store findings in a searchable research repository, tagged by feature and theme, so they inform future work.
Triangulate with product analytics and support data to see how widespread a problem is.

Best Practices

Test Early and Often

Test sketches and clickable prototypes before code is written, when changes are cheap. See Prototyping.

Define Success Before Testing

Agree on research questions, tasks, and success criteria in a test plan, so results are judged against intentions rather than impressions.

Separate Observation From Interpretation

Record what happened first; interpret during synthesis with others, which reduces individual bias.

Close the Loop

Retest fixed problems in the next round to confirm the change actually worked.

Common Mistakes

Leading Tasks

Testing With Colleagues

Findings Without Priorities

FAQ

How many users do I need for a usability test?

For qualitative, iterative testing, about five participants per user group usually uncover the most important problems. Quantitative benchmarks need larger samples, often 20 or more per group.

What is the difference between moderated and unmoderated testing?

In moderated tests, a facilitator guides the session and can ask follow-up questions. In unmoderated tests, participants complete tasks on their own through a tool, which is faster and cheaper but provides less insight into why problems occur.

What is the System Usability Scale?

A ten-item questionnaire that produces a usability score from 0 to 100. It is quick to administer and widely used to compare designs over time; scores around 68 are commonly considered average.

How do I synthesize usability findings?

Capture atomic observations, group them into themes with affinity mapping, describe each problem with evidence, rate severity, and write recommendations that the team can prioritize and act on.

Can usability testing replace analytics?

No. Testing explains why people struggle with a small sample; analytics shows how often and where problems occur at scale. Use both together.