The Ideal Usability Test Sample Size

Many businesses choose to test the usability of their website to confirm that the designs that internal stakeholders think are great, will indeed have the same reaction when used by their real audiences. Website usability testing can be done at various stages of a website design or redesign project – from comparing designs against each other, to testing a website prototype or even testing a beta website. Since findings from a usability test are used to influence the final design, one of the most common questions businesses ask is “what is the ideal usability test sample size?” The answers will undoubtedly surprise you.

High five
It seems intuitive that it should take a lot of testers to identify everything with any kind of statistical significance. In fact, this is not true, according to several trusted experts. A 2012 study by the Neilson Norman Group, a highly regarded user experience consultant, found that five testers is the go-to number when testing qualitative design properties. A more recent article from UXMatters.com clarifies that "there is a 95% chance that a group of 5 participants will find between 66.5% and 100% of the problems."

Plan more for quantitative studies
There are usability testing needs where a larger sample size is indeed warranted. This includes quantitative studies (aiming at statistics, not insights), card sorting tests (typically used for information architecture) and eyetracking studies. For each of these, the recommended number of users ranges from 15 to 40, according to the Neilson Norman Group. Additionally, when testing design preference (using comparative usability studies), a sample size of 20 is a safe bet. Ideally, your team or agency should be monitoring results as they come in to identify significant patterns or trends as they happen. If the results are consistent, adding more users to the study shouldn’t statistically influence your data.

More users may make you feel better, but won’t impact data
While five testers is sufficient for most usability studies, the reality is that most businesses won’t be convinced. It’s more likely that to feel satisfied with the usability testing investment and avoid inevitable credibility judgment by higher-ups, a larger sample size will be desired. While you are certainly welcome to include 20 or even 30 users in your usability study, be aware that most times you will be collecting data that isn’t helping in your decision-making and is likely to result in more project expense. If you are the rare company who really wants to spend more, consider spending it on additional types of usability testing for your project – not on adding more users to your test.

• Website

---