Nielsen ranked research methods by expert score, not measured ROI

What we checked

• Is the ranking an empirical ROI study of research methods, or expert judgment using a simple additive formula? confirmed. Nielsen states the four scores are his estimates, calibrated by four decades of practice. Value equals breadth plus depth plus remote suitability minus cost, with a theoretical range of -7 to +24. He invites readers to quibble with any single number and re-sort the list for their own product. [1]
• Do cheap, remote-friendly methods dominate the ranking while prestige lab methods sit at the bottom? confirmed. Moderated usability testing scores 18 and user interviews score 16. Product analytics and unmoderated testing tie at 13. Eyetracking scores 0; biometric testing is the only negative value, at -2. Nielsen’s default program is to run the top four continuously and ration prestige hardware. [1]

What we found

• Nielsen scores each method on breadth of scope (1-10), depth of insights (1-10), remote suitability (1-5), and cost (1-10, where expensive methods score high). Value equals breadth plus depth plus remote suitability minus cost; his 50 methods span -2 to +18 against a theoretical range of -7 to +24. He states the scores are his estimates and invites readers to adjust the numbers and re-sort. [1]
• Moderated usability testing ranks first (breadth 9, depth 8, remote 5, cost 4, value 18). Nielsen calls it the workhorse, recommends five users per round, and says sessions live or die on realistic tasks rather than click-by-click scripts. User interviews rank second (value 16), with an explicit ban on hypothetical demand questions. Product analytics and unmoderated testing tie at value 13; the default program is to run those four continuously. NN/g’s method cheat sheet independently says that if a team can do only one activity to improve an existing system, qualitative think-aloud usability testing is the most effective choice. [1][2]
• Eyetracking scores 0 (cost 9) and biometric testing scores -2, the only negative value. Nielsen calls eyetracking the Faberge egg of UX research and says attention is not comprehension. NN/g independently rates eyetracking as high cost and high difficulty, typically needing 30 or more participants and specialized equipment. A/B testing ranks 44th (value 6) because it answers narrow questions, explains nothing about why a winner won, and can trap teams on a local maximum. [1][3]
• The ranking is not a measured ROI study. NN/g maps about 20 methods on three axes (attitudinal versus behavioral, qualitative versus quantitative, and context of use) plus product-development phase, without a single value score. UX Tigers is Nielsen’s own site for work since 2023, so the 50-method list is a collapse of that long-standing map into insight-per-dollar, not a new measurement of team outcomes. [1][4][5]
• The five-user rule comes from Nielsen and Landauer’s 1993 model, restated in 2000: with a typical problem-discovery proportion of 31 percent per user, five users find about 85 percent of problems, and three rounds of five beat one study of 15. NN/g estimates a five-participant remote moderated study in the United States at about $415 plus 32 researcher hours on the low end, versus about $250 plus 11 hours for unmoderated. Microsoft’s 2002 RITE case study, which Nielsen ranks 10th at value 11, reported 31 problems and 30 fixes on the Age of Empires II tutorial, a 97 percent impact ratio. [6][7][8][1]
• Wizard of Oz testing ranks 32nd (value 8). Nielsen says it tests demand and interaction design for expensive functionality, AI features especially, before a line of code exists. NN/g’s 2024 guide says Wizard of Oz lowers investment risk into complex technologies such as generative AI and is useful for conversational UIs, learning-algorithm recommendations, and real-time lookups. Accessibility testing ranks 37th (value 7) because specialist recruiting raises cost, even though Nielsen says the findings tend to be severe and structural. [1][9]

What it means

Builders should treat the ranking as a resource-allocation heuristic, not as a quality ranking of evidence. A default program of moderated testing, interviews, analytics, and unmoderated testing is a defensible core. Pair analytics with qualitative follow-up, because analytics show what thousands of users do and never explain why. Do not ask interview or survey respondents whether they would use a feature; reconstruct specific past episodes and verify claims with behavioral data. For unbuilt AI, run Wizard of Oz or concierge tests before committing model and engineering cost, even though those methods sit in the middle of the table.

Do not drop A/B tests, accessibility studies, or field work because they score poorly. A/B testing is the only method on the list that Nielsen says proves causation at scale; it is a poor default, not a useless method. Recalibrate the spreadsheet for hardware, safety-critical, and regulated products, where a remote-suitability score of 1 is a constraint rather than a defect. Keep inspection methods as cheap supplements, which is Nielsen’s own position on heuristic evaluation, but do not substitute them for user data.

Still open

• No independent dataset tests whether Nielsen’s 50 scores predict insight per dollar across product types.
• How the scores should change for agentic AI, multimodal products, or evaluation of model behavior rather than graphical usability is unspecified.
• Inter-rater reliability of the four criteria is unreported.
• Whether remote suitability belongs in a value formula for hardware and safety-critical work is a design choice, not a measured result.