Respond 2017

Think Aloud User Testing - Challenging Established Status Quo

About this presentation

At Loop11, our team has been discussing the potential influence and impact of the think aloud method of user testing on the natural usage behaviour of participants. Does the method distract or change the way in which users naturally interact? Are we introducing bias into the testing process by making respondents verbalise their inner monologue? There have been previous research studies that do cite an impact on behaviour, however with the growth of rapid online testing and the availability of quantitative metrics, the relative merits and drawbacks of employing this approach have become even more pertinent. Our team formed its own hypothesis, that the think aloud method would extend task completion times and influence the depth to which participants explore navigation and content. We were also concerned that the approach would lead to a skew towards more vocal (possibly extroverted) attitudinal behavioural types and under-representation of others. To put the approach to the test, our team designed a comparative testing scenario to pit think aloud testing against testing carried out without the need to verbalise thoughts. When running the same set of tasks and questions, on the same website, but using two different usability testing tools, one prioritising think aloud, we recorded significant difference in the testing results - though not exactly in ways we had expected! Surprisingly, for the most part we found no real discernible difference in time on task. However, an interesting result from the three studies was that the greater the level of anonymity, the lower the NPS but the higher the SUS. Does this mean that if a user knows they are being recorded, and their responses will be rated, then they are more likely to award a higher NPS score? Within this limited sample, it would appear so. How test participants are sourced also impacts results. Providing incentives for performing tasks in a certain manner (very articulate and deliberate) leads to further skewing of results. Rating of participants perpetuates this phenomenon. Participants should be incentivised for completing the study in a natural manner, not what pleases the viewing experience.