Day 1 UI 9: Honing Your Usability Testing Skills

October 11, 2004

I spent the bulk of the day in an all day techniques practicum with Ginny Riddish working on refining usability testing techniques including issue identification and task development for usability test scripts.

One of the many interesting items that came up as part of the session were in the areas of preparing and holding usability tests. The big item that sort of made my jaw drop, particularly with the numerous discussions I have had in my group, is that you really can't get good usability test results with 6 or less people in your sample. That really slaps in the face of conventional wisdom of several folks including Virzi and John Lewis at IBM.

The caveat in all this is the frequency of testing. The results that Virzi found in 1992, where 90% of usability problems were identified with 3 -5 people was derived from software interfaces and not web applications. That really is a important distinction. Riddish maintains that 6 or less participants in a usability test are only effective if you test more than just once in a software development cycle. If you are resting your laurels on one test on a web product, you cannot rely on a small sample size. The key to usability testing, especially in small sample sizes, is in iterative testing.

Another major pitfall to Usability Testing, which is something I have seen elsewhere, is the lack of focus and scope on what is being tested. It is absolutely essential to be very focused on what it is that is going to be tested--a prototype, a navigation system, or a working product, and to be very specific up front with what are some of the issues that you are concerned about BEFORE going into testing. There are instances where you will end up with mutliple issues and concerns that you will want to test, but it is often wise to segment those items into groups and then run multiple usability tests to test each of those groups.

Finally, test early and test often. This really is not news, but it reinforces what is so common in web site and web application development. All too often, Usability testing is called in midway through a project or just before a product is going to launch to fix a "concern" after it has been coded. It is much more efficient and prudent to test before there is a problem. This can be very effective even if the product is a web site that is going to be redesigned, but has not been put through that process yet. A usability evaluation of the "old site" can be invaluable in identifying issues with the site that stakeholders are not even aware of. All too often, even in a complete redesign, there are features that either have issues or just work well that are either carried over or just dropped without any comprehension of actual issues experienced by the users of the site. Early testing can help identify these types of "features" that should be examined and refined further before going into the site redesign process.