In part 1 of this series, I discussed the benefits of usability testing on enterprise software projects and outlined a general approach for integrating the practice of usability testing into a typical Scrum project (see diagram below). In this post, I’ll lay out a 3-step process for performing a usability test on an enterprise project, describe some recognized best practices, and highlight how it can can hook into the standard elements of Scrum (e.g. burndown, backlog, etc.). Here we go…
Step 1: Plan for the Test
Preparation is the trickiest part of usability testing. If good users are not found, relevant tasks not defined, or environments not set-up, the results of the test will be specious at best, or non-existent at worst! It is crucial, therefore, to develop the right foundation for test:
Find the Users: For the typical B2C web site, representative users can be found at any street corner or coffee shop. For many enterprise projects, however, users are not so easily located or accessed. Because users of enterprise software may have specific domain knowledge, work in different departments, or reside in different geographic offices, procuring access to them or aligning schedules can be a significant challenge. If possible, it is important to surmount these obstacles – real users will give more accurate feedback, and will be generally more receptive of the system when it is eventually rolled out, becoming advocates of it rather than opponents to it.
When it is not possible to find real users, representative users (e.g. mapping to defined personas), or even non-representative users, can be substituted with some success; many usability issues can be uncovered by users with no domain experience, a fresh set of eyes the most salient requirement.
It is well documented that usability testing requires no more than 4-6 users per round. Usability issues are typically universal and so are found early – there are diminishing returns for testing with any more than 6 users (i.e. they all tell you about the same problems!).
Plan the Tasks: Not every feature of an application can or should be usability tested. As noted above, it is important, therefore, to identify for usability testing only those features which are prominent, complex, or have a high cost of user error. Once these features are identified, a script of user tasks can be created. Note that tasks should clear, concise, and worded in a way that is independent of the design of the application (e.g. not “click ‘add item to cart’ and click the ‘purchase’ button” but rather “buy a widget”). Lastly, it is important both to run through the script yourself, and also ask another team member to do the same, and ensure that the script takes no more than 15-30 minutes to execute (with the assumption that a user unfamiliar with the application could take 3x this long).
Write the Post-Test Questionnaire: Depending on the application it may be useful to learn about not only the user’s performance (i.e. can they use the system) but also their preference (i.e. do they like it). Surprisingly, it has been found that the two do not always correlate – users may have problems using the application but say the like it, or use it fine but say they hate it. For this reason, it is important to supplement the observational data (i.e. notes from watching the user) with data from a post-test questionnaire. Note that demographic, qualitative, and quantitative data are important – so ask both open-ended questions like “how did you feel about XYZ?” and “on a scale of 1-5, was the system usable?”. Lastly, the late usability expert Randy Pausch advocated always asking at least one question after a usability test: “what were the two worst things about the system?”, guaranteeing that the user will be forced to shed light on at least two tangible usability flaws (and not try to avoid hurting your feelings).
Set up the Environment, Data, and Access: Finally, a stable and perhaps isolated version of the system must be available for the test participants to use. In many cases this will be the normal QA environment, but depending on the frequency of builds or transiency of the data, this may or may not be adequate. If it is not, alternatives may be either to deploy the system on your local machine or to create a special, temporary environment specifically for usability testing. The feasibility of these options obviously depends on the complexity of the application and dependent sub-systems, serivces, or data stores – a simple single web-app to single database may be easy to configure and deploy, whereas a complex system that leverages LDAP, multiple databases, an ERP, and a handful of external web-services would not be.
Additionally, for some usability test scripts, it will be necessary to have each user test with a fresh, pre-configured, non-confidential set of data (for example, maybe a user is asked to find the user with an outstanding balance, and so such a user must exist). If this is the case, a set of SQL set-up and tear-down scripts may be necessary, or perhaps just a script for manually re-creating the data via the user interface. In any case, it is imperative to understand the data dependencies for the test up-front, to ensure that tests go smoothly, and users find issues related to usability and not bad data conditions.
Lastly, in many cases the test subject will log in as a pre-configured, sample user, however in some cases it will be helpful for the user to log in as himself and see “his” data during the test. In this case, it is important to ensure the user has appropriate access control for the QA environment – and it’s not until he tries to log in that you discover that he doesn’t have authorization.
Step 2: Perform the Test
If planning is done correctly, actually performing the usability test should be a snap – your job is simply to make the subject feel comfortable, stand back, and watch what they do! Here are a few things to keep in mind.
Introduce Yourself and the Test: When meeting the user, it’s important to set a very informal, relaxed, and open atmosphere early, so that the user feels comfortable sharing how they feel about the system. To that end, be sure to mention the following simple points:
- It is the system that is being tested, not them.
- The system has issues, but it’s difficult for developers to see them because we are too close to the problem. This is why their help is so important.
- Negative feedback is actually more helpful than positive. No feelings will be hurt if they are frustrated or don’t like something.
- Their help is greatly appreciated, and will contribute to a better product.
- If they get stuck you can jump in and help, but otherwise you’ll stay quiet and let them drive.
After introductions, give them the test script and have them briefly skim the tasks. Convey to them that although the tasks should only take 30-45 minutes, they should feel comfortable taking their time.
Observe, Ask Questions, but Don’t Help: Once the user has begun, it’s important to let them use the system as they wish, even if it means them navigating down false paths (the whole point is to see how they would use it if you weren’t there!). As they complete the tasks, encourage them to talk, and be sure to ask probing, but non-revealing, questions, like “I saw that you clicked the ‘filter’ button but were surprised with the result – what did you expect would happen there?”. If the user is hopelessly stuck, then it may be necessary to give them hints, but otherwise remain an observer.
Take Notes: As the user completes the tasks, take copious notes of their actions, comments, and frustrations – these notes will trigger your memory later and fuel the Usability Observations list that you assemble. Many would argue that it is necessary to also video record usability test sessions, however, the benefit is rarely worth the cost in my opionion (even though it is relatively cheap): project stakeholders seldom take the time to actually view a usability test video session, users may be apprehensive and less honest when being recorded, and the set-up and storage of the recordings can be frustrating and time-consuming.
Step 3: Analyze the Results
Once the tests have been completed, the data must be analyzed and then synthesized into a set of actionable recommendations. To do so, two artifacts can be created, an objective Usability Observations List and a subjective Usability Analysis & Recommendation document.
Create a Usability Observations list: Go back through your notes, and record discrete usability issues that users encountered in a simple spreadsheet. The list should be objectively and tactfully worded, stating, for example, that the user “did not know what the word ‘invoice’ meant” and not “‘invoices’ should be called ‘bills’”. Consensus should be noted for each item (e.g. “3 out 6 users”), and each should be classified either as a “usability bug” (which would later be entered in an issue tracking system to be implemented during the current sprint) or as a “feature request” (which would be added to the product backlog, to be prioritized for a future sprint).
Create a Usability Analysis & Recommendation document: Finally, it will be helpful for project stakeholders to receive a high-level analysis of the usability test, summing up overall impression of the site’s usability (using both quantitative and qualitative data as useful), the areas of the system that were most and least usable, and a set of recommendations for future action (e.g. new features to the backlog, additional rounds of usability testing, etc.). This document may captured in a Wiki, stored in document control (e.g. SharePoint), or just sent via email.
That’s it! Again, I’d like to hear your thoughts on usability testing on enterprise projects. Please share!
