Usability

Usability Benchmarking Downloadable Content

• Complete Usability Benchmarking Toolkit (.zip, 1.11MB)
• Usability Benchmarking Study Planner (.pdf, 670KB)
• Usability Benchmarking ePrescribe Scenarios (.pdf, 259KB)
• Usability Benchmarking Subject Reference Sheet (.pdf, 28.6KB)
• ASQ and PSSUQ Questionnaires (.pdf, 44.7KB)
• ASQ and PSSUQ Questionnaires (with supplementary questions) (.pdf, 57KB)
• Post Scenario Interview Questions (.pdf, 19KB)
• Benchmarking Analysis Workbook (.xls, 71KB)
• Usability Analysis Workbook - Normal Condition (.xls, 88KB)
• Usability Analysis Workbook - Think-Aloud Condition (.xls, 88KB)
• System Automation Analysis Workbook (.xls, 16KB)

Overview

Usability Benchmarking (UB) studies capture data pertaining to a system’s performance and usability (learnability, ease of use, efficiency, safety, and effectiveness) by recording the actions taken by system users during the fulfillment of realistic use case scenarios. These recordings can then be analyzed to produce quantitative system use benchmarks (i.e. the amount of time required to complete a system task) and system usability statistics.
During Usability Benchmarking studies, system users are tested and recorded over two phases of use case scenario testing (normal condition testing and ‘Think Aloud’ testing). For each tested user, the user is first recorded completing the normal condition scenarios under conditions similar to real practice. After each normal condition scenario test, the user fills out a brief usability questionnaire (the After Scenario Questionnaire or ASQ). Once all normal condition tests are completed, the user fills out a second type of usability questionnaire (the Post-Study System Usability Questionnaire or PSSUQ).
The second phase of user testing involves having the user complete the ‘Think Aloud’ scenarios, while verbally providing information about their current actions (thinking aloud).

The normal condition and ‘Think Aloud’ test data is collected for each system user selected for the study.

An optional third phase of usability benchmarking, system automation testing, requires an additional round of tests to be conducted by a single system user. During system automation testing, the tested user completes a series of system actions that are designed to elicit automated system responses (i.e. warning messages, calculations, etc.). The system responses are then recorded and checked for appropriateness.

The audio/video recordings and questionnaire results from the normal condition, ‘Think Aloud’, and (optionally) system automation tests are then coded and analyzed by a research team to produce:

• System benchmarks that can be used for evaluating and comparing system performance (i.e. task completion time, number of actions required to fulfill a task, etc.);
• A list of system usability problems and errors (that were encountered by the users during the tests);
• Comparisons between the normal condition and ‘Think Aloud’ system test flows.

Disclaimer

The material in this section is still under development.  We will be testing and evaluating the methods defined in this section throughout the course of this year.  Please stay posted to this site for important content updates.

Top