Notes

Strengths and weaknesses

Strengths

• Testing system learnability by exploration - most new system users learn how to use a system by self-exploring the system's functions. In such circumstances, learning gains will only achieved by the user if the system is intuitive and has a clearly visible system status. Such aspects are analyzed in detail in CWs.
• Streamlining human computer interaction - CWs break down the user actions required to achieve a system goal. With this achieved, it becomes clear whether or not such processes can be made more efficient (i.e. by remodelling the interaction process).
• Systematically produced results - the Cognitive Walkthrough process is systematic and reproducible, which enables other evaluators to re-test Walkthrough results.
• Uncovering the misconceptions made by system designers about the system users - a computer system is designed by the developers for the users. Often times, system developers can create systems that are tailored to their own preferences (as opposed to the user's preferences). CW uncovers these misconceptions so that the computer systems can be tailored more towards the actual system users.
• Flexibility of analyst use - CWs can be conducted individually or in groups by HCI experts or even the system developers themselves.

Weaknesses

• Based on assumptions - a CW study is based entirely on the assumptions that the analysts make about the mindset of the users as they operate the system under study.
• Limited usability aspects tested - CWs primarily test the usability aspects relating to the learnability of a system (i.e. "Is the information clearly presented?", "Is it clear to the user what to do next?", etc.), and can miss other aspects of system usability, such as interface consistency or recurring problems (Wharton, Rieman, Lewis, & Polson, 1994).
• Difficulties in learning/conducting CW studies - CW studies follow a well defined, systematic process, which can be somewhat more effortful to learn at first than other usability engineering methods.

Return to tabs

Plan

Select the system and tasks to analyze

Select the system and tasks to analyze

Cognitive Walkthroughs should be conducted on specific system design aspects or components. It is important to have a well defined evaluation boundary drawn for a CW to prevent scope creep. CW studies go into incredible detail for each system task; therefore, a complete evaluation of a large system could get out of hand very easily. The chosen system component that is evaluated should be representative of the whole system, so that an approximation of the entire system's usability can be formed.

CW studies can be performed on systems at any stage of their developmental cycle. It is possible to perform a CW on a paper prototype (where a written description of each system interface screen is given in detail); however, it is much easier for the analyst to perform a CW on a system with an already completed interface.

Return to tabs

Identify the user population

Identify the user population

During CW studies, the evaluators will be forced to make predictions about:

• How the users will use the interface?
• How the users will react to the interface responses?
• What goals the users will form during their operation of the system?
• What system actions will the users find easy, difficult, problematic, etc.?

To do this effectively, the evaluators will have to have clear descriptions of the intended system user population. This information can be drawn from the user profile set that was created in phase 2 of the Rapid Response study design (see Phase 2. Know the Users).

Return to tabs

Select the evaluators

Select the evaluators

Cognitive Walkthroughs can be performed by a single evaluator or by groups of evaluators (working collaboratively). Cognitive Walkthrough evaluators are normally the peers of the system's developers; however, it is also possible for developers to conduct self-CW-evaluations on their own products. In either case, the selected evaluators need to be able to put themselves into the mindset of the intended target system users, and thus forget their own personal biases.

Return to tabs

Goals, sub-goals, and actions

Goals, sub-goals, and actions

The written task descriptions are defined and sequenced in a hierarchical structure that is composed of goals, sub-goals, and actions. In order to achieve a system goal, the user must perform certain actions. A goal in a system task can be broken into numerous sub-goals (i.e. an ultimate goal may be to fill a prescription order, but in order to do that, the user must achieve various sub-goals, such as: logging into the system, loading the appropriate patient, selecting the correct medication, etc.). Sub-goals also require the completion of actions in order be fulfilled (i.e. for the sub-goal "Login to the system", the user may have to perform the following actions: click on the EHR launch shortcut; click on the user name text box; type the user name; click on the password box; type the user password; and click on the "OK" button). All goals and sub-goals can be composed of more sub-goals. The goals, and the actions required to attain a goal or sub-goal can be unordered (meaning they can be completed in any order), or they can be ordered (meaning they must be completed in a certain order to attain the desired goal).

Return to tabs

Describe the user's initial goals

Describe the user's initial goals

The first step in creating a task description is to create a written description of the user's initial goals (what they hope to achieve by using the system during the evaluation). This description should paint the picture of the user's initial system goal (i.e. to load their patient's record into their new EHR system), the user's background (i.e. a 60 year old male physician with less than 3 months experience with electronic record systems), their current mindset (i.e. stressed and in a rush, due to the fact that it is the middle of a busy day), as well as the course of events that brought the user to the need of accomplishing this goal (i.e. "A patient has just entered the physician's office. The physician would like to open the patient's health record file for his reference during the exam. The physician is sitting at his desk (where a desktop contains the new EHR system that he has been using for the past 2 months. The EHR system is currently turned on; however, the physician is logged out of the system. The physician's computer is currently on the EHR 'Login' screen").

Return to tabs

Select the task granularity

Select the task granularity

The most challenging aspect of defining the goals, sub-goals, and actions in a CW study is determining the level of granularity for the system task descriptions. If, for example, the task descriptions are too granular (i.e. the actions are divided by keystrokes: Action 1: Type "T", Action 2: Type "y", etc.), then both writing and following the immensely detailed task descriptions would be quite cumbersome. If, on the other hand, the descriptions are too abstract (i.e. only having one action and one goal: Goal 1: "Fill a prescription order", Action 1: "Enter the system and fill the prescription order"), then much will be missed during the analysis stage.

Return to tabs

Sequence the task descriptions

Sequence the task descriptions

The action sequence developed for the evaluation should be the most efficient means of accomplishing the defined system goal. If the best possible task sequence is used, and the system is still found to be inefficient during the evaluation, then it can be concluded that the system will need to undergo changes. On the other hand, if an inferior task sequence is used in the walkthrough, then nothing can be learned, because the users should have performed the task in a different way (meaning if their current task method was flawed, then they should have used the superior task method).

When sequencing Cognitive Walkthrough task instructions, the sub-goals and actions should be indented from their parent goals, and the ordered: goals, sub-goals, and actions should be numbered (by order of their occurrence) in the instruction set. An example set of sequenced goals, sub-goals, and actions (both ordered and unordered) is as follows:

• Ultimate Goal: Fill a prescription for a patient using the prescription function of the EHR system
  1. Sub-goal (ordered) - Login to the system

     a. Action sequence (ordered): click on the EHR launch shortcut

     b. AND THEN: click on the user name text box

     c. AND THEN: type the user name

     d. AND THEN: click on the password box

     e. AND THEN: type the user password

     f. AND THEN: click on the "OK" button

  2. Sub-goal (ordered) - Select the patient

     a. Action sequence (ordered): click on the "File"->"Patient Search" menu item

     b. AND THEN: type the patient's name into the blank textbox

     c. AND THEN: click on the "SEARCH" button

     d. Sub-goal: Determine which patient to select

     • Action (unordered): Lookup the patient's ID# from the patient's card
     • Action (unordered): Lookup the returned patient's ID#s from the EHR screen
     • Action (ordered) - AND THEN: determine which patient ID# on the EHR matches with the ID# from the patient's card
     • Action (ordered) - AND THEN: select "OK" next to the patient ID# that matches the one found on the patient's card on the EHR

  3. Sub-goal (ordered) - Select the medication

     a. Action sequence (ordered): click on the "File"->"Medication Search" menu item

     b. AND THEN: type the medication's name into the blank textbox

     c. AND THEN: click on the "SEARCH" button

     d. AND THEN: click on the "Order Medication" button (next to the returned medication name)

     e. AND THEN: select the appropriate dose from the "Dose" drop-box

     f. AND THEN: select the "Fill prescription order" button

To summarize this task description: the ultimate goal of the task was to fill a prescription order. In order to accomplish this goal, three sub-goals had to be completed. To successfully accomplish the first sub-goal ("Login to the system"), the user performed six ordered actions. The user then performed the second goal (named "Select the patient"), which was composed of three ordered tasks, and a secondary sub-goal ("determine which patient to select"). The secondary sub-goal had two unordered actions (that could have been completed in any sequence), followed by two ordered actions that were completed in sequence (ordered actions). The third and final sub-goal of the ultimate goal ("Select the medication"), was composed of six ordered actions. The successful completion of the three sub-goals, in order, fulfills the requirements needed to complete the ultimate goal.

Return to tabs

System response descriptions for prototype systems

System response descriptions for prototype systems

If a fully operational system is used in the CW, then a written description of the task goals, sub-goals, and actions will suffice. If, however, a paper prototype system is used in the Cognitive Walkthrough, then written descriptions of the system's behavior, both before and after each action, will also have to be written in this stage. Such system descriptions will have to include imagery of what the system interface looks like, as well as any system prompts (i.e. sounds, pop-up menus, etc.) that appear during the user's interaction with the system. For example, when starting the third sub-goal ("Select the medication") of the sequence defined in "Sequence the task descriptions", the following description would have supplemented the goal-action sequence for a paper prototype system:

Sub-goal (ordered) - Select the medication

• System state description - The electronic patient record for the selected patient is opened to the starting patient dashboard view. There are four components in the dashboard menu: "Patient Demographics", "Current Medications", "Known Allergies", and "Previous consultation summary". Above the patient dashboard screen are the main interface menu items: the "File" menu (which consists of the "Patient Search" and "Medication Search" items), and the "Exit" menu. There are no other windows or user prompts open at this time.

Action sequence (ordered): click on the "File"->"Medication Search" menu item

• System state description - The "Medication Search" window opens overtop of the patient dashboard display. Within the "Medication Search" window are the following options: a label that states "Please enter the medication name", which is to the left of a blank text box with a blinking cursor within it, a button that is labeled "SEARCH", and a window exit button.

Action - AND THEN: type the patient's name into the blank textbox

• System state description - etc.

Return to tabs

Do

Cognitive Walkthrough study

Cognitive Walkthrough study

After the system components, users, evaluators, and system tasks have been defined, it is then possible to conduct the Cognitive Walkthrough study. During this DO phase of a CW, the chosen evaluator/s step through the chosen system components in the mindset of the chosen user groups, by following the defined system tasks. As CW evaluators step through the system, they look for potential usability problems that might occur once the system is under use by actual system users.

There are two types of usability problems that can occur in a Cognitive Walkthrough: 'Goal Problems', which occur when the user doesn't know what to do or is attempting to do the wrong thing (i.e. a user reads the screen prompt which states "Press any key", and they user then has the goal to push the "ANY" key (the wrong goal), but of course, they can't find it), and 'Action Problems', which occur when the user correctly knows what they need to do, they just don't know how to do it (i.e. if the user is given the prompt "The file is saved. You may now exit the program", the user may know that they need to exit the program (the appropriate goal), they just might not know how to exit the program (the appropriate action)). The following sections describe the process of checking for such usability problems in A CW study:

The Cognitive Walkthrough Evaluation Cycle

 

Return to tabs

Identify potential 'Goal Problems'

Identify potential 'Goal Problems'

In the first task step of the CW process, the evaluators must determine their current situation and system goals based on the written description of the initial goals (created in the previous section). For all other steps in the CW process, the evaluators' goals will be generated based on the feedback that they receive from the system, after each action they perform. The goals that the evaluators form from the system feedback can then be compared to the goals that are stated in the sequenced task description set. If there are any discrepancies between what the evaluators understand as the goals (from the perspective of the target system user), and what the task description defines as the user goals, then a 'Goal Problem' has occurred, meaning the user has formed a goal that is not aligned with what the system was designed for.

An example of such a 'Goal Problem' (from the task sequence defined in the Sequence the task descriptions section), could be that after successfully logging into the system (sub-goal 1), the evaluator is brought to the main EHR menu, which only has two menu options 'File' and 'Exit'. Under the 'File' option, there are two more options, 'Patient Search' and 'Medication Search'. At this point, according to the defined task set, the evaluator is next supposed to have the goal of selecting the patient, so that they can then order their medication. An example of a potential problem, in this step, would be if the evaluator notices that there's no indication on the system's interface that the user should select 'Patient Search' before 'Medication Search', meaning the user could potentially create the wrong goal at this stage by searching for the medication before the patient. If it were required of the system user to select the patient before the medication, then this could pose as a potential problem (as the user may attempt to search for the medication first, only to find out that they need to find the patient, and then redo their work in finding the correct medication). For this reason, the evaluator would mark that a potential 'Goal Problem' exists when stepping between sub-goal 1 (logging into the system) and sub-goal 2 (selecting the patient). The evaluator could also make note of a potential solution at this point, such as only displaying the 'Patient Search' menu option when a patient is not yet selected.

The following form was adapted from (Polson P. , Lewis, Rieman, & Wharton, 1992) as data collection tool to be used by the evaluator when determining if there are any potential 'Goal Problems' in the current CW step:

Form 1: Cognitive Walkthrough Data Collection Tool - Goal Problems- adapted from (Polson P. , Lewis, Rieman, & Wharton, 1992)

Cognitive Walkthrough Data Collection Form - Part A - Goal Problems
Task:	Action #:
1. Goal structure for this step.
1.1 Correct goals. What are the appropriate goals for this point in the interaction?
1.2 Incorrect goals. What other goals could possibly be formed (incorrectly) at this point in the interaction?
1.3 Mismatch with likely goals. What percentage of users will form the incorrect goals, based on the analysis at the end of the previous step? Compare the correct goals with the incorrect goals. How many users would have been brought off track by selecting the incorrect goals? Check the analysis from the previous step. Is there any way that this goal could not have been reached by the system users? a) 0% b) 25% c) 50% d) 75% e) 100%
1.4 Potential Solutions. What are some possible strategies that can be used to prevent such goal mismatches from happening?

Cognitive Walkthrough Data Collection Form Part-A

Return to tabs

Identify potential 'Action Problems'

Identify potential 'Action Problems'

Once all possible 'Goal Problems' have been noted for a step, the evaluator can assume that the user determined the correct goal (as defined in the task description), and search for any 'Action Problems'. In this analysis step, the evaluator will look at the given task goal (i.e. select the patient from the EHR system), and determine whether or not the system's interface will provide enough information to the user (i.e. via prompts, labels, etc.) to be able to achieve that goal. The primary question in this stage of analysis is whether or not the user will notice the correct course of action to take based on the feedback given by the system from the previous action. The evaluator's job is to tell the story of the user, regarding which actions the user would take, and why the user would choose to take such actions.

The following form was adapted from (Polson P. , Lewis, Rieman, & Wharton, 1992) as data collection tool to be used by the evaluator when determining if there are any potential 'Action Problems' in the current CW step:

Form 2: Cognitive Walkthrough Data Collection Tool - Action Problems - adapted from (Polson P. , Lewis, Rieman, & Wharton, 1992)

Cognitive Walkthrough Data Collection Form - Part B - Action Problems
2. Choosing and executing the action
2.1 Correct actions. What are the appropriate actions for this point in the interaction?
2.2 Incorrect actions.What other actions could possibly be performed (incorrectly) at this point in the interaction? What percentage of users will choose these incorrect actions? a) 0% b) 25% c) 50% d) 75% e) 100%
2.3 Apparency. Is the correct choice of action apparent? If not, what percentage of users will miss this correct course of action? a) 0% b) 25% c) 50% d) 75% e) 100%
2.4 Label. What is the label description that is associated with the correct action? If there was a label provided: 2.4.1 Link of label to action. How apparent and clearly marked is it? What percentage of users would miss this label? a) 0% b) 25% c) 50% d) 75% e) 100% 2.4.2 Link of label to goal. Is the correct action associated with the current task goal? If so, then how? If not, then what percentage of users would miss this association between the labeled action and the goal? a) 0% b) 25% c) 50% d) 75% e) 100% If there wasn't a label provided: 2.4.3 No label. How will the users be able to determine which action to take in order to achieve their goal? What percentage of users will not take this correct action? a) 0% b) 25% c) 50% d) 75% e) 100%
2.5 Time-out. If there's a time-out feature in this step (the user only has a certain amount of time to complete the task before the system stops the process), does the system provide enough time to the user? How many users might have trouble with this time-out feature? a) 0% b) 25% c) 50% d) 75% e) 100%
2.6 Hard to do. Is this action physically difficult to perform? What percentage of users might have problems physically completing this task? a) 0% b) 25% c) 50% d) 75% e) 100%

Cognitive Walkthrough Data Collection Form Part-B

Return to tabs

Re-define the current goal structure

Re-define the current goal structure

After the user's actions have been analyzed by the CW evaluator, the process is repeated by analyzing the system for potential goal or action problems for the next step in the defined system task description. Before this can happen, the system feedback from the previous actions needs to be analyzed in order to re-define the user's goal structure. At this point, the evaluator has to ask:

• If the correct action was performed, will the user attain appropriate feedback from the system to confirm that positive gains have been made towards their ultimate goal?

Answering this question will help the evaluator re-define their current goal structure, which will in-turn determine their next course of action.

The following form was adapted from (Polson P. , Lewis, Rieman, & Wharton, 1992) as data collection tool to be used by the evaluator when determining if there are any modifications in the goal structure after the previous CW step:

Form 3: Cognitive Walkthrough Data Collection Tool - Modifications of the goal structure - adapted from (Polson P. , Lewis, Rieman, & Wharton, 1992)

Cognitive Walkthrough Data Collection Form - Part - C Modifications of the Goal Structure
3. Modification of the goal structure.
3.1 System Response. Assuming that the correct action was performed in the previous stage, what was the system's response?
3.2 Clear Progress. What percentage of users will see that they have made progress towards achieving their goal? a) 0% b) 25% c) 50% d) 75% e) 100%
3.3 Accomplished goals. List all current goals that have been accomplished: Has the system provided enough feedback to indicate to the user that these goals have been accomplished? If not, what percentage of users will not realize that such goals have been accomplished? a) 0% b) 25% c) 50% d) 75% e) 100%
3.4 Incomplete goals that look accomplished. Are there any current goals that aren't accomplished but may appear accomplished (due to the system feedback)? If so, which goals? What is the system feedback that leads the user to believe that these goals are accomplished? What percentage of users may falsely believe that such goals have been accomplished? a) 0% b) 25% c) 50% d) 75% e) 100%
3.5 Sub-goals. Are there any super-goal sub-goal structures in the current step? If so, are there any super-goals that may falsely seem accomplished due to the completion of a sub-goal component (even if there're more sub-goals to complete before the super-goal is accomplished)? How many users may falsely believe that a super-goal is completed due to the completion of one of its sub-goal components? a) 0% b) 25% c) 50% d) 75% e) 100%
3.6 New goals in response to prompts. Does the system provide any prompts to indicate that new goals should be formed by the user? If so, what are the prompts, and what are the goals? What is the percentage of users that will understand these prompts and form the new goals? a) 0% b) 25% c) 50% d) 75% e) 100%
3.7 Other new goals. Are there any other new goals that need to be formed that aren't addressed by the system prompts? If so, what are these goals and what percentage of users will form them? a) 0% b) 25% c) 50% d) 75% e) 100%

Cognitive Walkthrough Data Collection Form Part-C

Return to tabs

Study

Analyze and Compile Results

If multiple evaluators were used in the walkthrough to generate several reports on potential goal and action problems, then these reports need to be compiled into a super-problem-list. To do this, it is usually best to hold a group meeting with all of the system evaluators, so that everyone can discuss their results, resolve any discrepancies, and come up with possible solutions for the found problems. It is imperative that in this compiled problem document, that all user knowledge requirements and assumptions, as well as all task descriptions are clearly defined (to better justify the results).

Analyze the action sequence

On top of generating a list of all system learnability problems, Cognitive Walkthroughs can also help with generating more streamlined computer systems. As part of the CW procedure, the best-case-scenario activity procedure had to be defined for each investigated system goal. These activity procedures (such as the one defined in the Sequence the task descriptions section) can also be analyzed for efficiency in the CW STUDY phase. If it's determined that a given goal requires too many actions (i.e. if it takes 15 mouse clicks to reach the "About Us" section of a company Web page), then the process can be streamlined (i.e. by providing a link on the company's home page, thus reducing the required number of mouse clicks, and streamlining the overall user information request process).

Return to tabs

Act

Iterative input into design

The formal super-problems-list document and the streamlined system action sequences should be given to the system developers so that they can be incorporated into the next phase of system design. Once these changes have been made to the system, it becomes possible to re-run the CW study, so that the revised system can be tested for improvements in system learnability and efficiency.

Return to tabs

PDSA Summary

Plan

Select the system and tasks to be analyzed

• Specific system components should be chosen to minimize the scope of the CW study

Identify the user population

• A detailed analysis of the user population needs to be made in order to create accurate assumptions about their behavior during the CW evaluation
• This can be drawn from the user profiling stage of the Rapid Response study

Select the evaluators

• CW can be performed by groups or by individuals
• CW can be performed by the system's developers, by the developer's peers, or by trained HCI experts

Create task descriptions

• The task descriptions provide the CW evaluator with detailed step-by-step instructions of actions to perform during the evaluation in order to achieve some defined system goal
• A goal is what the user wants to achieve
• A goal can be made up of sub-goals
• Goals and sub-goals are achieved by performing actions
• Selecting the granularity of the actions can be difficult
  • It should be written in the same detail as a tutorial for a first time system user
• A written description of the user's initial goals and the events that lead them up to those goals should be provided to the system evaluator
• The task instruction set should be structured hierarchically (from goals, to sub-goals, to actions), and numerically ordered according to their occurrence in the task
• Paper prototype systems require written descriptions of the system's interface and responses to the users' actions

Do

At the beginning of each task, evaluators must re-evaluate their goal structure, based on the feedback from the system interface (i.e. if the system interface changed from the login screen to the patient select screen, the user will most likely have to change their current goal from "Login to the EHR" to "Select the appropriate Patient").

The evaluators then inspect each task step for possible 'Goal Problems'.

After determining all potential user goal problems, the evaluators assume that the user formed the appropriate task goal, and inspect the system for any potential 'Action Problems'.

At the end of each task, the evaluators must re-evaluate their goal structure, based on the feedback from the system interface.

This process is then repeated for each task step until the ultimate system goal has been achieved.

Study

The 'Study' phase of a CW involves compiling a super-list of all the usability problems found in each of the walkthroughs.

• This list should be generated collaboratively, through group discussions about each of the found problems by the evaluators that found them.

The action sequences required to fulfill system goals should also be analyzed as part of the STUDY stage of a CW.

• All action sequences should be streamlined to assure that the system goals can be accomplished in the most efficient means possible.

Act

Provide the system learnability and efficiency suggestions to the system's developers so that the improvements can be incorporated into the next phase of the system's design.

CW studies can be repeated for each new system design, thus causing iterative improvements.

Return to tabs

References

References

• Polson, P., Lewis, C., Rieman, J., & Wharton, C. (1992). Cognitive Walkthroughs: A Method for Theory-Based Evaluation of User Interfaces. International Journal of Man-Machine Studies , 36 (5), 741-773.
• Wharton, C., Rieman, J., Lewis, C., & Polson, P. (1994). The cognitive walkthrough method: a practitioner's guide. In J. Nielsen, & R. L. Mack, Usability inspection methods (pp. 105-140). New York: John Wiley & Sons, Inc.

Return to tabs