HCI 3: Introduction to Cognitive Walkthrough
Darryn Lavery
edited by Philip Gray
Usability evaluation methods involving users of the target population, such as think-aloud protocol analysis, are known as empirical techniques. These techniques can be expensive as an implementation in the form of a prototype is needed. Furthermore, test users are required which may be in short supply. These test users could quickly be over-used, particularly if the evaluator is trying to assess how easy a system is to learn or to use.
An alternative is to use analytic methods which can be applied in the absence of users and an implemented design. These methods are not intended to replace empirical methods, but to make them more effective. The GOMS task analysis method and the associated Keystroke Level Model (KLM) has been used effectively for analysing performance on well-defined routine tasks. Using GOMS and KLM, an evaluator can predict performance times for benchmark tasks. However, the results of applying these types of methods can only be understood through comparison with usability targets or against another design. Hence, these methods cannot be used to improve an initial design.
Alternative forms of analytic methods, often known as inspection methods, try to improve a design by predicting potential usability problems. A usability problem is an aspect of the system or a demand on the user which makes it unnecessarily unpleasant, inefficient, difficult or impossible for the user to achieve their goals. Once identified, these problems can be fixed by the design team, as with problems found with think aloud studies.
Simple inspection methods can consist of a check list of guidelines or design principles, for example be consistent, provide feedback etc. Using this check list, an evaluator examines the system or design for breaches of these principles, which may indicate problems.
A more structured approach is to use (design dependent) task analysis. One example and the one we will be examining, is Cognitive Walkthrough, which focuses on usability problems concerned with how easy the system is to learn to use.
To perform a walkthrough, an evaluator or group of evaluators must specify a scenario. A scenario is a story about how a person would use a system to carry out a task. The task is described in terms of its goals, actions and system feedback. The evaluator(s) steps through each of the goals and actions, trying to justify whether or not the user would try to achieve the goal or carry out the action, by answering a set of questions.
Performing a Cognitive Walkthrough proceeds along the following three steps:
1. Create A Scenario
2. Perform The Walkthrough
3. Record the problems found
These will now be described in detail.
A scenario is a story about how someone could solve a task with a particular system. Scenarios should include a description of who the person is, what they are trying to do and why, how the designer thinks they could achieve the goal with the system
Cognitive Walkthrough requires that the scenario be described in terms of the goals the user needs to solve the task, the actions needed to satisfy those goals and the system feedback in response to user actions. If there are several ways of achieving the goal or subgoal, these must be described separately.
As an example, consider the goal of making a copy of some text using a macintosh word processor. The main goal can be decomposed into three envisioned subgoals: copying the text to the clipboard (B), selecting the point to copy the text to (E), and pasting the text from the clipboard. Subgoal B can be further decomposed into the goals of selecting the text to copy (C) and the issuing of a copy command (D).
Goals for making a copy of the text
The main and subgoals have been alphabetically ordered in a particular to make the Cognitive Walkthrough analysis easier. Note the following:
1. The main goal is always "A".
2. "B" is always the first subgoal of "A".
3. The order in which the user will generate the envisioned subgoals is alphabetic.
4. For any subgoal, the next subgoal generated will be the next alphabetically. For example, after generating the subgoal of "E" (selecting the point to copy to), the next subgoal would be "F" (issuing the paste command).
5. If a subgoal is not simple, then its first (sub)subgoal will be the next alphabetically. For example, subgoal "B" (copying the text to the clipboard) is not simple and can be decomposed into further subgoals, the first one being "C" (selecting the text to copy).
Subgoals "C", "D", "E" and "F" can not be further decomposed into other subgoals, and require a set of actions to satisfy them. The following method will satisfy subgoal "D".
|
Method to achieve subgoal D: Issuing Copy Command Main goal and active subgoals (current user subgoals from the main goal to this subgoal): A) Make a copy of the text B) Copy text to clipboard D) Issue copy command Comments: Assumes text has been selected for
copying |
|
|
|
|
|
User Action |
System Feedback |
Comments |
|
D1 |
Move mouse pointer to "Edit" menu and hold down mouse
button |
"Edit" menu revealed |
see Screenshot.... |
|
D2 |
Select "Copy" command |
"Copy" command flashes "Edit" menu disappears |
Text is now copied to the clipboard |
Once one or more scenarios have been defined, a Cognitive Walkthrough can be applied by stepping through the goals and actions, and answering a set of questions. The walkthrough team could consist of one person, for example the designer of the system or it could be in a group which could include the designer(s), users from the target population, usability specialists, software engineers etc.
The evaluator will step through each of the subgoals, actions and examine resultant system feedback in their envisioned order. The evaluator will answer a set of questions trying for example to justify whether or not the user will try to achieve the correct subgoal or action. Negative answers to these questions should be recorded as they suggest potential usability problems with the system.
Once an evaluator has answered a negative question and recorded it, the evaluator will assume that the question has been answered positively and move onto the next question.
These questions will now be described, followed by guidance how to apply them.
Different questions address the subgoals, actions and system feedback. These questions are now described. A summary of this list is available as a checklist.
Two questions are applied to each of the subgoals:
1) Will the user try to achieve the right subgoal?
This question determines if the user will try to achieve the right subgoal. For example, in some systems the user must select a printer before they print a document. However, the user may not think of selecting a printer.
The user may try to achieve the right subgoal
If the user does not try to achieve the right subgoal, they may try to achieve the next subgoal or an inappropriate subgoal or give up and hence suffer task failure.
2) What knowledge is needed to achieve the right subgoal? Will the user have this knowledge?
The user may want to achieve the right subgoal (question 1), but does not have the appropriate knowledge. To use a system requires knowledge of the task domain and the system supporting the task and can include procedural, conceptual, knowledge about the current system state and knowledge about the objects manipulated by the task. Good interactive system design should reduce the amount of knowledge needed by the user.
The user may have the knowledge
If the user does not have the correct knowledge, then they may be able to continue with the task. The user may lack procedural knowledge i.e. how to achieve a subgoal but will continue in the hope of finding a method. Whether or not the user can determine the correct method will be assessed by answering the other questions.
Sometimes, users will require knowledge such as filenames, passwords, input parameters etc. The user may try to achieve a subgoal not knowing this information only to get stuck when the system requires the user to enter this information -- they may guess the information or look it up, for example in a manual. Users familiar with the task may not try to achieve the subgoal until they have the necessary information.
Actions are addressed by two questions.
3) Will the user notice that the correct action is available?
The user may notice that the correct action is available:
If the user does not notice that the correct action is available, they may select an inappropriate action, or they may give up their current goal or subgoal and hence suffer task failure.
4) Will the user associate the correct action with the subgoal they are trying to achieve?
Assuming that the user notices that the correct action is available, this question assesses if the user will believe that this is the correct action to take. According to the theory underlying Cognitive Walkthrough, when users form a goal they select actions which they believe will help solve their goal or get them closer to it. The theory suggests that users pick actions which are `related' to their current goal, for example if the user is trying to print a document they may look for a "Print" or "Document" menu or action.
The user may make the association
If the user does not make the association, they may select an inappropriate action, or they may give up their current goal or subgoal, and hence suffer task failure.
The system should provide feedback in response to each action. The feedback should be observable, understandable and indicate that the user is making progress towards their subgoal or goal. Feedback is assessed by the following three questions.
5) Will the user perceive the feedback?
The user may perceive the feedback
If there is no feedback or if the user does not perceive the feedback, then the user may be uncertain that their action(s) had any effect. They may try the action again, try another action, carry on despite the uncertainty or even give up.
6) Will the user understand the feedback?
There is no guarantee that the user will understand the feedback for example a message, even if it is perceived. The user may understand the feedback
If the user does not understand the feedback and so what happened, the user may become anxious. They may continue despite the anxiety or give up. Alternatively, the user could misunderstand something which causes difficulties later.
7) Will the user see that progress is being made towards solution of their task in relation to their main goal and current subgoals?
More importantly, feedback should provide an indication of the user's progress towards achieving their goal. Ideally, feedback in response to the last action of a method should indicate achievement of that subgoal, and in some cases the achievement of parent subgoals.
Sometimes, feedback can mislead the user into believing their main goal has been achieved without achieving other subgoals. For example, early cash machines supplied the money before returning the card. People then believed that their main goal of obtaining money was achieved -- they then often forgot to take their card!
The user may perceive progress
If the user does not see that progress is being made but still understands the feedback. the user may try other actions to determine that their earlier actions had the desired effect. If the user is misled into believing that goals have been satisfied they may not try to achieve other subgoals.
The walkthrough starts with the main goal "A". It is up to the evaluator whether or not they analyse goal "A" or not as it can be assumed that the user will be trying to achieve this goal, and assessments of knowledge will be made when analysing the subgoals. The analysis starts with the first subgoal "B" (copying the text in the clipboard in the example below) is examined to determine if the user would try to achieve that subgoal and if they had the appropriate knowledge through questions 1 & 2.
The Goal hierarchy for copying and pasting text
Once "B" has been analysed, "C" is analysed with questions 1 & 2. As "C" can not be further decomposed into further subgoals, it has a method of actions to achieve it. Each of the actions and resultant feedback are analysed with questions 3 to 7. Once the actions have been analysed, analysis moves to the next subgoal, "D" etc.
When any question is answered negatively, it indicates a potential usability problem. The evaluator should describe the problem in such a way to convince others (such as the design team) that it is a problem and deserves fixing. Therefore, the problem the user may encounter should be described in as much detail as possible.
When describing the problem the evaluator should describe when the problem could occur (what would the user be trying to do?), what the interaction problem was, why the problem occurred (why did the analyst answer no to the question) and what the consequences of the interaction problem would be, for example what would the user try do next?
Edited by Philip Gray, 5 Jan 1997