Evaluation Methods in HCI Design
Evaluation Methods in HCI Design
INTRODUCING EVALUATION
Abbas Moallem, Ph.D.
Chapters: 12 + 13 +15 7
Session 12
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
Slide 3
OVERVIEW
Explain the key concepts used in evaluation.
Introduce different evaluation methods.
Show how different methods are used for different
purposes at different stages of the design process and
in different contexts of use.
Introduce and explain the DECIDE framework.
Discuss the conceptual, practical, and ethical issues
involved in evaluation.
Show how evaluators mix and modify methods.
Discuss the practical challenges
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
Slide 9
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
Slide 10
KEY POINTS
Evaluation & design are closely integrated in usercentered design.
Some of the same techniques are used in evaluation
as for establishing requirements but they are used
differently
(e.g. observation interviews & questionnaires).
Three types of evaluation: laboratory based with
users, in the field with users, studies that do not
involve users
The main methods are: observing, asking users,
asking experts, user testing, inspection, and
modeling users task performance, analytics.
Dealing with constraints is an important skill for
evaluators to develop.
[Link]
10
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
Slide 11
TYPES OF EVALUATION
Expert(Inspection base) Evaluations
Cognitive Walkthrough
Heuristic Evaluation
Any settings not involving users,
Exmaple: consultants critique, predict, analyze & model
aspects of the interface analytics.
User Evaluation
Controlled settings involving users, eg usability
testing & experiments in laboratories and living
labs.
Natural settings involving users, eg field studies to
see how the product is used in the real world.
[Link]
11
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
Slide 12
[Link]
12
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
Slide 13
[Link]
13
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
Slide 14
[Link]
14
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
Slide 15
[Link]
15
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
Slide 16
[Link]
16
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
10
Slide 17
[Link]
17
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
11
Slide 18
[Link]
18
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
12
Slide 19
KEY POINTS
Many issues to consider before conducting an
evaluation study.
These include: goals of the study; involvment or
not of users; the methods to use; practical &
ethical issues; how data will be collected, analyzed
& presented.
The DECIDE framework provides a useful
checklist for planning an evaluation study.
[Link]
19
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
13
Slide 20
INSPECTIONS METHODS
Walkthroughs involve stepping through a
pre-planned scenario noting potential
problems.
Experts use their knowledge of users &
technology to review software usability
Heuristic evaluation is a review guided by a
set of heuristics.
Expert critiques (crits) can be formal or
informal reports.
.
[Link]
20
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
14
Slide 21
COGNITIVE WALKTHROUGH
Cognitive Walkthrough is a method for evaluating user
interface by analyzing the mental processes required by
users.
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
15
Slide 22
COGNITIVE WALKTHROUGHS
Focus on ease of learning.
Designer presents an aspect of the design &
usage scenarios.
Expert is told the assumptions about user
population, context of use, task details.
One or more experts walk through the design
prototype with the scenario.
Experts are guided by 3 questions.
[Link]
22
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
16
Slide 23
THE 3 QUESTIONS
Will the correct action be sufficiently
evident to the user?
Will the user notice that the correct action
is available?
Will the user associate and interpret the
response from the action correctly?
As the experts work through the scenario
they note problems.
[Link]
23
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
17
Slide 24
PLURALISTIC WALKTHROUGH
Variation on the cognitive walkthrough
theme.
Performed by a carefully managed team.
The panel of experts begins by working
separately.
Then there is managed discussion that leads
to agreed decisions.
The approach lends itself well to
participatory design.
[Link]
24
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
18
Slide 25
PERFORMING A COGNITIVE
WALKTHROUGH
Choose a specific task from the suite of tasks the
interface is intended to support.
Determine one or more correct sequences of actions for
that task.
Examine these sequences in the context provided by
the interface.
Assess whether a hypothetical user would be able to
select an appropriate action at each point.
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
19
Slide 26
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
20
Slide 27
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
21
Slide 28
DISADVANTAGES OF COGNITIVE
WALKTHROUGH
Relies on analysis rather on user testing.
Provides a detailed examination of a particular task
rather than an overview of the interface.
Provides no quantitative data.
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
22
Slide 29
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
23
Slide 30
COGNITIVE WALKTHROUGH
PREPARATION
Define assumed user background
General knowledge
Computer knowledge
Task knowledge
Choose a sample task
Important
Realistic
Specify the correct action sequence(s) for the task
Determine the interface state along the sequence(s)
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
24
Slide 31
COGNITIVE WALKTHROUGH
ANALYSIS
For each action answer the following questions:
Will the user be trying to achieve the right effect?
Will the user notice the correct action is available?
Will the user associate the correct action with the
desired effect?
If the correct action is performed, will the user see
that progress is being made?
Based on the yes or no answer:
Explain why a user would choose that action
Explain why a user would not choose
that action
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
25
Slide 32
COGNITIVE WALKTHROUGH
FOLLOW-UP
Suggest where the design is likely to fail and why.
Provide specific guidance for each problem.
Indicate which the problems may be superficial and
where profound changes are needed.
Report the designers view of the interface and
eventual difference with the users view (if any).
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
26
Slide 33
HEURISTIC EVALUATION
Developed Jacob Nielsen in the early 1990s.
Based on heuristics distilled from an
empirical analysis of 249 usability
problems.
These heuristics have been revised for
current technology.
Heuristics being developed for mobile
devices, wearables, virtual worlds, etc.
Design guidelines form a basis for
developing heuristics.
[Link]
33
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
27
Slide 34
HEURISTIC EVALUATION
A systematic inspection of a user interface design by
interface specialists to determine the usability, based on
their experience, guidelines, and standards.
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
28
Slide 35
ADVANTAGES OF A HEURISTIC
EVALUATION
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
29
Slide 36
DISADVANTAGES OF A HEURISTIC
EVALUATION
Relies on analysis rather on user testing
Relies on the judgment of the evaluator and his/her
level of expertise
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
30
Slide 37
PERFORMING A HEURISTIC
EVALUATION
Use multiple evaluators for the best results. A single
evaluator will miss most of the usability problems in
an interface.
Each evaluator should inspect the interface on their
own.
An evaluation session generally lasts one or two
hours. Longer sessions are needed for more
complicated systems.
Evaluators individually decide how they want to
proceed with evaluating the interface.
Each evaluator provides a report or
recorded version.
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
31
Slide 38
HEURISTIC EVALUATION
PRINCIPLES
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
32
Slide 39
FEEDBACK
The system should always keep users informed about
what is going on through appropriate feedback within
reasonable time.
System feedback should be expressed in the users
language to guide and provide effective feedback.
Feedback must be provided in case of system failure.
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
33
Slide 40
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
34
Slide 41
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
35
Slide 42
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
36
Slide 43
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
37
Slide 44
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
38
Slide 45
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
39
Slide 46
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
40
Slide 47
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
41
Slide 48
DISCOUNT EVALUATION
Heuristic evaluation is referred to as discount
evaluation when 5 evaluators are used.
Empirical evidence suggests that on average 5
evaluators identify 75-80% of usability problems.
[Link]
48
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
42
Slide 49
[Link]
49
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
43
Slide 50
[Link]
50
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
44
Slide 51
[Link]
51
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
45
Slide 52
Clarity
Minimize unnecessary complexity & cognitive load
Provide users with context
Promote positive & pleasurable user experience
[Link]
52
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
__________
AM/Session12/SJSU/ISE217
46