IPDET
Module 8:
Selecting and Constructing
Data Collection
Instruments
Introduction
Data Collection Strategies
Characteristics of Good Measures
Quantitative and Qualitative Data
Tools for Collecting Data
IPDET 2009
22
Data Collection
Strategies
No one best way: decision depends on:
What you need to know: numbers or stories
Where the data reside: environment, files,
people
Resources and time available
Complexity of the data to be collected
Frequency of data collection
Intended forms of data analysis
IPDET 2009
33
Rules for Collecting
Data
Use multiple data collection methods
Use available data, but need to know
how the measures were defined
how the data were collected and cleaned
the extent of missing data
how accuracy of the data was ensured
IPDET 2009
44
Rules for Collecting
Data
If must collect original data:
be sensitive to burden on others
pre-test, pre-test, pre-test
establish procedures and follow them
(protocol)
maintain accurate records of
definitions and coding
verify accuracy of coding, data input
IPDET 2009
55
Structured Approach
All data collected in the same way
Especially important for multi-site
and cluster evaluations so you can
compare
Important when you need to make
comparisons with alternate
interventions
IPDET 2009
66
Use Structured Approach
When:
need to address extent questions
have a large sample or population
know what needs to be measured
need to show results numerically
need to make comparisons across
different sites or interventions
IPDET 2009
77
Semi-structured
Approach
Systematic and follow general
procedures but data are not
collected in exactly the same way
every time
More open and fluid
Does not follow a rigid script
may ask for more detail
people can tell what they want in their
own way
IPDET 2009
88
Use Semi-structured
Approach when:
conducting exploratory work
seeking understanding, themes,
and/or issues
need narratives or stories
want in-depth, rich, backstage
information
seek to understand results of data
that are unexpected
IPDET 2009
99
Characteristics of
Good Measures
Is
Is
Is
Is
the
the
the
the
IPDET 2009
measure
measure
measure
measure
relevant?
credible?
valid?
reliable?
11
00
Relevance
Does the
measure
capture what
matters?
IPDET 2009
Do not measure
what is easy
instead of what
is needed
11
11
Credibility
Is the measure believable? Will it
be viewed as a reasonable and
appropriate way to capture the
information sought?
IPDET 2009
11
22
Internal Validity
How well does
the measure
capture what it
is supposed to?
IPDET 2009
Are waiting
lists a valid
measure of
demand?
11
33
Reliability
A measures
precision and
stability- extent
to which the
same result
would be
obtained with
repeated trials
IPDET 2009
How reliable are:
birth weights of
newborn
infants?
speeds
measured by a
stopwatch?
11
44
Quantitative Approach
Data in numerical form
Data that can be precisely
measured
age, cost, length, height, area, volume,
weight, speed, time, and temperature
Harder to develop
Easier to analyze
IPDET 2009
11
55
Qualitative Approach
Data that deal with description
Data that can be observed or self-reported,
but not always precisely measured
Less structured, easier to develop
Can provide rich data detailed and
widely applicable
Is challenging to analyze
Is labor intensive to collect
Usually generates longer reports
IPDET 2009
11
66
Which Data?
If you:
Then Use:
- want to conduct statistical
analysis
- want to be precise
- know what you want to measure
- want to cover a large group
Quantitati
ve
- want narrative or in-depth
information
- are not sure what you are able to
measure
- do not need to quantify the
results
IPDET 2009
Qualitati
ve
11
77
Obtrusive vs.
Unobtrusive Methods
Obtrusive
data collection
methods that directly
obtain information
from those being
evaluated
e.g. interviews, surveys,
focus groups
IPDET 2009
Unobtrusive
data collection
methods that do not
collect information
directly from
evaluees
e.g., document analysis,
GoogleEarth,
observation at a
distance, trash of the
stars
11
88
How to Decide on Data
Collection Approach
Choice depends on the situation
Each technique is more
appropriate in some situations
than others
Caution: All techniques are subject
to bias
IPDET 2009
11
99
Triangulation to
Increase Accuracy of
Triangulation of methods
Data
collection of same information using
different methods
Triangulation of sources
collection of same information from a
variety of sources
Triangulation of evaluators
collection of same information from more
than one evaluator
IPDET 2009
22
00
Data Collection Tools
Participatory Methods
Records and Secondary Data
Observation
Surveys and Interviews
Focus Groups
Diaries, Journals, Self-reported Checklists
Expert Judgment
Delphi Technique
Other Tools
IPDET 2009
22
11
Tool 1: Participatory
Methods
Involve groups or communities
heavily in data collection
Examples:
community meetings
mapping
transect walks
IPDET 2009
22
22
Community Meetings
One of the most common
participatory methods
Must be well organized
agree on purpose
establish ground rules
who will speak
time allotted for speakers
format for questions and answers
IPDET 2009
22
33
Mapping
Drawing or using existing maps
Useful tool to involve stakeholders
increases understanding of the community
generates discussions, verifies secondary
sources of information, perceived changes
Types of mapping:
natural resources, social, health, individual
or civic assets, wealth, land use,
demographics
IPDET 2009
22
44
Transect Walks
Evaluator walks around community
observing people, surroundings,
and resources
Need good observation skills
Walk a transect line through a map
of a community line should go
through all zones of the community
IPDET 2009
22
55
Tool 2: Records and
Secondary
Data
Examples of sources:
files/records
computer data bases
industry or government reports
other reports or prior evaluations
census data and household survey data
electronic mailing lists and discussion groups
documents (budgets, organizational charts,
policies and procedures, maps, monitoring
reports)
newspapers and television reports
IPDET 2009
22
66
Using Existing Data
Sets
Key issues: validity, reliability,
accuracy, response rates, data
dictionaries, and missing data
rates
IPDET 2009
22
77
Advantage/Challenge:
Available Data
Advantages
Often less expensive and
faster than collecting the
original data again
Challenges
There may be coding errors or
other problems. Data may not
be exactly what is needed.
You may have difficulty
getting access. You have to
verify validity and reliability
of data
IPDET 2009
22
88
Tool 3: Observation
See what is happening
traffic patterns
land use patterns
layout of city and rural areas
quality of housing
condition of roads
conditions of buildings
who goes to a health clinic
IPDET 2009
22
99
Observation is Helpful
when:
need direct information
trying to understand ongoing
behavior
there is physical evidence, products,
or outputs than can be observed
need to provide alternative when
other data collection is infeasible or
inappropriate
IPDET 2009
33
00
Degree of Structure of
Observations
Structured: determine, before the
observation, precisely what will be
observed before the observation
Unstructured: select the method
depending upon the situation with no
pre-conceived ideas or a plan on what to
observe
Semi-structured: a general idea of what
to observe but no specific plan
IPDET 2009
33
11
Google Earth
Maps and satellite images for
complex or pinpointed regional
searches
Has an Advanced version and an
Earth Outreach version
Web site for Google Earth
[Link]
IPDET 2009
33
22
Ways to Record
Information from
Observations
Observation
guide
printed form with space to record
Recording sheet or checklist
Yes/no options; tallies, rating scales
Field notes
least structured, recorded in
narrative, descriptive style
IPDET 2009
33
33
Guidelines for Planning
Observations
Have more than one observer, if
feasible
Train observers so they observe the
same things
Pilot test the observation data
collection instrument
For less structured approach, have a
few key questions in mind
IPDET 2009
33
44
Advantages and
Challenges:
Observation
Advantage
Collects data on actual vs. selfs
reported behavior or
perceptions. It is real-time vs.
retrospective
Challenges Observer bias, potentially
unreliable; interpretation and
coding challenges; sampling can
be a problem; can be labor
intensive; low response rates
IPDET 2009
33
55
Tool 4: Surveys and
Interviews
Excellent for asking people about:
perceptions, opinions, ideas
Less accurate for measuring
behavior
Sample should be representative
of the whole
Big problem with response rates
IPDET 2009
33
66
Structures for Surveys
Structured:
Precisely worded with a range of predetermined responses that the respondent can
select
Everyone asked exactly the same questions in
exactly the same way, given exactly the same
choices
Semi-structured
Asks same general set of questions but
answers to the questions are predominantly
open-ended
IPDET 2009
33
77
Structured vs.
Semi-structured
harder to develop
StructureSurveys
easier to complete
d
easier to analyze
more efficient when working with large
numbers
easier to develop: open ended questions
Semistructure more difficult to complete: burdensome for
people to complete as a self-administrated
d
questionnaire
harder to analyze but provide a richer
source of data, interpretation of openIPDET 2009ended responses subject to bias
33
88
Modes of Survey
Administration
Telephone surveys
Self-administered questionnaires
distributed by mail, e-mail, or
websites
Administered questionnaires,
common in the development context
In development context, often issues
of language and translation
IPDET 2009
33
99
Mail / Phone / Internet
Surveys
Literacy issues
Consider accessibility
reliability of postal service
turn-around time
Consider bias
What population segment has
telephone access? Internet access?
IPDET 2009
44
00
Advantages and
Challenges of Surveys
Advantage Best when you want to know
s
what people think, believe, or
perceive, only they can tell you
that
Challenges People may not accurately recall
their behavior or may be
reluctant to reveal their behavior
if it is illegal or stigmatized. What
people think they do or say they
do is not always the same as
44
IPDET 2009
what they actually do.
11
Interviews
Often semi-structured
Used to explore complex issues in
depth
Forgiving of mistakes: unclear
questions can be clarified during the
interview and changed for subsequent
interviews
Can provide evaluators with an
intuitive sense of the situation
IPDET 2009
44
22
Challenges of
Interviews
Can be expensive, labor intensive,
and time consuming
Selective hearing on the part of
the interviewer may miss
information that does not conform
to pre-existing beliefs
Cultural sensitivity: e.g., gender
issues
IPDET 2009
44
33
Tool 5: Focus Groups
Type of qualitative research where
small homogenous groups of
people are brought together to
informally discuss specific topics
under the guidance of a moderator
Purpose: to identify issues and
themes, not just interesting
information, and not counts
IPDET 2009
44
44
Focus Groups Are
Inappropriate when:
language barriers are
insurmountable
evaluator has little control over the
situation
trust cannot be established
free expression cannot be ensured
confidentiality cannot be assured
IPDET 2009
44
55
Focus Group Process
Phase
Openin
g
Action
Ice-breaker; explain purpose; ground rules;
introductions
2 Warm- Relate experience; stimulate group
up
interaction; start with least threatening and
simplest questions
3 Main
body
4 Closur
Move to more threatening or sensitive and
complex questions; elicit deep responses;
connect emergent data to complex, broad
participation
End with closure-type questions; summarize
e
and refine; present theories, etc; invite final
44
IPDET 2009
comments or insights; thank participants66
Advantages and
Challenges of Focus
Groups
Advantage
Can be conducted relatively quickly and
s
easily; may take less staff time than indepth, in-person interviews; allow
flexibility to make changes in process
and questions; can explore different
perspectives; can be fun
Challenge
s
Analysis is time consuming; participants
not be representative of population,
possibly biasing the data; group may be
influenced by moderator or dominant
group members
IPDET 2009
44
77
Tool 6: Diaries and SelfReported Checklists
Use when you want to capture
information about events in
peoples daily lives
Participants capture experiences in
real-time not later in a
questionnaire
Used to supplement other data
collection
IPDET 2009
44
88
Step
Guidelines for Diaries
or Journals
Process
1 Recruit people face-to-face
encourage participation, appeal to altruism,
assure
confidentiality, provide incentive
2 Provide a booklet to each participant
cover page with clear instructions, definitions,
example
short memory-joggers, explain terms, comments
on last
page , calendar
3 Consider the time-period for collecting data
IPDET 2009
44
99
Self-reported
Checklists
Cross between a questionnaire and a
diary
The evaluator specifies a list of
behaviors or events and asks the
respondents to complete the checklist
Done over a period of time to capture
the event or behavior
More quantitative approach than diary
IPDET 2009
55
00
Advantages and Challenges of
Diaries and Self-reported
Checklists
Advantage Can capture in-depth, detailed data that might be
otherwise forgotten
s
Can collect data on how people use their time
Can collect sensitive information
Supplements interviews provide richer data
Challenge
s
IPDET 2009
Requires some literacy
May change behavior
Require commitment and self-discipline
Data may be incomplete or inaccurate
Poor handwriting, difficult to understand phrases
55
11
Tool 7: Expert Judgment
Use of experts,
one-on-one or as a
panel
E.g., Government
task forces,
Advisory Groups
IPDET 2009
Can be structured
or unstructured
Issues in
selecting experts
55
22
Selecting Experts
Establish criteria for selecting
experts not only on recognition as
expert but also based on:
areas of expertise
diverse perspectives
diverse political views
diverse technical expertise
IPDET 2009
55
33
Advantages and
Challenges of Expert
Judgment
Advantage
Fast, relatively inexpensive
s
Challenges Weak for impact evaluation
May be based mostly on
perceptions
Value of data depends on how
credible the experts are
perceived to be
IPDET 2009
55
44
Tool 8: Delphi Technique
Enables experts to engage remotely in a
dialogue and reach consensus, often about
priorities
Experts asked specific questions; often
rank choices
Responses go to a central source, are
summarized and fed back to the experts
without attribution
Experts can agree or argue with others
comments
Process may be iterative
IPDET 2009
55
55
Advantages and
Challenges of Delphi
Advantage Allows participants to remain anonymous
Technique
s
Is inexpensive
Is free of social pressure, personality
influence, and individual dominance
Is conducive to independent thinking
Allows sharing of information
Challenges May not be representative
Has tendency to eliminate extreme
positions
Requires skill in written communication
Requires time and participant
55
commitment
IPDET 2009
6
6
Other Measurement
Tools
-
scales (weight)
tape measure
stop watches
chemical tests :
i.e. quality of
water
IPDET 2009
- health testing
tools:
i.e. blood pressure
- aptitude and
achievement
tests
-citizen report
cards
55
77
Data Collection
Summary
Choose more than one data
collection technique
No best tool
Do not let the tool drive your work
but rather choose the right tool to
address the evaluation question
IPDET 2009
55
88
A Final Note.
I never guess. It is a capital mistake
to theorize before one has data.
Insensibly one begins to twist facts and
theories,
instead of theories to suit facts.
--Sir Arthur Conan Doyle
Question
s?
IPDET 2009
55
99
