[Link] Suppiah Shanmugam, PhD.
TIMSS International Results
TIMSS Released Items
TIMSS Technical Report
[Link]
Personal Communication with IEA
TIMSS
Trends in International Mathematics and Science
Study
International assessment on Maths/Science for
grade 4/Grade 8
Conducted by International Association for
Evaluation of Educational Achievement (IEA)
Every four years since 1995 (TIMSS 2011- 5th cycle)
Development and Administration
Maths/Sc Test Booklets
Student/Teacher/School/Curriculum Questionnaires
Maths Questionnaires
Student Questionnaire(S)
Home Experience
School Experience
Teacher Questionnaire (T)
Education
Professional Development
Teaching Experience
School Questionnaire(P)
Availability of Resources
Types of Programmes
Learning Environment
Curriculum Questionnaire(C)
Organisation of Maths/Sc Curriculum
Content of Maths/Sc Curriculum
3
Student Testing Time
Student Achievement Booklet Part 1
45 minutes
Break
Student Achievement Booklet Part 2
45 minutes
Break
Student Questionnaire
30 minutes
Student Achievement Booklet
28 item blocks
14 Maths
14 Sc
Assigned to 14 Booklets
Each booklet has 4 blocks
2 maths
2 Sc
Each block has 12- 18 items and in two booklets
(Linking)
MCQ
Structured Response Questions
-
Provide explanation
Support an answer with reasons
Numerical evidence
Draw diagram
Display data
One block- score points of 18 (on average)
Maths/Sc blocks alternately begin (balance position
effect)
TIMSS Mathematics Assessment
Framework
TIMSS 2011 Mathematics Framework
is organized around two dimensions
a content dimension
specifying the domains or subject matter to be assessed
within mathematics (number, algebra, geometry, and
data and chance)
Has several topic areas
Each topic area is presented as a list of objectives
a cognitive dimension
specifying the domains or thinking processes to be
assessed (knowing, applying, and reasoning)
The cognitive domains describe the sets of behaviors
expected of students as they engage with the
mathematics content.
TIMSS Maths Content Domain
Number (30%)
Algebra
(30%)
Whole Num
Fractions/Decimals
Integers
Ratio/Proportion/
Percent
Patterns
Algebraic Expressions
Equation/Formula and Function
Geometry
(20%)
Data & Chance
(20%)
G Shapes
Data organisation &representation
G Measurement
Data Interpretation
Location & Movement Chance
Cognitive Domain
Knowing Using mathematics
Depends on mathematical knowledge
(35%)
Familiarity with mathematics concept
Facts - factual knowledge that provides the basic
language of mathematics, and the essential
mathematical facts and properties that form the
foundation for mathematical thought.
Procedures- entails recall of sets of actions and
how to carry them out & computational
procedures and tools
Knowledge of concepts - make connections
between elements of knowledge, judge the validity
of mathematical
statements and methods, and create mathematical
representations
Cognitive Domain
Applying
(40%)
set in reallife situations
purely
mathematical
questions
emphasis
more familiar
and routine
tasks
application of mathematical tools in a
range of contexts.
The facts, concepts, and procedures often
are very familiar to the student, with the
problems being routine ones.
apply mathematical knowledge of facts,
skills, and procedures or understanding of
mathematical concepts to create
representations
Problem solving is central but the problem
settings are more routine in the
implemented curriculum.
have been standard in classroom exercises
textbook problems
Cognitive Domain
Reasoning
(25%)
for logical, systematic thinking.
includes intuitive and inductive reasoning
based on patterns and regularities that can be
novelty of the context
the complexity of the
situation,
any solution to the
problem must involve
several steps,
drawing on knowledge
and understanding
from different areas of
mathematics
involve transfer of
knowledge and skills to
new situations
Interactions among
reasoning skills
used to arrive at solutions to non-routine
problems.
Non-routine problems - problems that are
very likely to be unfamiliar to students.
They make cognitive demands over and
above those needed for solution of routine
problems, even when the knowledge and
skills required for their solution have been
learned.
Cognitive Domain
Knowing
(35%)
Applying
(40%)
Reasoning
(25%)
12
TOPIC AREA FRACTIONS AND DECIMALS
CONTENT DOMAIN NUMBER
COGNITIVE DOMAIN KNOWING
13
TOPIC AREA FRACTIONS AND DECIMALS
CONTENT DOMAIN NUMBER
COGNITIVE DOMAIN KNOWING
14
TOPIC AREA FRACTIONS AND DECIMALS
CONTENT DOMAIN NUMBER
COGNITIVE DOMAIN APPLYING
15
TOPIC AREA FRACTIONS AND DECIMALS
CONTENT DOMAIN NUMBER
COGNITIVE DOMAIN APPLYING
16
TOPIC AREA RATIO, PROPORTION AND PERCENT
CONTENT DOMAIN NUMBER
COGNITIVE DOMAIN APPLYING
17
TOPIC AREA GEOMETRIC SHAPES
CONTENT DOMAIN GEOMETRY
COGNITIVE DOMAIN REASONING
18
TOPIC AREA INTEGER
CONTENT DOMAIN NUMBER
COGNITIVE DOMAIN REASONING
19
TOPIC AREA GEOMETRIC SHAPES
CONTENT DOMAIN GEOMETRY
COGNITIVE DOMAIN REASONING
20
TOPIC AREA DATA
CONTENT DOMAIN DATA AND CHANCE
COGNITIVE DOMAIN REASONING
21
INTERNATIONAL BENCHMARK ADVANCED (625)
-reason with information, draw conclusions, make generalizations, and solve linear
equations.
-solve a variety of fraction, proportion, and percent problems and justify
conclusions.
-express generalizations algebraically and model situations.
-solve a variety of problems involving equations, formulas, and functions.
-reason with geometric figures to solve problems and with data from several sources
or unfamiliar representations to solve multi-step problems
Content Domain: Geometry
Cognitive Domain: Reasoning
Description: Solves a word
problem involving filling a
three-dimensional shape with
rectangular solids
22
INTERNATIONAL BENCHMARK HIGH (550)
-apply understanding and knowledge in complex situation
-use information from several sources to solve problems involving different
types of numbers and operations.
-relate fractions, decimals, and percents to each other.
-show basic procedural knowledge related to algebraic expressions.
-use properties of lines, angles, triangles, rectangles, and rectangular prisms to
solve problems.
-analyse data in a variety of graphs.
Peter, James, and Andrew each had 20 tries at throwing balls
into a basket. Complete the missing boxes below.
Name
Number of
Percentage of
Successful Shots Successful
Shots
Peter
10 out of 20
James
15 out of 20
Andrew
out of 20
50 %
80%
Content Domain:
Number
Cognitive Domain:
Knowing
Description: Given
the part and the
whole, can express
the part as a
percentage, and
given the whole and
the %, can find the
part
23
INTERNATIONAL BENCHMARK INTERMEDIATE (475)
-apply basic mathematical knowledge in a variety of situations.
-solve problems involving decimals, fractions, prop,%
-understand simple algebraic relationships.
-relate a two-dimensional drawing to a three-dimensional object.
-read, interpret, and construct graphs and tables.
-recognise basic notions of likelihood.
What does xy + 1 mean?
o a Add 1 to y, then multiply by x.
o b Multiply x and y by 1.
o c Add x to y, then add 1.
od Multiply x by y, then add 1.
Content Domain: Algebra
Cognitive Domain:
Knowing
Description: Knows the
meaning of a simple
algebraic expression
involving multiplication
and addition
24
INTERNATIONAL BENCHMARK LOW (440)
Have some knowledge of whole numbers and decimals,
operations, and basic graphs
42.65 + 5.748 =
Answer: ________
Content Domain:
Number
Cognitive Domain:
Knowing
Description: Adds a
two-place and a
three-place decimal
25
Malaysian Students Performance
26
Administration in Malaysia
45 countries
Four countries in SEA
(Indonesia, Malaysia, Singapore, Thailand)
180 Schools
5733 Form Two students
Test Booklets
Bahasa Malaysia and English
Student Questionnaire
Bahasa Malaysia
Maths Teacher Questionnaire
English
School Questionnaire
English
27
Malaysia Performance Trend
Only in Grade 8
Participated since 1999
1999- 16th (39 countries) - BM
2003- 10th (48 countries) - BM
2007- 20th (46 countries) - BM/ENG
2011- 26th (45 countries) - BM/ENG
Recorded as one of the countries that declined greatly (40
points or more)
540
520
519
508
500
480
474
460
440
440
420
400
1999
2003
2007
2011
28
Performance Trend- Content Domain
500
494
480
474
474
460
459
455
451
2011
440
440
430
432
429
420
400
380
Number
Algebra
Geometry
Data & Chance
Num Geometry Data & Chance / Algebra
Num Overall
Geometry / Data & Chance / Algebra Overall
Overall
29
Performance Trend - Cognitive Domain
480
475
470
477
474
473
466
465
460
455
450
445
440
444
435
440
439
430
2007
2011
426
425
420
415
Knowing
Applying
Reasoning
Overall
Low performance in Reasoning
Not much difference between Knowing/Applying
30
Performance Trend -Content Domain
470
by gender
Girls
465
460
Boys
460
Overall
455
450
451
445
440
441
440
438
435
430
435
432
430
428
425
420
425
419
422
415
410
Number
Algebra
Geometry
Data&Chance
Girls Overall Boys
31
Performance Trend- Cognitive Domain
by gender
460
455
Girls
456
Boys
450
445
Overall
445
444
440
439
435
430
432
431
432
425
420
420
415
Knowing
Applying
Reasoning
Girls Overall Boys
32
ACHIEVEMENT FOR MALAYSIA AT INTERNATIONAL BENCHMARK
100
90
80
70
60
1999
93
2003
93
82
65
2007
70
2011
66
50
50
40
36
36
30
30
20
13
18
10
10
6
2
0
Low
Intermediate
High
Advanced
At every benchmark, there is a decline
33
Percentile points for Malaysia
700
630
608
600
589
562
507
500
455
416
400
388
530
507
475
440
373
2011
2007
2003
319
300
290
200
100
0
5th
25th
50th
75th
95th
34
PERCENTILE POINTS for MALAYSIA & SINGAPORE
in TIMSS 2011
800
734
713
700
672
620
600
589
559
560
507
494
500
453
440
Singapore
Malaysia
400
373
319
300
290
200
100
0
5th
10th
25th
50th
75th
90th
95th
35
TIMSS Mathematics Items
36
Question Types and Scoring Procedures
2 question formats- MCQ & constructed-response
Each MCQ is worth 1 score point.
Constructed-response questions worth 1 or 2 score
points, depending on the nature of the task and the
skills required to complete it.
The choice of item format depends on the mathematics
or being assessed, and the format that best enables
students to demonstrate their proficiency.
Multiple-Choice Questions (MCQ)
MCQ provide students with 4 response options.
Select one correct response
can be used to assess any of the behaviors in the cognitive
domains.
allow valid, reliable, and economical measurement of a wide
range of content in a relatively short testing time.
do not allow for students explanations or supporting statements
less suitable for assessing students ability to make more
complex interpretations or evaluations.
linguistic features need to be appropriate.
questions are written clearly and concisely.
response options also are written to minimise the reading
load of the question.
The options that are incorrect are written to be
plausible, but not deceptive.
Constructed-Response Questions (CRQ)
students are required to construct/give a written
response, rather than select a response from a set of options
allow students to provide explanations, support an answer
with reasons or numerical evidence, draw diagrams, or
display data,
well-suited for assessing aspects of knowledge and skills
that require students to explain phenomena or interpret
data based on their background knowledge and experience
real-world setting - the setting is familiar to students.
need to prepare a Scoring Rubric
Scoring Rubric
describes the essential features of appropriate and complete
responses.
focus on evidence of the type of behavior the question assesses.
describe evidence of completely correct, partially correct and
completely incorrect responses.
student responses at each level of understanding provide
important guidance to those who will be rating the students
responses.
In scoring, the focus is solely on students achievement with
respect to the topic being assessed, not on their ability to write
well.
students need to communicate clearly
Types of Items- Grade 8
Types of Items- Grade 8
Scoring Rubric
Types of Items- Grade 8
Types of Items- Grade 8
Types of Items- Grade 8
TIMSS Item Writing Process
47
Item Writing Process and Guidelines
What should the student know?
What should the student be able to do?
consider the timing, grade appropriateness, difficulty
level, potential sources of bias (cultural, gender, or
geographical, ).
Make sure that item validity is not affected by factors that
unnecessarily increase the difficulty of the item, such as
unfamiliar
or
difficult
vocabulary, grammar, directions, contexts, or stimulus
materials
be sensitive to the possibility of unintentionally placing
particular groups of students at an unfair disadvantage
diagrams and graphs are drawn accurately (to scale unless
otherwise noted), and are correctly and fully labeled.
MCQ
Stem is the initial part of the item in which the task is defined.
Options refer to the entire set of labeled response choices presented
under the stem.
Key is the correct response option.
Distracters are the incorrect response options.
ask a direct question with only one correct answer, and provide plausible
distracters
the question must be able to stand alone, and be answerable without
the response options
do not include extraneous information that can confuse students
avoid questions for which a wrong method yields the correct answer
(e.g., a question about a circle with a radius of 2, since computing either
area or circumference get 4)
avoid writing items where students can work backwards from the
response options to find the correct answer (e.g., solving for x in an
equation).(CRQ is more appropriate)
CRQ
Write
a full credit
answer in terms of the
language, knowledge, a
nd skills that student
could be expected to
possess & determine
whether to allocate 1 or
2 score points.
Develop
a
specific
scoring guide
Take Home information
Each of the items needs to contribute to the overall
mathematics test
Some relatively easy items and some challenging items
Avoid items that almost all students or almost no
students are able to answer correctly
Checklist- MCQ
Checklist-CRQ
Each item
1The Content Domain, topic area, and objective the
item measures
2. The Cognitive Domain
3. The item number (1, 2, 3, etc.)
4. The key (multiple-choice items only) or
5. The scoring guide
