Chapter 2: Getting to Know Your Data

 Data Objects and Attribute Types

 Basic Statistical Descriptions of Data

 Data Visualization

 Measuring Data Similarity and Dissimilarity

 Summary

1
 Measuring the Dispersion of Data
 Quartiles, outliers and boxplots
 Quartiles: Q1 (25th percentile), Q3 (75th percentile)
 Inter-quartile range: IQR = Q3 – Q1
 Five number summary: min, Q1, median, Q3, max
 Boxplot: ends of the box are the quartiles; median is marked; add
whiskers, and plot outliers individually
 Outlier: usually, a value higher/lower than 1.5 x IQR
 Variance and standard deviation (sample: s, population: σ)
 Variance: (algebraic, scalable computation)
1 n 1 n 2 1 n 1 n 1 n 2
2
s
n 1
i1 (xi  x)
2

n 1[

i1 xi  ( i1
2
i) ]    (x i   ) 
2

N i1
2

N  i1 x
i  
2


n  x
 Standard deviation s (or σ) is the square root of variance s2 (or σ2)

2
 Boxplot Analysis
 Five-number summary of a distribution
 Minimum, Q1, Median, Q3, Maximum
 Boxplot
 Data is represented with a box
 The ends of the box are at the first and third
quartiles, i.e., the height of the box is IQR
 The median is marked by a line within the
box
 Whiskers: two lines outside the box
extended
to Minimum and Maximum
 Outliers: points beyond a specified outlier
threshold, plotted individually
3
 Visualization of Data Dispersion: 3-D Boxplots

March 30, 2021 Data Mining: Concepts and Techniques 4

 Properties of Normal Distribution
Curve
 The normal (distribution) curve
 From μ–σ to μ+σ: contains about 68% of the

measurements (μ: mean, σ: standard

deviation)
 From μ–2σ to μ+2σ: contains about 95% of it
 From μ–3σ to μ+3σ: contains about 99.7% of it

5
Graphic Displays of Basic Statistical
Descriptions
 Boxplot: graphic display of five-number summary
 Histogram: x-axis are values, y-axis repres. frequencies
 Quantile plot: each value xi is paired indicating
with fi
that approximately 100 fi % of data are  xi
 Quantile-quantile (q-q) plot: graphs the quantiles of
one univariant distribution against the corresponding
quantiles of another
 Scatter plot: each pair of values is a pair of coordinates
and plotted as points in the plane
6
 Histogram
Analysis
 Histogram: Graph display of
tabulated frequencies, shown as 4
0
bars
 It shows what proportion of cases 3
5
fall into each of several categories
 Differs from a bar chart in that it is 3
0
the area of the bar that denotes the 20
value, not the height as in bar 2
1
5
5
charts, a crucial distinction when the10
categories are not of uniform width
5
 The categories are usually specified
0
as non-overlapping intervals of 1000 3000 5000 7000 9000
some variable. The categories (bars) 0 0 0 0 0

must be adjacent

7
Histograms Often Tell More than Boxplots

 The two histograms

shown in the left may
have the same boxplot
representation
 The same values
for: min, Q1,
median, Q3, max
 But they have rather
different data
distributions

8
 Quantile-Quantile (Q-Q) Plot
 Graphs the quantiles of one univariate distribution against the
corresponding quantiles of another
 View: Is there is a shift in going from one distribution to
another?
 Example shows unit price of items sold at Branch 1 vs. Branch 2 for
each quantile. Unit prices of items sold at Branch 1 tend to be
lower than those at Branch 2.

9
 Scatter
plot
 Provides a first look at bivariate data to see clusters of
points, outliers, etc
 Each pair of values is treated as a pair of coordinates and
plotted as points in the plane

10
Positively and Negatively Correlated Data

 The left half fragment is positively

correlated
 The right half is negative correlated

11
Uncorrelated Data

12
 Chapter 2: Getting to Know Your Data

 Data Objects and Attribute Types

 Basic Statistical Descriptions of Data

 Data Visualization

 Measuring Data Similarity and Dissimilarity

 Summary

13
 Data Visualization
 Why data visualization?
 Gain insight into an information space by mapping data onto graphical
primitives
 Provide qualitative overview of large data sets
 Search for patterns, trends, structure, irregularities, relationships among
data
 Help find interesting regions and suitable parameters for further
quantitative analysis
 Provide a visual proof of computer representations derived
 Categorization of visualization methods:
 Pixel-oriented visualization techniques
 Geometric projection visualization techniques
 Icon-based visualization techniques
 Hierarchical visualization techniques
 Visualizing complex data and relations
14
 Pixel-Oriented Visualization Techniques
 For a data set of m dimensions, create m windows on the screen, one
for each dimension
 The m dimension values of a record are mapped to m pixels at the
corresponding positions in the windows
 The colors of the pixels reflect the corresponding values

(a) Income (b) Credit Limit (c) transaction volume (d) age
15
 Laying Out Pixels in Circle Segments
 To save space and show the connections among multiple dimensions,
space filling is often done in a circle segment

(a) Representing a data record

(b) Laying out pixels in circle segment
in circle segment
16
Geometric Projection Visualization Techniques
 Visualization of geometric transformations and projections
of the data
 Methods
 Direct visualization
 Scatterplot and scatterplot matrices
 Landscapes
 Projection pursuit technique: Help users find meaningful
projections of multidimensional data
 Prosection views
 Hyperslice
 Parallel coordinates
17
 Scatterplot Matrices

Used by ermission of M. Ward, Worcester Polytechnic Institute

Matrix of scatterplots (x-y-diagrams) of the k-dim. data [total of (k2/2-k) scatterplots]

18
 Parallel Coordinates
 n equidistant axes which are parallel to one of the screen axes and
correspond to the attributes
 The axes are scaled to the [minimum, maximum]: range of the
corresponding attribute
 Every data item corresponds to a polygonal line which intersects each
of the axes at the point which corresponds to the value for the
attribute

• • •

Attr. 1 Attr. 2 Attr. 3 Attr. k

19
Parallel Coordinates of a Data Set

20
 Icon-Based Visualization Techniques

 Visualization of the data values as features of icons

 Typical visualization methods
 Chernoff Faces
 Stick Figures
 General techniques
 Shape coding: Use shape to represent certain
information encoding
 Color icons: Use color icons to encode more
information
 Tile bars: Use small icons to represent the relevant
feature vectors in document retrieval
21
 Chernoff Faces
 A way to display variables on a two-dimensional surface, e.g., let x be
eyebrow slant, y be eye size, z be nose length, etc.
 The figure shows faces produced using 10 characteristics--head
eccentricity, eye size, eye spacing, eye eccentricity, pupil size,
eyebrow slant, nose size, mouth shape, mouth size, and mouth
opening): Each assigned one of 10 possible values, generated using
Mathematica (S. Dickson)
 REFERENCE: Gonick, L. and Smith, W. The
Cartoon Guide to Statistics. New York:
Harper Perennial, p. 212, 1993
 Weisstein, Eric W. "Chernoff Face." From
MathWorld--A Wolfram Web Resource.
[Link]/[Link]

22
 Hierarchical Visualization Techniques

 Visualization of the data using a hierarchical

partitioning into subspaces
 Methods
 Dimensional Stacking
 Worlds-within-Worlds
 Tree-Map
 Cone Trees
 InfoCube

23
 Dimensional
Stacking
attribute
attribute 4
2
attribute
3
attribute 1

 Partitioning of the n-dimensional attribute space in 2-D

subspaces, which are ‘stacked’ into each other
 Partitioning of the attribute value ranges into classes.
The important attributes should be used on the outer
levels.
 Adequate for data with ordinal attributes of low cardinality
 But, difficult to display more than nine dimensions
 Important to map dimensions appropriately 24
 Dimensional
Stacking
Used by permission of M. Ward, Worcester Polytechnic Institute

Visualization of oil mining data with longitude and latitude mapped to the
outer x-, y-axes and ore grade and depth mapped to the inner x-, y-axes
25
 Tree-Map
 Screen-filling method which uses a hierarchical partitioning
of the screen into regions depending on the attribute values
 The x- and y-dimension of the screen are partitioned
alternately according to the attribute values (classes)

MSR Netscan Image

Ack.: [Link] 26
Tree-Map of a File System (Schneiderman)

27
 InfoCub
e
 A 3-D visualization technique where hierarchical
information is displayed as nested semi-transparent
cubes
 The outermost cubes correspond to the top level
data, while the subnodes or the lower level data
are represented as smaller cubes inside the
outermost cubes, and so on

28
 Three-D Cone Trees
 3D cone tree visualization technique
works well for up to a thousand nodes or
so
 First build a 2D circle tree that arranges
its nodes in concentric circles centered on
the root node
 Cannot avoid overlaps when projected to
2D
 G. Robertson, J. Mackinlay, S. Card. “Cone
Trees: Animated 3D Visualizations of
Hierarchical Information”, ACM SIGCHI'91
 Graph from Nadeau Software Consulting
website: Visualize a social network data set
that models the way an infection spreads
from one person to the next
29
 Visualizing Complex Data and Relations
 Visualizing non-numerical data: text and social networks
 Tag cloud: visualizing user-generated tags
 The importance of
tag is represented
by font size/color
 Besides text data,
there are also
methods to visualize
relationships, such as
visualizing social
networks

Newsmap: Google News Stories in 2005

 Chapter 2: Getting to Know Your Data

 Data Objects and Attribute Types

 Basic Statistical Descriptions of Data

 Data Visualization

 Measuring Data Similarity and Dissimilarity

 Summary

31
 Similarity and Dissimilarity
 Similarity
 Numerical measure of how alike two data objects are

 Value is higher when objects are more alike

 Often falls in the range [0,1]

 Dissimilarity (e.g., distance)

 Numerical measure of how different two data objects

are
 Lower when objects are more alike

 Minimum dissimilarity is often 0

 Upper limit varies

 Proximity refers to a similarity or dissimilarity

32
 Data Matrix and Dissimilarity Matrix
 Data matrix
 n data points with p
 x 11 ... x1f ... x 1p 
dimensions 
... ... 
... ... ...
 Two modes  x i1 
... xif ... ip 
 ... x ... 
 x ... ...
x nf ... 
n1 x np
 Dissimilarity matrix  ... ... 
 0 
 n data points, but  d(2,1) 0 
registers only the  
 d(3,1) d (3,2) 0 
distance 
 A
 
triangular : : :
... 0
matrix d d (n,2) 
 Single mode (n,1)
... 33
 Proximity Measure for Nominal Attributes

 Can take 2 or more states, e.g., red, yellow, blue,

green (generalization of a binary attribute)
 Method 1: Simple matching
 m: # of matches, p: total # of variables

d ( i , j )  p p m
 Method 2: Use a large number of binary attributes
 creating a new binary attribute for each of the
M nominal states

34
 Proximity Measure for Binary
Attributes Object j
 A contingency table for binary data
Object i

 Distance measure for symmetric

binary variables:
 Distance measure for
asymmetric
binary variables:
 Jaccard coefficient (similarity
measure for asymmetric binary
variables):
 Note: Jaccard coefficient is the
same as “coherence”:

35
 Dissimilarity between Binary Variables
 Example
Name Gender Fever Cough Test-1 Test-2 Test-3 Test-4
Jack M Y N P N N N
Mary F Y N P N P N
Jim M Y P N N N N
 Gender is a symmetric attribute
 The remaining attributes are asymmetric binary
 Let the values Y and P be 1, and the value N 0
0  1
d ( jack , mary )   0.33
2  0  1
1  1
d ( jack , jim)   0.67
1  1  1
1  2
d ( jim, mary )   0.75
1  1  2
36
 Standardizing Numeric Data
 Z-score: z  x

X: raw score to be standardized, μ: mean of the population, σ:



standard deviation
 the distance between the raw score and the population mean in
units of the standard deviation
 negative when the raw score is below the mean, “+” when
above
 An alternative way: Calculate the mean absolute deviation
s f n1 (| x1 mf || x2 f mf |...| xnf mf
where f
m f  n1|)(x1 f  x2 f  ...  nf
)
xif  f
.

x zif  m s
 standardized measure (z- f
 score
Using ): absolute deviation is more robust than using standard
mean
deviation

37
 Example:
Data Matrix and Dissimilarity
Matrix
Data Matrix
point attribute1 attribute2
x1 1 2
x2 3 5
x3 2 0
x4 4 5

Dissimilarity Matrix
(with Euclidean Distance)
x1 x2 x3 x4
x1 0
x2 3.61 0
x3 5.1 5.1 0
x4 4.24 1 5.39 0

38
 Ordinal Variables

 An ordinal variable can be discrete or continuous

 Order is important, e.g., rank
 Can be treated like interval-scaled
 replace x
if by their rank
r i f {1,...,M f }
 map the range of each variable onto [0, 1] by replacing
i-th object in the f-th variable by
 1
zi f  r if
M f
 1
 compute the dissimilarity using methods for interval-
scaled variables

39
 Attributes of Mixed
Type
 A database may contain all attribute types
 Nominal, symmetric binary, asymmetric binary, numeric,

ordinal
 One may use a weighted formula to combine their effects

 pf  ( f )d
 1 ij
d(i, j)  (f)
ij
 pf  1ij
 f is binary or nominal:( f )
dij(f) = 0 if xif = xjf , or dij (f) = 1 otherwise
 f is numeric: use the normalized distance
 f is ordinal
 Compute ranks r and 1
if
zif  r
 Treat z as interval-scaled
if
M 1
if
f

40
 Cosine Similarity
 A document can be represented by thousands of attributes, each
recording the frequency of a particular word (such as keywords) or
phrase in the document.

 Other vector objects: gene features in micro-arrays, …

 Applications: information retrieval, biologic taxonomy, gene feature
mapping, ...
 Cosine measure: If d1 and d2 are two vectors (e.g., term-frequency
vectors), then
cos(d1, d2) = (d1  d2) /||d1|| ||d2|| ,
where  indicates vector dot product, ||d||: the length of vector
d
41
 Example: Cosine Similarity
 cos(d1, d2) = (d1  d2) /||d1|| ||d2|| ,
where  indicates vector dot product, ||d|: the length of vector d

 Ex: Find the similarity between documents 1 and 2.

d1 = (5, 0, 3, 0, 2, 0, 0, 2, 0, 0)
d2 = (3, 0, 2, 0, 1, 1, 0, 1, 0, 1)

d1d2 = 5*3+0*0+3*2+0*0+2*1+0*1+0*1+2*1+0*0+0*1 = 25
||d1||=
(5*5+0*0+3*3+0*0+2*2+0*0+0*0+2*2+0*0+0*0)0.5=(42)0.5
= 6.481
||d2||=
(3*3+0*0+2*2+0*0+1*1+1*1+0*0+1*1+0*0+1*1)0.5=(17)0.5
= 4.12
cos(d1, d2 ) = 0.94 42