Basic IR: Modeling
Basic IR Task:
Match a subset of documents to the user’s
query
Slightly more complex:
and rank the resulting documents by predicted
relevance
The derivation of relevance leads to different
IR models.
Concepts:
Term-Document Incidence
Imagine matrix of terms X documents with 1 when
the term appears in the document and 0 otherwise.
search segment select semanti …
c
MIR 1 0 1 1
AI 1 1 0 1
…
Queries satisfied how?
Problems?
Concepts:
Term Frequency
To support document ranking, need
more than just term incidence.
Term frequency records number of
times a given term appears in each
document.
Intuition: More times a term appears in
a document the more central it is to the
topic of the document.
Concept: Term Weight
Weights represent the importance of a
given term for characterizing a document.
wij is a weight for term i in document j.
Mapping Task and Document
Type to Model
Index Full Text Full Text +
Terms Structure
Searching Classic Classic Structured
(Retrieval)
Surfing Flat Flat Structure Guided
(Browsing) Hypertext Hypertext
IR Models
Set Theoretic
Fuzzy
Extended Boolean
Classic Models
boolean Algebraic
U vector
Generalized Vector
Retrieval: probabilistic
s Lat. Semantic Index
e Adhoc Neural Networks
r Filtering
Structured Models
Probabilistic
Non-Overlapping Lists
T Inference Network
Proximal Nodes
a Belief Network
s Browsing
k Browsing
Flat
Structure Guided
Hypertext from MIR text
Classic Models: Basic Concepts
Ki is an index term
dj is a document
t is the total number of docs
K = (k1, k2, …, kt) is the set of all index terms
wij >= 0 is a weight associated with (ki,dj)
wij = 0 indicates that term does not belong to doc
vec(dj) = (w1j, w2j, …, wtj) is a weighted vector
associated with the document dj
gi(vec(dj)) = wij is a function which returns the weight
associated with pair (ki,dj)
Classic: Boolean Model
Based on set theory: map queries with
Boolean operations to set operations
Select documents from term-document
incidence matrix
Pros:
Cons:
Exact Matching Ignores…
term frequency in document
term scarcity in corpus
size of document
ranking
Vector Model
Vector of term weights based on term
frequency
Compute similarity between query and
document where both are vectors
vec(dj) = (w1j, w2j, ..., wtj) vec(q) =
(w1q, w2q, ..., wtq)
Similarity is the cosine of the angle between
the vectors.
Cosine Measure
j
cos()
dj
dj q
Sim(d , q )
dj q
q
t
w i, j
wi ,q
t
i 1
t
Since wij > 0 and wiq > 0, i, j
w 2
i 1
i ,q
w 2
i 1
0 <= sim(q,dj) <=1
from MIR notes
How to Set Wij Weights?
TF-IDF
Within document: Term-Frequency
tf measures term density within a document
Across document: Inverse Document
Frequency
idf measures informativeness or rarity of term
across corpus.
n
idf i log
df i
TF * IDF Computation
wi ,d tf i ,d log(n / df i )
tf i ,d frequency of term i in document d
n total number of documents
df i the number of documents that contain term i
What happens as number of occurrences in a document
increases?
What happens as term becomes more rare?
TF * IDF
TF may be normalized.
tf(i,d) = freq(i,d) / max(freq(l,d))
IDF is computed
normalized to size of corpus
as log to make TF and IDF values
comparable
IDF requires a static corpus.
How to Set Wi,q Weights?
1. Create Vector directly from query
2. Use modified tf-idf
freq (i, q) n
Wi ,q 0.5 0.5 * * log
max( freq (i, q)) df i
The Vector Model:
Example
k1 k2 k3
d1 2 0 1 k2
k1
d2 1 0 0 d7
d6
d3 0 1 3 d2
d4 2 0 0 d4 d5
d3
d5 1 2 4 d1
d6 1 2 0
d7 0 5 0
k3
q 1 2 3
from MIR notes
The Vector Model: k1
k2
Example (cont.) d2 d6
d7
d4 d5
d3
d1
1. Compute Tf-IDF Vector for each document k3
For first document:
K1: ((2/2)*(log (7/5)) = .33
K2: (0*(log (7/4))) =0
K3: ((1/2)*(log (7/3))) = .42
for rest:
[.34 0 0], [0 .19 .85], [.34 0 0], [.08 .28 .85],
[.17 .56 0], [0 .56 0]
from MIR notes
The Vector Model: k1
d7
k2
Example (cont.) d2
d4
d6
d5
d3
d1
k3
2. Compute the Tf-IDF for the query [1 2 3]:
K1: (.5 + ((.5 * 1)/3))*(log (7/5)))
K2: (.5 + ((.5 * 2)/3))*(log (7/4)))
K3: (.5 + ((.5 * 3)/3))*(log (7/3)))
which is: [.22 .47 .85]
The Vector Model: k1
d7
k2
Example (cont.) d2
d4
d6
d5
d3
d1
k3
3. Compute the Sim for each document:
D1:
D1*q = (.33 * .22) + (0 * .47) + (.42 * .85) = .43
|D1| = sqrt((.33^2) + (.42^2)) = .53
|q| = sqrt((.22^2) + (.47^2) + (.85^2)) = 1.0
sim = .43 / (.53 * 1.0) = .81
D2: .22 D3: .93 D4: .23
D5: .97 D6: .51 D7: .47
Vector Model
Implementation Issues
Sparse TermXDocument matrix
Store term count, term weight, or
weighted by idfi ?
What if the corpus is not fixed (e.g., the
Web)? What happens to IDF?
How to efficiently compute Cosine for
large index?
Heuristics for Computing
Cosine for Large Index
Select from only non-zero cosines
Focus on non-zero cosines for rare (high idf)
words
Pre-compute document adjacency
for each term, pre-compute k nearest docs
for a t term query, compute cosines from query
to union of t pre-computed lists, choose top k
The TFIDF Vector Model:
Pros/Cons
Pros:
term-weighting improves quality
cosine ranking formula sorts documents
according to degree of similarity to the query
Cons:
assumes independence of index terms
