Vector Space Model - Example: Tf-idf Weights

Example: Tf-idf Weights

In the classic vector space model proposed by Salton, Wong and Yang the term specific weights in the document vectors are products of local and global parameters. The model is known as term frequency-inverse document frequency model. The weight vector for document d is, where

w_{t,d} = \mathrm{tf}_{t,d} \cdot \log{\frac{|D|}{|\{d' \in D \, | \, t \in d'\}|}}

and

  • is term frequency of term t in document d (a local parameter)
  • is inverse document frequency (a global parameter). is the total number of documents in the document set; is the number of documents containing the term t.

Using the cosine the similarity between document dj and query q can be calculated as:

In a simpler Term Count Model the term specific weights do not include the global parameter. Instead the weights are just the counts of term occurrences: .

Read more about this topic:  Vector Space Model

Famous quotes containing the word weights:

    “... the prevalent custom of educating young women only for marriage, and not for the duties and responsibilities consequent on marriage—only for appendages and dead weights to husbands—of bringing them up without an occupation, profession, or employment, and thus leaving them dependent on anyone but themselves—is an enormous evil, and an unpardonable sin.”
    —Harriot K. Hunt (1805–1875)