Skip to content
Back to Generative AIAll lessons

Generative AI

TF-IDF

O(n · m) time to score one document, where n is the number of unique words in it and m is the number of documents in the collection (or O(n) if document frequencies are already computed) · O(v) space for a vocabulary of v words.

The idea, in plain English

Imagine you flip through a pile of documents, looking for the words that make one of them special. Not just any word that shows up a lot, but a word that is common in this document and rare everywhere else, like a fingerprint. TF-IDF stands for Term Frequency times Inverse Document Frequency. It is a score that finds exactly those words. A word that is frequent here but also frequent in every other document, like 'the', gets a near-zero score. It is not special. A word that is frequent here but rare in the rest gets a high score. That is what makes this document distinct.

How it works

  1. 1For each word in the document you are scoring, compute its Term Frequency, or TF. This is how often the word shows up in this document, divided by the document's total word count.
  2. 2For each word, compute its Inverse Document Frequency, or IDF. Take the total number of documents, divide it by how many of them contain that word at least once, then take the logarithm of that ratio. A word that appears in every document gets an IDF near zero. A word that appears in only one document gets a high IDF.
  3. 3Multiply TF by IDF for each word. That is its TF-IDF score. Sort the words by that score to see which ones best describe this particular document.

When you'd use it

Use this for search engines, keyword extraction, and finding the most important or distinctive words in a document compared to a larger collection. It is an old idea, from before embeddings took over, but it is still a widely used building block.

Common beginner mistakes

  • Don't forget the IDF half and rank words by raw frequency (TF) alone. That just surfaces filler words like 'the' and 'and', which show up everywhere and say nothing distinctive about this document.
  • Don't compute IDF once and never update it. If you add new documents to the collection, every existing document's TF-IDF scores can shift, because IDF depends on the whole collection, not just one document.

Try it — edit and run

Click the code to edit · press ⌘/Ctrl+↵ to run

Editable code. Tab and Shift+Tab indent. Press Escape, then Tab, to move focus out of the editor.

Expected output — hit Run to try it
Documents:
0: the cat sat on the mat
1: the dog sat on the rug
2: the cat and the dog are friends

Scoring document 2: "the cat and the dog are friends"
and: tf=0.14 idf=1.10 tf-idf=0.16
are: tf=0.14 idf=1.10 tf-idf=0.16
friends: tf=0.14 idf=1.10 tf-idf=0.16
cat: tf=0.14 idf=0.41 tf-idf=0.06
dog: tf=0.14 idf=0.41 tf-idf=0.06
the: tf=0.29 idf=0.00 tf-idf=0.00

Most distinctive word: and

Note: 'And', 'are', and 'friends' all tie for the highest score, since each appears in only one document. 'Cat' and 'dog' tie right behind them. Ties are broken alphabetically, so both languages always print the words in the same order.

Not sure this is the right topic? See the learning paths → or where this leads →