Word Frequency and N-grams, Explained

After reading this you will know how word and phrase counting works, how stop-word filtering changes the picture, and how to read a frequency list without drawing false conclusions.

What word frequency counts, and one quick example

Word frequency is the simplest useful measurement in text analysis: take a document, split it into words, and count how many times each word appears. An n-gram extends the idea to short phrases. A bigram is two words in a row, a trigram is three.

Take this sentence: The cat sat on the mat and the cat slept. Lowercase it and strip punctuation, and you get 10 word tokens. Count them and the appears 3 times, cat appears 2 times, and every other word (sat, on, mat, and, slept) appears once. The most common bigram is the cat, which occurs twice.

That is the whole mechanism. The value is in what the counts reveal: repetition you did not notice, filler you can cut, and the vocabulary that actually defines a text.

When frequency analysis helps, and when it misleads

Frequency counting answers a narrow question well: which words and phrases repeat, and how often. That is genuinely useful when you want to find your own verbal tics, check whether a keyword appears often enough to matter, or compare the vocabulary of two documents.

It is a poor tool for meaning. Counting cannot tell that bank in "river bank" differs from bank in "savings bank". It cannot tell that good and excellent are related. It treats run, runs and running as three separate words. Any conclusion about topic or sentiment drawn from raw counts is a guess dressed up as a number.

Frequency is not importance. In almost every English document the top word is the, which carries no topic information at all. High count and high meaning are different things, and confusing them is the most common analysis error.

The counting rules, precisely

Two rules decide every number this tool produces. First, tokens are normalized: text is lowercased and punctuation is stripped, so Word, word and word. all count as the same token word. Second, n-grams do not cross sentence boundaries.

The second rule matters more than it looks. Consider the end. The start. Without the boundary rule you would count the bigram end the, a phrase that never occurred as a phrase. By stopping at the period, the tool refuses to invent it.

For a sentence with w word tokens, the number of n-grams inside that sentence is:

g = \max(0,\; w - n + 1)

Here w is the word count of the sentence, n is the n-gram size (2 for bigrams, 3 for trigrams), and g is how many n-grams that sentence yields. A 10-word sentence gives 10 - 2 + 1 = 9 bigrams and 10 - 3 + 1 = 8 trigrams. A 2-word sentence gives 1 bigram and 0 trigrams. Because the total is summed per sentence, a document of many short sentences produces fewer n-grams than its raw word count alone would suggest.

Stop words, and what filtering does to the list

Stop words are extremely common function words such as the, and, of, to, a, and in. They dominate raw counts and carry little topic signal. The idea of filtering them out goes back to Hans Peter Luhn's work at IBM in 1958, which argued that both the most frequent and the rarest words are poor discriminators, and that the useful vocabulary sits in between.

Filtering does two things. It removes stop words from the single-word list, so the first content word rises to the top. It also removes any n-gram whose words are all stop words, or trims phrases anchored on them, so that surviving phrases carry more meaning. The trade-off is real: filtering not can flip the sense of a phrase, and filtering to be destroys a famous quotation. Use it to explore vocabulary, not to preserve exact wording.

Worked example on the sample text

Run the demo with this text: The cat sat on the mat. The cat sat on the dog. The dog ran.

Normalize and split into 3 sentences. Token counts are 6, 6 and 3, for 15 word tokens total.
Count single words. the appears 5 times, cat 2, sat 2, on 2, dog 2, mat 1, ran 1.
Count bigrams, stopping at each period. Sentence one gives 5 bigrams, sentence two gives 5, sentence three gives 2, for 12 bigrams. the cat occurs twice, cat sat twice, sat on twice, the dog twice.
Count trigrams the same way: 4 + 4 + 1 = 9 trigrams. the cat sat, cat sat on and sat on the each occur twice.
Turn on stop-word filtering. the, on and ran drop from the top word list, leaving cat (2), dog (2), sat (2) and mat (1) as the content vocabulary.

Notice that the went from the clear leader to absent. That is the whole point of filtering: the raw list describes grammar, the filtered list describes subject matter.

The tall bar for the vanishes after filtering, while the content words keep their counts.

Reading and interpreting the results

A frequency list tends to follow Zipf's law, named after linguist George Kingsley Zipf, who described it in 1935 and 1949. It says that if you rank words by frequency, the count of the word at rank r is roughly proportional to 1/r:

f(r) \approx \frac{f(1)}{r}

Here f(r) is the frequency of the word at rank r, and f(1) is the frequency of the most common word. If the top word occurs 1000 times, the pattern predicts the second word near 500, the third near 333, and the tenth near 100. Real texts fit this loosely, but the shape is reliable: a few words are very common and a long tail occurs once. When you see the steep drop, that is Zipf, not something special about your text.

Two practical readings follow. A very long tail of once-only words signals rich or uncontrolled vocabulary. A repeated bigram or trigram that is not a fixed phrase (say in order to appearing 30 times in a short essay) is a padding pattern worth editing.

Common mistakes

The errors below produce numbers that look authoritative and are wrong.

Comparing raw counts across texts of different lengths: A word appearing 40 times in a 40,000-word book (0.1%) is rarer than one appearing 5 times in a 500-word memo (1%). Compare rates, not counts.
Treating word forms as distinct topics: analyze, analyzed and analysis count separately, so a text about analysis can look less focused than it is. The tool does not stem words.
Reading meaning into stop words: the topping the list tells you the text is English prose, nothing more.
Forgetting the sentence boundary: If your text has no periods (a list of fragments, for instance), the whole thing is one sentence and n-grams will span the entire input.

Related tools on this site

Frequency analysis pairs naturally with a few neighbors. Use the Word & Character Counter first to get totals, sentences and reading time, which give you the denominators for rate comparisons. Once you know your vocabulary, the Readability Scorer tells you how hard the text is to read. If your counts look strange, invisible or look-alike characters may be splitting tokens; the Unicode Inspector shows every codepoint. To see which phrases changed between two drafts, use the Text Diff Checker, and for letter-level puzzles the Anagram & Palindrome Checker works on rearrangements rather than counts.

Frequently asked questions

Why does "word" and "Word." count as the same word?

Before counting, text is lowercased and stripped of surrounding punctuation. That merges Word, word and word. into one token so capitalization and trailing marks do not inflate your list.

Why don't my bigrams include phrases that span a period?

N-grams stop at sentence boundaries. The phrase end the from the end. The start is never counted, because those two words never appeared as a phrase within one sentence.

Should I turn stop-word filtering on or off?

Turn it on to see the content vocabulary and topic-bearing phrases. Turn it off when exact wording matters, such as checking a repeated phrase that includes not or to be.

Is my text sent anywhere?

No. The analysis runs entirely in your browser, so nothing you paste leaves your device.

Why is the total number of bigrams less than the word count?

Each sentence of w words yields only w - 1 bigrams, and the last word of every sentence starts no bigram. More sentences means fewer n-grams for the same word total.

Try it yourself in the tool