Word frequency data

See general page on comparing our lists to others (lists 5,000-60,000)

No other word frequency list of English besides ours contains an accurate listing of 100,000 or more words. There are at least four main problems with other lists, and all other lists suffer from at least one of these:

1. Their corpus is too small. We use the 450 million word Corpus of Contemporary American English (COCA), and the words that occur at the end of the list (e.g. words #90,000-100,000) occur about 16-17 times each. If you had a 100 million word corpus like the British National Corpus (1/4 to 1/5 the size of COCA), each of these words would occur about 4 times. At that point, many of these words are just "noise" -- there's no way to know if they appear in one or two texts by chance, or if other words should be in the list instead. And very small corpus like the American National Corpus or some of the "corpora" for the lists at Wiktionary are completely inadequate for a 100,000 (or even a 20,000) word list.

2. Their "corpus" is not balanced. If you have a corpus based on just newspapers (which is easy to create), you'll just get "newspaper language", and the same holds for fiction or any other genre. For example, see verbs in COCA that do appear in a corpus with fiction but rarely in a newspaper-only corpus  (undress, sigh, glance, stare, suppose, kiss, smile, etc). Most of the corpora from the Wiktionary lists are based on one narrow genre. And corpora that are based just on web-accessible texts are perhaps the worst -- there's no way to know what's in there, and what is left out (e.g. fiction). COCA is almost perfectly balanced between spoken, fiction, popular magazines, newspapers, and academic -- a wide range of English.

3. Their "corpus" is not tagged for part of speech. There are more than 140 words that occur both as a noun and as a verb at least 10,000 times in COCA, including words like state, work, place, head, point, end, study, face, care, use, control, and experience. But what is their frequency as a noun and as a verb? Or what about words like [school, room, father, party, oil, programs], or [serve, save, spoke, pulls, fail, joins]. Which are mostly nouns and which are mostly verbs? It's crucial to know the part of speech of a word in order to know how it is used in the language, but many other word lists don't show this.

4. Their word list is not carefully corrected. Take a look at the word lists from the British National Corpus (we downloaded the unlemmatized / all.num.gz file). As is shown in #1 above on this page, their words #90,000-100,000 would occur about four times each. Take a look at these files of nouns, verbs, adjectives, and adverbs that occur four times in their list. How accurate is this list? While we can't claim absolute 100% accuracy for our list (but see a sample of 5,000 completely randomly-selected words), we would argue that it is at least 99% correct -- for even these low-frequency words (e.g. #80,000-100,000), and very close to 100% for more common words (e.g. #1-40,000) -- and we believe that this is much higher than any other list.