Word frequency data


Our data is based on two different corpora: the 14 billion word iWeb corpus, and the 560 million word Corpus of Contemporary American English (COCA; 450 million words when the wordlists were created). COCA is the only large corpus of English that is both up-to-date (the latest texts are from late 2017) and which is  based on a wide range of genres (e.g. spoken, fiction, newspapers, magazines, and academic writing). Most of the following refers to the COCA word lists.

Why worry about what corpus is used? After all, there are many English word lists and frequency lists out on the Web (see in particular the  British National Corpus and the American National Corpus). Some are good, and others are very poor in quality. Not all frequency lists are created equal.

One should be very, very suspicious of word lists that are taken from messy web data, outdated texts, or corpora that are too small to effectively model what is happening in the real world. Or worse, word lists that don't give you any idea what they are based on. As the saying goes: "garbage in (bad texts), garbage out (frequency lists)".

Here's some questions you might ask yourself as you consider downloading or purchasing a word list:

Depth and accuracy. Why do so many wordlists on the web contain just the top 1000-3000 words of English? Why not the top 20,000 or 60,000? It's because even a bad corpus (the collection of texts that the word lists are based on) can produce a moderately accurate list for the very most frequent words. But because the corpus is neither deep nor balanced enough, you start getting messy data for medium and lower frequency words. Ask to see samples of the top 20,000 or 60,000 words (e.g. every 7th or 10th word). If they don't have it, then you should be very, very suspicious of that word list.

Genres. Does the corpus contain texts from a wide variety of genres -- spoken, fiction, popular magazines, newspapers, and academic journals? Frequency lists that are based on just one of these may only contain 40-50% of the words from a more balanced corpus. The COCA data is based on the Corpus of Contemporary American English, which is almost perfectly balanced across genres.

Size. COCA contained (when the word frequency was created) more than 450 million words, and each of the top 20,000 words occurs at least 300 times. In a small 10-20 million word corpus, some of these words would occur just 7-8 times. At that point, the lower frequency words might make it into the list "by chance", whereas others are left out. No such problem with COCA. (And iWeb is 25 times as large as COCA).

How recent is it? Language change happens. If the word list is based on 15-20 year-old texts (or much worse, 100 year old public domain novels), then it will be missing many of the words from the modern language. COCA is based on texts from 1990-2017 (20 million words each year) and iWeb was collected in 2017 -- or in other words, virtually right up to the current time.

Are they just word forms? Do you really want to see the individual frequency of shoe and shoes, or realize, realizes, realized, and realizing? Do you want to have the combined frequency of watch as a verb (they watch TV) and watch as a noun (his watch broke)? If the lists are simply taken from pages that are "scraped" from the web, they will just provide long lists of words, without grouping them meaningfully (e.g. shoe/shoes), or separating them when necessary (e.g. watch as a noun and as a verb). Both the COCA and iWeb word lists show the lemma (e.g. decide = decide, decides, decided, deciding) and group by part of speech (e.g. watch as a noun and as a verb).


Summary. There are many word frequency lists out on the web. Some are just OK, and some are truly bad. The frequency lists that we have created are the only ones available anywhere that are based on a large, recent, and balanced corpus of English.