In 1975 a US Navy research team fitted three readability formulas to the same 531 enlisted sailors, the same comprehension test and the same 18 passages. From the grades printed in their report, we found the three still disagree by a median of 1.5 grades a passage, and by as much as 4.0 grades on one of them. One paragraph run through five formulas comes back with five grades. Each formula reads a different surface feature of the text, and a grade level (a year of US schooling) only predicts how one group of readers did on one test. Even the best fit is loose. Flesch-Kincaid, the grade formula from that study, missed the measured reading grade of its own passages by 2.09 grades on average, on our calculation from the report's table.
Each formula counts a different mix of surface features
The source is Kincaid, Fishburne, Rogers and Chissom's 1975 report, which tested personnel at four technical training schools on passages from Navy training manuals. A passage was given the grade at which half the readers scored 35 percent or better on a cloze test, where readers fill in deleted words. The report treats that bar as equivalent to the 75 percent correct on multiple-choice questions used by Rudolf Flesch in 1948. So Kincaid's formulas and Flesch's share roughly one standard of "understood".
What differs is the input. Flesch-Kincaid weighs words per sentence and syllables per word, while the Automated Readability Index counts characters per word. The Navy's own "Fog Count", a different formula from Robert Gunning's Fog index, sorts words into easy and hard. Fitted to identical data, those three spread 2 grades or more on 4 of the 18 passages.
SMOG is the formula built on a stricter bar. G. Harry McLaughlin's 1969 paper took the grade of readers "showing complete comprehension" of standard test lessons, and SMOG counts only polysyllables, words of three or more syllables. On the Navy passages our reference implementation puts SMOG about two grades above the measured grade on average. That is indicative at best: the passages run 5 to 15 sentences on our count, and McLaughlin's own web page says the formula "was normed on 30-sentence samples".
Flesch-Kincaid missed its own Navy passages by about two grades
On the passages it was built from, its best case, the formula's mean absolute error against the measured grade was 2.09 grades; its worst miss was 4.3 grades, on passage 12. It landed within a grade and a half of the measured grade on only a third of the passages, and the report's two other new formulas did no better.
The measured grade is itself an estimate. Each sailor read nine of the eighteen passages, and reading ability came from the comprehension section of the Gates-MacGinitie test, which the authors say gave higher estimates than the full test would have. Passage 6 shows why agreement proves little. The three formulas put it between grades 11.2 and 12.3, and the measured grade was 16.0.
Florida's sentence rule lifts its statute's reading-ease score
Florida's section 627.4145 deems an insurance policy readable if, among layout conditions, its text scores at least 45 on Flesch Reading Ease, a scale of 0 to 100 on which higher means easier. It also defines a sentence unusually, stating that "A unit of words ending with a period, semicolon, or colon, excluding headings and captions, shall be counted as one sentence."
We scored the statute's own first two subsections, 258 words, with our reference implementation, which counts syllables from the CMU Pronouncing Dictionary. Ending sentences only at full stops, question marks and exclamation marks gives 4 sentences and a score of 8.2. Florida's rule finds 10 sentences and gives 47.5. We did not apply the statute's exclusions for defined words and medical terms, and the rule governs policies, which the statute's own text is not. Across our 24 English texts, 18 of them Navy passages, the rule changed the score on only 9.
A syllable counter's mistakes surface in Fog and SMOG
Flesch-Kincaid averages syllables over every word, so counting errors dilute. Swapping the dictionary for a simple vowel-group rule moved its grade by 0.34 on average across our 24 texts, and by 1.43 at most. Fog and SMOG hinge on a threshold instead, so one extra syllable can turn an ordinary word into a "complex" one.
Until 12 October 2026 our reading level calculator over-counted the syllables in almost every word ending in a consonant plus "le", 1,708 of the 1,768 such words in the CMU dictionary, so "People were able to settle in the middle of the little table." scored Fog 21.5 and SMOG 15.9 where the correct figures are 4.8 and 3.1. Our text statistics and word counter pages carried their own counting code, and the three disagreed by up to 13.1 Flesch points on the same text. The revision of that date moves all three onto one shared counting module and counts SMOG as McLaughlin specified. It also warns on texts under 30 sentences, shows the five grades as a range rather than an average, and declines Malay, Vietnamese and Chinese text.
Malay bus directions grade high because English formulas count syllables
We wrote nine sentences of plain bus directions in English and translated them into Malay. Our reference implementation gives the English a Flesch-Kincaid grade of 3.7 and the Malay 15.0, a gap the formula creates by treating word length as difficulty: at similar sentence lengths, the Malay averages 2.2 syllables a word on our count against 1.2 in the English. A single word shows the mechanism, perhentian ("stop") counting as three syllables on our counter where the English "stop" counts as one. Our Chinese version gave the formulas no words to count at all.
Formulas built for these languages exist, among them a Malay formula from Khadijah Rohani Md. Yunus's University of Miami dissertation of the early 1980s and CRIE, a Chinese text analyser published by Sung and colleagues in 2016. Janan and Wray (Malaysian Journal of Learning and Instruction, 2014) ran six formulas over 64 English texts for children aged 6 to 11 and found "significantly different readability indices for the same text". They argue the lesson applies to Malay, though they tested no Malay text.
Plain-language guidance seldom names a formula
The US Plain Writing Act of 2010 defines plain writing as "clear, concise, well-organized" and names no grade or formula. GOV.UK's guidance, in its March 2025 archived copy, told writers to aim "for a 9 year old reading age" without naming a formula; the current guidance pages we searched no longer carry the phrase. Microsoft's Word help does name its formulas, suggesting a Flesch-Kincaid score "around 7.0 to 8.0" for most documents, and Florida pairs its 45 with its own sentence rule.
In Singapore, SpeakEasy, a Hack for Public Good 2026 project page dated 29 January 2026 and updated on 29 September, reports that across five key government sites "only 13% of content is written at lower secondary level or below". It names no formula. It does quote a UX agency describing the same trouble: "Different tools give different results."
View author profile → · Editorial policy
