Notes › Tech Notes › Character frequency or word frequency: one Chinese sentence, a 23-row board and an empty table

Character frequency or word frequency: one Chinese sentence, a 23-row board and an empty table

Keyword work on a Chinese draft almost always starts the same way: paste the whole body into the frequency tool, switch to word frequency, wait for a ranking. A total of 0 words plus one line reading No data is hard to connect with the 25 Chinese characters you just pasted — every character is sitting right there in the box.

An empty table is not broken: extraction only accepts ASCII letters

Line 152 of the template sets the extraction regex to /[a-zA-Z]+/g, so a word is a run of ASCII letters. Hand the same sentence 中文根本不按词切分所以字频统计会把每个汉字单独计数 to Text Frequency Counter and the two tabs return two different things. Character frequency: 25 characters, 23 rows, top row 字 (zi) at 2 hits and 8.0%, second row 计 (ji) also 8.0%, every other listed character 1 hit at 4.0%, and the 23 displayed shares add up to exactly 100.0%. Word frequency: 0 words. Insert a space between every character and it is still 0 — the tool does not split on whitespace, so hand-tokenising cannot rescue it.

Same 25-char Chinese sentence Char frequency: 23 rows zi 8.0% ji 8.0% zhong 4.0% wen 4.0% Other CJK chars: 1 each Word frequency: 0 words No data Total words 0 Spaced out: still 0

Fig. 1: the two rankings for one 25-character Chinese sentence, rows labelled by pinyin

Han characters, punctuation and full-width spaces share this ranking

Character mode skips only four code points: space, newline, tab and carriage return (line 123 of the template). Everything else counts. Finding 。 in the top 5 is no surprise: in a 43-character paragraph it lands 3 hits at 7.0%. The full-width space is not even on the skip list — in the 5-character sample A B C it takes the top row with 2 hits at 40.0%.

The denominator carries those characters too. The 58-character mixed sample shipped in the template counts only 49 of them, and its 41 shares sum to just 83.5%: the missing 16.5% is whitespace that was skipped yet left in the denominator. Counting happens in UTF-16 code units, so 50 emoji plus abc measure 103 in length and only 53 code points — one emoji is split into 2 rows of 50 hits each at 48.5%.

The stop-word switch moves the denominator, not just the ranking

The 117-character, 24-word English sample built into the template exposes these definitions best. With the stop-word field empty the ranking has 17 rows, the top row is the at 4 hits and 16.7%, and the rows add up to 100.3% — the surplus is rounding residue from 17 lines of toFixed(1). Turn on the default table of 30 words and the top row becomes fox at 2 hits and 8.3%; ranks 1 through 4, fox, over, dog and again, tie, distinct words fall from 17 to 11, and the share total drops to 62.6%.

Same 117-char sample the 16.7% fox 8.3% over 8.3% dog 8.3% again 8.3% Stopwords OFF: 17 rows top the, 16.7% Shares total 100.3% Stopwords ON: 11 rows top fox, 8.3% Shares total 62.6%

Fig. 2: top 5 rows and share totals with stop words off and on

That 62.6% is not a retention rate: the denominator is still the 24 words before filtering, the 11 rows only account for 15 hits, and 24 minus 15 = 9 is what the filter erased. The switch does not shift ranks, it swaps occupants — the four words that tied one tier below the leader are lifted into the top block, and the single rank 1 row, the, disappears.

Case sensitive on, and the stop-word table half-stops working

All 30 entries are lower-cased when the table loads, but matching looks them up in the original casing. With Case sensitive ticked, the sentence-initial The slips past the filter, the top row becomes The at 2 hits and 8.3%, and the summary still reports 30 stop words excluded. Clear the stop words and keep the same tick, and the the family splits into two rows, The 2 and the 2, at 8.3% each, while the merged view is a single row at 4 hits and 16.7% — neither row reaches half of its real share.

On a tie, the three modes use three different orders

All three modes share one comparator that sorts by hit count descending, and the sort is stable, so a tie is decided entirely by where the array came from before sorting — and the three sources differ:

  • Character mode reads Object.keys, which pulls integer-like keys to the front in numeric order: for the input 1234567890 the top row is 0, and b a a b 5 5 z returns 5, b, a, z — the digit 5 is pushed into rank 1 (15.4%).
  • Word mode also reads Object.keys, but keys there can only be letter runs, so ties keep pure first-appearance order: in zebra apple zebra apple zebra at 2 hits and 50.0% precedes apple, and alphabetical order would reverse them.
  • Keyword mode follows the order you typed the list, ignoring which keyword appears first in the text.

Words that contain digits collapse as a group: the 5000 numbered tokens from w0 to w4999 leave a single row w at 5000 hits and 100.0%, so version lists and SKU lists merge into one row.

The keyword column reports a percentage that is not density

Its numerator is hits and its denominator is characters. In the 43-character default sample the phrase 人工智能 appears 3 times and the panel shows 7.0%; measured against the 12 characters it really occupies it is 27.9%, a factor of 4. The list is not de-duplicated either: write one keyword on 3 rows plus a second one on 1 row and the tool reports 4 keywords and 10 total hits, while the text holds only 4. Matching is non-overlapping: looking for aa inside aaa reports 1 hit at 33.3% with 2 real positions, and inside aaaa it reports 2 hits at 50.0% with 3 real. There is no top-N switch either, so 1000 keyword rows render 1000 rows.

ModeWhat counts as a rowShare denominatorWho leads a tieMeasured on Chinese
Character frequencyAny non-whitespace character, punctuation and full-width spaces includedText length, whitespace left inside itDigits cut in, the rest by first appearance25 characters yield 23 rows, top row 8.0%
Word frequencyA run of ASCII lettersWord count before filtering, 24First-appearance orderThe same sentence gives 0 words
KeywordEvery row of your list, no de-duplicationCharacter count, not occupied charactersOrder written in the list3 hits shown as 7.0%, real occupancy 27.9%

What this ranking answers, and what it cannot

It answers relative frequency. In a fixed 463-character English passage the top row really is e (50 hits), which agrees with the single-letter frequency tables, but the tool prints 10.8%, letters alone over 379 letters give 13.2%, and the textbook figure of 12.7% matches neither — the whole gap is the 78 whitespace characters and 6 punctuation marks left in the denominator.

It cannot answer anything about Chinese words: a word is an ASCII letter run, so Chinese never reaches this board, and the keyword column uses characters as its denominator. If you still want to use it, run the keyword list through Text Sort Dedupe so repeated rows cannot inflate the total hit count. If all you wanted is how long a draft is, word and character counting uses another definition and has nothing to do with ranking.

Q: Why does Chinese always come back as 0 words?

A word is whatever /[a-zA-Z]+/g captures, a run of ASCII letters, and Han characters never enter that class. Typing a space after every character still yields 0, because it does not split on whitespace. Accented letters act as separators instead, so café leaves a single row caf.

Q: The shares do not add up to 100%, is the math wrong?

All three causes show up in measurement. With stop words empty, 17 rows sum to 100.3% because every row keeps one decimal place. In character mode, 41 rows sum to only 83.5% because whitespace is skipped yet stays in the denominator. With stop words on, 11 rows sum to 62.6% because the denominator is still the 24 words before filtering. Whether shares can be added depends on whether numerator and denominator describe the same set.

Q: On a tie, whose row is the leader?

It depends on the mode. In character mode ASCII digits cut in by numeric order, so the leader of 1234567890 is 0; word mode uses first appearance, so zebra precedes apple; keyword mode copies how you wrote the list. None of the three is alphabetical, so on a tie do not treat the first row as the only answer.

📎 Related articles

Merged line counts never add up: a missing trailing newline, a swallowed BOM and an unreliable order 2026-10-26 Excel pasted into JSON: dates, long numbers and precision are already wrong the moment you paste 2026-10-26 The decoder only echoes the raw text: EAN-13 check digit, UPC-A leading zero, CODE128 mod 103 2026-10-26 QR code decoding tested: the error-correction level cannot beat an occlusion, 2 pixels per module is the real line 2026-10-26 CSS minify left the calc() spaces alone, but the beautifier split .btn:hover in two 2026-10-26