Notes › Resources › Two Whole Public-Domain Classics Fed into the Text Reader: Line Counts at 81474 and 78497 Bytes, the Code Unit and Code Point Gap, and the GBK Misread Replacement-Character Bill

Two Whole Public-Domain Classics Fed into the Text Reader: Line Counts at 81474 and 78497 Bytes, the Code Unit and Code Point Gap, and the GBK Misread Replacement-Character Bill

Two public-domain classics went in whole: the Tang file at 81474 bytes shows 29218 code units against 29217 code points, the Jiuzhang file at 78497 bytes reads 27071 on both scales, plus the U+FFFD tally of a whole-file GBK misread.

What that character field really counts when a whole book goes in

Drag a whole classic onto the text reading page and the info bar answers with four fields at once: file name, size, line count, character count. On small samples these numbers always huddle together; it takes a whole book to pull them apart. That is exactly the question this round settles with two real books instead of toy inputs.

Both TXT files here are whole public-domain classics saved back from the resource pages: the Three Hundred Tang Poems copy at 81474 bytes and the Nine Chapters on Arithmetic copy at 78497 bytes, fed as stored into the text reading page with no slicing in between. The conclusion up front: that character field counts UTF-16 code units, not bytes and not code points; on the Tang file code units run 1 ahead of code points, on the Jiuzhang file not a single unit extra. As for the four rulers, each has its own definition: bytes are the Buffer length, code units are the string length, code points are [...text].length, and Han characters are counted by the Basic Block regex.

The extra 1 of code units over code points

The Tang file measures 29218 code units against 29217 code points, and the interface shows the former. The extra 1 comes from one character from Extension F, code point U+2CB3B, sitting at code point index 7519: in UTF-16 it takes 2 code units but only 1 code point, so the readout runs one ahead of the eye. Screening the Jiuzhang file for surrogate pairs returns 0, and code units and code points agree at 27071. That Extension F character lies outside the Basic Block regex, so it was never counted into the 22123 Han characters. None of this is the page misreading; the rulers are simply graduated differently, and two real books set both cases - code units unequal to code points, and code units equal to code points - on the table at the same time.

Two whole books side by side Tang Jiuzhang Segments 2992 / 1305 Nonempty 2024 / 888 Bytes 81474 78497 Units 29218 27071 Points 29217 27071 Han 22123 20874

Figure 1: on every measure the upper bar is Tang, the lower bar Jiuzhang; only in the two cells of code units and code points do the Tang numbers differ by 1, while Jiuzhang shows the same value twice.

The decision chain stops at step two

The first 3 bytes of both files measure 227,128,144, which is the UTF-8 encoding of 【, not a BOM, so the BOM branch misses; then one strict UTF-8 pass decodes the whole text, the chain halts at step two and returns utf-8, and neither the strict GBK step nor the lenient fallback is ever entered, so the encoding field shows auto-detect (自动检测): UTF-8. The size field takes the KB rung and prints 79.6 KB and 76.7 KB. The line-count field, 2992 and 1305, includes the trailing empty segment: 2024 and 888 nonempty, neither file has a CR, and both end in a newline. The longest line runs 529 code units at line 829, and 216 code units at line 534.

The bill for reading the whole file as GBK

This machine cannot produce GBK bytes, so the direction of a real GBK file decoded as UTF-8 stays marked unmeasured, for want of a GBK encoder here; what can be shown is the reverse, handing the whole UTF-8 byte stream to the lenient gbk decoder. The Tang file comes out at 44212 code units containing 1915 U+FFFD characters; the Jiuzhang file at 41116 code units containing 736. The character-count field jumps from 29218 to 44212, while every other field in the bar keeps its reading.

The segment count does not move, still 2992 and 1305, because LF is a legal single byte under GBK and never joins a two-byte pair. The misread inflates the character count and leaves the line arithmetic untouched.

The render ceiling trips first, not the size gate

A single book sits 47008 segments (Tang) and 48695 segments (Jiuzhang) short of 50000. Tripping the line takes splicing: 16 books give 47857 segments and do not trip it, 17 books reach 50848 segments and the tail line appears, "... file too large, showing only the first 50000 lines (50848 in total)"; for Jiuzhang, 38 books at 49553 segments do not trip it, 39 books reach 50857 segments and it does. At that same moment the spliced file is only 1385058 and 3061383 bytes, 13.21% and 29.20% of the size gate at 10485760; a single book sits at 0.777% and 0.749%. What trips first is the render ceiling, not the size limit.

Distance to 50000 segments and byte share Tang 2992 gap 47008 Jiuzhang 1305 gap 48695 50000 Splice 17 = 50848 segments bytes 1385058, 13.21% Splice 39 = 50857, 29.20% gate 10485760

Figure 2: the segment side has already tripped its line while the byte side has traveled only 13.21% and 29.20% of its gate.

The books' own name ledger

Now put the rulers back on the books themselves. Counting title lines in the Tang file with /^【[^】]*】$/ reaches 320 poems; no volume headline stands on its own line (the volume-head regex hits 0), and there are 1424 full stops. The Jiuzhang file has 9 volume heads and 226 numbered problems, per volume: Fangtian 35, Sumi 42, Cuifen 18, Shaoguang 22, Shangong 26, Junshu 26, Yingbuzu 18, Fangcheng 17, Gougu 22, which adds back to exactly 226, with 1166 full stops. These chapter names are, in this article, only strings counted out of the file, rendered here in pinyin, and they are not expanded further.

Search matches only strings that truly exist in the file

  • This edition's 《静夜思》 (Quiet Night Thought) reads 「床前看月光」 and 「举头望山月」, 1 hit each
  • the circulating edition's 「床前明月光」 and 「举头望明月」 hit 0 in this 81474-byte file
  • 202 hits for 「术曰」 (the method says), 253 for 「答曰」 (the answer says), and 1 for 「今有田广十五步」 in the Jiuzhang file
  • 〔一〕 hits 9, because all nine volumes renumber from 〔一〕; keeping only the prefix 〔一 raises the count to 98

A query first passes through meta-character escaping before being joined into a regex, and that escape class actually covers 14 ASCII meta-characters; the hit counts match an eyeball substring count case by case. The lesson is plain: what the search box reports is the file's own text, edition and all.

Two sets of rulers, checked against the wording

Everything above is measured at the pure Node string layer; the interface layer is a second set of rulers. The string layer can settle two things along the way: escHtml is a no-op on these two books, /&/g and "/g hit 0 in both full texts, so no entity expansion changes the length before the DOM; and the line number equals the split index plus 1, checked position by position across 1 to 2992 and 1 to 1305, all equal. The rendering of 2992 lines in a real browser, sticky line numbers and scrolling, the number of highlight spans, and what lands on the clipboard are all marked unmeasured - no browser was opened this round.

The wording was also checked against the readings of two real books: taking this page's intro and summary from line 1497 on in config/tools.php, plus the template's drag sentence "supports UTF-8 / GBK / Big5 / Shift-JIS, up to 10MB", item by item - of 8 verdicts, 6 [agree], 0 [conflict], two partly agree. One is "displays file size, line count, character count": the character count is given in code units, and the Tang file's 29218 is not its code point count 29217. The other is that the sentence lists four encodings while auto-detection only ever emits two exits, utf-8 and gbk; Big5 and Shift-JIS require a manual pick, and a manual pick on these two UTF-8 files would produce a wide field of replacement characters.

Common questions

Q: The character count is one above the characters I counted - is the page misreading?

That field counts UTF-16 code units. The Tang file holds 1 Extension F character (code point U+2CB3B) taking 2 code units but only 1 code point, so 29218 sits 1 above the code point count 29217; the Jiuzhang file has 0 surrogate pairs, and 27071 is identical on both rulers. Either way the text itself is untouched.

Q: Would a whole book be stopped by the 10MB line?

Not these two: 81474 and 78497 bytes are 0.777% and 0.749% of 10485760. If a line trips first, it is the 50000-segment render ceiling: only 17 spliced books at 50848 segments, or 39 at 50857 segments, trigger the tail notice, and by then the bytes reach only 13.21% and 29.20%.

Q: If I switch the dropdown to GBK, do the line counts go wrong too?

The character count changes, the line count does not. Read as GBK, the Tang file yields 44212 code units with 1915 replacement characters and the Jiuzhang file 41116 with 736, while the segment count stays 2992 and 1305, because LF is a legal single byte under GBK and takes no part in pairing.

📎 Related articles

Three Hash Implementations on One Page: Two Public-Domain TXTs as Byte Corpus, Measuring the Block Ledger and Encoding Mutations of file-md5 Character by Character 2026-11-09 Shenglü Qimen and Liweng Duiyun Fed Whole Into the Text Diff Page: 154 Common Lines, 665 Added, 401 Removed, and the Stats Line Gives Only Two Numbers 2026-11-08 Measuring a Pinyin Table With Liweng Duiyun: 7160 Hits, 21 Off-Table Chars, 232 Polyphones 2026-11-07 Does the Thousand Character Classic Hold 1000 Distinct Characters: Word Count Measures 996 in Range, 1000 Real Han, 995 Distinct 2026-11-05 Pasting All 120 Chapters of Dream of the Red Chamber Into a Character Frequency Counter: Why Rank 1 Is the Comma at 59,260 Hits 2026-11-04