Bytes are not judged wrong; they are claimed by the first decoder that succeeds
Open a log exported by an old system and it is full of boxes and question marks, so the instinct is that the software picked the wrong encoding. The detection code is actually honest: it tries several decoders in a fixed order and takes whichever succeeds first, and every attempt is strict, with the decoder running in fatal mode so an invalid byte for that encoding throws immediately. All it guarantees is that these bytes are valid under this encoding, not that this was the encoding they were saved in.
The four steps are: (1) if the start matches a BOM (EF BB BF) it is judged utf-8, the three bytes are stripped and decoding proceeds, so the character count does not grow by one; (2) decode strictly as utf-8; (3) if that fails, decode strictly as gbk; (4) if both fail, fall back to utf-8 in lenient mode, turning bad bytes into replacement characters. Each sample stops at one step:
| Sample bytes | Stops at | Judged as | Decoded |
|---|---|---|---|
E4 B8 AD E6 96 87 E6 B5 8B E8 AF 95 | step 2 succeeds | utf-8 | sample Chinese text, 4 characters |
| the string above plus EF BB BF | step 1 matches | utf-8 | sample Chinese text, 15 bytes |
D6 D0 CE C4 B2 E2 CA D4 | step 2 throws, step 3 succeeds | gbk | sample Chinese text, 8 bytes |
82 B1 82 F1 82 C9 82 BF 82 CD | step 2 throws, step 3 succeeds | gbk | garbled Han text, 5 characters |
E4 B8 AD FF | steps 2 and 3 throw, falls to step 4 | utf-8 | one character plus a replacement mark, 2 characters |
The fourth row shows the problem best: those 10 bytes are the Shift-JIS encoding of the Japanese word for hello. There is no BOM, strict utf-8 throws, and then gbk gets its turn: the same bytes are also a valid GBK two-byte sequence, so auto judges gbk and the screen shows five Han characters that look plausible but mean nothing. Only switching to shift-jis by hand gives the Japanese greeting back, again 5 characters. Detection has just two exits, utf-8 and gbk, and auto never returns the other two, so the claim of Shift-JIS support only holds when you pick it yourself.
The same Chinese sentence costs half as much: UTF-8 stores 3 bytes per character for 12 bytes in total and GBK stores 2 bytes per character for 8, with 4 characters on 1 line either way. Change the declaration and the characters change: those 8 bytes read as big5 give a different set of four Han characters, while the 12 bytes read as gbk give 6 characters, one of them in a private-use code point that shows as an empty box on screen. None of it is what you wanted to read.
Figure 1 | One Chinese sentence of 4 characters on 1 line stored three ways: 12 bytes, 8 bytes, and 15 bytes with a BOM while the character count is still 4.
Bytes, characters, lines: three rulers that never agree
Each of the three numbers uses its own ruler: size comes from file.size before decoding, character count from the string length in UTF-16 code units, and line count from the number of segments split on \n. The line containing one U+1F600 separates them best: 6 code units, 5 code points, 16 UTF-8 bytes, all three different, and the character count in the UI is 1 higher than a human would count. The emoji takes 2 code units and 4 bytes, while a Han character takes only 1 code unit and 3 bytes.
Splitting recognises only \n: a\r\nb\r\nc yields 3 segments, and the first two each keep an invisible \r that is still counted as a character. One extra newline at the end produces 4 segments, so the text still has 3 lines while the UI says 4; an old file with bare CR never splits at all and the whole document crowds into line 1. The line number is i plus 1, starting at 1, so line N is numbered N.
| Form | Bytes | Segments = lines | Segments keeping \r |
|---|---|---|---|
a\r\nb\r\nc | 7 | 3 | 2 |
a\r\nb\r\nc\r\n | 9 | 4 | 3 |
a\rb\rc | 5 | 1 | 0 |
Two limits, and the one you hit first is often not the 10MB
The size threshold is written 10 * 1024 * 1024, or 10485760 bytes, while the label says 10MB maximum: that is the 1024 convention, exactly 10 MiB, and 10.48576 MB under the 1000 convention. The test is strictly greater, so 10485760 gets through and 10485761 is rejected with a message that the file exceeds the 10MB limit, and both show 10.00 MB in the info bar.
Rendering has another limit and only draws the first 50000 segments. A file of 50000 lines at 60 bytes each, 59 characters plus one \n, is 3000000 bytes = 2.8610 MiB, just 28.61% of the size cap, so it passes the size check, but splitting on \n gives 50001 segments, the extra one being the empty tail, and it still exceeds the rendering limit. The same content without the trailing newline is 2999999 bytes and 50000 segments, which does not trigger it.
Figure 2 | The cap of 10485760 bytes is 10 MiB; the 3000000-byte sample is only 28.61% of it, yet the 50000-segment rendering limit is what it hits first.
When it truncates, the page footer says, word for word, that the file is too large and only the first 50000 lines are shown out of 50002, while the line count in the info bar says the full 50002. Two numbers disagree, so when reading a log you first have to know which one you are looking at.
Where the size display rolls over
- 1023 shows as 1023 B and 1024 as 1.0 KB: the byte range has no decimals and the KB range keeps 1.
- 1048575 shows as 1024.0 KB rather than 1.00 MB, and only 1048576 rolls over; 1073741823 shows as 1024.00 MB and 1073741824 as 1.00 GB.
Search highlighting counts the escaped string
Line content is escaped before it enters the DOM, with 4 replacements, one more than json-to-csv because of double quotes, and matching runs against that escaped string. Take a sample line such as line <b>bold</b> & more: searching for <b> gives 0 hits though the original has 1, searching for & gives 5 hits though the original has a single &, the extra 4 being the < and > that escaping wrote, and searching for LINE gives 1 hit thanks to the gi flags. Once the 2 double quotes become entities, searching for a double quote gives 0 hits. The highlight tag also cuts entities in half, throwing every position off.
These conclusions can be checked again and again in one place. The dropdown in Text File Reader has 6 entries (auto, utf-8, gbk, gb2312, big5, shift-jis); auto uses only the first two, the other three take effect only when chosen by hand, and gb2312 never takes part in detection. Load the same file and change only the declaration, watching the same bytes land on different characters, and you have the fastest way to understand this chain. The message that the encoding is unsupported and UTF-8 was used as a fallback cannot be reached either: in Chrome 154 all 5 real encoding values construct a decoder successfully, so it is a dead path. To check which character a run of bytes maps to by code point, pair it with a Chinese and Unicode converter; to split an oversized log and read it piece by piece, splitting large text is less trouble than fighting that 50000-segment limit.
Q: Why does switching to GBK work when auto does not?
Two cases. Genuine GBK Chinese (D6 D0 CE C4 B2 E2 CA D4) always throws at step 2 and succeeds at step 3, so auto was never wrong; what cannot be rescued is the Japanese file, which already passes the GBK step, so the chain never reaches shift-jis or big5, and those two exits do not exist in the code.
Q: Why are only the first 50000 lines shown when the info bar says 50002?
Two different measures: rendering takes only the first 50000 segments, while the line count in the info bar comes from a separate split on \n with no 50000 limit. Take the total from the info bar, and use the difference between the two numbers to see how much is missing.
Q: Why does searching for an angle-bracket tag find nothing when it is visible on the page?
That angle bracket on the page is the escaped form, and in the string used for matching it is already an entity beginning with <; conversely & gives 5 hits while the original has 1. To find it, search for the ordinary text next to it that was not escaped.
Q: Can the encoding be identified from the bytes alone?
Not uniquely. The same bytes can be valid under several encoding rules, and the detection chain only collapses many solutions into one; the only external evidence left is a BOM, the source system and known fields.