Notes › Tech Notes › Ascending Is Not Code-Point Order: localeCompare('zh') Pinyin Collation and Three Duplicate-Line Denominators

Ascending Is Not Code-Point Order: localeCompare('zh') Pinyin Collation and Three Duplicate-Line Denominators

Six lines that matched neither order

The list is 6 lines: 苹果、香蕉、橘子、安卓、北京、上海. Sorted ascending it returns 安卓 北京 橘子 苹果 上海 香蕉 — not the code-point order I expected, 上海 北京 安卓 橘子 苹果 香蕉, and not stroke order either. It is pinyin order: ān běi jú píng shàng xiāng.

I ran it in Text Sort Dedupe, whose comparator is hard-coded at line 170 of the template: a.localeCompare(b, 'zh'), i.e. the browser ICU Chinese collation. The label "smallest Unicode code point first" does not hold, and neither does "Chinese last": put 苹果, zebra, Apple, 香蕉 in one column and ascending returns 苹果 香蕉 Apple zebra. The Han block sits entirely ahead of Latin, while code-point order would be Apple zebra 苹果 香蕉.

It does not wobble either. 300 re-sorts returned 300/300 identical results, and the 16 pairwise cells contain no transitivity break: a total order that is stable but simply different from code-point order.

item10 still sorts before item2

Numbered lists draw the most objections. item10, item2, item1, item21, item100 ascending returns item1 item10 item100 item2 item21. Plain numbers sort by character position as well: 10, 9, 100, 2, 0 returns 0 10 100 2 9.

Line 170 never passes {numeric:true}, and no sensitivity or caseFirst either. Used to a spreadsheet you expect item2 ahead of item10. Nothing is mis-sorted here; the natural-sort tier is just off, and padding zeroes by hand is the only way to get it.

Case splits more finely. banana, Banana, apple, Apple, cherry ascend as apple Apple banana Banana cherry whether or not "case sensitive" is ticked. Turn dedupe on and Apple, apple, APPLE, banana, Banana with the box unticked keeps 2 lines and reports 3 duplicate lines. The toLowerCase() at line 122 feeds only the dedupe key, never the comparator at line 170.

"By length" counts code units, not glyphs or bytes

The a.length - b.length at line 181 counts UTF-16 code units. ab, one 4-byte emoji, a, 苹果, abc, é ordered by ascending length return a(1) ab(2) emoji(2) 苹果(2) é(2) abc(3). One glyph of emoji ties with a 2-byte ab, a 6-byte 苹果 ties as well, and lines of equal length get no secondary comparator.

Dedupe compares exact code points and never trims

apple, apple with a trailing space, apple with a trailing Tab, Apple with a leading space, then 1 blank line: keep-first on those 5 lines returns 5 result lines and 0 duplicate lines. Sameness is exact code-point equality; not one leading or trailing blank is erased.

Blank lines, though, hang on trim. a, a blank line, a half-width space, U+3000, Tab, b — 6 lines, and switching on "remove blank lines" leaves 2, with the total falling from 6 to 2 too. Splitting recognises only \n. Pasting on Windows is safe because the textarea has already normalised CRLF to LF: 4 lines apple, banana, apple, cherry measure total 4, result 3, duplicates 1.

A line named constructor vanishes entirely

Line names are the one thing that deletes data in silence. 9 lines of input, apple, constructor, toString, valueOf, hasOwnProperty, isPrototypeOf, __proto__, propertyIsEnumerable, banana: keep-first outputs only apple and banana, while the stats read total 9, result 2, duplicates 7.

The cause sits at lines 120-121, const seen = {} and const duplicates = {}: reading a key on a plain object literal climbs the prototype chain, so if (!seen[k]) at line 128 is false forever for 6 of those names, and __proto__ finds a truthy Object.prototype as well. All 7 lines of that kind produce 0 lines of output; 4 lines of constructor, constructor, toString, toString under "duplicates only" print 0 lines and report 0 duplicate lines. Field-name lists and translation key tables collide with this most often; intersecting two lists is a separate problem.

The same x, x, x, y, y, z gives three duplicate-line counts

Dedupe modeOutputDuplicate linesFormula
Keep first / keep lastx y z3Difference (line 130)
Duplicates onlyx y2Equals result lines (line 154)
Unique onlyz5Difference (line 164)

Swap in 7 lines, apple, Banana, apple, cherry, Banana, apple, date, and the three counts stay 3, 2, 5. Only keep-first and keep-last answer "how many lines went away"; duplicates only reports how many distinct values repeated. The total is lines.length at line 114, taken after blank lines are removed, not the raw line count in the box; picking "no dedupe" hides the third stat altogether (line 192).

Duplicate lines: same x,x,x,y,y,z Keep first 3 Duplicates only 2 Unique only 5 Fig. 1: duplicate-line denominator per mode

No ceiling, only a full recompute on every keystroke

The template is 198 lines / 7897 bytes, with no length cap and no format allow-list: a single line of 1000000 characters still outputs 1000000 bytes. Measured on Node v24.19 with an input of 100000 lines and 669999 bytes: 68 ms without dedupe, 710 ms ascending, 273 ms ascending plus keep-first, so pairwise localeCompare runs roughly 10 times slower than not sorting. The box recomputes everything on each keystroke (oninput at line 53), so a long list feels like typing through lag rather than hitting an error.

Time ms: same 100000-line input No dedupe 68 Sort + keep first 273 Sort ascending 710 Fig. 2: sorting multiplies this step

The shuffle is sound, it just cannot be reproduced

It is textbook Fisher-Yates (lines 176-179): 4 elements run 2400 times produced all 24 permutations, the rarest 84 times and the commonest 123 against an expected 100, so the distribution has no defect. The trouble is that the order comes from Math.random, so two clicks that disagree cannot be replayed. The algorithm trade-off is in the shuffle a list piece.

Q: Can I choose code-point order or stroke order instead?

No. The locale at line 170 is fixed at 'zh', and there is no switch between pinyin, code point and natural order. Getting item2 first means padding zeroes yourself.

Q: The export has fewer lines than the input. What do I check first?

Three places: blank lines removed by trim (U+3000 and Tab count as blank); Apple merged with apple once "case sensitive" is unticked; and line names hitting prototype properties such as constructor, where the whole line is judged a duplicate. The first two are definitions, the last is data gone.

Q: Which duplicate-line count should a report cite?

For "how many lines went away", use keep-first or keep-last (the total minus the result count at line 130). For "how many values repeated", use duplicates only, which equals the result line count (line 154). On one and the same input those are 3 and 2.

📎 Related articles

Merged line counts never add up: a missing trailing newline, a swallowed BOM and an unreliable order 2026-10-26 Excel pasted into JSON: dates, long numbers and precision are already wrong the moment you paste 2026-10-26 The decoder only echoes the raw text: EAN-13 check digit, UPC-A leading zero, CODE128 mod 103 2026-10-26 QR code decoding tested: the error-correction level cannot beat an occlusion, 2 pixels per module is the real line 2026-10-26 Character frequency or word frequency: one Chinese sentence, a 23-row board and an empty table 2026-10-26