Notes › Tech Notes › Is Pasted-Back SQL Byte-Identical After Formatting: 8 String Syntaxes Cross-Checked, 2 Unclosed Backticks Paired Silently

Is Pasted-Back SQL Byte-Identical After Formatting: 8 String Syntaxes Cross-Checked, 2 Unclosed Backticks Paired Silently

format-sql promises that quoted content is never altered. This audit measures the boundary: the word-start regex counts U+00A0 and above as word characters, 24 of 24 cross-check cells match byte-for-byte, two unclosed backticks pair silently, and the stats line reports UTF-16 code units.

What the page promises and what this audit keeps

The format-sql template measures 40858 bytes and 1113 lines on the live page. It ships its own lexer, and everything from tokenizing to minifying runs inside the page: the 31326 characters of <script> source are read back whole and fed verbatim into node:vm, and none of the 44 top-level functions are altered. The page sells the promise that "formatting only rearranges whitespace; it never changes the contents of strings, comments, or identifiers." This article measures that promise's edges: delimiting, prefix breaking, the cross-check, failure messages, minimal separation, statistics baseline.

Word-start count: the range from U+00A0 to U+FFFF opens a word

The regex behind word-start detection treats the whole range from U+00A0 through U+FFFF as word-start characters, so Chinese, full-width symbols and non-breaking spaces all fall inside; the accounting splits into three layers. A bare Chinese column name (SELECT 中文列名 FROM 表) cuts into 4 tokens, the name staying one whole word token instead of being sliced character by character; in SELECT ()a(x) full-width parentheses behave as word-internal characters and merge into the preceding word, becoming a single word. The third layer is the non-breaking space: the lexer tests whitespace before word scanning, so an NBSP only acts as a gap at line start; inside a word it counts as word-internal — the run SELECT+NBSP+1+NBSP+FROM+NBSP+t cuts into just 1 word token, and neither SELECT nor FROM gets recognized as a keyword.

sqIsWordStart word-start regex From U+00A0 on, whole range starts words before range Chinese column name · 1 word token full-width parens · merged into previous word NBSP-joined run · only 1 word cut

Figure 1 — word-start ledger: one judgment class covers the range; Chinese, full-width and NBSP each cut their own way.

Prefix strings: 10 spellings concatenate into one token, one space breaks them

The prefixed spellings E'x', N'y', X'4F', B'01', U&'z' plus the matching five lowercase families make 10 prefix forms in total: each lands whole inside one string token, and sandwiched between SELECT and FROM the total token count is 4 in all cases. Breaking was measured too: one space between prefix and quote and E 'x' degrades into two pieces, a word plus a string; U& "x" shatters further, cutting straight into 3 fragments. Both breakages only change the delimiting — nothing inside the quotes was touched.

The 24-cell verbatim cross-check: 0 rewrites

Eight string spellings (doubled single-quote escape, backslash escape, doubled backslash, a cross-line case, the U& and X prefixed strings and so on) against three modes — beautify, minify, normalized whitespace — produce a grid of 24 cells. In every cell the input slice appears verbatim inside that mode's output: the check comes out 24/24 true with 0 rewrites. Backtick, square-bracket and double-quote identifier styles are likewise fully preserved across all three modes.

Failure ledger: 6 paths keep their own messages; backticks are the exception

Unclosed delimiters and unbalanced parentheses make 6 failure paths in total, each with its own error message:

TriggerMessage as printed
One backtick missing` quote not closed
Single quote left open' quote not closed
Square bracket missing its tailBracketed identifier not closed, missing ]
Block comment missing its tailBlock comment not closed, missing */
Extra opening parenthesisParentheses not paired, 1 left paren still open
Extra closing parenthesisRight parenthesis without a matching left paren

Row and column math: the line number counts segments of the pre-start slice split on newlines; the column is the start index minus the most recent newline index. In the three-line sample the unclosed quote starts at position 26 with the previous newline at 15, reported near line 3, character 11. When an error fires, result and panel are wiped to zero; under all 6 paths the five statistics keep only the input character count at its true value. The exception: two unclosed backticks in one input pair up. SELECT `x plus a newline plus FROM `y raises no error: one cross-line identifier token swallows the middle line into the quoted content, the beautified output runs 3 lines, statement count 1, silent throughout.

Line comment rewritten as block comment: the star-slash leaks through

When the minify mode packs a line comment -- into a /* */ shell, it does not escape a star-slash inside the body. Measured: --x*/y produces /*x*/y*/; re-lexing cuts 7 tokens, the comment block covers only the front half, and y with the trailing star-slash enter the statement layer as a word and an operator. The page's hint line, written to stop a comment "eating the statement behind it", collides with reality in this cell. Between keeping and dropping comments the demo sample differs by 117 bytes, while the line count differs by 0.

Minimal separation in minify: the sqHazard table holds only 3 rules

Whether two tokens keep a space after minifying is decided by sqHazard, whose rule table carries 3 entries: keep one between two operator characters, between a slash and a star, and between a star and a slash. Running 13 crafted inputs through the minifier, only 4 come back byte-for-byte identical; re-lexing those 13 strings leaves the token sequence of 11 unchanged and changes 2. In SELECT a : : b, once the colons split apart, the second colon pulls b into a variable token and the word count drops from 5 to 4; SELECT $$abc$$ FROM t minifies until SELECT and the following string fuse into 1 word: the hazard rules do not guard the dollar sign as a word-internal character. A letter touching an equals sign is judged no-space, and 1 -1 compressed into 1-1 still restores to 3 tokens.

The switches move bytes without moving judgment: on the same demo sample, indent widths of 2, 4 and 8 spaces plus Tab give outputs of 1304 / 1510 / 1922 / 1201 bytes while the line count stays at 72; turning off operator spaces saves 25 bytes, leading commas cost 17 more; IN-wrap thresholds of 1, 3 or 6 give byte-identical output, while 99 changes it to 1432 bytes across 62 lines.

sqHazard rule table · only 3 rules operator characters adjacent · keep slash-then-star, star-then-slash · keep 13 crafted strings minified · 4 byte-identical re-lexed · 11 unchanged · 2 changed colon case · next word absorbed as variable dollar string · fused with keyword into 1 word letter touching = judged no-space · 1 -1 into 1-1 restores

Figure 2 — minimal separation: a table of 3 rules; after minifying, 4 strings stay verbatim and re-lexing changes 2.

The statistics line reports UTF-16 code units, not bytes

The statistics block has 5 items: statement count, result rows, result characters, maximum indent level and input characters. Both character counts report UTF-16 code units, not bytes: the demo output reports 1376 characters but really weighs 1510 bytes; the input reports 967 characters against a real 1101 bytes — both off by 134, the whole gap from non-ASCII characters. Result rows split on newlines, so a string newline still adds one: a cross-line string, once minified, measured at 2 rows. Two pieces of the page copy track the measurement: the $$ dollar-quoting really is unimplemented, cutting into punctuation, punctuation, word as the page itself says; but the ? placeholder lands inside an operator token, since the lexical table counts it as an operator character, contradicting the claim that it is not treated as one — the measurement wins. The idempotency ledger has two cells: the 1510-byte beautified output, fed through beautify once more, is byte-for-byte identical, and the 931-byte minified output is equally idempotent. Timing, rendering and downloaded content were not measured in this batch.

Common questions

Q: If I paste the formatted SQL back and run it again, is it byte-identical?

Both modes are idempotent: the demo's 1510-byte beautified output, fed back through beautify, is reproduced byte for byte, and re-minifying the 931-byte output changes nothing.

Q: After an unclosed-quote error, are my previous results still there?

No. All 6 unclosed-delimiter paths clear the result and the panel, leaving only the input character count; close the delimiter and recompute to restore everything.

Q: Can I use the statistics line's character count as a byte count?

No — it reports UTF-16 code units, sitting 134 away from the real byte figure on the demo sample. To fit a byte budget, convert from the encoding yourself.

Every figure here was measured on the engine behind the SQL beautifier and compressor page. The sibling JS beautifier and compressor page has string states of its own; its numbers are not reused here. Comment placement and IN wrapping were measured in an older article, SQL comments and IN wrapping, and are not redone.

📎 Related articles

84 Field Names Across 7 Tabs: 2nd_field Comes in 6 Spellings and 5 Switches Move Only the Java Column 2026-11-20 Excel to JSON trusts only two numeric regexes: 0012 becomes 12, 1e5 stays a string, and the 8-cell switch matrix lands between 99 and 192 bytes 2026-11-20 A YAML alias is a copy: 10 references expand to 697 characters, and on a name clash own beats merged 2026-11-20 3.7 becomes 3 and 1e3 becomes 1: this fraction calculator truncates at the input, then switches display rules at 14 characters 2026-11-14 The cut lands on the last dot only: archive.tar.gz yields gz, photo. and .gitignore share one cell, and the rebuilt column drops a dot 2026-11-14