Last week I took on a reference list pulled out of a PDF, more than 180 entries. At first glance every entry was there; after pasting it into an editor and turning on "show whitespace" it was nothing but problems: line breaks inherited from the typesetting sat in the middle of each entry, a whole string of spaces sat inside a title, and an entirely invisible character hid at the very start, which sent the first and last entries to bizarre positions when sorting. I planned to delete them by hand, and by entry 40 it was a mess, because my first move was "delete all empty lines": the rows made of spaces immediately became real empty lines, and everything I had already fixed started changing again.
Invisible characters are more than one kind
The lesson that stuck with me: whitespace is not "one thing" but several different characters that look alike and come from different places and need different handling. Only after telling them apart do you know which switch to flip.
| Character | Code point and notation | Where it usually comes from | How it looks on screen |
|---|---|---|---|
| Half-width space | U+0020, " " in code |
Typed directly on the keyboard | Narrow gap, the easiest to spot |
| Full-width space | U+3000 |
Chinese typesetting, spaces typed with a Chinese input method | About one Han character wide |
| Non-breaking space | U+00A0, often written in HTML |
Indentation on a page and "do not break the line here" spacing | Looks like a normal space but counts toward length |
| Tab character | Tab, U+0009, written \t in code |
Column separation when copying from Excel, code indentation | Jumps to the next tab stop, width varies |
| Line break | LF U+000A i.e. \n, CR U+000D, CRLF on Windows |
Ends of lines; every typeset PDF line leaves a trace | Splits one record into several lines |
| BOM | U+FEFF, the UTF-8 bytes are EF BB BF |
Added at the start of a file by some editors when saving | Completely invisible, but distorts the first line |
Copy a column built from paths such as C:\Users\name\a.txt and the problem is usually compound: tabs between columns, CRLF at line ends, and backslashes inside the paths themselves. A blanket "delete spaces" does nothing useful here; you have to name each character.
Why the order changes the result
Cleaning is not idempotent; each step rewrites the input of the next. Three specific traps I hit:
Converting full-width spaces to half-width has to come before stripping spaces. If you strip half-width spaces first and convert afterwards, the newly created spaces escape that round and the text keeps a pile of gaps that look already removed.
Removing empty lines has to come after trimming the ends. A line made of spaces or tabs is not empty to a tool; it contains two characters. Trim the ends of each line first and those lines collapse to zero length, so the next pass can remove them. The time I did it backwards, I removed empty lines first, the count did not budge, and I assumed the switch was broken.
Removing all line breaks has to be the last step. The idea of "start and end of each line" no longer exists once everything is one line. Trimming after joining only touches the two ends of the whole block, and the spaces that used to sit at line ends stay where they were.
As for "remove all spaces", I file it under dangerous: code indentation, Markdown alignment and columns copied from Excel all live on spaces, and removing them all glues a column of data into one lump. When I need it, I try ten lines first.
The fixed routine I use now
- Turn on show whitespace and confirm which character you are up against. Do not skip this, or everything after it is guesswork.
- Convert full-width spaces to half-width.
- Remove tabs (if you need the column structure, convert them to spaces instead of deleting them).
- Trim spaces at the start and end of every line.
- Merge consecutive empty lines into one, or remove empty lines entirely; pick one, because turning on both tends to eat separators you wanted to keep.
- Check the counts. Text Cleaner updates its statistics as it works, and the change in line count should equal the empty lines I expected to remove; if it does not, go back a step.
- Confirm no invisible characters remain: the U+00A0 behind
is outside the reach of every switch above, so I always check for it separately.
Structural work only starts after cleaning. To add numbers and prefixes to each entry I use Line Operations; to compare this list against another one, or sort it and drop duplicate entries, I use Text Sort and Dedupe. The order is always clean first, transform second, because trailing spaces make two identical entries look different and the dedupe result goes wrong.
One class of problem cleaning cannot fix
Two entries from that reference list never came back: the PDF pagination cut one record in half across a page break, and the second author's name was split top and bottom. Removing empty lines only deletes characters; it cannot know which two half lines were originally one sentence.
My approach is to stitch them together by hand and accept that this step needs a person. The same goes for words hyphenated across a line break, like com- then puter in a PDF: delete the line break and you get com-puter, still wrong, so the hyphen has to go too. These are cases of content being chopped up, not of extra characters.
Q: I removed empty lines, so why does the export still have a pile of them?
Because those lines are not empty; they contain spaces or tabs. Trim the ends of each line first so they collapse into real empty lines, then remove empty lines. Swap the order and nothing gets deleted.
Q: I copied a column of paths from Excel and it pasted as one blob. What happened?
Excel uses tabs between columns and CRLF between rows. If you want one entry per line, process rows only and replace tabs with your own separator; if you want to keep the table structure, leave the tabs alone.
Q: There is a character at the start of the first line I cannot delete. What now?
That is a BOM (U+FEFF), an encoding-level issue, and the text cleaner has no switch for it. I re-save the file with a no-BOM encoding rather than fighting it with delete.
Q: Is it not easier to tick every cleaning option at once?
Easier but uncontrollable. Full-width conversion and space removal create work for each other, empty-line removal and trim depend on each other, and once everything is on you cannot tell which option changed the numbers. I would rather work in two or three rounds, one or two switches at a time, and confirm the effect in the statistics.
Q: Can cleaning delete characters from the body text?
The options only target whitespace and tags, so they should not touch the body. But "remove all spaces" and "remove HTML tags" are powerful, and with mixed Chinese and English content or content that carries markup I always test on a short excerpt before running the whole batch.
Back to that list: 180-something entries cleaned down to 178, and the two missing ones I stitched back by hand. What I remember is not where a particular switch sits but this judgment: whitespace problems need classification first and ordering second, because with the wrong order the same set of options produces a completely different result, and content chopped up by typesetting is beyond the tool's reach, so do not expect one more checkbox to bring it back.