Quotes and angle brackets breaking the page: HTML entity encoding traps

Last year I helped a friend who runs a technical blog migrate comments off an old site. The bodies were already stored in escaped form, I treated them as plain text, and the export script encoded whole blocks again. The result was that a pile of comments on the new site showed literal symbols instead of words: <p>, &. Readers assumed the layout was broken. Fixing it took far longer than expected, because every comment had to be checked one at a time to see how many times it had been encoded and whether it needed one decode or two. Since then I trust one rule about escaping: work out which layer it happens in first, then decide whether to touch anything.

Angle brackets are structure, not content

In HTML source, a less-than sign marks the start of a tag, a greater-than sign marks the end, and an ampersand marks the start of a character reference. So when you want those three characters to appear on the page as text, you have to write them another way so the parser does not read them as structure. That is the whole reason for encoding: not a vague claim about being safer, but the fact that a character in that position would be parsed as syntax.

The reverse matters too. An entity written somewhere is turned back into a character before parsing, and attribute values are where this gets subtle. My test comes down to two questions: is this content meant to be read by a person, or executed by the browser? In the first case, angle brackets and quotes need encoding; in the second, they must not be.

Common entities at a glance

The table below is the one taped to the edge of my monitor. The middle column is what you type for the browser, and the right column is what finally appears on the page. The last column is the decimal code point, which is how numeric references are written; when you cannot remember a name, the code point still gets you there.

The character easiest to get wrong in that table is the ampersand itself. It is the character that opens a reference, so to show an ampersand on the page the source has to say &. Write a bare ampersand followed by letters and the browser swallows those letters as an entity name, and characters disappear from the page for no visible reason.

How double escaping piles up

That migration was exactly this. Encoding is one function call: do it twice and the text wears two layers, while rendering only takes one off. List the layers and I can work backwards from any comment to how many times it was encoded.

The pattern is that every extra encode adds another amp; in front of the literal text. So & on a page means the source carries one ampersand layer too many. Literal tag text such as <p> means the tag was encoded twice: <p> encoded once becomes <p>, and encoding that string again turns its ampersand into &, so the screen shows the literal form <p>.

When fixing it I do not decode the entire database once. I check each record for how many times it was encoded, strip only the extra layers, and encode once at output. Raw characters in storage and escaping at the output layer is the only way to stop it coming back; storing escaped text hands the responsibility to whoever writes next, and it will happen again.

When to encode and when to decode

My list is short. Code snippets that should appear as text, anything a user types before it goes into HTML, third-party content pasted into a template: encode. Text from an API or an RSS feed that arrives with & or ' and needs to be stored or counted: decode. Storing into a database for another application to read: store the decoded characters. Showing on a page: encode once at the moment of display.

Attribute context is another place that goes wrong. Say you write a title attribute on an input and the value contains a bare double quote, as in title="he said "hello""". The first inner quote closes the attribute early, and the rest is parsed as a mess of extra attributes. In that position the " has to be written as ". That is also why, when I use HTML Entity Encoder, I check how it handles quotes: only when angle brackets and quotes are both covered do I paste the result straight into an attribute.

There is another case: putting a chunk of HTML inside JavaScript. When single quotes, double quotes and line breaks are mixed in one string, hand-written escaping goes wrong easily. The tool offers a conversion from HTML to a JS string, either concatenated or as a template literal, and I use it for snippets that go into embedded tooltips; it beats counting backslashes by hand.

Do not encode the readable text as well

Encoding options usually come in two scopes: only special characters, or every non-ASCII character too. The second turns Chinese into a run of &#xxxxx;. It looks the same on the page, but the source grows visibly, readability drops to zero, and diffs become impossible to read. Unless the output pipeline truly accepts only ASCII, I encode special characters only. Reading code points and handling full-width versus half-width forms is a different job: Text Encoder converts character numbers and width forms, and has nothing to do with whether the browser parses something as syntax.

URL Encoder and Decoder is easy to confuse with this. In a URL, & separates parameters, so a literal ampersand inside a parameter value uses the percent form %26, not &. Put an HTML entity into a query string and the server receives the literal characters &. Conversely, %3C in HTML body text is displayed as %3C and never treated as a tag. The two rule sets govern different layers, and mixing them is the second most common trap I see.

Questions that come up often

Q: The page shows &. How many times do I decode?

Work backwards from what is displayed: & on screen means the source holds the result of two encodes. One decode gives text in the form &, and a second decode gives the original character. I do not guess; I decode once and look. If & is still showing, one layer remains; if the ampersand itself appears, it is done.

Q: Why does &nbsp still work without the semicolon?

Some legacy entity names are recognised in certain contexts, but that is not behaviour to rely on. The semicolon marks the end of a reference, so I always write it, especially in text where letters follow the ampersand; drop the semicolon and the parser swallows those letters as well.

Q: Is encoding user input once enough?

It depends where the value finally lands. In an HTML text node, once is enough. In an attribute value, quotes need handling as well as angle brackets. If the text also goes into a URL or JavaScript, it has to be encoded again by the rules of that layer. Encoding in one place and expecting it to cover every layer is how trouble starts.

Q: My editor shows a wall of entities. Is the document broken?

Not necessarily. Many rich text editors and CMSs store escaped text, and that is a normal storage format. The real problem is an application that reads it assuming plain text, because then it encodes a second time. I confirm the storage convention before deciding whether to decode.

Q: Can decoding restore things that should have stayed escaped?

Yes. If a piece of text is meant to display the entity notation itself, for example a page that must show the literal characters <, decoding turns it into <, and once that goes into HTML it may be read as a tag. So I only decode content I know is stored in escaped form; anything of unknown origin is treated as text first.

Back to those comments. I rewrote the export so the database stores raw characters and rendering encodes once, restored the double-encoded rows by their layer count, and no literal entities appeared again. The test got simpler too: when you see extra symbols, count the layers instead of decoding again straight away, and before every encode ask which layer the text ends up in, whether it is text for a person to read or a position that will be parsed as syntax. Answer that correctly and & has no reason to appear.