URL Encoding Bugs: Double Encoding, Plus Signs and Broken Query Strings

A URL may only contain a limited set of characters, and several of them have a special meaning. The question mark starts the query, the ampersand separates parameters, the equals sign joins a name to its value and the hash starts a fragment. If your data contains one of those characters, it must be replaced by a percent sign followed by two hexadecimal digits, for example %26 for an ampersand. Characters outside ASCII are first converted to UTF-8 bytes and each byte is written this way, so é becomes %C3%A9.

The most common mistake is encoding a complete URL, which also encodes the colon, the slashes and the question mark and produces a string that is no longer an address. The right approach is to encode each value separately before putting it into the URL.

• ▸Use component encoding for a single query value or path segment. In JavaScript this is encodeURIComponent.
• ▸Use whole URL encoding only to clean up a complete address that has unencoded spaces or accents, and understand that it leaves the structural characters alone. In JavaScript this is encodeURI.
• ▸Better still, use the platform's URL and query builders: the URL and URLSearchParams classes in JavaScript, urllib.parse in Python and similar in other languages. They handle the rules for you.

If you encode a string that is already encoded, the percent sign itself is encoded as %25. A space becomes %20 the first time and %2520 the second. If you see %25 followed by two hex digits in a URL, something encoded twice. It usually happens when one layer encodes a value and a framework or a redirect encodes the finished URL again.

The reverse also happens: decoding twice turns a literal %2B in the data into a plus sign, and then into a space. The rule is simple. Every value should be encoded exactly once on the way out and decoded exactly once on the way in.

In an HTML form submission the content type is application/x-www-form-urlencoded, and spaces are written as plus signs. In a URL path, and in the strict URL standard for query values, a space is %20. Many servers accept both in the query string, but a plus sign in a path means a literal plus. Trouble arrives when data containing a real plus sign, such as a phone number or an email with plus addressing, is not encoded and the server reads it as a space. Encode the plus as %2B whenever it is data.

• ▸Path and query text should be percent encoded UTF-8. Browsers show you the decoded form in the address bar, but the request on the wire is encoded.
• ▸Domain names with non-ASCII letters use a different scheme called punycode, written with an xn-- prefix, not percent encoding.
• ▸Normalize Unicode before encoding when you compare or sign URLs, since é can be one character or an e followed by a combining accent, which encode differently.

A redirect parameter that itself contains a URL must be encoded so its ampersands and question marks are not read as part of the outer URL. OAuth redirect URIs must match the registered value after decoding, so a trailing slash or an encoding difference can produce a redirect mismatch error. Signed URLs, such as pre-signed storage links, compute a signature over the exact encoded string, so re-encoding the URL in any way, even equivalently, invalidates the signature. Pass those URLs around untouched.

• ▸Print the final URL exactly as sent, not a decoded version.
• ▸Look for %25 sequences, which indicate double encoding.
• ▸Compare a working request with a failing one character by character.
• ▸Paste the value into a URL encoder and decoder to see what each layer is doing to it.