This is a fourteen-byte JSON document:
{"a":"\uDEAD"}Node parses it. Python parses it. PHP refuses it. All three are behaving correctly, and the reason is that "valid JSON" turns out not to be a single predicate.
The specification says so itself
RFC 8259 requires UTF-8 on the wire: "JSON text exchanged between systems that are not part of a closed ecosystem MUST be encoded using UTF-8." But its grammar for an escape is just %x75 4HEXDIG — four hex digits, with no rule that a leading surrogate must be followed by a trailing one.
The specification names the hole, and it names the mechanism:
"the ABNF in this specification allows … bit sequences that cannot encode Unicode characters; for example, "\uDEAD" … when a library truncates a UTF-16 string without checking whether the truncation split a surrogate pair … implementations might … suffer fatal runtime exceptions."
⚠️ The warning about a library truncating a UTF-16 string describes chunked streaming. Tool-call arguments from a model are assembled from deltas in exactly that way.
Three runtimes, three behaviours
| Runtime | What happens |
|---|---|
| Node | Parses. JSON.stringify round-trips it faithfully. isWellFormed() returns false, and writing the value as UTF-8 silently produces ef bf bd — the replacement character. |
| Python | Parses. json.dumps succeeds. Encoding the value raises UnicodeEncodeError. |
| PHP | Refuses at parse. json_last_error() is 10: "Single unpaired UTF-16 surrogate in unicode escape". |
The control on PHP is that the same parser accepts {"a":"𝄞"}, a correctly paired astral character, with no error. It is discriminating on pairing, not on being unusual.
Python needs one correction that is easy to get wrong, and we got it wrong first. It is often said that Python raises on output here. By default it does not: json.dumps escapes the surrogate back to ASCII as \udead, and encoding that is fine. The exception arrives when you encode the string value itself, or when you dump with ensure_ascii=False:
json.dumps(v).encode("utf-8") → ok
v["a"].encode("utf-8") → UnicodeEncodeError
json.dumps(v, ensure_ascii=False).encode("utf-8") → UnicodeEncodeErrorThe distinction matters because a web framework writing a response body typically uses one of the two forms that raise.
The bytes
Node's replacement character is U+FFFD, and its UTF-8 encoding derives directly:
U+FFFD → 1110 1111 10 111111 10 111101 = ef bf bdPython, asked to encode the surrogate anyway with surrogatepass, produces ed ba ad — which is not UTF-8 at all but WTF-8, the same derivation applied to a code point UTF-8 is not allowed to represent.
Moving the same logical document through Node and Python yields two different byte sequences on the wire. One of them substitutes U+FFFD, losing the original value permanently, without reporting an error.
Where this reaches a token counter
tiktoken carries a hand-written workaround for exactly this, and the maintainers' comment concedes what it costs: "Technically, this introduces a place where encode + decode doesn't roundtrip a Python string", because "we use errors="replace" to handle weird things like lone surrogates".
Reproduced on version 0.14.0 with the o200k encoding, the string A\uDEADB encodes to three tokens and decodes back as A�B — the round-trip returns false. The control is a correctly paired character in the same test, which round-trips exactly.
The reference token counter measures a repaired string, not the original input. The one-character difference occurs in the component responsible for deciding whether a request fits the context window.
What this does not mean
None of these behaviours is a bug. Node preserves the input, Python refuses to encode an invalid string, and PHP rejects it at parse. Each choice is defensible and they cannot all be simultaneously satisfied.
RFC 8259 is not careless either. It states the hole, names the mechanism, and warns that behaviour is unpredictable. A pairing constraint would require a semantic rule, which the grammar cannot express. Retrofitting one would invalidate documents already in circulation — the same reasoning that gave JavaScript a well-formed stringify rather than a stricter parse.
⚠️ And we did not demonstrate the thing the specification suggests. RFC 8259 names UTF-16 truncation as the cause in general; we did not obtain any particular vendor's stream splitting a surrogate pair, and this piece does not claim one does. The gap is real and we are leaving it open rather than closing it by implication.
What to do with it
Validate for well-formedness, not for parse success. That JSON.parse returned is not evidence the document can be transported.
Node str.isWellFormed() / str.toWellFormed()
Python s.encode("utf-8") inside a try
PHP handle JSON_ERROR_UTF16 from json_last_error()In a multi-runtime pipeline, the component that first rejects a payload may be correctly identifying something an upstream service passed on. A PHP service refusing what a Node service happily forwarded is the PHP service catching something, not breaking.
And if you are counting tokens to decide whether a request fits, know that the counter may be measuring a repaired copy of your input.
Where this comes from, and what will date it
The requirement and the warning are quoted from RFC 8259 itself, a 28,360-byte document. The three runtime behaviours were executed here rather than cited — Node 26.4.0, Python 3.9.6 and PHP 8.4.24 — and the byte sequences were re-derived from the code points rather than copied. The tokenizer behaviour is from tiktoken 0.14.0, with the maintainers' comment quoted from its own source.
⚠️ For disclosure, our own JSON formatter and JSON repair tool both route through a parser that accepts this document, and both report it valid. That is correct under the grammar, and we are not calling it a defect — but a validity verdict that says nothing about well-formedness is exactly the gap described above, and noting it beside the verdict would help.
This dates very slowly. RFC 8259 is stable, the runtimes' choices are long-standing, and the tokenizer claim is pinned to a version. What would date it is a runtime changing its mind, which would be news in itself.
View author profile → · Editorial policy
