Guides / Unicode output preflight

JSON to CSV: Find Unpaired Surrogates Before UTF-8 Output

Short answer: Before creating a UTF-8 CSV, check the decoded field values for surrogate code points U+D800 through U+DFFF. Refuse an unpaired surrogate with its record, field and position. Keep a valid JSON surrogate pair as its decoded emoji, and keep literal backslash text unchanged. Do not hide failures with replacement or ignore encoding modes.

A JSON parser can return a string that the next encoding step cannot write as ordinary UTF-8. The useful boundary here is decoded field to output bytes, not whether the source contains a backslash. This is a local Python preflight recipe, not a measurement of the browser converter.

Parsing and encoding are separate checks

RFC 8259 section 8.2 describes unpaired UTF-16 surrogate escapes that its grammar can admit and warns about interoperability. A paired escape is different: the JSON representation "\ud83d\ude42" can represent the smiling emoji. Python's Unicode HOWTO explains the separate conversion from Unicode strings to encoded bytes and its error-handling choices.

Our contract is an array of exact id/text string objects whose decoded values can be encoded with strict UTF-8. It is deliberately narrower than accepting every bit sequence the JSON grammar can describe. No numeric, nested-row or source-spelling repair is performed.

Seven original runs with exact inputs

Executed locally on 2026-10-28. For each accepted case, we read the CSV back and compared every field against the parsed records. Rejections happened before destination creation, and the source bytes remained unchanged in every run. The error uses one-based record numbers and zero-based decoded Python string indexes; these are not source-byte positions or visual grapheme counts.

paired-escape

[{"id":"E1","text":"\ud83d\ude42"}]

Observed CSV:

id,text
E1,🙂

literal-emoji

[{"id":"E1","text":"🙂"}]

Observed CSV:

id,text
E1,🙂

visible-backslash

[{"id":"E2","text":"\\ud800"}]

Observed CSV:

id,text
E2,\ud800

lone-high

[{"id":"H1","text":"A\ud800B"}]

Observed rejection:

Surrogate rejected: record=1, field=text, character_index=1, code_point=U+D800

lone-low

[{"id":"L1","text":"\udfff"}]

Observed rejection:

Surrogate rejected: record=1, field=text, character_index=0, code_point=U+DFFF

reversed-pair

[{"id":"R1","text":"\ude42\ud83d"}]

Observed rejection:

Surrogate rejected: record=1, field=text, character_index=0, code_point=U+DE42

surrogate-id

[{"id":"X\ud800","text":"valid text"}]

Observed rejection:

Surrogate rejected: record=1, field=id, character_index=1, code_point=U+D800

The paired-escape and literal-emoji inputs produced identical CSV bytes. The emoji's UTF-8 bytes were F0 9F 99 82. The visible-backslash case retained backslash-u-d-8-0-0 as ordinary field text. A regex that rejects every occurrence of that spelling in raw source would reject legitimate text along with malformed values.

Report the bad value without copying its contents into logs

for index, char in enumerate(value):
    point = ord(char)
    if 0xD800 <= point <= 0xDFFF:
        raise ValueError(
            f"Surrogate rejected: record={record}, field={field}, "
            f"character_index={index}, code_point=U+{point:04X}")
value.encode("utf-8", errors="strict")

This diagnostic exposes the location and code point, not the full user-supplied field. Fix the producer or recover the original text from a trustworthy source. Do not guess a missing half of a pair. Encoding with errors='replace' or errors='ignore' changes what is written; encoding with surrogatepass is not this recipe's ordinary UTF-8 interchange contract.

The range scan applies to decoded Python strings. Do not run this same range test on individual UTF-16 code units in JavaScript and conclude that every valid emoji is bad: the string representations differ. This guide's observed behavior and indexes come from the downloadable Python implementation.

Download and reproduce

Download the complete exporter, paired-escape fixture, literal-emoji fixture, lone-high rejection fixture, literal-backslash fixture and exact emoji CSV.

python unicode-scalar-csv-export.py scalar-paired-escape.json emoji-new.csv
python unicode-scalar-csv-export.py scalar-lone-high.json rejected-new.csv

Choose new destinations in a trusted directory. The script checks every record and field before opening the output with exclusive creation. It refuses existing destinations, identical source/output paths, duplicate keys and bare nonstandard numeric constants. Its 1,000,000-byte cap is a chosen demonstration policy; it reads the document in memory and does not provide a durable transaction or a general concurrency guarantee.

import argparse, csv, json
from pathlib import Path

def reject_constant(token):
    raise ValueError("Non-JSON constant: " + token)

def unique_keys(pairs):
    result = {}
    for key, value in pairs:
        if key in result:
            raise ValueError("Duplicate key: " + key)
        result[key] = value
    return result

def decode(raw):
    return json.loads(raw, parse_constant=reject_constant,
                      object_pairs_hook=unique_keys)

def export(source, target):
    source, target = Path(source), Path(target)
    if source.resolve() == target.resolve():
        raise ValueError("Source and destination must differ")
    if source.stat().st_size > 1_000_000:
        raise ValueError("Demo byte cap exceeded")
    data = decode(source.read_text(encoding="utf-8"))
    if not isinstance(data, list):
        raise ValueError("Expected a records array")
    for record, row in enumerate(data, start=1):
        if not isinstance(row, dict) or set(row) != {"id", "text"}:
            raise ValueError("Expected exact id/text objects")
        if not all(isinstance(row[k], str) for k in ("id", "text")):
            raise ValueError("id and text must be strings")
        for field in ("id", "text"):
            value = row[field]
            for index, char in enumerate(value):
                point = ord(char)
                if 0xD800 <= point <= 0xDFFF:
                    raise ValueError(
                        f"Surrogate rejected: record={record}, field={field}, "
                        f"character_index={index}, code_point=U+{point:04X}")
            value.encode("utf-8", errors="strict")
    with target.open("x", encoding="utf-8", newline="") as f:
        writer = csv.writer(f, lineterminator="\n")
        writer.writerow(["id", "text"])
        writer.writerows((row["id"], row["text"]) for row in data)

if __name__ == "__main__":
    p = argparse.ArgumentParser()
    p.add_argument("source")
    p.add_argument("target")
    args = p.parse_args()
    try:
        export(args.source, args.target)
    except (ValueError, OSError, RecursionError) as e:
        p.exit(1, str(e) + "\n")

What a passing result does not prove

This check establishes encodable string values within the demonstrated schema. It does not normalize Unicode, sanitize spreadsheet formulas, or decide which characters a particular receiver permits. In particular, NUL can be UTF-8-encodable yet violate a receiving text-column contract; see the separate NUL handoff gate.

The earlier source-spelling workflow retains source bytes and includes surrogate rejection as a boundary case. Today's workflow is the dedicated record/field diagnostic before output creation, not another receipt or escape-spelling article. The UTF-16 byte-decoding guide addresses malformed input bytes, a different boundary from escaped surrogate values in otherwise readable JSON text.

After export, compare parsed CSV fields against accepted records. A successful encoder alone is not evidence of correct row selection or downstream import behavior.