Guides / Source decoding

UTF-16 JSON to UTF-8 CSV: Declare Byte Order, Preserve Field Characters

Short answer: Decode UTF-16 source bytes before parsing JSON. Use a leading UTF-16 BOM to select byte order, or require an explicit UTF-16LE/UTF-16BE declaration when the BOM is absent. Export the resulting fields as UTF-8 CSV. Remove only a recognized source signature at byte offset zero, never U+FEFF characters inside field values.

This is a local Python source-decoding recipe with a fixed id/text string schema. It is not another browser file-size test or an automatic detector for arbitrary encodings. The input encoding and CSV output encoding are separate choices.

A source marker and a field character are different

Our original records contain text beginning with U+FEFF and a second field containing Japanese text plus an emoji. The first field, shown with an explicit escape for inspection, is "\ufeffKeep". That character belongs to the value even though its encoded spelling can resemble a file signature.

Python's encoding documentation distinguishes byte-order signatures from decoded text. Its codec table provides UTF-16LE and UTF-16BE separately. We use an explicit input policy rather than guessing the byte order from a successful-looking decode.

Observed results from eight original fixtures

Executed locally on 2026-10-25 with the downloadable exporter. Byte counts and leading hexadecimal bytes below come from our generated files. Successful CSVs were read back with csv.DictReader and compared field by field with the original records; unsuccessful cases created no CSV destination.

The BOM-marked little-endian and big-endian files, plus explicitly declared BOM-less little-endian input, produced identical UTF-8 CSV bytes. This equality was checked across those three outputs. It is not a claim that every byte sequence can be decoded safely by trying codecs until one works.

Inspect the successful CSV without hiding U+FEFF

The following is a diagnostic representation of its text, so the invisible character and line terminators are escaped. It is not the literal spelling stored in the CSV:

'id,text\nU1,\ufeffKeep\nU2,東京 🙂\n'

The first text field retains U+FEFF followed by Keep. Its UTF-8 field prefix is EF BB BF 4B 65 65 70, after the header and identifier. None of the successful files starts with a UTF-8 BOM: that is the chosen output policy, not deletion of the same bytes wherever they appear.

The decoder contract

• Default mode requires FF FE or FE FF at the very start.
• Explicit LE or BE permits no signature, but rejects a conflicting signature.
• Recognized UTF-32 signatures are rejected before interpreting a UTF-16 prefix.
• Only the recognized leading two-byte signature is removed; all remaining bytes are strictly decoded.

Strict decoding raises on malformed input instead of inserting replacement characters or ignoring bytes. See the official bytes.decode documentation. A successful decode still must pass JSON parsing and the record schema. Do not delete zero bytes, strip all U+FEFF characters or use errors='ignore' to make an error disappear.

Download and reproduce

Download the complete standalone exporter, BOM-marked LE input, BOM-marked BE input, BOM-less LE input and exact expected UTF-8 CSV. Keep the input files as raw downloads, rather than copying a browser-rendered preview and calling that a byte-for-byte fixture.

python utf16-json-csv-export.py utf16-bom-le.json le-new.csv
python utf16-json-csv-export.py utf16-bom-be.json be-new.csv
python utf16-json-csv-export.py utf16-declared-le.json declared-new.csv --input-encoding utf-16-le

Use new output names. The script validates all records before exclusive destination creation, refuses an existing output and requires different source/destination paths. It uses a chosen 1,000,000-byte demo cap and loads the document in memory. This is not a streaming tool, a benchmark or a crash-durable file transaction. Review untrusted fields before spreadsheet import; correct encoding does not neutralize spreadsheet formulas.

import argparse, csv, json
from pathlib import Path

def reject_constant(token):
    raise ValueError("Non-JSON constant: " + token)

def unique_keys(pairs):
    result = {}
    for key, value in pairs:
        if key in result:
            raise ValueError("Duplicate key: " + key)
        result[key] = value
    return result

def decode(raw):
    return json.loads(raw, parse_constant=reject_constant,
                      object_pairs_hook=unique_keys)

def decode_utf16(raw, encoding):
    if raw.startswith((b"\xff\xfe\x00\x00", b"\x00\x00\xfe\xff")):
        raise ValueError("UTF-32 is outside this UTF-16 recipe")
    mark = None
    if raw.startswith(b"\xff\xfe"):
        mark = "utf-16-le"
    elif raw.startswith(b"\xfe\xff"):
        mark = "utf-16-be"
    if encoding == "utf-16":
        if mark is None:
            raise ValueError("Missing UTF-16 BOM; declare byte order")
        codec = mark
    else:
        codec = encoding
        if mark is not None and mark != codec:
            raise ValueError("BOM conflicts with declared byte order")
    payload = raw[2:] if mark is not None else raw
    return payload.decode(codec, errors="strict")

def export(source, target, encoding="utf-16"):
    source, target = Path(source), Path(target)
    if source.resolve() == target.resolve():
        raise ValueError("Source and destination must differ")
    if source.stat().st_size > 1_000_000:
        raise ValueError("Demo byte cap exceeded")
    raw = source.read_bytes()
    data = decode(decode_utf16(raw, encoding))
    if not isinstance(data, list):
        raise ValueError("Expected a records array")
    for row in data:
        if not isinstance(row, dict) or set(row) != {"id", "text"}:
            raise ValueError("Expected exact id/text objects")
        if not all(isinstance(row[k], str) for k in ("id", "text")):
            raise ValueError("id and text must be strings")
    with target.open("x", encoding="utf-8", newline="") as f:
        writer = csv.writer(f, lineterminator="\n")
        writer.writerow(["id", "text"])
        writer.writerows((row["id"], row["text"]) for row in data)

if __name__ == "__main__":
    p = argparse.ArgumentParser()
    p.add_argument("source")
    p.add_argument("target")
    p.add_argument("--input-encoding", choices=["utf-16", "utf-16-le", "utf-16-be"], default="utf-16")
    args = p.parse_args()
    try:
        export(args.source, args.target, args.input_encoding)
    except (ValueError, OSError, RecursionError) as e:
        p.exit(1, str(e) + "\n")

Where this workflow stops

This recipe exports only exact id/text string objects. It does not normalize Unicode, reinterpret numeric columns or automatically repair a producer's encoding declaration. A missing BOM is not proof of LE; select an explicit mode only when the producer or file specification establishes that order. A wrong BOM-less declaration can sometimes produce valid text, so successful decoding alone is not evidence that the declaration was correct.

For UTF-8 source signatures and CSV output signatures, see the separate UTF-8 BOM guide. For previously observed browser file shapes, see file input measurements. Today's independent workflow covers a declared UTF-16 decoding boundary and refusal cases rather than repeating those browser observations.

After decoding, verify every parsed CSV field. Compare character values as well as visible rendering: an invisible U+FEFF can survive correctly while a screenshot makes two strings appear identical.