Guides / Handoff policy

NUL in JSON to CSV: Reject the Handoff, Do Not Delete the Character

Short answer: If the receiving system forbids NUL, inspect decoded JSON field values for U+0000 before writing the CSV. Reject the export with a record, field and character position, and keep the source unchanged. Do not silently delete the character or assume CSV quoting makes it acceptable to the receiver.

An escaped \u0000 is permitted inside a JSON string. That does not mean every downstream text column can store its decoded value. This page implements a deliberate NUL-free handoff contract for a local Python exporter, not a universal claim that all CSV software rejects NUL.

Check the receiving contract first

RFC 8259 section 7 requires JSON control characters to be escaped in source text. Parsing "A\u0000B" therefore gives a value with a NUL between A and B; it is not the same thing as an empty string or a JSON null.

A concrete receiving constraint appears in PostgreSQL 18's character-type documentation: code zero cannot be stored in those character types. That is a documented database restriction, not a PostgreSQL import we measured here. Check your actual receiver rather than generalizing this policy to every spreadsheet, database or CSV dialect.

Our export gate and observed locations

We executed six original local cases on 2026-10-27. The schema is an array of exact id/text string objects. Records below are numbered from one; character_index starts at zero within the decoded Python string, counts Unicode code points and is not a source-byte offset or a displayed grapheme count.

plain

Exact input:

[{"id":"A1","text":"AB"}]

Observed CSV:

id,text
A1,AB

nul-at-start

Exact input:

[{"id":"A2","text":"\u0000AB"}]

Observed rejection:

NUL rejected: record=1, field=text, character_index=0

nul-after-emoji

Exact input:

[{"id":"A3","text":"\ud83d\ude42A\u0000B"}]

Observed rejection:

NUL rejected: record=1, field=text, character_index=2

nul-in-id

Exact input:

[{"id":"X\u0000Y","text":"AB"}]

Observed rejection:

NUL rejected: record=1, field=id, character_index=1

literal-backslash

Exact input:

[{"id":"A4","text":"A\\u0000B"}]

Observed CSV:

id,text
A4,A\u0000B

existing-output

Exact input:

[{"id":"A1","text":"AB"}]

Observed rejection:

[Errno 17] File exists: 'F:\\CodexWork\\jsontocsvtool\\drafts\\2026-10-27-nul-handoff\\existing-output.csv'

In the emoji case, the decoded sequence begins with the emoji, then A, then NUL, so the reported index is 2. The literal-backslash case contains the six visible characters backslash-u-0-0-0-0 and is accepted. We parsed both successful CSVs back and compared all fields to their source records. The sources remained byte-for-byte unchanged in all six runs. NUL refusals created no output file; the existing-output case retained its previous contents.

Why deletion changes the data

Original value 1, diagnostic spelling: 'A\x00B'
Original value 2:                      'AB'
After deleting NUL from both:          'AB', 'AB'

Our local deletion comparison produced identical results from those two different values. An export that strips NUL would erase this distinction. Keeping the original document and refusing the handoff gives the producer a chance to correct the source or agree on a separate, reversible representation.

The rejection message intentionally identifies a location without dumping the whole field. Actual records can contain private text. Do not send raw failed records to public logs automatically, and do not describe an altered review copy as the unchanged source.

Reproduce before your next handoff

Download the complete NUL-free exporter, emoji/NUL rejection input, literal-backslash input and exact accepted CSV.

python reject-nul-csv-export.py nul-handoff-nul-after-emoji.json rejected-new.csv
python reject-nul-csv-export.py nul-handoff-literal-backslash.json accepted-new.csv

Use fresh output names in a trusted directory. This script checks all records before exclusive output creation. It rejects non-string fields, duplicate keys, bare nonstandard numeric constants, identical source/destination paths and an existing destination. Its 1,000,000-byte input cap is a chosen demo policy. It loads the document in memory and does not claim crash durability or protection against concurrent directory changes.

import argparse, csv, json
from pathlib import Path

def reject_constant(token):
    raise ValueError("Non-JSON constant: " + token)

def unique_keys(pairs):
    result = {}
    for key, value in pairs:
        if key in result:
            raise ValueError("Duplicate key: " + key)
        result[key] = value
    return result

def decode(raw):
    return json.loads(raw, parse_constant=reject_constant,
                      object_pairs_hook=unique_keys)

def export(source, target):
    source, target = Path(source), Path(target)
    if source.resolve() == target.resolve():
        raise ValueError("Source and destination must differ")
    if source.stat().st_size > 1_000_000:
        raise ValueError("Demo byte cap exceeded")
    data = decode(source.read_text(encoding="utf-8"))
    if not isinstance(data, list):
        raise ValueError("Expected a records array")
    for record, row in enumerate(data, start=1):
        if not isinstance(row, dict) or set(row) != {"id", "text"}:
            raise ValueError("Expected exact id/text objects")
        if not all(isinstance(row[k], str) for k in ("id", "text")):
            raise ValueError("id and text must be strings")
        for field in ("id", "text"):
            value = row[field]
            position = value.find("\x00")
            if position != -1:
                raise ValueError(
                    f"NUL rejected: record={record}, field={field}, "
                    f"character_index={position}")
            value.encode("utf-8", errors="strict")
    with target.open("x", encoding="utf-8", newline="") as f:
        writer = csv.writer(f, lineterminator="\n")
        writer.writerow(["id", "text"])
        writer.writerows((row["id"], row["text"]) for row in data)

if __name__ == "__main__":
    p = argparse.ArgumentParser()
    p.add_argument("source")
    p.add_argument("target")
    args = p.parse_args()
    try:
        export(args.source, args.target)
    except (ValueError, OSError, RecursionError) as e:
        p.exit(1, str(e) + "\n")

Keep the boundary narrow

The gate checks NUL in both demonstrated fields; it is not a recursive scanner for arbitrary nested data, nor does it reject every other control character. A tab or line break requires its own receiving policy. For quoted multiline fields, use the line-break guide. For null, missing fields and empty strings, use the field-state guide; NUL is a character inside a string, not one of those field states.

The CSV writer still needs proper quoting and newline="", as described in Python's writer reference. Quoting defines record and field boundaries; the receiver's allowed-character rules remain separate. Untrusted spreadsheet formulas are also outside this NUL gate.

When an export passes, compare parsed fields and record counts. A passing NUL check establishes only this contract. It does not prove that every downstream type conversion or application rule will preserve the data.