Guides / Initial UTF-8 signature

Remove the JSON File's Initial BOM, Not Characters Inside Its Fields

The same bytes have different roles at different positions

Our original input starts with EF BB BF before the opening array bracket. It also contains that byte sequence inside the second record's note. The first is the transport prefix accepted by this recipe; the second encodes a character in field content. Deleting every occurrence confuses those roles and changes a value.

File start: EF BB BF 5B ...
Record B2 note: "inside\ufefffield"
Preserved field: inside + U+FEFF + field
Incorrect global deletion: insidefield

The displayed escape makes the field character visible. The downloadable fixture contains its UTF-8 bytes directly. A separate executed fixture uses the JSON escape \ufeff, which also decodes to the same field character.

RFC 8259, section 8.1 requires UTF-8 for JSON exchanged outside a closed ecosystem, prohibits adding a BOM to network-transmitted JSON, and permits parsers to ignore an initial one. This does not turn a character inside a string into removable transport metadata.

Why a plain UTF-8 decode can still fail

Decoding bytes with utf-8 retains the initial character. In our Python 3.12.14 execution, passing that decoded string to json.loads raised:

Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)

This is a result for the decoded-string call shown here, not a claim that every JSON parser rejects BOM-prefixed bytes. The Python JSON reference documents the initial-BOM behavior. Different parser entry points and implementations can handle byte input differently.

raw = Path('bom-input.json').read_bytes()
text = raw.decode('utf-8-sig', errors='strict')
rows = json.loads(text)
# Do not use text.replace('\ufeff', '') on the document.

Python's utf-8-sig codec reference states that decoding skips an optional UTF-8 signature at the beginning. Encoding with that codec adds a signature, so choosing it for input does not mean choosing it for output. Our CSV writer uses ordinary UTF-8.

Original example: preserve B2 after CSV read-back

[{"id":"B1","note":"ordinary"},
 {"id":"B2","note":"inside\ufefffield"}]

Download all files into one folder and run Python 3.12 or later:

python bom-to-csv.py bom-input.json new-output.csv
python bom-verify.py

On 2026-10-14, Python 3.12.14 exported the two original records and reconstructed the exact ordered strings using csv.DictReader. B2 retained the sequence U+0069 U+006E U+0073 U+0069 U+0064 U+0065 U+FEFF U+0066 U+0069 U+0065 U+006C U+0064. The checker reads the supplied measured CSV; change its filename when checking your own newly exported file.

We also executed the tempting global replacement before parsing. It produced valid JSON and a changed B2 value, insidefield. Parsing success therefore did not establish data preservation. The evidence download contains that failed equality comparison and the successful export's code points.

Test the boundary, not just a friendly example

One leading prefix and no prefix both pass. Two consecutive prefixes fail because only the first is skipped; a remaining character at the document start is not valid JSON syntax. A prefix placed after a space is not at byte offset zero and also fails. Invalid UTF-8 and the UTF-16 fixture fail this deliberately UTF-8-only recipe. A BOM signature is not a universal encoding detector.

The escaped field-character case passes with the character preserved. None of the rejected fixtures creates a CSV. These are our particular executed cases, not exhaustive testing of all encodings or malformed input.

Keep input decoding separate from CSV policy

The exporter accepts only a JSON array with exactly string fields id and note. It rejects duplicate keys, nonstandard constants, values that do not satisfy that contract and strings that cannot be encoded as UTF-8. It reads at most 1,000,001 bytes and rejects above a 1,000,000-byte demonstration policy. That policy is not a measured converter limit or a total-memory guarantee.

Validation completes before exclusive CSV output creation. Existing output is never overwritten. A disk failure during writing can still leave a partial output. The original JSON file remains available for inspection; source-byte prefix handling does not silently clean, normalize or trim its field values.

Our CSV has a plain id,note header without an initial BOM. If your receiving spreadsheet requires a different CSV encoding policy, choose it separately and read the file back with that explicit policy. CSV quoting does not prevent spreadsheet formula execution in arbitrary untrusted fields. This example is a local Python export workflow, not a tested claim about the browser converter's BOM handling.

The same input contract in three images

For another transformation that can change string equality, see Unicode normalization. That guide concerns NFC/NFD transformations; this page concerns only a document prefix versus a field character. Use row and field verification after export, or open the JSON to CSV converter for inspectable input.