Guides / Parse diagnostics

JSON to CSV Error Positions: Character Index or UTF-8 Byte Offset?

Short answer: A Python JSONDecodeError position indexes the decoded document, not its original UTF-8 bytes. For unchanged UTF-8 input without a BOM, map it with len(text[:error.pos].encode("utf-8")). Keep character index, byte offset, line and column labelled separately, especially when non-ASCII text appears before the error.

This guide helps locate a failed JSON input before CSV conversion. It does not repair the document or claim that every converter reports positions using Python's convention. A browser, editor or other parser may use a different indexing scheme.

One error, several coordinate systems

The official JSONDecodeError reference defines pos as an index into the document and provides lineno and colno. In this recipe, character_index and utf8_byte_offset start at zero; line and column are the parser's one-based values. They are not interchangeable with visual cursor columns or a browser's UTF-16 code-unit indexes.

A UTF-8 emoji uses more bytes than its decoded Python string index advances. Conversely, the ASCII source spelling \ud83d\ude42 occupies its own source characters. The error's position refers to the document string supplied to the parser, not to the value that a preceding JSON escape would represent.

Seven original fixture results

We executed the read-only inspector locally on 2026-10-29. The fixtures preserve their original bytes, including CRLF. The malformed examples contain a trailing comma or an unfinished array; the last fixture is a valid comparison. The table records actual parser results, not estimated editor positions.

Exact fixture text follows. CRLF is shown through an escaped diagnostic representation in that case, so two line-ending bytes are not silently rendered as one:

ascii

[{"text":"A",}]

japanese

[{"text":"東京",}]

emoji

[{"text":"🙂",}]

escaped-emoji

[{"text":"\ud83d\ude42",}]

crlf

'[\r\n{"text":"🙂",}\r\n]'

end-of-input

[{"text":"🙂"}

valid

[{"text":"🙂"}]

The emoji and Japanese cases have a larger byte offset than character index. In the ASCII and ASCII-escaped cases they match. In the unfinished-array case, the reported byte offset equals the file length: it identifies the end boundary, not a byte that can be read there. Each failed case also checked that the computed byte prefix equals the original source prefix.

Map the exact input, not a cleaned copy

raw = path.read_bytes()
text = raw.decode("utf-8", errors="strict")
# after json.loads(text) raises JSONDecodeError:
byte_offset = len(text[:error.pos].encode("utf-8"))

Do not normalize newlines, remove whitespace or reserialize the JSON before making this calculation. Such changes produce a different coordinate space. The inspector reads bytes first so CRLF is retained. It explicitly refuses a UTF-8 BOM rather than returning an unlabelled offset relative to a stripped payload. For UTF-16, decompressed data or text extracted from an envelope, define a separate mapping to the actual input bytes.

Download the read-only inspector

Download the full inspector, emoji error fixture, ASCII-escaped error fixture, raw CRLF fixture and end-boundary fixture.

python json-error-offset-inspect.py offset-emoji.json

It prints a JSON diagnostic and exits with status 1 for a parser syntax error, 0 when this parser returns successfully, or 2 for the handled input/decoding failures. It never creates a CSV, rewrites the input or prints a source-text excerpt. Its SHA-256 and source-byte count identify which exact file the coordinates describe; they do not authenticate the producer.

import argparse, hashlib, json
from pathlib import Path

def inspect(source):
    path = Path(source)
    if path.stat().st_size > 1_000_000:
        raise ValueError("Demo input cap exceeded")
    raw = path.read_bytes()
    if raw.startswith(b"\xef\xbb\xbf"):
        raise ValueError("This mapping requires UTF-8 without a BOM")
    text = raw.decode("utf-8", errors="strict")
    report = {"source_sha256": hashlib.sha256(raw).hexdigest(),
              "source_bytes": len(raw)}
    try:
        json.loads(text)
    except json.JSONDecodeError as error:
        byte_offset = len(text[:error.pos].encode("utf-8"))
        assert raw[:byte_offset] == text[:error.pos].encode("utf-8")
        report.update(syntax_parsed=False, message=error.msg,
                      character_index=error.pos, utf8_byte_offset=byte_offset,
                      line=error.lineno, column=error.colno,
                      at_end=error.pos == len(text))
        return report, 1
    report.update(syntax_parsed=True)
    return report, 0

if __name__ == "__main__":
    p = argparse.ArgumentParser()
    p.add_argument("source")
    args = p.parse_args()
    try:
        report, status = inspect(args.source)
        print(json.dumps(report, ensure_ascii=True, sort_keys=True))
        raise SystemExit(status)
    except (ValueError, OSError, RecursionError) as error:
        p.exit(2, str(error) + "\n")

Use the location as evidence, not automatic permission to patch

A parser position is where parsing failed; the cause can begin earlier. Check the producer's syntax and contract before changing anything. The 1,000,000-byte input cap is this example's chosen policy, not a converter benchmark. A successful json.loads call also is not proof of the required record schema, unique keys or strict rejection of nonstandard constants.

The guide's scope is locating a syntax failure in exact source bytes. For escape spelling versus decoded fields, see source retention. For UTF-16 source decoding, see the declared byte-order workflow. After correcting the input through an agreed producer policy, convert and verify the exported fields; a corrected diagnostic alone does not establish a complete CSV export.