Also known as: JSON Lines, NDJSON, newline-delimited JSON, ldjson
JSONL — JSON Lines, also called NDJSON — is one JSON object per line. Brutally simple, ubiquitous in ML datasets and log shipping, friendly to streaming and append-only writes.
JSONL is JSON, one object per line, with \n as the record separator. The whole spec fits in one sentence — and that’s the point. There is no schema header, no compression layer, no offset index, no metadata. Just {...}\n{...}\n{...}\n. Every Python script, every awk one-liner, every streaming consumer, every cloud-storage tool already handles it. It’s the format that won by being too dumb to lose.
{"id": 1, "text": "first record", "label": "a"}
{"id": 2, "text": "second record", "label": "b"}
{"id": 3, "text": "third record", "label": "c"}The shape of JSONL exactly matches “structured records produced one at a time and consumed in order.” That’s most of the ML data lifecycle:
The line boundary is load-bearing. It lets you wc -l to count records, head -n 1000 to sample, split -l to shard, cat *.jsonl to merge. The price is no built-in schema, no compression, no column projection. For most ML datasets that price is fine.
Three signals that you’ve outgrown it:
Below ~50 GB or single-team ownership, stay on JSONL. The simplicity wins.
json.dumps(obj) with no indent= argument.\n inside string values is the silent killer. A naive producer that writes obj["text"] = "line1\nline2" will produce a JSONL line that splits into two when a downstream reader does .split("\n"). Always escape newlines (json.dumps does this by default; CSV-to-JSONL converters often don’t)..jsonl.gz and .jsonl.zst are the conventions. pyarrow.json can stream-decompress them. Don’t store raw JSONL above ~10 GB if you have any control over the producer; the 5-10× compression ratio is free.application/x-ndjson is the most-supported MIME type.| Property | JSONL | Parquet | CSV |
|---|---|---|---|
| Schema-on-write | ❌ | ✅ | ❌ |
| Column projection | ❌ | ✅ | ❌ |
| Streaming-friendly | ✅ | partial | ✅ |
| Edit by hand | ✅ | ❌ | ✅ |
| Append at O(1) | ✅ | ❌ | ✅ |
| Storage efficiency | poor | excellent | poor |
| Universal tooling | ✅ | mostly | ✅ |
JSONL wins on every column where simplicity helps. Parquet wins on every column where scale helps. Pick by where your dataset is in its lifecycle.
Because a JSON array forces every reader to parse the whole file before yielding the first element — which means you can't stream it, you can't append to it, and you can't process it line-by-line in a shell pipeline. JSONL flips all three. Each line is a self-contained JSON document; readers consume one line at a time; producers append at O(1); head -n 1000 works.
Once your dataset crosses ~50 GB AND you scan only a few fields per query, switch to Parquet. JSONL parses every byte of every record even when you only need two columns out of fifty; Parquet skips the rest of the row entirely. The transition point is when your training step becomes I/O-bound — typically a few hundred GB for ML datasets, sooner for analytics.
Both use JSONL. Each line is a JSON object with a messages array ({role: 'user'|'assistant'|'system', content: '...'}). OpenAI's fine-tuning format expects {messages: [...]} per line; Anthropic's batch API expects one request per line with custom_id for joining results. The ecosystem converged on JSONL because every loader, sharder, and shuffler already handled it.
