SYNX 3.7: SYNXL brings typed record streams to AI datasets

SYNX 3.7 is out. It adds one operator to the language and a whole new file format next to it: SYNXL — a record-stream format for datasets, built for the two jobs JSONL and CSV are usually pressed into and handle badly.

The short version: declare your fields once, then stream typed records. Nesting and multiline text come along for the ride, and malformed rows get reported instead of quietly disappearing.

Why JSONL and CSV hurt on datasets

Open any JSONL training set and look at what you are actually paying for. Every record repeats every key, every brace, every quote. On a corpus of a few hundred thousand chat samples, the field names alone are a measurable share of the tokens you feed a model — and they carry no information after the first line.

Multiline text is worse. A prompt, a code sample, a model answer — all of it collapses into \n-escaped noise that no human reads and every diff mangles.

CSV avoids the repetition but gives up almost everything else: no types, no nesting, and the unresolved question every CSV pipeline eventually hits — is an empty cell an empty string, or a missing value? Add a comma to a field and you are in quoting rules that differ per dialect.

SYNXL in one screen

!synxl 1
!fields id[type:int] ; score[type:float] ; messages[block]

1 ; 0.91
  messages
    - role system
      content You are a helpful assistant.
    - role user
      content |+
          def f(x):
              return x + 1

2 ; 0.74
  messages
    - role user
      content Привет

The field list is declared once. Records start at column zero, scalar fields are separated by ;, and any field marked [block] opens up underneath the record in full SYNX — lists, nested objects, multiline text. That is what makes chat-shaped data expressible directly instead of flattened into an escaped string.

That example projects to ordinary JSON:

[{"id":1,"messages":[{"content":"You are a helpful assistant.","role":"system"},
  {"content":"def f(x):\n    return x + 1","role":"user"}],"score":0.91}, …]

What the format actually guarantees

• Types where they belong. [type:int], [required], [enum:a|b|c] live in the header, not in a sidecar schema file and not re-inferred per row.

• Empty is not nothing. An empty field is null; an empty string is written "". The CSV ambiguity is gone by construction.

• No silent drops. SYNX skips malformed structure quietly. SYNXL reports it — missing fields, surplus fields, failed casts, unknown block keys — each with a record index and a line number. A dataset that loses a column without telling you is worse than one that fails loudly.

• Records are independent. A record boundary is decidable from a single byte, so the format appends safely, shards cleanly, and parses in parallel.

• Schema evolution mid-file. Gained a column? Write a new !fields line and keep appending. Earlier records stay valid and keep their original shape.

Built for files that do not fit in memory

The 16 MiB input cap that SYNX applies to a document applies per record in SYNXL — datasets are routinely gigabytes, and records are independent, so a whole-file cap protects nothing a per-record cap does not.

The streaming reader holds exactly one record live. Measured with the CLI on a 142 MB file of 2.4 million records, peak memory stays flat at roughly 10 MB across validate, parse and split — reading the same file into memory would cost at least its own size.

The other half of 3.7: |+

The language itself gained exactly one thing: |+, an indent-preserving multiline opener. Plain | trims every continuation line, which is fine for prose and destructive for anything whitespace-significant. |+ locks a base indent on the first content line and preserves everything past it — so code, ASCII diagrams and prompt scaffolds survive intact inside a value.

SYNX 3.6 remains the frozen interoperability baseline. 3.7 is additive: a 3.6 parser stays conformant for every document that avoids 3.7-only constructs.

Reading it from code

SYNXL ships in four implementations today, with the same semantics behind each:

// Rust
let doc = synx_core::synxl::parse_lines(&text)?;
for record in &doc.records { /* … */ }

// TypeScript
import { parseSynxl, streamSynxlFile } from '@aperturesyndicate/synx-format'
for (const record of streamSynxlFile('dataset.synxl')) { /* … */ }

# Python
import synx_native as synx
for record in synx.synxl_stream_file("dataset.synxl"):
    train(record["messages"])

And from the shell, including converters in both directions:

synx synxl parse dataset.synxl --format ndjson
synx synxl validate dataset.synxl --constraints
synx synxl convert corpus.jsonl          # jsonl → synxl
synx synxl split dataset.synxl -n 500000 # shards that are valid on their own

Availability

3.7.1 is published on crates.io, npm, PyPI and NuGet.

SYNXL itself is implemented in Rust, TypeScript, Python and the CLI. The native C++, Dart, .NET, Go, Java and Swift parsers support SYNX 3.7 including |+, but do not read .synxl yet — until they do, convert with the CLI or read it from one of the four.

Specification and conformance

SYNXL is versioned on its own axis — format version 1, independent of the language version — because a SYNXL document is not a SYNX document: its top level is a sequence of records rather than an object.

The normative text is SYNXL-1-NORMATIVE.md, and the contract behind it is a suite of 67 conformance cases derived from that text independently of any implementation. The Rust runner additionally re-reads every accepted case through all three of its readers and fails if they disagree.

Full documentation and a live playground: synx.aperturesyndicate.com

What's next

SYNXL support for the remaining parsers, starting with Go and .NET. The specification is stable and the conformance suite is written, so the second wave is mechanical rather than exploratory — which is exactly the point of writing the tests before the implementations.