TOON Format Reference

TOON (Token-Oriented Object Notation) is a compact encoding of the JSON data model designed for LLM consumption. It reduces token usage by approximately 66% compared to JSON while maintaining lossless round-trips.

This document describes the TOON format as implemented in epanetparser (SPEC v4.1 compliance).

Overview

TOON provides five array forms and an object form for nested structures:

  1. Inline — Primitive arrays on header line (e.g., tags[3]: a,b,c)

  2. List — Non-uniform arrays, one - item per element

  3. Tabular — Uniform object arrays, declare fields once then stream rows

  4. Keyed Tabular — Objects of uniform objects, rows carry their own key

  5. Object — Nested objects using indentation

Design Goals

  • Token Efficiency: Minimize LLM context usage for structured data

  • Human Readable: Familiar syntax (YAML-like indentation, CSV-like tables)

  • LLM Guardrails: Explicit lengths and field declarations prevent hallucination

  • Deterministic: Same input always produces same output

  • Streaming Friendly: Tabular form enables incremental parsing

Quick Example

JSON:

{
    "is_valid": false,
    "counts": {"ERROR": 2, "WARNING": 1},
    "issues": [
        {"code": "E_X", "severity": "ERROR", "message": "msg1"},
        {"code": "W_Y", "severity": "WARNING", "message": "msg2"}
    ]
}

TOON:

is_valid: false
counts: {ERROR: 2, WARNING: 1}
issues[2]{code,severity,message}:
E_X,ERROR,msg1
W_Y,WARNING,msg2

TOON Forms in Detail

Inline Form (Primitive Arrays)

Used for arrays of primitives (strings, numbers, booleans, null) when the array has 10 or fewer items and the total line width is under 120 characters.

# Inline array of numbers
values[4]: 1,2,3,4

# Inline array of strings
tags[3]: alpha,beta,gamma

# Inline array with null
optional[2]: value,null

# Inline array of booleans
flags[3]: true,false,true

List Form (Non-Uniform Arrays)

Used when array elements are not uniform objects or when inline form limits are exceeded. Each element prefixed with -.

mixed:
- 42
- "hello"
- true
- null

objects:
- name: Alice
  age: 30
- id: 123
  role: admin

Tabular Form (Uniform Object Arrays)

The most token-efficient form for arrays of objects with identical keys. Fields declared once in header, then one row per object.

users[3]{id,name,role}:
1,Alice,admin
2,Bob,user
3,Carol,user

# With nested objects in cells (inline encoding)
items[2]{id,metadata}:
1,{created:2024-01-01,active:true}
2,{created:2024-01-02,active:false}

Keyed Tabular Form (Uniform Object Maps)

Used for objects whose values are uniform objects. Each row includes its key.

users[2]{id,name}:
alice,1,Alice
bob,2,Bob

Object Form (Nested Objects)

Standard indentation-based key-value pairs for nested structures.

server:
  host: localhost
  port: 8080
  ssl: true
  routes[2]{path,handler}:
    /api/users,handle_users
    /api/health,handle_health

String Quoting Rules

Strings are quoted only when necessary:

  • Unquoted: Alphanumeric plus _, -, ., /, : (no spaces, no special chars)

  • Quoted: Contains spaces, ,, :, [, ], {, }, #, ', ", or starts with -

unquoted: hello_world
quoted: "hello, world"
quoted: "contains: colon"
quoted: "starts with -dash"

Escaping in quoted strings: \\ → \\\\, " → \", newline → \n.

Special Values

  • null — JSON null

  • true / false — Booleans (lowercase)

  • Numbers — Integers and floats as-is (NaN/Inf become null)

ValidationReport TOON Encoding

The encode_validation_report function produces optimized TOON for validation reports with a fixed field order and tabular issues array.

Field Order

  1. is_valid (boolean)

  2. counts (inline object: ERROR, WARNING, INFO)

  3. issues (tabular array with fixed column order)

  4. Any additional fields

Issues Column Order

code,message,severity,rule_id,ruleset_key,component_type,component_name,attribute,component_data,context,category,fix_suggestion,validation_layer,simulation_metadata

Example Output

is_valid: false
counts: {ERROR: 1, WARNING: 2, INFO: 0}
issues[3]{code,message,severity,rule_id,ruleset_key,component_type,component_name,attribute,component_data,context,category,fix_suggestion,validation_layer,simulation_metadata}:
E_NODE_ELEVATION_MISSING,"Junction does not define elevation",ERROR,rule_junction_has_elevation,epanet_core,WNTREPANETNode,J1,elevation,{node_type: Junction, name: J1, elevation: null, base_demand: 0}, {ruleset: epanet_core, component_subtype: Junction},Parameter,Provide the junction elevation in meters.,static,{}
W_NODE_COORDINATES_MISSING,"Missing coordinates, node will not be displayed",WARNING,warn_node_missing_coord,epanet_core,WNTREPANETNode,J1,coordinates,{node_type: Junction, name: J1, elevation: 100, base_demand: 0}, {ruleset: epanet_core, component_subtype: Junction},Topology,Assign X,Y coordinates to the node for map display.,static,{}
E_UNKNOWN_CURVE_REFERENCE,"Links reference undefined curves: P1 pump_curve_name <C9>",ERROR,rule_link_curves_exist,epanet_core,network,,pump_curve_name,{}, {ruleset: epanet_core},Network,Ensure all link curve references (pump_curve_name, efficiency_curve_name, headloss_curve_name) point to existing curves.,static,{}

Note that component_data and context are encoded as inline objects within their tabular cells.

CLI Usage

# TOON output for validation
epanetparser validate -f network.json --toon-output

# With custom ruleset
epanetparser validate -f network.json --toon-output --ruleset milp

# Mutually exclusive with --json-output and --pretty-output
# Only one output format may be specified

Python API

from epanetparser.core.toon import encode, encode_validation_report
from epanetparser.core.validation import ValidationReport

# Encode any JSON-compatible value
data = {"items": [{"id": 1, "name": "A"}, {"id": 2, "name": "B"}]}
toon_str = encode(data)

# Encode validation report with optimized structure
report = network.validate()
toon_str = report.as_toon()
# Equivalent to:
toon_str = encode_validation_report(report.as_dict())

TOONEncoder Options

The TOONEncoder class accepts configuration options:

from epanetparser.core.toon import TOONEncoder

encoder = TOONEncoder(
    indent="  ",           # Indentation string (default: two spaces)
    delimiter=",",         # Field delimiter: ",", "\t", or "|" (default: comma)
    max_inline_items=10,   # Max items for inline array form (default: 10)
    max_inline_width=120,  # Max character width for inline form (default: 120)
)
toon_str = encoder.encode(data)

Delimiter Options

The default delimiter is comma (,). For data containing commas, use tab (\t) or pipe (|):

encoder = TOONEncoder(delimiter="\t")
# Uses tabs for field separation

encoder = TOONEncoder(delimiter="|")
# Uses pipes for field separation

Implementation Notes

  • Tabular detection: An array uses tabular form if all elements are objects with identical keys. Empty arrays with key "issues" also use tabular form with predefined ValidationReport issue fields.

  • Empty arrays: Rendered as [] (inline) or key[0]{fields}: (tabular).

  • Nested structures: Objects and arrays within tabular cells are encoded inline using the same rules recursively.

  • Field ordering: encode_validation_report enforces a specific key order for deterministic output.

See Also