Data I/O & Formats

T’s data layer is built on Apache Arrow: DataFrames are backed by Arrow tables, and T provides fast native readers and writers for the most common tabular formats. This guide explains how to load and save data, which format to pick, and what to do when you need a format T does not read natively.

---

Choosing a Format

Format	Functions	Best For	Notes
CSV	read_csv() / write_csv()	Small-to-medium data, interchange with other tools	Human-readable; type inference; fast native path when using defaults
Parquet	read_parquet() / write_parquet()	Long-term storage, archival, large datasets, sharing with external analytics tools	Compressed, columnar, type-preserving; smaller on disk
Arrow IPC	read_ipc() / write_ipc()	Pipeline intermediates, cross-runtime exchange, fastest round trips	Zero-copy; no compression; also known as Feather v2
JSON	t_read_json() / t_write_json()	Config, nested lists and dicts	See the Serializers guide for pipeline usage

For a full decision walkthrough between Parquet and Arrow IPC, jump to Parquet vs Arrow IPC: How to Choose.

All three tabular readers (read_csv, read_parquet, read_ipc) accept a local path or a URL. For reproducible downloads inside pipelines, use fetchurl with a pinned SHA-256 hash instead — see Downloading Data from URLs.

---

Downloading Data from URLs

read_csv, read_parquet, and read_ipc all accept a URL in place of a local path. How the download is handled depends on where you are running T.

In the REPL or a script

The reader downloads the file to a temporary location, parses it, and removes the temporary file. This is convenient for exploration, but the contents of a URL can change between runs, so it is not reproducible.

df = read_csv("https://example.com/data.csv")

If you want to keep a copy on disk, use fetchurl, which downloads immediately via curl:

fetchurl("https://example.com/data.csv", output = "data.csv")

In a pipeline (reproducible downloads)

Pipeline builds run in the Nix sandbox, where downloads must be content-addressed. Use fetchurl to create a node backed by Nix’s builtins.fetchurl with a pinned sha256, so every build fetches exactly the same bytes:

p = pipeline {
  raw = fetchurl("https://example.com/data.csv", sha256 = "sha256-abc123...")
  data = read_csv(raw) |> mutate($x > 0)
}

To obtain the hash for a URL, use prefetch. It runs nix-prefetch-url, returns the SHA-256 hash, and seeds the file in the Nix store so the later fetchurl build finds it cached and does not re-download:

hash = prefetch("https://example.com/data.csv")
print(hash)  -- e.g. "sha256-abc123..."

p = pipeline {
  raw = fetchurl("https://example.com/data.csv", sha256 = hash)
  data = read_csv(raw) |> mutate($x > 0)
}

populate_pipeline(p, build = true)

-- The downloaded asset is available at p.raw.path after the build:
content = read_file(p.raw.path)

Notes on fetchurl nodes:

• fetchurl defaults to serializer = ^bin, so the node artifact is the raw downloaded bytes, ready for downstream T nodes to read.
• Pass serializer = ^text for plain-text assets (e.g. a JSON config file) that should be materialized as text.
• In pipeline mode, sha256 is required — omitting it raises a TypeError.

See Fetching Remote Assets in the Pipeline Tutorial and the fetchurl / prefetch reference pages for more details.

---

CSV

read_csv() reads a CSV file into a DataFrame. For standard files it uses a native Arrow CSV reader (GArrowCSVReader), which is significantly faster and more memory-efficient than a pure OCaml parser for large datasets.

When the Native Fast Path Is Used

The native Arrow reader is used automatically when you call read_csv() with its default parameters:

• Comma separator (,)
• No skipping of the header or lines
• No automatic column name cleaning

If you pass any non-default option (separator = ";", skip_lines = 5, clean_colnames = true, …), T falls back to a pure OCaml CSV parser. This keeps full compatibility with complex CSV layouts while providing maximum speed for standard ones.

df = read_csv("data.csv")                     -- native fast path
df = read_csv("data.csv", separator = ";")    -- OCaml parser
df = read_csv("https://example.com/data.csv") -- URL: see Downloading Data from URLs

NA Handling

The native reader recognizes the following values as NA:

• NA
• na
• N/A
• Empty fields

This keeps the native reader consistent with the OCaml fallback parser.

Fallback Behavior

If the native Arrow reader fails (e.g. malformed file or unsupported encoding), T falls back automatically:

1. Native attempt: read with the Arrow CSV reader.
2. Warning: a warning is printed to stderr.
3. OCaml fallback: the file is re-read with the pure OCaml parser.

Note that the fallback is more memory-intensive and may hit Out_of_memory errors for files larger than a few gigabytes on systems with limited RAM.

---

Parquet

Parquet is the recommended format for large-scale data in T because it is:

• Binary and columnar, with built-in compression.
• Type-preserving, avoiding the overhead of type inference.
• Fast to ingest, using a native parquet-glib reader that is quicker than CSV reading.
• Zero-copy loadable into memory for large datasets.

df = read_parquet("data.parquet")
write_parquet(df, "data.parquet")

---

Arrow IPC (Feather v2)

Arrow IPC files (also known as Feather v2) give the fastest read/write round trip and preserve the native Arrow storage model exactly. They are the ideal choice for passing large DataFrames between runtimes.

df = read_ipc("data.arrow")
write_ipc(df, "data.arrow")

The .arrow and .feather extensions are interchangeable — both denote the same Arrow IPC (Feather v2) byte format, and read_ipc/write_ipc accept any filename regardless of extension. Use whichever reads more clearly for your artifact.

Within a pipeline, DataFrames passed between T, R, Python, and Julia nodes use Arrow IPC through the ^ipc serializer — see the Serializers guide and the Pipeline Tutorial for details.

---

Parquet vs Arrow IPC: How to Choose

Parquet and Arrow IPC are both columnar, type-preserving, Arrow-native binary formats. Every T runtime (T, R, Python, Julia) can read and write both, and neither requires schema inference on load. The choice between them comes down to one question: is this data a live hand-off, or a durable artifact?

• Arrow IPC serializes Arrow’s in-memory columnar representation directly to disk. There is no extra encoding or compression step, so it is the fastest format to write and read by a wide margin. In exchange, files are uncompressed by default and often several times larger than Parquet.
• Parquet is a storage-optimized layout. It applies block compression (snappy by default) and organizes data into row groups with column-level metadata, so files are smaller and readers can skip columns they do not need. Compression costs time at write time, but pays off at rest and for scans.

Side-by-side comparison

	Arrow IPC (^ipc)	Parquet (^parquet)
File extension	.arrow (also .feather)	.parquet
Compression	None by default	Snappy by default
Relative file size	Larger (uncompressed)	Typically 4–10× smaller for numeric data
Write speed	Fastest (memory dump)	Slower (compression + encoding)
Read speed	Fastest	Very fast; can skip whole columns
Schema / type fidelity	Exact	Exact
Zero-copy / memory-map friendly	Yes	No (must decode row groups)
Cross-runtime support (T/R/Python/Julia)	Symmetric	Symmetric
Column pruning on read	No	Yes (column metadata + row groups)
External tool ecosystem	Limited (Feather readers)	Broad (Spark, DuckDB, dask, pandas, Athena/BigQuery, …)
Typical lifetime	Minutes to hours	Days to years

Choose Arrow IPC when

• The data is moving between nodes. Within a pipeline, use ^ipc to pass DataFrames between T, R, Python, and Julia nodes — it is T’s default choice for exactly this reason.
• You want the fastest possible round trip. If a node reads, transforms, and writes data as an intermediate step, IPC avoids paying compression cost twice.
• The artifact is short-lived or disposable — a staging table, a scratch output, a cache.
• You are memory-mapping large data and want to avoid decoding overhead.

Choose Parquet when

• You are persisting a result. Write final outputs, snapshots, and “published” datasets to Parquet so they stay small and readable for years.
• Disk space or transfer cost matters. Parquet’s compression shrinks files dramatically, which also speeds up copying and uploads.
• You are sharing data with the wider analytics ecosystem. Spark, DuckDB, dask, pandas, BigQuery, and Athena all read Parquet natively; IPC is mostly confined to the Arrow ecosystem.
• Downstream readers benefit from column pruning. Analytics workloads that read a few columns from a wide table are much faster with Parquet.
• You plan to append or scan subsets repeatedly.

Decision flow

Is the data only needed while the pipeline runs?
├─ Yes  → use ^ipc (fastest, no compression)
└─ No (you keep or share it) →
     Is it a durable artifact / shared with external tools?
     ├─ Yes → use ^parquet (compressed, columnar, interoperable)
     └─ No  → use ^ipc (speed) or ^parquet (smaller files) as you prefer

Rule of thumb

Use ^ipc/read_ipc/write_ipc for anything that moves through a pipeline, and ^parquet/read_parquet/write_parquet for anything you keep, ship, or store. If you are unsure and the file is small, either works — the formats are interchangeable at the API level (read_parquet/write_parquet and read_ipc/write_ipc are drop-in replacements for each other).

[!TIP] In a pipeline you can use both freely: pass data between nodes with ^ipc for speed, and add a final node that materializes the result to Parquet with write_parquet() for storage. This gives you fast intermediates and small, shareable outputs.

---

Formats T Does Not Read Natively

T does not ship parsers for proprietary office or statistical formats such as Excel (.xlsx/.xls), SPSS (.sav), SAS, Stata (.dta), or similar. Do not look for a read_excel() in T — it does not exist.

For these formats, use R or Python inside a pipeline node and let T orchestrate the conversion:

1. Declare the package you need in tproject.toml — e.g. readxl or openxlsx (R), pandas or polars (Python) — under [r-dependencies] or [py-dependencies], then run t update. See the First Pipeline guide for declaring dependencies.
2. Read the file in a node and hand the result onward via a serializer:

p = pipeline {
  raw = rn(
    command = <{
      library(readxl)
      read_excel("data.xlsx")
    }>,
    serializer = ^ipc
  )

  clean = node(
    command = raw |> filter($amount > 0),
    deserializer = ^ipc
  )
}

3. Save from a node the same way — write the file with the package of your choice inside an R or Python node, or convert with write_csv()/write_parquet() in T for downstream consumption.

This keeps T’s standard library small and focused while giving you access to the full ecosystem of R and Python readers, writers, and converters.

---

Recommendations for Large Data

For datasets exceeding 2–3 GB:

1. Prefer Parquet for storage: convert CSVs to Parquet before reading them into T (e.g. with R’s arrow::write_parquet() or Python’s pandas.DataFrame.to_parquet()). See Parquet vs Arrow IPC: How to Choose for when to reach for IPC instead.
2. Use standard CSVs: if you must use CSV, stick to the default comma separator and no leading comment lines to stay on the native fast path.
3. Mind memory limits: the pure OCaml fallback path is bounded by OCaml’s heap and string-size limits — still large on 64-bit systems, but less efficient than Arrow’s memory mapping.

---

Related Guides

• Performance — how the Arrow backend works, vectorization, and the native vs. fallback paths
• Serializers in T — ^ipc, ^csv, ^json, and custom serializers for pipelines
• Pipeline Materialization — how node artifacts are stored and read back
• Function Reference — read_csv, write_csv, read_parquet, write_parquet, read_ipc, write_ipc