Interactive Pipeline Debugging in T-Lang

T-Lang provides a unified, powerful interactive debugger designed to solve the challenges of debugging polyglot pipeline nodes executing in sandboxed environments (like Nix derivations).

When a pipeline node fails or behaves unexpectedly, you can spin up an interactive subshell (or language REPL) pre-configured with all upstream dependencies bound as environment variables. This allows you to inspect variables, dry-run functions, and step through your code line-by-line using real execution data.

The Challenge

In data pipelines, code blocks in guest runtimes (like Python, R, or Julia) run inside isolated sandboxes. Upstream dependencies are materialized in the read-only /nix/store and passed downstream.

If a python node like data_cleanup crashes: 1. Nix stdout/stderr logs might only show a generic stack trace. 2. The exact variables and raw files generated by upstream steps are locked in the store. 3. Writing mock inputs locally is tedious and error-prone.

The Solution: t debug

The T-Lang debugger intercepts the targeted node, resolves all of its upstream dependencies, locates their latest materialized /nix/store paths, sets up the environment variables, and launches the runtime’s interactive REPL directly.

You can trigger the debugger in two ways: 1. From the Terminal (CLI): Use t debug <node> 2. From the T REPL (Interactive): Use debug_node(p.node_name)

Debugging Dependencies Live in tproject.toml

Debugging a node uses the project environment declared in tproject.toml. If a runtime package needed to deserialize an upstream artifact is missing, the debugger will start in that runtime without the package it needs.

Use t doctor to spot missing runtime packages and add them to the relevant dependency section in tproject.toml:

After updating tproject.toml, run t update and re-enter nix develop so the debug environment is rebuilt.

Do not install these packages manually inside the language package manager if you want reproducible debugging. Add them to tproject.toml instead.

Common examples:

1. Using the Command Line Interface (CLI)

To debug a node directly from your shell, run:

t debug <node_name>

By default, this looks for the pipeline defined in src/pipeline.t and targets the specified node.

If your pipeline is defined in a custom script, specify the file path first:

t debug src/my_custom_pipeline.t data_cleanup

What Happens Next?

T-Lang will evaluate your script up to the requested node, gather all upstream build paths, set up the environment, and drop you into the corresponding REPL. For example, targeting a Python node will output:

==================================================
Debugging Node: Y (Runtime: Python)
==================================================
Environment variables set for dependencies:
  - dataset_np = /nix/store/5fcfj6wfh...-pipeline_output/dataset_np

Starting interactive Python REPL...
Tip: Load upstream dependencies in Python using:
  import tlang
  dataset_np = tlang.read_node("dataset_np")
Press Ctrl+D or exit to return to T REPL.
==================================================
>>>

2. Using the T REPL

If you are already inside an active interactive T-Lang REPL session, you can debug any node in your pipeline using the debug_node function:

# Assume p is your built pipeline
debug_node(p.data_cleanup)

This starts the same subshell environment. Once you exit the subshell (using Ctrl+D, exit(), or q()), control is cleanly returned back to your T REPL session.

How It Works (Under the Hood)

Pristine Environments & Custom Node Variables

To keep the subshell clean and avoid environment pollution, T-Lang does not inject Nix store dependency paths directly as environment variables in the spawned subprocess. Instead, it prints them clearly under the Upstream dependencies header on startup, along with language-specific companion package loading tips.

However, if you have configured custom node-specific environment variables in your pipeline (via p_env_vars / un_env_vars), T-Lang programmatically extracts and propagates these variables directly into the debugger subshell process environment.

Custom Prompts and R Quiet Mode

The interactive subshells automatically apply customized prompt configurations so you always know you are in a debugger session:

Runtime Safety Validation Guard

T-Lang interactive debugging is supported exclusively for interactive REPL-capable runtimes (Python, R, and Julia). Attempting to debug any other runtime (such as Quarto or Bash) will be intercepted immediately and will raise a descriptive ValueError instead of dropping you into a raw shell or crashing:

T> debug_node(p.report)
Error(ValueError: "[L1:C1] debug_node: only R, Python, and Julia nodes are supported for interactive debugging. Node 'report' has unsupported runtime 'Quarto'.")

Step-by-Step Debugging Walkthrough

Let’s say your Python node fails because of a column type mismatch.

  1. Launch the Debugger:

    t debug process_features

  2. The Python REPL Opens: The py> prompt is shown, and all upstream dependencies and their paths are listed.

  3. Inspect Inputs:

    py> import tlang
py> df = tlang.read_node("raw_df")
py> df.dtypes

  4. Locate the Bug: You realize a column is parsed as a string instead of a float.

  5. Exit the Debugger: Press Ctrl+D. You are returned back to your T REPL session, ready to fix the source code!

Reference Table: Runtime Subshells

Node Runtime Spawned Subshell Prompt Loading Command
Python python -i py> import tlang; dep = tlang.read_node(\"dep\")
R R --no-save --quiet r> library(tlang); dep <- read_node(\"dep\")
Julia julia -i -e 'include(\"/path/to/.t_debug_startup.jl\")' jl> using tlang; dep = read_node(\"dep\")