# API Reference Package-oriented guide to T's standard library. > **Coverage**: Package overview and worked examples > **Exhaustive per-function reference**: [`docs/reference/index.md`](reference/index.md), generated from source docstrings > **Auto-loaded**: All standard packages are automatically available in every T session --- ## Table of Contents - [Shell Interaction](#shell-interaction) — Executing commands from T - [Core Package](#core-package) — Basic functional utilities - [Base Package](#base-package) — Errors, NA, assertions - [Math Package](#math-package) — Mathematical functions - [Stats Package](#stats-package) — Statistical functions - [DataFrame Package](#dataframe-package) — CSV I/O and DataFrame operations - [Colcraft Package](#colcraft-package) — Data manipulation verbs and window functions - [Chrono Package](#chrono-package) — High-performance date and time manipulation - [Strcraft Package](#strcraft-package) — Modern string manipulation - [Lens Package](#lens-package) — Composable access and update lenses - [Pipeline Package](#pipeline-package) — Pipeline introspection - [Explain Package](#explain-package) — Introspection and debugging tools - [Testcraft Package](#testcraft-package) — Unit-testing primitives - [Propcraft Package](#propcraft-package) — Property-based testing primitives --- For generated one-page documentation for every exported function, including newer Chrono, string, join, to_factor, and helper APIs, use the [Function Reference](reference/index.md). --- ## Core Syntax: Lists, Dictionaries, and Blocks These forms are **distinct and non-overlapping**: - `[a, b, c]` → List literal - `[key: value, ...]` → Dictionary literal - `{ ... }` → Block statement (control-flow/body syntax), **not** a dictionary ### Bracket literal rule (`[...]`) When parsing a bracket literal, T applies this rule: 1. Parse comma-separated top-level items. 2. If **any** top-level item is `key: value`, the whole literal is treated as a dictionary. 3. Otherwise, it is treated as a list. This means: ```t [] -- empty List [:] -- empty Dict [1, 2, 3] -- List [name: "alice"] -- Dict [name: "alice", age: 32] -- Dict ``` ### Disallowed mixed forms A single bracket literal cannot mix dictionary entries and plain expressions: ```t [name: "alice", 12] -- Parse error [name:] -- Parse error ``` ### Braces are blocks `{ ... }` is reserved for block syntax (e.g., in control flow and pipeline/intent constructs). It is not used for general dictionary literals. An empty brace block `{}` parses as an empty block (`Block []`) and evaluates to `NA` at runtime. Braces are never used for dictionary literals; dictionaries always use the bracket (`[...]`) syntax described above. --- ## Shell Interaction ### Shell Escape (`?<{ ... }>`) T provides first-class support for executing shell commands using the `?<{ }>` syntax. - **As a Statement**: Prints output directly to `stdout`. - **As an Expression**: Captures `stdout` as a `String`. - **Error Handling**: Non-zero exit codes produce a `ShellError` containing `stderr`. - **Working Directory**: The `cd` command is special-cased to change the interpreter's working directory. **Examples:** ```t ?<{ls -la}> -- Prints directory listing files = ?<{ls}> -- Captures filenames in a string ?<{cd /tmp}> -- Changes working directory to /tmp ``` --- ## Core Package Fundamental functional programming utilities. ### `print(value)` Print a value to standard output. **Parameters:** - `value` — Any value to print **Returns:** The printed value (for chaining) **Examples:** ```t print(42) -- 42 print("Hello, T!") -- Hello, T! print([1, 2, 3]) -- [1, 2, 3] x = 10 |> print |> \(v) v * 2 -- Prints 10, returns 20 ``` --- ### `pretty_print(value)` Pretty-print a value with detailed formatting (for DataFrames, structures, etc.). **Parameters:** - `value` — Any value **Returns:** The value (for chaining) **Examples:** ```t pretty_print(df) -- Formatted DataFrame output ``` --- ### `type(value)` Get the type name of a value as a string. **Parameters:** - `value` — Any value **Returns:** `String` — Type name **Examples:** ```t type(42) -- "Int" type(3.14) -- "Float" type(true) -- "Bool" type("hello") -- "String" type([1, 2]) -- "List" type([x: 1]) -- "Dict" type(NA) -- "NA" type(error("x")) -- "Error" type(df) -- "DataFrame" ``` --- ### `to_integer(value)` Convert a value to an integer robustly. Handles strings with spaces, percentages, commas, and recognizes 'TRUE'/'FALSE'. Also propagates vectorization over Collections. **Parameters:** - `value` — Any value (String, Bool, Float, List, Vector) **Returns:** `Int`, `NA`, or a Collection of `Int`/`NA` **Examples:** ```t to_integer("12 300") -- 12300 to_integer("TRUE") -- 1 to_integer("FALSE") -- 0 to_integer("15%") -- 15 to_integer("3,14") -- 3 to_integer(3.14) -- 3 to_integer("hello") -- NA(Int) to_integer(["1", "2"]) -- [1, 2] ``` --- ### `to_float(value)` / `to_float(value)` Convert a value to a float robustly. `to_float` is an alias for `to_float`. Handles strings with spaces, percentages, commas, and recognizes 'TRUE'/'FALSE'. Also propagates vectorization over Collections. **Parameters:** - `value` — Any value (String, Bool, Int, List, Vector) **Returns:** `Float`, `NA`, or a Collection of `Float`/`NA` **Examples:** ```t to_float("3,14") -- 3.14 to_float("15%") -- 15.0 to_float(" 1 200.5 ") -- 1200.5 to_float("TRUE") -- 1.0 to_float("F") -- 0.0 to_float(42) -- 42.0 to_float("hello") -- NA(Float) to_float(["1", "2"]) -- [1.0, 2.0] ``` --- ### `to_symbol(value)` Convert a string name into a `Symbol` so it can be injected into quoted code with `!!`. Existing symbols pass through unchanged. **Parameters:** - `value` — String or Symbol **Returns:** `Symbol` **Examples:** ```t to_symbol("mpg") -- mpg to_expr(select(df, !!to_symbol("mpg"))) -- to_expr(select(df, mpg)) name = "result" to_expr(f(!!to_symbol(name) := 42)) -- to_expr(f(result = 42)) ``` --- ### `args(fn)` Returns a dictionary of parameter names and their expected types for a function. --- ### `is_error(value)` Returns `true` if the value is an Error object. --- ### `get(target, selector = NA, default = NA)` Unified retrieval for variables, collection elements, pipeline nodes, or lens focuses. When called with a single string or symbol argument inside an NSE data verb (`mutate`, `filter`, `summarize`, …), `get()` checks the **data mask** first: if the name matches a column in the current row or DataFrame, the column value is returned; otherwise it falls back to the global environment. **Examples:** ```t get("salary") -- variable lookup get(list, 0) -- indexing get(df, col_lens("mpg")) -- lens focus get(val, 0) -- fallback if val is NA/Error -- Data-mask aware: column lookup inside mutate x = 42 df = dataframe(a = [1, 2, 3]) df |> mutate(b = \(row) get("a")) -- column "a" from data mask df |> mutate(c = \(row) get("x")) -- "x" not a column, falls back to global ``` --- ### `ifelse(condition, true_val, false_val, missing = NA, out_type = NA)` Vectorized conditional selection. --- ### `case_when(...formulas, .default = NA)` Vectorized multi-condition switch. Uses `condition ~ value` formulas. --- ### `identical(a, b)` Deep equality check. Works for collections and complex objects. --- ### `node_when(condition, value)` Static conditional for pipeline nodes. Evaluated at pipeline construction time. Returns `value` if `condition` is truthy, otherwise excludes the node from the DAG. `node_when` is only meaningful as the direct value of a node binding inside a `pipeline { }` block. Using the result outside that context (arithmetic, `is_na()`, etc.) is unsupported. ### `node_fork(...condition_value_pairs, .default = ...)` Static multi-way branch for pipeline nodes. Returns the value for the first truthy condition-value pair. If no condition matches and `.default` is provided, that value is included in the pipeline; if `.default` is omitted the node is excluded from the DAG entirely (null marker behaviour, not `NA`). `node_fork` is only meaningful as the direct value of a node binding inside a `pipeline { }` block. Using the result outside that context is unsupported. --- ### `eval(expr)` / `to_expr(x)` / `to_exprs(...)` ### `quo(x)` / `quos(...)` / `enquo(p)` / `enquos(...)` Metaprogramming and quotation utilities. --- ### `body(fn)` / `source(fn)` Inspect function implementation. --- ### `run(cmd)` Execute a shell command and return its stdout as a string. --- ### `cat(...values, sep = " ", file = NA, append = false)` Print values to stdout or a file without a trailing newline (unless specified). --- ### `getwd()` / `exit(code = 0)` Environment and process control. --- ### `file_exists(path)` / `dir_exists(path)` / `list_files(path, pattern = NA)` ### `read_file(path)` / `read_lines(path)` File system introspection and reading. --- ### `path_join(...)` / `path_abs(path)` ### `path_basename(path)` / `path_dirname(path)` ### `path_ext(path)` / `path_stem(path)` Cross-platform path manipulation. --- ### `show_plot(plot)` Display a built or unbuilt R/Python/Julia plot node (depends on the environment's plot viewer). --- ### `env()` Returns a list of all variable names currently in the environment. --- ### `length(collection)` Get the number of elements in a collection. **Parameters:** - `collection` — List, Vector, or String **Returns:** `Int` — Number of elements **Examples:** ```t length([1, 2, 3]) -- 3 length("hello") -- 5 length([]) -- 0 ``` --- ### `head(collection, n)` Get the first element(s) of a collection. For DataFrames, returns the first `n` rows (default 5). For Lists, returns the first element. **Parameters:** - `collection` — List or DataFrame - `n` (optional) — Number of rows for DataFrames (default: 5); not used for Lists **Returns:** Single element (for Lists) or DataFrame (for DataFrames) **Examples:** ```t head([1, 2, 3, 4, 5]) -- 1 head(df) -- first 5 rows head(df, 3) -- first 3 rows head(df, n = 10) -- first 10 rows ``` --- ### `tail(collection, n)` For DataFrames, returns the last `n` rows (default 5). For Lists, returns all elements except the first. **Parameters:** - `collection` — List or DataFrame - `n` (optional) — Number of rows for DataFrames (default: 5); not used for Lists **Returns:** List (for Lists) or DataFrame (for DataFrames) **Examples:** ```t tail([1, 2, 3, 4, 5]) -- [2, 3, 4, 5] tail(df) -- last 5 rows tail(df, 3) -- last 3 rows tail(df, n = 10) -- last 10 rows ``` --- ### `map(collection, fn)` Apply a function to each element of a collection. **Parameters:** - `collection` — List or Vector - `fn` — Function to apply: `\(x) ...` **Returns:** List (or Vector) of results **Examples:** ```t map([1, 2, 3], \(x) x * x) -- [1, 4, 9] map(["a", "b"], \(s) s + "!") -- ["a!", "b!"] map([1, 2, 3], \(x) x + 10) -- [11, 12, 13] ``` --- ### `filter(collection, predicate)` Keep only elements that satisfy a predicate. **Parameters:** - `collection` — List or Vector - `predicate` — Function returning Bool: `\(x) ...` **Returns:** List (or Vector) of matching elements **Examples:** ```t filter([1, 2, 3, 4, 5], \(x) x > 3) -- [4, 5] filter([1, 2, 3], \(x) x % 2 == 0) -- [2] filter(["a", "ab", "abc"], \(s) length(s) > 1) -- ["ab", "abc"] ``` --- ### `sum(collection)` Sum all numeric elements. **Parameters:** - `collection` — List or Vector of numbers **Returns:** `Int` or `Float` — Sum **Examples:** ```t sum([1, 2, 3, 4, 5]) -- 15 sum([1.5, 2.5, 3.0]) -- 7.0 sum([]) -- 0 ``` --- ### `seq(start, end, step = 1)` Generate a sequence of numbers. **Parameters:** - `start` — Starting value - `end` — Ending value (inclusive) - `step` (optional) — Increment (default: 1) **Returns:** List of numbers **Examples:** ```t seq(1, 5) -- [1, 2, 3, 4, 5] seq(0, 10, 2) -- [0, 2, 4, 6, 8, 10] seq(5, 1, -1) -- [5, 4, 3, 2, 1] ``` --- ### `float_seq(start, end, n = 100)` Generate a sequence of evenly-spaced floats. **Parameters:** - `start` — Starting value (Float or Int) - `end` — Ending value (Float or Int) - `n` (optional) — Number of values (default: 100) **Returns:** List of evenly-spaced floats. **Examples:** ```t float_seq(0, 1, 5) -- [0.0, 0.25, 0.5, 0.75, 1.0] float_seq(start = 0, end = 1, n = 5) ``` --- **Parameters:** - `value` — Any value **Returns:** `Bool` — true if value is an Error **Examples:** ```t is_error(42) -- false is_error(error("msg")) -- true is_error(1 / 0) -- true ``` --- ### `getwd()` Returns the current working directory of the T interpreter. **Returns:** `String` — Working directory path --- ### `file_exists(path)` Check if a regular file exists at the given path. Returns `false` for directories. **Parameters:** - `path` — File path (String or Symbol) **Returns:** `Bool` --- ### `dir_exists(path)` Check if a directory exists at the given path. **Parameters:** - `path` — Directory path (String or Symbol) **Returns:** `Bool` --- ### `read_file(path)` Read the entire contents of a file as a string. **Parameters:** - `path` — File path (String or Symbol) **Returns:** `String` or `Error(FileError)` --- ### `list_files(path, pattern)` List files and directories in a given path. **Parameters:** - `path` (optional) — Directory to list (default: ".") - `pattern` (optional) — Regex pattern to filter filenames **Returns:** `List[String]` or `Error(FileError)` --- ### `env(name)` Get the value of an environment variable. **Parameters:** - `name` — Environment variable name (String or Symbol) **Returns:** `String` or `NA` if not found --- ### `exit(code)` Exits the T interpreter. **Parameters:** - `code` (optional) — Exit code integer (default: 0) --- ### `path_join(...)` Join multiple path segments using the system-specific separator. **Parameters:** - `...` — One or more path segments (String or Symbol) **Returns:** `String` --- ### `path_basename(path)` Get the filename/last component of a path. **Parameters:** - `path` — Path string **Returns:** `String` --- ### `path_dirname(path)` Get the directory portion of a path. **Parameters:** - `path` — Path string **Returns:** `String` --- ### `path_ext(path)` Get the file extension (including the dot). Returns `NA` if no extension is found. **Parameters:** - `path` — Path string **Returns:** `String` or `NA` --- ### `path_stem(path)` Get the filename without its extension. **Parameters:** - `path` — Path string **Returns:** `String` --- ### `path_abs(path)` Resolves a relative path to an absolute path against the current working directory. **Parameters:** - `path` — Path string **Returns:** `String` --- ### `rm(...)` Remove one or more variables from the environment by name. Supports bare symbols (R-style selective removal), strings, and lists of names via the `list` parameter. **Parameters:** - `...` — One or more variable symbols or strings identifying variables to remove. - `list` (optional) — A List of strings or symbols to remove. **Returns:** `NA` **Examples:** ```t x = 10; y = 20 rm(x, y) -- Removes x and y z = 30 rm("z") -- Removes z vars = ["a", "b"] rm(list = vars) -- Removes variables 'a' and 'b' ``` --- ## Base Package Error handling, NA values, and assertions. ### `error(message)` / `error(code, message)` Create an error value. **Parameters:** - `message_or_code` — Error message (if 1 arg) or error code (if 2 args) - `message` (optional) — Error message (if 2 args) **Returns:** `Error` value **Examples:** ```t error("Something went wrong") error("ValueError", "Invalid input") e = error("custom error") error_msg(e) -- "custom error" ``` --- ### `error_code(err)` Get the error code from an Error value. **Parameters:** - `err` — Error value **Returns:** `String` — Error code **Examples:** ```t e = 1 / 0 error_code(e) -- "DivisionByZero" e2 = error("TypeError", "msg") error_code(e2) -- "TypeError" ``` --- ### `error_msg(err)` Get the error message from an Error value. **Parameters:** - `err` — Error value **Returns:** `String` — Error message **Examples:** ```t e = error("Something broke") error_msg(e) -- "Something broke" e2 = 1 / 0 error_msg(e2) -- "Division by zero" ``` --- ### `warning_msg(node)` Get the warning message from a completed computed node (if any exists). Downstream nodes automatically inherit warnings from ancestor nodes, with each upstream warning prefixed by its source node name for clear provenance. **Parameters:** - `node` — ComputedNode value **Returns:** `String` — Warning message, or an empty string `""` if there are no warnings. Upstream warnings are prefixed with `"Ancestor node '' reported following warning: "`. Multiple warnings are joined with `". Furthermore, "`. **Examples:** ```t p = pipeline { a = suppress_warnings(node()) } build_pipeline(p) warning_msg(p.a) -- Returns warning message string or "" ``` --- ### `error_context(err)` Get additional context from an Error value (if available). **Parameters:** - `err` — Error value **Returns:** `Dict` — A dictionary of related context data. **Examples:** ```t error_context(e) -- Additional debugging information ``` --- ### `error_chain(err1, err2)` Explicitly chains two Error values together to preserve their provenance. This sets `err2` as the underlying cause in `err1`'s context. **Parameters:** - `err1` — The primary or outer Error value. - `err2` — The underlying cause Error value. **Returns:** `Error` — The chained Error value. **Examples:** ```t err1 = error("Primary calculation failed") err2 = error("KeyError", "Missing key 'x'") chained = error_chain(err1, err2) error_context(chained)$cause -- Returns err2 ``` --- ### `set_seed(seed)` Initializes the global random number generator with a given integer seed, enabling reproducible random draws from `sample()` and `slice_sample()`. **Parameters:** - `seed` — Integer seed value. **Returns:** `NA` **Examples:** ```t set_seed(42) sample([1, 2, 3, 4, 5], n = 3) ``` --- ### `with_seed(seed, thunk)` Runs a one-parameter lambda (the argument is ignored) with the global random number generator scoped to `seed`, then restores the previous RNG state. Outer random draws are unaffected, making this the natural way to scope determinism to a single expression — e.g. a reproducible `prop_for_all` run or a reproducible sample. **Parameters:** - `seed` — Integer seed value to scope the RNG to. - `thunk` — One-parameter lambda evaluated under `seed`. The argument is ignored; the lambda form makes the body lazy. **Returns:** The result of evaluating `thunk` (any type). **Examples:** ```t with_seed(42, \(u) sample([1, 2, 3, 4, 5], n = 3)) with_seed(42, \(u) prop_for_all(prop_gen_int_range(0, 100), \(x) x >= 0)) ``` **Notes:** - `with_seed` is exception-safe: if the thunk raises, the RNG state is still restored. - Nesting works: `with_seed(1, \(u) with_seed(2, \(v) ...))` restores the outer seed after the inner scope. --- ### `sample(x, n = 1, replace = false)` Draw a random sample of size n from a Vector or List, with or without replacement. **Parameters:** - `x` — `Vector` or `List` of values to sample from. - `n` (optional) — Number of elements to draw. Default `1`. - `replace` (optional) — Whether to sample with replacement. Default `false`. **Returns:** `Vector` or `List` (matching input type) **Examples:** ```t sample([10, 20, 30, 40, 50], n = 2) sample([1, 2, 3], n = 5, replace = true) ``` --- ### `assert(condition)` / `assert(condition, message)` Assert that a condition is true; error if false. **Parameters:** - `condition` — Boolean expression - `message` (optional) — Custom error message **Returns:** `true` if condition holds **Examples:** ```t assert(2 + 2 == 4) -- true assert(1 > 2) -- Error(AssertionError) assert(false, "Custom message") -- Error(AssertionError: Custom message) ``` --- ### `assert_file_exists(path)` / `assert_file_exists(path, message)` Assert that a regular file exists at `path`. **Parameters:** - `path` — File path - `message` (optional) — Custom assertion message **Returns:** `true` if the file exists **Examples:** ```t assert_file_exists("output.csv") assert_file_exists("report.html", "report generation failed") ``` --- ### `assert_dir_exists(path)` / `assert_dir_exists(path, message)` Assert that a directory exists at `path`. **Parameters:** - `path` — Directory path - `message` (optional) — Custom assertion message **Returns:** `true` if the directory exists **Examples:** ```t assert_dir_exists("results") assert_dir_exists("artifacts", "artifact directory was not created") ``` --- ### `assert_size_of_file(path, size)` / `assert_size_of_file(path, size, message)` Assert that a regular file exists and has the expected size in bytes. **Parameters:** - `path` — File path - `size` — Expected size in bytes - `message` (optional) — Custom assertion message **Returns:** `true` if the file exists and matches the expected size **Examples:** ```t assert_size_of_file("output.csv", 128) assert_size_of_file("report.html", 0, "report should be empty") ``` --- ### `assert_non_empty_file(path)` / `assert_non_empty_file(path, message)` Assert that a regular file exists and contains at least one byte. **Parameters:** - `path` — File path - `message` (optional) — Custom assertion message **Returns:** `true` if the file exists and is non-empty **Examples:** ```t assert_non_empty_file("output.csv") assert_non_empty_file("plot.png", "plot was not written") ``` --- ### `NA` Untyped missing value constant. **Examples:** ```t x = NA is_na(x) -- true ``` --- ### `na_int()` / `na_float()` / `na_bool()` / `na_string()` Create typed NA values. **Returns:** Typed NA value **Examples:** ```t na_int() -- NA(Int) na_float() -- NA(Float) na_bool() -- NA(Bool) na_string() -- NA(String) ``` --- ### `is_na(value)` Check if a value is NA. **Parameters:** - `value` — Any value **Returns:** `Bool` — true if value is NA **Examples:** ```t is_na(NA) -- true is_na(na_int()) -- true is_na(42) -- false is_na("hello") -- false ``` --- ### `serialize(value, path)` Serializes a value to a `.tobj` file. **Parameters:** - `value` — Any value to serialize - `path` — Output file path (String) **Returns:** `NA` **Seealso:** `deserialize` --- ### `deserialize(path)` Deserializes a value from a `.tobj` file. **Parameters:** - `path` — Input file path (String) **Returns:** `Any` — The deserialized value **Seealso:** `serialize` --- ### `t_write_json(value, path)` Serializes a T value to a JSON file. This is used as the universal baseline for object transport between runtimes. **Parameters:** - `value` — Any value to serialize - `path` — Path to the destination file (String) **Returns:** `NA` --- ### `t_read_json(path)` Deserializes a T value from a JSON file. Automatically handles type conversion for scalars, lists, and dictionaries. **Parameters:** - `path` — Path to the JSON file (String) **Returns:** `Any` — The deserialized value --- ### `fetchurl(url, sha256?, output?, dest?)` Downloads a file from a URL. In the REPL, wraps curl for immediate download. Inside a pipeline, creates a node that uses Nix's `builtins.fetchurl` to fetch the asset into the Nix store, making it available downstream. **Parameters:** - `url` — The URL to download (String) - `sha256` (optional) — Expected SHA-256 hash (String). Required in pipeline mode. - `output` (optional) — Output file path for REPL mode (String). Defaults to basename of URL. - `dest` (optional) — Output directory for REPL mode (String). Defaults to current directory. **Returns:** `String` (REPL mode) — The path to the downloaded file. `Node` (pipeline mode) — A pipeline node configured to fetch the URL via Nix. **Examples:** ```t -- REPL mode: download to current directory data = fetchurl("https://example.com/data.csv", output = "data.csv") -- Pipeline mode: fetch via Nix builtins.fetchurl p = pipeline { raw = fetchurl("https://example.com/data.csv", sha256 = "abc123..."); result = read_csv(raw) |> mutate(...); } build_pipeline(p) ``` --- ### `prefetch(url)` Downloads a URL and computes its SHA-256 hash. Useful for obtaining the hash needed by `fetchurl` in pipeline mode. **Parameters:** - `url` — The URL to prefetch (String) **Returns:** `String` — The SHA-256 hex digest of the downloaded content. **Examples:** ```t hash = prefetch("https://example.com/data.csv") print(hash) -- e.g. "abc123..." ``` --- ## Math Package Mathematical functions operating on scalars and vectors. Most functions are vectorized over Collections. ### `sqrt(x)` Square root. **Parameters:** - `x` — Number (Int or Float) **Returns:** `Float` **Examples:** ```t sqrt(4) -- 2.0 sqrt(2) -- 1.41421356237 sqrt(0) -- 0.0 sqrt(-1) -- Error (negative input) ``` --- ### `abs(x)` Absolute value. **Parameters:** - `x` — Number **Returns:** Same type as input **Examples:** ```t abs(-5) -- 5 abs(3.14) -- 3.14 abs(0) -- 0 ``` --- ### `log(x)` / `log10(x)` / `log2(x)` Logarithm functions. `log` is natural logarithm (base e). **Parameters:** - `x` — Number (must be positive) **Returns:** `Float` **Examples:** ```t log(10) -- 2.30258509299 log10(100) -- 2.0 log2(1024) -- 10.0 ``` --- ### `exp(x)` Exponential function (e^x). **Parameters:** - `x` — Number **Returns:** `Float` **Examples:** ```t exp(0) -- 1.0 exp(1) -- 2.71828182846 ``` --- ### `pow(base, exponent)` Power function (base^exponent). **Parameters:** - `base` — Number - `exponent` — Number **Returns:** `Float` **Examples:** ```t pow(2, 10) -- 1024.0 pow(9, 0.5) -- 3.0 ``` --- ### `sin(x)` / `cos(x)` / `tan(x)` Standard trigonometric functions (input in radians). --- ### `asin(x)` / `acos(x)` / `atan(x)` / `atan2(y, x)` Inverse trigonometric functions. `atan2` returns the angle whose tangent is y/x. --- ### `sinh(x)` / `cosh(x)` / `tanh(x)` ### `asinh(x)` / `acosh(x)` / `atanh(x)` Hyperbolic and inverse hyperbolic functions. --- ### `floor(x)` / `ceiling(x)` / `ceiling(x)` Rounding to integers. `ceiling` and `ceiling` are aliases. --- ### `round(x, digits = 0)` / `signif(x, digits = 6)` Rounding to decimal places or significant figures. --- ### `trunc(x)` / `sign(x)` Truncate fractional part or get the sign (-1, 0, 1) of a value. --- ### `ndarray(data, shape = NA)` / `reshape(array, shape)` Create or reshape N-dimensional arrays. `ndarray` can infer shape from nested lists. --- ### `shape(array)` / `ndarray_data(array)` Get NDArray dimensions (as a List) or flat data (as a List of Floats). --- ### `matmul(a, b)` / `inv(matrix)` / `transpose(matrix)` Linear algebra operations on 2D NDArrays. --- ### `diag(x)` / `kron(a, b)` / `cbind(a, b)` Matrix creation and manipulation. `diag` extracts the diagonal from a 2D array or creates a diagonal matrix from a 1D array. --- ### `iota(n)` Returns a Vector of length `n` filled with `1.0` (a ones vector). Useful for initializing weights or masks. **Examples:** ```t pow(2, 3) -- 8.0 pow(10, 2) -- 100.0 pow(4, 0.5) -- 2.0 (square root) pow(2, -1) -- 0.5 ``` --- --- ## Stats Package Statistical functions for data analysis. Most functions handle missingness via an `na_rm` parameter. ### Descriptive Statistics #### `median(x, na_rm = false, weights = NA)` / `mean(x, na_rm = false, weights = NA)` #### `min(x, na_rm = false)` / `max(x, na_rm = false)` / `range(x, na_rm = false)` Basic descriptive statistics. `mean` and `median` also accept optional non-negative observation weights. `range` returns a List of [min, max]. --- #### `var(x, na_rm = false, weights = NA)` / `sd(x, na_rm = false, weights = NA)` / `cv(x, na_rm = false, weights = NA)` Variance, standard deviation, and coefficient of variation (sd/mean). These also accept optional non-negative observation weights. The weighted `sd`/`var` path uses the weighted population denominator (`sum(weights)`), while the unweighted path uses sample formulas. --- #### `iqr(x, na_rm = false, weights = NA)` / `mad(x, na_rm = false)` Interquartile range and Median Absolute Deviation (scaled by 1.4826). `iqr` accepts optional non-negative observation weights. --- #### `fivenum(x, na_rm = false, weights = NA)` Tukey's five-number summary (min, lower-hinge, median, upper-hinge, max), with optional non-negative observation weights. --- #### `skewness(x, na_rm = false, weights = NA)` / `kurtosis(x, na_rm = false, weights = NA)` Skewness and excess kurtosis, with optional non-negative observation weights. --- #### `trimmed_mean(x, trim = 0.1, na_rm = false, weights = NA)` Mean calculated after trimming a fraction of observations from each end. Optional weights affect the trim cut points and the retained mean. --- #### `quantile(x, p, na_rm = false, weights = NA)` Compute quantile/percentile (p between 0 and 1), with optional non-negative observation weights. --- #### `mode(x)` Return the most frequent value. Values are compared with type-aware, NaN-aware equality, so different types never collide. Ties break deterministically by first occurrence in input order. Does not currently support `na_rm`. --- ### Data Transformation #### `normalize(x)` / `standardize(x)` / `scale(x)` Rescale or center numeric data. `scale` and `standardize` compute z-scores. `normalize` scales to [0, 1]. --- #### `winsorize(x, limits = [0.05, 0.05], na_rm = false, weights = NA)` Clamp values using one limit or a two-element vector `[lower_tail_fraction, upper_tail_fraction]`, with each fraction in `[0, 0.5)`. Optional weights affect the cut points only; zero-weight observations remain in the output. --- #### `huber_loss(actual, predicted, delta = 1.0)` Compute the Huber loss between two vectors. --- ### Distributions (CDFs) #### `pnorm(x)` / `pt(x, df)` / `pf(q, df1, df2)` / `pchisq(q, df)` Cumulative Distribution Functions for Normal, Student-t, F, and Chi-squared distributions. --- ### Quantile Functions (Inverse CDFs) #### `qnorm(p, mean=0, sd=1)` / `qt(p, df)` / `qf(p, df1, df2)` / `qchisq(p, df)` Quantile (inverse cumulative probability) Functions for Normal, Student-t, F, and Chi-squared distributions. - `qnorm(p, mean = 0, sd = 1)` — normal quantile with optional `mean` and `sd` named args. - `qt(p, df)` — Student t quantile. - `qf(p, df1, df2)` — F quantile. - `qchisq(p, df)` — Chi-squared quantile. --- ### Modeling #### `lm(data, formula, weights = NA)` Fit a linear regression model. Without weights this is OLS; with `weights` it performs weighted least squares. The formula is a `Formula` value such as `mpg ~ wt + hp`. --- #### `summary(model)` / `fit_stats(model)` `summary(model)` returns a `Dict` containing a `_tidy_df` DataFrame plus metadata; `fit_stats(model)` returns a `DataFrame` of model-level metrics. --- #### `predict(data, model)` / `score(data, model)` Perform vectorized prediction on new data. `score` is an alias. --- #### `add_diagnostics(model, data)` / `add_diagnostics(model, data)` Augment data with per-observation diagnostics: `.fitted`, `.resid`, `.hat`, `.sigma`, `.cooksd`, and `.std.resid`. --- #### `anova(model1, model2, ...)` Compare multiple nested models using an ANOVA table. --- #### `coef(model)` / `residuals(model)` / `vcov(model)` / `df_residual(model)` Extract model components. --- #### `wald_test(model, terms)` Perform a Wald test for a joint hypothesis on coefficients. --- #### `t_read_onnx(path)` / `t_read_pmml(path)` Import pre-trained models from ONNX or PMML formats for native scoring. Julia nodes can also consume `^onnx` artifacts through `ONNXRunTime.jl`; ONNX export from Julia remains explicitly unsupported. --- ### `cut(x, breaks, ...)` / `poly(x, degree, ...)` Basis functions for modeling. --- --- ## DataFrame Package CSV I/O and DataFrame introspection. ### `to_dataframe(data)` Constructs a DataFrame from either a list of rows (Dictionaries) or a Dictionary of columns (Vectors/Lists). **Parameters:** - `data` — List of Dictionaries (row-wise) or a single Dictionary (column-wise) **Returns:** `DataFrame` **Examples:** ```t -- Column-wise df = to_dataframe([x: [1, 2], y: [3, 4]]) -- Row-wise df = to_dataframe([ [name: "Alice", age: 30], [name: "Bob", age: 25] ]) ``` --- ### `read_csv(path, separator = ",", skip_lines = 0, skip_header = false, clean_colnames = false)` Read a CSV file into a DataFrame. **Parameters:** - `path` — File path (String) - `separator` (optional) — Column separator (default: ",") - `skip_lines` (optional) — Number of lines to skip at start (default: 0) - `skip_header` (optional) — If true, treat first row as data (default: false) - `clean_colnames` (optional) — If true, normalize column names (default: false) **Returns:** `DataFrame` --- ### `read_parquet(path)` / `write_parquet(to_dataframe, path)` Read or write Parquet files using the native parquet-glib reader/writer. Prefer Parquet for compressed, long-term storage of large datasets or when sharing data with external analytics tooling. For the fastest possible round trip (no compression), use `read_ipc` / `write_ipc` instead. See [Parquet vs Arrow IPC: How to Choose](data-formats.md#parquet-vs-arrow-ipc-how-to-choose). --- ### `read_ipc(path)` / `write_ipc(to_dataframe, path)` Read or write Arrow IPC files (Feather v2). IPC is the fastest format to read and write (no compression), ideal for pipeline intermediates and cross-runtime exchange. For compressed, long-term storage of large datasets, use `read_parquet` / `write_parquet` instead. See [Parquet vs Arrow IPC: How to Choose](data-formats.md#parquet-vs-arrow-ipc-how-to-choose). --- ### `write_csv(to_dataframe, path, separator = ",")` Write a DataFrame to a CSV file. --- ### `nrow(to_dataframe)` / `ncol(to_dataframe)` Get number of rows or columns. --- ### `colnames(to_dataframe)` Get column names as a List of strings. --- ### `clean_colnames(x)` Standardizes column names using a snake_case convention. Works on DataFrames or Lists of strings. --- ### `glimpse(to_dataframe)` Prints a summary of the DataFrame structure, including dimensions, column names, types, and first few values. --- ### `pull(to_dataframe, column)` Extracts a single column as a Vector. --- ### `to_array(to_dataframe, columns = NA)` Converts numeric columns of a DataFrame to a matrix (NDArray). --- ### `ncol(to_dataframe)` Get number of columns. **Parameters:** - `to_dataframe` — DataFrame **Returns:** `Int` — Column count **Examples:** ```t ncol(df) -- 5 ``` --- ### `colnames(to_dataframe)` Get column names. **Parameters:** - `to_dataframe` — DataFrame **Returns:** List of Strings **Examples:** ```t colnames(df) -- ["name", "age", "dept", "salary"] ``` --- ### `glimpse(to_dataframe)` Get a compact overview of a DataFrame, showing column names, types, and example values. Similar to dplyr's `glimpse()`. **Parameters:** - `to_dataframe` — DataFrame **Returns:** Dict with `kind`, `nrow`, `ncol`, and `columns` (list of column summaries) **Examples:** ```t glimpse(df) -- {`kind`: "to_dataframe", `nrow`: 100, `ncol`: 4, `columns`: ["name ...", "age ...", ...]} ``` --- ### `clean_colnames(to_dataframe)` / `clean_colnames(names)` Normalize column names to safe identifiers. **Parameters:** - `to_dataframe` — DataFrame, OR - `names` — List of Strings **Returns:** DataFrame with cleaned names, OR List of cleaned Strings **Transformations:** 1. Symbol expansion: `%` → `percent`, `€` → `euro`, `$` → `dollar`, etc. 2. Diacritics removal: `café` → `cafe` 3. Lowercase 4. Non-alphanumeric → `_`, collapse runs 5. Prefix digits with `x_`: `1st` → `x_1st` 6. Empty → `col_N` 7. Collision resolution: `_2`, `_3`, etc. **Examples:** ```t clean_colnames(["Growth%", "MILLION€", "café"]) -- ["growth_percent", "million_euro", "cafe"] clean_colnames(["A.1", "A-1"]) -- ["a_1", "a_1_2"] (collision resolved) df2 = clean_colnames(df) -- DataFrame with cleaned column names ``` --- ## Colcraft Package Data manipulation verbs and window functions. ### Data Verbs #### `select(to_dataframe, ...columns)` Select columns by name. Supports dollar-prefix NSE syntax and the selection helpers `starts_with`, `ends_with`, `contains`, `everything`, `where`, `all_of`, `any_of`, and `matches`. `matches` compiles its pattern with PCRE2 in UTF-8 mode, so `.` matches a code point and `\p{...}` Unicode property classes are supported (e.g. `select(df, matches("^\\p{L}+$"))`); `contains` performs a plain literal substring search. **Parameters:** - `to_dataframe` — DataFrame - `...columns` — Column references (`$name`) **Returns:** DataFrame with selected columns **Examples:** ```t df |> select($name, $age) df |> select($dept) ``` --- #### `filter(to_dataframe, predicate)` Filter rows by condition. Supports NSE expressions with dollar-prefix column references. **Parameters:** - `to_dataframe` — DataFrame - `predicate` — NSE expression (`$age > 25`) **Returns:** DataFrame with matching rows **Examples:** ```t df |> filter($age > 30) df |> filter($dept == "Engineering") df |> filter($salary > 50000 and $active == true) ``` --- #### `mutate(to_dataframe, $col = expr)` / `mutate(to_dataframe, new_col, fn)` Add or transform a column. Supports `$col = expr` named-arg syntax with NSE. **Parameters (named-arg form):** - `to_dataframe` — DataFrame - `$col = expr` — Column name from `$col`, value from NSE expression **Parameters (positional form):** - `to_dataframe` — DataFrame - `new_col` — Column reference (`$bonus`) - `fn` — Function taking row dict: `\(row) ...`, OR - `value` — Constant value for all rows **Returns:** DataFrame with new/modified column **Examples:** ```t -- Named-arg NSE syntax df |> mutate($bonus = $salary * 0.1) df |> mutate($age_next_year = $age + 1) -- Positional NSE with lambda df |> mutate($bonus, \(row) row.salary * 0.1) -- Grouped mutate (broadcast group result) df |> group_by($dept) |> mutate($dept_size, \(g) nrow(g)) ``` --- #### `arrange(to_dataframe, column, direction = "asc")` Sort rows by column. Supports dollar-prefix NSE for column names. **Parameters:** - `to_dataframe` — DataFrame - `column` — Column reference (`$age`) - `direction` (optional) — "asc" or "desc" (default: "asc") **Returns:** Sorted DataFrame **Examples:** ```t df |> arrange($age) df |> arrange($salary, "desc") ``` --- #### `group_by(to_dataframe, ...columns)` Group by one or more columns. Supports dollar-prefix NSE for column names. **Parameters:** - `to_dataframe` — DataFrame - `...columns` — Column references (`$dept`) **Returns:** Grouped DataFrame **Usage:** ```t -- Use with summarize to aggregate df |> group_by($dept) |> summarize($avg_salary, \(g) mean(g.salary)) -- Use with mutate to broadcast group results df |> group_by($dept) |> mutate($dept_count, \(g) nrow(g)) ``` **Examples:** ```t df |> group_by($dept) df |> group_by($dept, $location) ``` --- #### `summarize(grouped_df, $col = expr)` / `summarize(grouped_df, new_col, fn)` Aggregate grouped data. Supports `$col = expr` named-arg syntax with NSE. **Parameters (named-arg form):** - `grouped_df` — Grouped DataFrame (from `group_by()`) - `$col = expr` — Column name from `$col`, aggregation from NSE expression (e.g. `sum($amount)`) **Parameters (positional form):** - `grouped_df` — Grouped DataFrame (from `group_by()`) - `new_col` — Column reference (`$count`) - `fn` — Aggregation function: `\(group) ...` **Returns:** DataFrame with one row per group **Examples:** ```t -- Named-arg NSE syntax df |> group_by($dept) |> summarize($count = nrow($dept)) df |> group_by($dept) |> summarize($avg_salary = mean($salary)) df |> group_by($region) |> summarize($total_sales = sum($sales), $n = nrow($region)) -- Positional NSE with lambda df |> group_by($dept) |> summarize($count, \(g) nrow(g)) ``` --- #### `ungroup(grouped_df)` Remove grouping from a DataFrame. **Parameters:** - `grouped_df` — Grouped DataFrame **Returns:** Ungrouped DataFrame **Examples:** ```t ungrouped = df |> group_by($dept) |> ungroup() ``` --- ### Join and Bind Functions #### `left_join(x, y, by = NA)` / `inner_join` / `full_join` / `semi_join` / `anti_join` Join two DataFrames. Join keys are normalized so that integer `1` and float `1.0` match, mirroring R's coercion-to-character semantics. **Parameters:** - `x`, `y` — DataFrames to join - `by` (optional) — Column(s) to join on. If omitted, uses common columns. **Returns:** Joined DataFrame --- #### `bind_rows(...)` / `bind_cols(...)` Combine multiple DataFrames by stacking rows or placing columns side-by-side. --- ### Wrangling Utilities #### `count(df, ...columns)` Count occurrences of unique values. --- #### `distinct(df, ...columns)` Keep only unique rows. Repeated `NaN` values collapse to a single row, consistent with `n_distinct`. --- #### `drop_na(df, ...columns)` Drop rows containing NA values in the specified columns. Works across all column types, including date, datetime, dictionary, and list columns. --- #### `replace_na(df, values)` Replace NA values with specified defaults. --- #### `rename(df, ...new_name = old_name)` Rename columns. --- #### `relocate(df, ...columns, before = NA, after = NA)` Change column order. --- #### `slice(df, ...indices)` / `slice_min(df, col, n = 1)` / `slice_max(df, col, n = 1)` / `slice_sample(df, n = 1, replace = false)` Subset rows by position, extreme values, or random sample. `slice_sample` draws a random sample of n rows with or without replacement. Use `set_seed()` for reproducible results. --- #### `pivot_longer(df, cols, names_to = "name", values_to = "value")` #### `pivot_wider(df, names_from = "name", values_from = "value")` Reshape DataFrames between long and wide formats. --- #### `separate(df, col, into, sep = "[^a-zA-Z0-9]+")` / `unite(df, col, ...from, sep = "_")` Split a column into multiple columns, or combine multiple columns into one. The `sep` pattern is compiled with PCRE2 in UTF-8 mode, so multibyte values are never torn. Split semantics match legacy `Str.split`: trailing empty tokens are dropped and an empty subject yields no tokens. --- ### Factor Manipulation #### `to_factor(x, levels = NA, ordered = false)` Create to_factor-encoded vectors. Derives unique levels alphabetically if `levels` is not provided. --- #### `levels(f)` Get labels from a to_factor. --- #### `fct_recode(f, ...new = old)` / `fct_relevel(f, ...levels, after = 0)` Rename or reorder to_factor levels. --- #### `fct_lump_n(f, n, other_level = "Other")` / `fct_lump_min` / `fct_lump_prop` Collapse infrequent levels into an "Other" category. --- #### `fct_infreq(f)` / `fct_rev(f)` / `fct_reorder(f, x, .desc = false)` Reorder levels by frequency, reversal, or summary of another vector. --- ### Aggregation Context #### `n()` Returns the number of rows in the current group. Only valid inside `summarize()`. --- #### `n_distinct(x)` Returns the number of unique non-NA values. --- ### Window Functions Window functions compute values across rows without collapsing them. #### Ranking Functions ##### `row_number(vector)` Assign unique row numbers. **Parameters:** - `vector` — Vector or List **Returns:** Vector of row numbers (1, 2, 3, ...), NA for NA positions **Examples:** ```t row_number([10, 30, 20]) -- Vector[1, 3, 2] row_number([3, NA, 1]) -- Vector[2, NA, 1] ``` --- ##### `min_rank(vector)` Minimum rank (gaps after ties). **Parameters:** - `vector` — Vector or List **Returns:** Vector of ranks **Examples:** ```t min_rank([1, 1, 2, 2, 2]) -- Vector[1, 1, 3, 3, 3] min_rank([3, NA, 1, 3]) -- Vector[2, NA, 1, 2] ``` --- ##### `dense_rank(vector)` Dense rank (no gaps). **Parameters:** - `vector` — Vector or List **Returns:** Vector of ranks **Examples:** ```t dense_rank([1, 1, 2, 2]) -- Vector[1, 1, 2, 2] dense_rank([10, 10, 20]) -- Vector[1, 1, 2] ``` --- ##### `cume_dist(vector)` Cumulative distribution (proportion ≤ value). **Parameters:** - `vector` — Vector or List **Returns:** Vector of Float (0.0 to 1.0) **Examples:** ```t cume_dist([1, 2, 3]) -- Vector[0.333..., 0.666..., 1.0] ``` --- ##### `percent_rank(vector)` Percent rank ((rank - 1) / (n - 1)). **Parameters:** - `vector` — Vector or List **Returns:** Vector of Float (0.0 to 1.0) **Examples:** ```t percent_rank([1, 2, 3]) -- Vector[0.0, 0.5, 1.0] ``` --- ##### `ntile(vector, n)` Divide into n groups. **Parameters:** - `vector` — Vector or List - `n` — Number of groups (Int) **Returns:** Vector of group numbers (1 to n) **Examples:** ```t ntile([1, 2, 3, 4], 2) -- Vector[1, 1, 2, 2] ntile([1, 2, 3, 4, 5], 3) -- Vector[1, 1, 2, 2, 3] ``` --- #### Offset Functions ##### `lag(vector, n = 1)` Shift values forward (add NA at start). **Parameters:** - `vector` — Vector or List - `n` (optional) — Number of positions (default: 1) **Returns:** Vector with shifted values **Examples:** ```t lag([1, 2, 3, 4]) -- Vector[NA, 1, 2, 3] lag([1, 2, 3, 4], 2) -- Vector[NA, NA, 1, 2] lag([1, NA, 3]) -- Vector[NA, 1, NA] ``` --- ##### `lead(vector, n = 1)` Shift values backward (add NA at end). **Parameters:** - `vector` — Vector or List - `n` (optional) — Number of positions (default: 1) **Returns:** Vector with shifted values **Examples:** ```t lead([1, 2, 3, 4]) -- Vector[2, 3, 4, NA] lead([1, 2, 3, 4], 2) -- Vector[3, 4, NA, NA] ``` --- #### Cumulative Functions NA propagates: once NA is encountered, all subsequent values become NA. ##### `cumsum(vector)` Cumulative sum. **Examples:** ```t cumsum([1, 2, 3, 4]) -- Vector[1, 3, 6, 10] cumsum([1, NA, 3]) -- Vector[1, NA, NA] ``` --- ##### `cummin(vector)` Cumulative minimum. **Examples:** ```t cummin([3, 1, 4, 1]) -- Vector[3, 1, 1, 1] ``` --- ##### `cummax(vector)` Cumulative maximum. **Examples:** ```t cummax([1, 3, 2, 5]) -- Vector[1, 3, 3, 5] ``` --- ##### `cummean(vector)` Cumulative mean. **Examples:** ```t cummean([2, 4, 6]) -- Vector[2.0, 3.0, 4.0] ``` --- ##### `cumall(vector)` Cumulative AND (all true so far?). **Examples:** ```t cumall([true, true, false]) -- Vector[true, true, false] ``` --- ##### `cumany(vector)` Cumulative OR (any true so far?). **Examples:** ```t cumany([false, true, false]) -- Vector[false, true, true] ``` --- ## Chrono Package High-performance date and time manipulation, inspired by R's `lubridate`. ### `to_date(value)` / `to_datetime(value)` Convert values to Date or Datetime types. **Parameters:** - `value` — String, Number, or Collection of values **Returns:** `Date` / `Datetime` / `Collection` **Examples:** ```t to_date("2023-05-15") -- 2023-05-15 to_datetime("2023-05-15 14:00:00") ``` --- ### `ymd(string)` / `mdy(string)` / `dmy(string)` / `ydm(string)` ### `ymd_h(string)` / `ymd_hm(string)` / `ymd_hms(string)` Parse strings into dates or datetimes using common layouts. **Parameters:** - `string` — Date or Datetime string **Returns:** `Date` / `Datetime` **Examples:** ```t ymd("2023-05-15") mdy("05-15-2023") ymd_hms("2023-05-15 14:30:05") ``` --- ### `parse_date(string, format)` / `parse_datetime(string, format, tz = "UTC")` Parse strings into temporal values using explicit `strptime`-style formats. **Parameters:** - `string` — Input string - `format` — Format string (e.g., "%Y-%m-%d") - `tz` (optional) — Timezone label for `parse_datetime` **Returns:** `Date` / `Datetime` --- ### `today()` / `now(tz = "UTC")` Get the current UTC date or datetime. **Returns:** `Date` / `Datetime` --- ### `year(x)` / `month(x, label = false)` / `day(x)` / `day(x)` ### `yday(x)` / `wday(x, label = false, week_start = 7)` / `week(x)` / `isoweek(x)` / `isoyear(x)` ### `quarter(x)` / `semester(x)` Extract calendar components from Date or Datetime values. **Parameters:** - `x` — Date or Datetime - `label` (optional) — If true, returns month/weekday names as strings. - `week_start` (optional) — Day the week starts on (1=Mon, 7=Sun). **Returns:** `Int` / `String` --- ### `hour(x)` / `minute(x)` / `second(x)` / `tz(x)` Extract time-of-day components or timezone labels from Datetime values. **Returns:** `Int` / `Float` / `String` --- ### `am(x)` / `pm(x)` Check whether a time is before or after noon. **Returns:** `Bool` --- ### `floor_date(datetime, unit)` / `ceiling_date(datetime, unit)` / `round_date(datetime, unit)` Round a date/datetime to the nearest unit boundary (year, month, day, hour, etc.). **Parameters:** - `datetime` — Date or Datetime - `unit` — Unit as string ("month", "day", "hour", etc.) **Returns:** Same as input type **Examples:** ```t floor_date(to_date("2023-05-15"), "month") -- 2023-05-01 ``` --- ### `make_date(year, month, day)` / `make_datetime(year, month, day, hour, min, sec, tz)` Construct temporal values from numeric components. --- ### `format_date(x, format)` / `format_datetime(x, format)` Format temporal values as strings using `strftime`-style patterns. --- ### `interval(start, end)` / `%within%(x, interval)` Construct temporal intervals and test membership. `%within%` can be used as an infix operator: `d %within% interval(start, end)` desugars to `%within%(d, interval(start, end))`. Any function name can be used infix with `%name%` syntax (R-style), resolving to the same-named builtin or user function. --- ### `years(n)` / `months(n)` / `weeks(n)` / `days(n)` / `hours(n)` / `minutes(n)` / `seconds(n)` Construct Period objects for date arithmetic. --- ### `is_date(x)` / `is_datetime(x)` / `is_period(x)` / `is_duration(x)` / `is_interval(x)` Type predicates for temporal values. --- ### `is_leap_year(x)` / `days_in_month(x)` Calendar helpers. --- ### `with_tz(x, tz)` / `force_tz(x, tz)` Update the timezone label of a Datetime value. --- ## Strcraft Package Modern string manipulation utilities, inspired by R's `stringr`. ### `str_replace(string, pattern, replacement)` / `replace_first(string, pattern, replacement)` Replace occurrences of a pattern. `str_replace` replaces **all** occurrences (global replace); `replace_first` replaces only the first occurrence. --- ### `str_detect(string, pattern)` / `contains(s, sub)` Check if a pattern or substring exists. `str_detect` compiles `pattern` as a PCRE2 regular expression in UTF-8 mode, so `.`, `[...]`, anchors, and `\p{...}` classes operate on Unicode code points — `str_detect("héllo", "^h.llo$")` is `true` and `str_extract_all("héllo", "\\p{L}")` returns `["h", "é", "l", "l", "o"]`. `contains` performs a plain substring search. --- ### `starts_with(s, prefix)` / `ends_with(s, suffix)` Check string boundaries. --- ### `str_extract(s, pattern)` / `str_extract_all(s, pattern)` Extract matching substrings using UTF-8-aware PCRE2 patterns (see `str_detect`). `str_extract` returns the first match; `str_extract_all` returns a List of all matches. --- ### `str_count(s, pattern)` / `str_nchar(s)` Count regex matches (PCRE2, UTF-8 aware — each code point matched by `.` counts once) or total characters. --- ### `str_trim(s)` / `trim_start(s)` / `trim_end(s)` Remove whitespace. --- ### `str_lines(s)` / `str_words(s)` / `str_split(s, sep)` Split strings into parts. `str_lines` splits on newlines; `str_words` splits on any whitespace. With an empty separator, `str_split(s, "")` splits into individual Unicode code points (multi-byte characters are never torn apart). --- ### `str_pad(s, width, side = "left", pad = " ")` Pad strings to a fixed width. Width is measured in characters (Unicode code points), not bytes. --- ### `str_trunc(s, width, side = "right", ellipsis = "...")` Truncate strings with an ellipsis. Width is measured in characters (Unicode code points); truncation never splits a multi-byte character in half. --- ### `str_flatten(values, collapse = "")` / `str_join(items, sep = "")` Combine multiple strings into one. --- ### `to_lower(s)` / `to_upper(s)` Unicode-aware case mapping (not locale-dependent): `to_upper("éàüç")` is `"ÉÀÜÇ"`, `to_lower("ÉÀÜÇ")` is `"éàüç"`, and full case folding is applied where Unicode requires it (e.g. `to_upper("ß")` is `"SS"`). Multi-character mappings never split UTF-8 code points. --- ### `str_repeat(s, n)` Repeat a string `n` times. --- ### `str_format(fmt, values)` / `str_sprintf(fmt, ...)` String interpolation and formatting. `str_format` uses `{name}` placeholders with a Dictionary or named List; `str_sprintf` uses C-style `%` specifiers. --- --- ## Lens Package Composable access and update lenses for dictionaries, lists, data frames, and pipeline inspection. For the full walkthrough and worked examples, see the [Lens guide](lens.md). For the generated per-function entries, see the [Function Reference](reference/index.md). ### `col_lens(name)` / `idx_lens(i)` / `row_lens(i)` Focus on a dictionary key/column, a list index, or a DataFrame row. --- ### `filter_lens(predicate)` Focus on elements matching a predicate (supports DataFrames, Lists, and Vectors). --- ### `node_lens(name)` / `node_meta_lens(name, field)` / `env_var_lens(node, var)` Focus on pipeline nodes, their metadata, or environment variables. --- ### `compose(...lenses)` Combine multiple lenses into a deep traversal. --- ### `get(data, lens)` / `set(data, lens, value)` / `over(data, lens, fn)` Read, write, or transform data at the focused location. --- ### `modify(data, ...pairs)` Apply a sequence of `(lens, function)` pairs to the same data structure. --- ## Pipeline Package Pipeline introspection and management. ### `pipeline(...)` Constructs a Pipeline from a Dictionary of named nodes or a List of node records. --- ### `build_pipeline(p, verbose = 0, pipeline_name = NA)` / `populate_pipeline(p, build = true)` Materialize a pipeline to Nix artifacts. `build_pipeline` is the primary entry point for full Nix builds and returns a `BuildLog` value (`nodes`, `duration`, `failed_nodes`, `out_path`). `populate_pipeline` can be used to generate the Nix expression without building (with `build = false`). Use `pipeline_name` to record a name in the build log for later disambiguation via `list_logs()`. --- ### `read_pipeline(p)` / `inspect_pipeline(p)` Returns a dictionary with node metadata and diagnostics summary. `inspect_pipeline` focuses on the DAG structure (edges). --- ### `read_node(node)` Retrieves the dynamically evaluated or built artifact of a node from an in-scope pipeline. Strictly expects a `ComputedNode` object (e.g. `p.node_name`). For reading from historical build logs without the pipeline in scope, use [`read_past_node(p.node_name, which_log = ...)`](#read_past_nodenode-which_log). --- ### `filter_node(p, predicate)` / `select_node(p, ...)` Subsetting nodes in a pipeline. `filter_node` keeps nodes matching a condition; `select_node` picks nodes by name. --- ### `mutate_node(p, ...)` / `rename_node(p, ...)` `mutate_node` modifies metadata fields on pipeline nodes (all nodes, or scoped to a subset via the `where` named argument). `rename_node` changes a node's label while preserving its dependency edges. **Mutable fields:** `noop` (Bool), `runtime` (String), `serializer` (String), `deserializer` (String), `deps` (List[String]), `functions` (List[String]), `include` (List[String]), `env_vars` (Dict), `args` (Dict), `shell` (String), `shell_args` (List[String]), `flake` (String). Unlike `set_pipeline_global_options`, `mutate_node` **replaces** list/dict fields entirely rather than combining them. Pass `NA` to clear an optional or list/dict field — except `deps`, which cannot be cleared with `NA` because dependency edges cannot be re-derived outside the original evaluation environment (mutate it to a concrete list instead). All mutated fields are marked `"node"`-sourced in provenance tracking. **Parameters:** - `p` — The pipeline to modify. - `...` — Metadata assignments as `$field = value` pairs. - `where` (optional) — Predicate scoping which nodes are updated. **Returns:** `Pipeline` — a new pipeline with updated node metadata. **Examples:** ```t p |> mutate_node($noop = true) p |> mutate_node($serializer = "pmml", where = $runtime == "R") p |> mutate_node($functions = ["utils.R"], $env_vars = NA) ``` --- ### `arrange_node(p, ...)` Reorders nodes in the pipeline definition (does not affect execution order, which is DAG-driven). --- ### `trace_nodes(p, node_names)` Returns a sub-pipeline containing only the specified nodes and all their recursive dependencies. --- ### `which_nodes(p, predicate)` Filter the richer node records from `read_pipeline(p).nodes` without manually writing `read_pipeline`, `compose`, or an explicit lambda. --- ### `errored_nodes(p)` Convenience wrapper returning the subset of node records whose `diagnostics.error` is not `NA`. --- ### `pipeline_to_ga(p, name = NA, pipeline_script = "src/pipeline.t", file = NA)` Generates a GitHub Actions CI workflow YAML to run the pipeline on push/PR events. It integrates with Cachix (`rstats-on-nix`) and caches built Nix artifacts inside the repository's `t-runs` branch as `.nar` archives. **Parameters:** - `p` — The Pipeline to configure. - `name` (optional) — Project name (auto-detected from `tproject.toml` if omitted). - `pipeline_script` (optional) — Path to the pipeline T script (default: `"src/pipeline.t"`). - `file` (optional) — Output file path. If specified, writes the YAML directly to that file (usually `.github/workflows/.yml`); if omitted, returns the workflow YAML content as a String. **Returns:** String (workflow YAML content or file write success message). **Examples:** ```t pipeline_to_ga(p) pipeline_to_ga(p, name = "my-project", file = ".github/workflows/ci.yml") ``` --- ### `pipeline_report(p, which_log = NA, file = NA, target = "ssh")` Generates a structured execution report summarizing the status of pipeline nodes, execution durations, error logs, and warnings. **Parameters:** - `p` — The Pipeline to report on. - `which_log` (optional) — Regex selector to report on a specific historical build log instead of the latest log. - `file` (optional) — Target output file path. Defaults to `_pipeline/pipeline_report_.md` (ssh) or `.html` (web). - `target` (optional) — Output format: `"ssh"` for Markdown format (default), or `"web"` for HTML format. **Returns:** String path to the generated report file. **Examples:** ```t pipeline_report(p) pipeline_report(p, target = "web", file = "report.html") ``` --- ### `set_pipeline_global_options(pipeline, functions = [:], include = [], env_vars = [:], serializer = NA, deserializer = NA, noop = false, args = [:], shell = NA, shell_args = [], flake = NA, dependencies = NA, runtimes = NA, nodes = NA)` Pure function that returns a new pipeline with the given defaults merged into target nodes. The original pipeline is not modified. By default the settings are merged into every node; pass `runtimes` and/or `nodes` to restrict the merge to a subset (union semantics when both are given). Omitting both scoping arguments (or passing `na()`) targets every node; an explicitly empty list (`nodes = []`) targets no nodes. Merge semantics vary per option: - **Combine (prepend)** — `functions`, `include`, `env_vars`, `args`, `shell_args`, `dependencies`. Global values come first; per-node values follow. For same-key dict entries (`env_vars`, `args`) the later per-node value wins. - **Override** — `serializer`, `deserializer`, `shell`, `flake`. A provided global value replaces every target node's per-node value entirely. - **Force-only** — `noop`. `noop = true` forces target nodes to no-op; `noop = false` has no effect and cannot un-set a per-node `noop = true`. **Parameters:** - `pipeline` — The input pipeline (positional or piped). - `functions` (optional) — Dict mapping runtime shorthands to function file paths. For example: `[rn: "functions.R", pyn: ["preproc.py", "utils.py"]]`. Each value can be a single path (String) or list of paths (List[String]). Runtime shorthands match node constructors: `rn` → R, `pyn` → Python, `jln` → Julia, `qn` → Quarto, `shn` → sh, `node` → T. If the same runtime shorthand appears more than once, all values are concatenated. Per-node `functions` arguments are appended after these global files. - `include` (optional) — String or List[String]. File paths to include in every node's sandbox. Per-node `include` arguments are appended after these global includes. - `env_vars` (optional) — Dict of environment variables for every node. Per-node `env_vars` override global values for the same key. - `serializer` (optional) — String, Symbol, or `^`-prefixed serializer name. Default serializer for every node; replaces any per-node serializer. `"default"` selects the runtime's default. - `deserializer` (optional) — String, Symbol, or `^`-prefixed deserializer name. Default deserializer for every node; replaces any per-node deserializer. `"default"` selects the runtime's default. - `noop` (optional) — Bool. If true, every node becomes a no-op. Setting false has no effect (it cannot un-set a per-node `noop = true`). - `args` (optional) — Dict of runtime/tool arguments for every node. Per-node `args` override global values for the same key. - `shell` (optional) — String. Shell interpreter for every node; replaces per-node `shell`. - `shell_args` (optional) — String or List[String]. Shell arguments, prepended before per-node `shell_args`. - `flake` (optional) — String. Nix flake reference for every node; replaces per-node `flake`. - `dependencies` (optional) — String or List[String]. Explicit dependencies, prepended before per-node `deps`. Note: per-node node constructors use the shorter `deps` argument name for the same field. - `runtimes` (optional) — String or List[String]. Scope the merge to nodes whose runtime is in this set. Accepts translated runtimes (`"R"`, `"Python"`) or constructor shorthands (`"rn"`, `"pyn"`). An unmatched runtime is a `TypeError`. Omitted (or `na()`) targets all nodes; an explicitly empty list (`runtimes = []`) targets no nodes. - `nodes` (optional) — String or List[String]. Scope the merge to exactly these node names. An unknown node name is a `TypeError`. When both `runtimes` and `nodes` are given, the target is their union. Omitted (or `na()`) targets all nodes; an explicitly empty list (`nodes = []`) targets no nodes. **Returns:** Pipeline — a new pipeline with the settings merged into the target nodes. **Examples:** ```t p = pipeline { a = rn(<{ ... }>), b = pyn(<{ ... }>) } q = set_pipeline_global_options(p, functions = [rn: "functions.R"], include = "shared/config.yaml", env_vars = [FOO: "bar"], serializer = ^json, noop = true ) # Scope to R nodes only q2 = set_pipeline_global_options(p, runtimes = ["rn"], serializer = ^ipc) # Scope to specific nodes q3 = set_pipeline_global_options(p, nodes = ["a"], noop = true) ``` --- ### `pipeline_node_options(pipeline, node)` Returns a Dict describing the fully resolved configuration of a single pipeline node, after any `set_pipeline_global_options` merges have been applied. This is the read-back companion to `set_pipeline_global_options`: what you merged in, you can read back out. The returned Dict has the following keys: - `name` — the node name (String) - `runtime` — one of `"T"`, `"R"`, `"Python"`, `"Julia"`, `"Quarto"`, `"sh"` (String) - `serializer` — e.g. `"default"`, `"pmml"` (String) - `deserializer` — e.g. `"default"`, `"pmml"` (String) - `noop` — whether the node is a no-op (Bool) - `deps` — names of nodes this node depends on (List of String) - `depth` — topological depth in the DAG (Int); roots are depth 0 - `command_type` — one of `"command"` or `"script"` (String) - `diagnostics` — node diagnostics (Dict) - `functions` — function files merged into the node (List of String) - `include` — included files (List of String) - `env_vars` — build environment variables (Dict) - `args` — runtime/tool arguments (Dict) - `shell` — shell interpreter, or NA when unset (String | NA) - `shell_args` — shell interpreter arguments (List of String) - `flake` — Nix flake path, or NA when unset (String | NA) - `provenance` — provenance of every resolved value, as a Dict with one entry per option key (`functions`, `include`, `env_vars`, `args`, `shell_args`, `serializer`, `deserializer`, `shell`, `flake`, `noop`, `dependencies`). Mergeable lists are grouped as `{ global: [...], node: [...] }`; scalar options are either `"global"`, `"node"`, or NA when unset. This is the source-provenance companion to `explain(p.node)`, telling you which values came from `set_pipeline_global_options` (global) and which were declared on the node itself. An unknown node name is a `TypeError` listing the valid node names. **Parameters:** - `pipeline` — The input pipeline (positional or piped). - `node` — The node name to read back (String). **Returns:** Dict — the resolved node configuration. **Examples:** ```t p = pipeline { a = rn(<{ ... }>), b = pyn(<{ ... }>) } q = set_pipeline_global_options(p, functions = [rn: "functions.R"], noop = true) pipeline_node_options(q, "a") # => { name = "a", runtime = "R", functions = ["functions.R"], noop = true, ... } ``` --- ### `pipeline_config_to_frame(p)` Produces a DataFrame with one row per node showing resolved configuration values **and** per-field provenance counts. Extends `pipeline_to_frame` with provenance columns so queries like "which nodes got their serializer from global options?" can be answered directly in T. **Columns:** - Identity: `name`, `runtime`, `depth`, `command_type` (as in `pipeline_to_frame`) - Resolved scalar values: `serializer`, `deserializer`, `noop`, `shell`, `flake` - Provenance source markers: `prov_serializer`, `prov_deserializer`, `prov_noop`, `prov_shell`, `prov_flake` — each `"global"`, `"node"`, or empty when unset - Total counts: `n_deps`, `n_funcs`, `n_incs`, `n_env_vars`, `n_args`, `n_shell_args` - Provenance counts: `n_