Custom Columns

Custom columns let you implement your own generation logic using Python functions. Use them for multi-step LLM workflows, external API integration, or any scenario requiring full programmatic control. For reusable, distributable components, see Plugins instead.

Quick Start

import data_designer.config as dd
@dd.custom_column_generator(required_columns=["name"])
def create_greeting(row: dict) -> dict:
    row["greeting"] = f"Hello, {row['name']}!"
    return row
config_builder.add_column(
    dd.CustomColumnConfig(
        name="greeting",
        generator_function=create_greeting,
    )
)

Function Signatures

Three signatures are supported. Parameter names are validated:

For full_column strategy, use df instead of row.

For LLM access without params, use generator_params: None:

@dd.custom_column_generator(required_columns=["name"], model_aliases=["my-model"])
def generate_message(row: dict, generator_params: None, models: dict) -> dict:
    response, _ = models["my-model"].generate(prompt=f"Greet {row['name']}")
    row["greeting"] = response
    return row

Model aliases are validated before generation starts. If an alias doesn’t exist in your config, an error is raised during the health check.

Preserving JSON values in artifacts

Set storage_encoding="json" on CustomColumnConfig when the generated column contains flexible JSON, such as dynamic dictionaries or mixed arrays:

dd.CustomColumnConfig(
    name="payload",
    generator_function=generate_payload,
    storage_encoding="json",
)

The generator still returns Python objects. DD encodes a storage copy and restores logical values for result loading, processors, and DD seed readers. External Parquet readers receive JSON strings; JSONL exports contain nested JSON. Encoding is fixed for the run and applies to every non-null cell, including scalar values. Only JSON-compatible values are accepted; non-string dictionary keys, non-finite numbers, and arbitrary Python objects raise errors rather than being silently converted.

Columns without this option retain their existing native Parquet behavior. Unsupported native writes raise an error identifying the affected column and the opt-in storage options.

Generation Strategies

Recommendation: Use cell_by_cell for LLM calls. The framework handles parallelization automatically. Use full_column only for vectorized operations that don’t involve LLM calls.

For full_column, set generation_strategy=dd.GenerationStrategy.FULL_COLUMN.

The Decorator

@dd.custom_column_generator(
    required_columns=["col1"],        # DAG ordering
    side_effect_columns=["extra"],    # Additional columns created
    model_aliases=["model1"],         # Required for LLM access
)

Models Dict

The third argument is a dict of ModelFacade instances, keyed by alias. You must declare all models required in your custom column generator in model_aliases - this populates the models dict and enables health checks before generation starts.

@dd.custom_column_generator(model_aliases=["my-model"])
def my_generator(row: dict, generator_params: None, models: dict) -> dict:
    model = models["my-model"]
    response, trace = model.generate(
        prompt="...",
        parser=my_custom_parser,  # optional, defaults to identity
        system_prompt="...",
        max_correction_steps=3,
    )
    row["result"] = response
    return row

This gives you direct access to all ModelFacade capabilities: custom parsers, correction loops, structured output, tool use, etc.

Other keyword arguments to generate() are sent with each model request: request fields such as temperature, max_tokens, or tool_choice, and an extra_body merged as described in Extra Body Precedence. tool_choice uses OpenAI’s format and is translated for Anthropic providers; with tool_alias, it applies only until the model’s first tool call. Arguments that aren’t request fields are dropped with a warning.

Configuration

Resizing (1:N and N:1)

Custom column generators must preserve row count. A CELL_BY_CELL generator returns one dict per input row, and a FULL_COLUMN generator returns a DataFrame with the same number of rows it received.

For expansion, filtering, aggregation, or deduplication, put the row-count-changing work at a workflow boundary. Use Workflow Chaining to run one stage, transform that stage’s output, and seed the next stage from the transformed rows.

Use cases:

• Expansion (1:N): Generate multiple variations per input between workflow stages
• Retraction (N:1): Filter, aggregate, or deduplicate records between workflow stages

Multi-Turn Example

@dd.custom_column_generator(
    required_columns=["topic"],
    side_effect_columns=["draft", "critique"],
    model_aliases=["writer", "editor"],
)
def writer_editor(row: dict, generator_params: None, models: dict) -> dict:
    draft, _ = models["writer"].generate(prompt=f"Write about '{row['topic']}'")
    critique, _ = models["editor"].generate(prompt=f"Critique: {draft}")
    revised, _ = models["writer"].generate(prompt=f"Revise based on: {critique}\n\nOriginal: {draft}")
    row["final_text"] = revised
    row["draft"] = draft
    row["critique"] = critique
    return row

Development Testing

Test generators with real LLM calls without running the full pipeline:

data_designer = DataDesigner()
models = data_designer.get_models(["my-model"])
result = my_generator({"name": "Alice"}, None, models)

In unit tests that mock model clients, use MagicMock(spec=ModelFacade) so async methods are auto-detected:

from unittest.mock import MagicMock
from data_designer.engine.models.facade import ModelFacade
mock_model = MagicMock(spec=ModelFacade)

Mocking only generate() will silently no-op because the bridge routes through agenerate().

See Also

• Plugins Overview