Processors

Processors are transformations that modify your dataset before or after columns are generated. They run at different stages and can reshape, filter, or augment the data.

Overview

Each processor:

• Receives the complete batch DataFrame
• Applies its transformation
• Passes the result to the next processor (or to output)

Processors can run at three stages, determined by which callback methods they implement:

A processor can implement any combination of these callbacks. The built-in processors use process_after_batch() by default.

Processor Types

🗑️ Drop Columns Processor

Removes specified columns from the output dataset. Dropped columns are saved separately in the dropped-columns directory for reference.

Configuration:

1
import data_designer.config as dd
2
3
processor = dd.DropColumnsProcessorConfig(
4
    name="remove_intermediate",
5
    column_names=["temp_calculation", "raw_input", "debug_info"],
6
)

Behavior:

• Columns specified in column_names are removed from the output
• Original values are preserved in a separate parquet file
• Missing columns produce a warning but don’t fail the build
• Column configs are automatically marked with drop=True when this processor is added

Use Cases:

• Removing intermediate columns used only for LLM context
• Cleaning up debug or validation columns before final output
• Separating sensitive data from the main dataset

🔄 Schema Transform Processor

Creates an additional dataset with a transformed schema using Jinja2 templates. The output is written to a separate directory alongside the main dataset.

Configuration:

1
import data_designer.config as dd
2
3
processor = dd.SchemaTransformProcessorConfig(
4
    name="chat_format",
5
    template={
6
        "messages": [
7
            {"role": "user", "content": "{{ question }}"},
8
            {"role": "assistant", "content": "{{ answer }}"},
9
        ],
10
        "metadata": "{{ category | upper }}",
11
    },
12
)

Behavior:

• Each key in template becomes a column in the transformed dataset
• Values are Jinja2 templates with access to all columns in the batch
• Complex structures (lists, nested dicts) are supported
• Output is saved to the processors-outputs/{name}/ directory
• The original dataset passes through unchanged

Template Capabilities:

• Variable substitution: {{ column_name }}
• Filters: {{ text | upper }}, {{ text | lower }}, {{ text | trim }}
• Nested structures: Arbitrarily deep JSON structures
• Lists: ["{{ col1 }}", "{{ col2 }}"]

Use Cases:

• Converting flat columns to chat message format
• Restructuring data for specific model training formats
• Creating derived views without modifying the source dataset

Using Processors

Add processors to your configuration using the builder’s add_processor method:

1
import data_designer.config as dd
2
3
builder = dd.DataDesignerConfigBuilder()
4
5
# ... add columns ...
6
7
# Drop intermediate columns
8
builder.add_processor(
9
    dd.DropColumnsProcessorConfig(
10
        name="cleanup",
11
        column_names=["scratch_work", "raw_context"],
12
    )
13
)
14
15
# Transform to chat format
16
builder.add_processor(
17
    dd.SchemaTransformProcessorConfig(
18
        name="chat_format",
19
        template={
20
            "messages": [
21
                {"role": "user", "content": "{{ question }}"},
22
                {"role": "assistant", "content": "{{ answer }}"},
23
            ],
24
        },
25
    )
26
)

Execution Order

Processors execute in the order they’re added. Plan accordingly when one processor’s output affects another.

Processor Plugins

You can extend Data Designer with custom processors via the plugin system. A processor plugin is a Python package that provides:

• A config class inheriting from ProcessorConfig with a processor_type: Literal["your-type"] discriminator
• An implementation class inheriting from Processor that overrides the desired callback methods
• A Plugin instance connecting the two

Once installed, plugin processors are automatically discovered and can be used with add_processor() like built-in processors.

1
from my_processor_plugin.config import MyProcessorConfig
2
3
builder.add_processor(
4
    MyProcessorConfig(
5
        name="my_processor",
6
        # ... plugin-specific parameters ...
7
    )
8
)

Entry point configuration in pyproject.toml:

1
[project.entry-points."data_designer.plugins"]
2
my-processor = "my_plugin.plugin:my_processor_plugin"

See the plugins overview for the full guide on creating plugins.

Configuration Parameters

Common Parameters

DropColumnsProcessorConfig

SchemaTransformProcessorConfig