> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/datadesigner/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/datadesigner/_mcp/server.

# Processors

Processors are transformations that modify your dataset before or after columns are generated. They run at different stages and can reshape, filter, or augment the data.

> **Tip**
>
> When to Use Processors
> Processors handle transformations that don't fit the "column" model: restructuring the schema for a specific output format, dropping intermediate columns in bulk, or applying batch-wide operations.

## Overview

Each processor:

* Receives the complete batch DataFrame
* Applies its transformation
* Passes the result to the next processor (or to output)

Processors can run at three stages, determined by which callback methods they implement:

| Stage            | When it runs                                 | Callback method              | Use cases                                              |
| ---------------- | -------------------------------------------- | ---------------------------- | ------------------------------------------------------ |
| Pre-batch        | After seed columns, before dependent columns | `process_before_batch()`     | Transform seed data before other columns are generated |
| Post-batch       | After each batch completes                   | `process_after_batch()`      | Drop columns, transform schema per batch               |
| After generation | Once, on final dataset after all batches     | `process_after_generation()` | Deduplicate, aggregate statistics, final cleanup       |

> **Full Schema Available During Generation**
>
> Each batch carries the full dataset schema during generation. Post-batch schema changes such as column dropping only alter past batches, so all columns remain accessible to generators while building follow-up batches.

> **Row-count changes**
>
> Data Designer enforces row-count invariance in generation-time processors: `process_before_batch()` and `process_after_batch()`. Run row-filtering or expansion logic in `process_after_generation()`, which operates on the final dataset and supports row-count changes, or put the transform at a workflow boundary.

> **Resume after process\_after\_generation**
>
> `process_after_generation()` runs once on the entire generated dataset, not once per buffer. It loads the final parquet dataset, applies the processor, deletes the previous parquet files, and writes a new chunked result. Because this can change row counts, schemas, and row-group boundaries, Data Designer treats a dataset as terminal for resume after this stage has completed. Re-running with the same target is a no-op; extending the dataset requires a fresh run.

A processor can implement any combination of these callbacks. The built-in processors use `process_after_batch()` by default.

## Processor Types

### 🗑️ Drop Columns Processor

Removes specified columns from the output dataset. Dropped columns are saved separately in the `dropped-columns` directory for reference.

> **Tip**
>
> Dropping Columns is More Easily Achieved via `drop = True`
> The Drop Columns Processor is different from others in the sense that it does not need to be explicitly added: setting `drop = True` when configuring a column will accomplish the same.

**Configuration:**

```python
import data_designer.config as dd

processor = dd.DropColumnsProcessorConfig(
    name="remove_intermediate",
    column_names=["temp_calculation", "raw_input", "debug_info"],
)
```

**Behavior:**

* Columns specified in `column_names` are removed from the output
* Original values are preserved in a separate parquet file
* Missing columns produce a warning but don't fail the build
* Column configs are automatically marked with `drop=True` when this processor is added

**Use Cases:**

* Removing intermediate columns used only for LLM context
* Cleaning up debug or validation columns before final output
* Separating sensitive data from the main dataset

### 🔄 Schema Transform Processor

Creates an additional dataset with a transformed schema using Jinja2 templates. The output is written to a separate directory alongside the main dataset.

**Configuration:**

```python
import data_designer.config as dd

processor = dd.SchemaTransformProcessorConfig(
    name="chat_format",
    template={
        "messages": [
            {"role": "user", "content": "{{ question }}"},
            {"role": "assistant", "content": "{{ answer }}"},
        ],
        "metadata": "{{ category | upper }}",
    },
)
```

**Behavior:**

* Each key in `template` becomes a column in the transformed dataset
* Values are Jinja2 templates with access to all columns in the batch
* Complex structures (lists, nested dicts) are supported
* Output is saved to the `processors-outputs/{name}/` directory
* The original dataset passes through unchanged

**Template Capabilities:**

* **Variable substitution**: `{{ column_name }}`
* **Filters**: `{{ text | upper }}`, `{{ text | lower }}`, `{{ text | trim }}`
* **Nested structures**: Arbitrarily deep JSON structures
* **Lists**: `["{{ col1 }}", "{{ col2 }}"]`

**Use Cases:**

* Converting flat columns to chat message format
* Restructuring data for specific model training formats
* Creating derived views without modifying the source dataset

## Using Processors

Add processors to your configuration using the builder's `add_processor` method:

```python
import data_designer.config as dd

builder = dd.DataDesignerConfigBuilder()

# ... add columns ...

# Drop intermediate columns
builder.add_processor(
    dd.DropColumnsProcessorConfig(
        name="cleanup",
        column_names=["scratch_work", "raw_context"],
    )
)

# Transform to chat format
builder.add_processor(
    dd.SchemaTransformProcessorConfig(
        name="chat_format",
        template={
            "messages": [
                {"role": "user", "content": "{{ question }}"},
                {"role": "assistant", "content": "{{ answer }}"},
            ],
        },
    )
)
```

### Execution Order

Processors execute in the order they're added. Plan accordingly when one processor's output affects another.

## Processor Plugins

You can extend Data Designer with custom processors via the [plugin system](/plugins/overview). A processor plugin is a Python package that provides:

* A **config class** inheriting from `ProcessorConfig` with a `processor_type: Literal["your-type"]` discriminator
* An **implementation class** inheriting from `Processor` that overrides the desired callback methods
* A **`Plugin` instance** connecting the two

Once installed, plugin processors are automatically discovered and can be used with `add_processor()` like built-in processors.

```python
from my_processor_plugin.config import MyProcessorConfig

builder.add_processor(
    MyProcessorConfig(
        name="my_processor",
        # ... plugin-specific parameters ...
    )
)
```

**Entry point configuration** in `pyproject.toml`:

```toml
[project.entry-points."data_designer.plugins"]
my-processor = "my_plugin.plugin:my_processor_plugin"
```

See the [plugins overview](/plugins/overview) for the full guide on creating plugins.

## Configuration Parameters

### Common Parameters

| Parameter | Type | Description                                                  |
| --------- | ---- | ------------------------------------------------------------ |
| `name`    | str  | Identifier for the processor, used in output directory names |

### DropColumnsProcessorConfig

| Parameter      | Type       | Description                   |
| -------------- | ---------- | ----------------------------- |
| `column_names` | list\[str] | Columns to remove from output |

### SchemaTransformProcessorConfig

| Parameter  | Type            | Description                                                            |
| ---------- | --------------- | ---------------------------------------------------------------------- |
| `template` | dict\[str, Any] | Jinja2 template defining the output schema. Must be JSON-serializable. |