> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/datadesigner/llms.txt.
> For full documentation content, see https://docs.nvidia.com/nemo/datadesigner/llms-full.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/datadesigner/_mcp/server.

# Processors

Processors are transformations that modify your dataset before or after columns are generated. They run at different stages and can reshape, filter, or augment the data.

When to Use Processors
Processors handle transformations that don't fit the "column" model: restructuring the schema for a specific output format, dropping intermediate columns in bulk, or applying batch-wide operations.

## Overview

Each processor:

* Receives the complete batch DataFrame
* Applies its transformation
* Passes the result to the next processor (or to output)

Processors can run at three stages, determined by which callback methods they implement:

| Stage            | When it runs                                 | Callback method              | Use cases                                              |
| ---------------- | -------------------------------------------- | ---------------------------- | ------------------------------------------------------ |
| Pre-batch        | After seed columns, before dependent columns | `process_before_batch()`     | Transform seed data before other columns are generated |
| Post-batch       | After each batch completes                   | `process_after_batch()`      | Drop columns, transform schema per batch               |
| After generation | Once, on final dataset after all batches     | `process_after_generation()` | Deduplicate, aggregate statistics, final cleanup       |

Each batch carries the full dataset schema during generation. Post-batch schema changes such as column dropping only alter past batches, so all columns remain accessible to generators while building follow-up batches.

The async engine (default) enforces row-count invariance in `process_before_batch()` and `process_after_batch()` — a processor returning a different row count raises `DatasetGenerationError`. Run row-filtering or expansion logic in `process_after_generation()`, which operates on the final dataset and supports row-count changes. The legacy sync engine (opt-out via `DATA_DESIGNER_ASYNC_ENGINE=0`) is permissive about row-count changes at all stages.

`process_after_generation()` runs once on the entire generated dataset, not once per buffer. It loads the final parquet dataset, applies the processor, deletes the previous parquet files, and writes a new chunked result. Because this can change row counts, schemas, and row-group boundaries, Data Designer treats a dataset as terminal for resume after this stage has completed. Re-running with the same target is a no-op; extending the dataset requires a fresh run.

A processor can implement any combination of these callbacks. The built-in processors use `process_after_batch()` by default.

## Processor Types

### 🗑️ Drop Columns Processor

Removes specified columns from the output dataset. Dropped columns are saved separately in the `dropped-columns` directory for reference.

Dropping Columns is More Easily Achieved via `drop = True`
The Drop Columns Processor is different from others in the sense that it does not need to be explicitly added: setting `drop = True` when configuring a column will accomplish the same.

**Configuration:**

```python
import data_designer.config as dd

processor = dd.DropColumnsProcessorConfig(
    name="remove_intermediate",
    column_names=["temp_calculation", "raw_input", "debug_info"],
)
```

**Behavior:**

* Columns specified in `column_names` are removed from the output
* Original values are preserved in a separate parquet file
* Missing columns produce a warning but don't fail the build
* Column configs are automatically marked with `drop=True` when this processor is added

**Use Cases:**

* Removing intermediate columns used only for LLM context
* Cleaning up debug or validation columns before final output
* Separating sensitive data from the main dataset

### 🔄 Schema Transform Processor

Creates an additional dataset with a transformed schema using Jinja2 templates. The output is written to a separate directory alongside the main dataset.

**Configuration:**

```python
import data_designer.config as dd

processor = dd.SchemaTransformProcessorConfig(
    name="chat_format",
    template={
        "messages": [
            {"role": "user", "content": "{{ question }}"},
            {"role": "assistant", "content": "{{ answer }}"},
        ],
        "metadata": "{{ category | upper }}",
    },
)
```

**Behavior:**

* Each key in `template` becomes a column in the transformed dataset
* Values are Jinja2 templates with access to all columns in the batch
* Complex structures (lists, nested dicts) are supported
* Output is saved to the `processors-outputs/{name}/` directory
* The original dataset passes through unchanged

**Template Capabilities:**

* **Variable substitution**: `{{ column_name }}`
* **Filters**: `{{ text | upper }}`, `{{ text | lower }}`, `{{ text | trim }}`
* **Nested structures**: Arbitrarily deep JSON structures
* **Lists**: `["{{ col1 }}", "{{ col2 }}"]`

**Use Cases:**

* Converting flat columns to chat message format
* Restructuring data for specific model training formats
* Creating derived views without modifying the source dataset

## Using Processors

Add processors to your configuration using the builder's `add_processor` method:

```python
import data_designer.config as dd

builder = dd.DataDesignerConfigBuilder()

# ... add columns ...

# Drop intermediate columns
builder.add_processor(
    dd.DropColumnsProcessorConfig(
        name="cleanup",
        column_names=["scratch_work", "raw_context"],
    )
)

# Transform to chat format
builder.add_processor(
    dd.SchemaTransformProcessorConfig(
        name="chat_format",
        template={
            "messages": [
                {"role": "user", "content": "{{ question }}"},
                {"role": "assistant", "content": "{{ answer }}"},
            ],
        },
    )
)
```

### Execution Order

Processors execute in the order they're added. Plan accordingly when one processor's output affects another.

## Processor Plugins

You can extend Data Designer with custom processors via the [plugin system](/plugins/overview). A processor plugin is a Python package that provides:

* A **config class** inheriting from `ProcessorConfig` with a `processor_type: Literal["your-type"]` discriminator
* An **implementation class** inheriting from `Processor` that overrides the desired callback methods
* A **`Plugin` instance** connecting the two

Once installed, plugin processors are automatically discovered and can be used with `add_processor()` like built-in processors.

```python
from my_processor_plugin.config import MyProcessorConfig

builder.add_processor(
    MyProcessorConfig(
        name="my_processor",
        # ... plugin-specific parameters ...
    )
)
```

**Entry point configuration** in `pyproject.toml`:

```toml
[project.entry-points."data_designer.plugins"]
my-processor = "my_plugin.plugin:my_processor_plugin"
```

See the [plugins overview](/plugins/overview) for the full guide on creating plugins.

## Configuration Parameters

### Common Parameters

| Parameter | Type | Description                                                  |
| --------- | ---- | ------------------------------------------------------------ |
| `name`    | str  | Identifier for the processor, used in output directory names |

### DropColumnsProcessorConfig

| Parameter      | Type       | Description                   |
| -------------- | ---------- | ----------------------------- |
| `column_names` | list\[str] | Columns to remove from output |

### SchemaTransformProcessorConfig

| Parameter  | Type            | Description                                                            |
| ---------- | --------------- | ---------------------------------------------------------------------- |
| `template` | dict\[str, Any] | Jinja2 template defining the output schema. Must be JSON-serializable. |