> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/datadesigner/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/datadesigner/_mcp/server.

# 🏗️ Architecture & Performance

Data Designer is an **orchestration framework** that coordinates synthetic data generation workflows. It is a **client** of LLM inference servers—it does not host models itself.

This guide explains the architecture, execution model, and how to tune performance for your specific use case.

***

## Separation of Concerns

```
┌─────────────────────────────────────┐          ┌─────────────────────────────────────┐
│         Data Designer               │          │       Inference Server(s)           │
│         (Orchestration)             │  HTTP    │       (LLM Hosting)                 │
│                                     │  ─────►  │                                     │
│  • Dataset workflow management      │          │  • Model weights and execution      │
│  • Column dependency resolution     │          │  • GPU allocation and scheduling    │
│  • Batching and parallelism         │          │  • Request queuing                  │
│  • Retry and error handling         │          │  • Token generation                 │
│  • Adaptive concurrency (AIMD)      │          │  • Rate limiting (optional)         │
│  • Data validation and quality      │          │                                     │
└─────────────────────────────────────┘          └─────────────────────────────────────┘
              ▲                                                    ▲
              │                                                    │
        Your workflow                                    Your infrastructure
         configuration                                    (or cloud API)
```

### What Data Designer Does

* **Orchestrates** the generation workflow across multiple columns
* **Resolves dependencies** between columns (DAG-based execution)
* **Batches** work into manageable chunks (`buffer_size`)
* **Parallelizes** LLM calls within batches (`max_parallel_requests`)
* **Adapts to rate limits** automatically via AIMD concurrency control
* **Handles errors** with retries and early shutdown logic
* **Validates** generated data against schemas and constraints

### What Data Designer Does NOT Do

* **Host models**: You must provide LLM endpoints
* **Manage GPUs**: Your inference server handles GPU allocation
* **Scale inference**: You must provision sufficient capacity
* **Impose rate limits**: Your server or API gateway sets rate limits (Data Designer *reacts* to them automatically)

***

## Execution Model

#### Async execution

Data Designer uses the **async engine**, which dispatches work at the cell level and overlaps independent columns. The configuration knobs documented below (`buffer_size`, `max_parallel_requests`, request-admission tuning, error handling) apply to that engine.

The async engine processes datasets in **row groups**, with **parallel** operations across ready cells and independent columns.

### How It Works

**Step 1: Split into row groups**

Your dataset is divided into row groups of `buffer_size` records. Row groups checkpoint independently so interrupted runs can resume from durable parquet files.

**Step 2: Schedule ready work**

Data Designer builds an execution graph from column dependencies. Cells and full-column tasks whose dependencies are satisfied can run concurrently, while order-dependent columns are protected by scheduler locks.

Example workflow:

```
Row group 1 (100 records)
│
├─► Column 1: category (Sampler)      ──── All 100 values generated
├─► Column 2: prompt (LLM Text)       ──── All 100 values generated
├─► Column 3: response (LLM Text)     ──── All 100 values generated
├─► Column 4: score (Expression)      ──── All 100 values computed
│
└─► Checkpoint row group to disk
    │
    ▼
Row group 2 (100 records)
    ...repeat...
```

**Step 3: Generate cells in parallel**

Within each column, cells are processed **in parallel** up to the configured limit:

| Column Type                          | Parallelism Control                                                            |
| ------------------------------------ | ------------------------------------------------------------------------------ |
| Sampler and other non-inference work | Engine-managed; `non_inference_max_parallel_workers` is not currently consumed |
| LLM (Text, Code, Structured, Judge)  | `max_parallel_requests`                                                        |
| Expression                           | Sequential (fast, CPU-bound)                                                   |

### Key Concepts

| Concept              | Description                                                                                                       |
| -------------------- | ----------------------------------------------------------------------------------------------------------------- |
| **Row groups**       | Records are split into checkpointable groups of `buffer_size`; multiple row groups can be in flight concurrently. |
| **Dependency graph** | Columns and cells become runnable when their upstream dependencies are complete.                                  |
| **Parallel cells**   | Individual cells are generated in parallel up to the configured limit.                                            |

### Concurrency Formula

For a given model, async request concurrency is bounded by:

```python
concurrent_requests = min(
    active_ready_model_cells,  # Ready cells across active row groups
    current_admission_limit,   # AIMD-managed limit (≤ max_parallel_requests)
    max_in_flight_tasks,       # Global scheduler task-lease ceiling
)
```

`active_ready_model_cells` depends on the column DAG and the active row-group horizon. Each active row group contributes up to `buffer_size` rows per ready model column. Other work may consume part of the global `max_in_flight_tasks` budget. `max_parallel_requests` sets the **per-model ceiling**. The actual limit (`current_admission_limit`) is managed at runtime by an AIMD (Additive Increase / Multiplicative Decrease) request-admission controller that reacts to rate-limit signals from the inference server:

* **During optional startup ramp**: when `startup_ramp_seconds` is greater than 0, a new request domain starts at one concurrent request and increases linearly toward `max_parallel_requests` over that duration.
* **On the first 429 in a burst**: the limit is reduced by a configurable factor (default: 25% reduction) and a cooldown is applied. Further 429s from already in-flight requests in the same burst do not reduce the limit again — they release their permits and hold the limit steady.
* **After consecutive successes**: the limit increases by 1 (by default) until it reaches the ceiling or a stabilized rate-limit threshold.

This means Data Designer automatically finds the right concurrency level for your server without manual tuning.

**Example**: With `buffer_size=100` and `max_parallel_requests=32`, Data Designer can send up to 32 requests in parallel. If `startup_ramp_seconds=30`, it starts at one request and climbs linearly toward 32 over 30 seconds. If the server returns 429s, startup ramp stops, concurrency drops automatically (e.g., to 24, then 18), and normal AIMD recovery takes over once the server catches up.

***

## Configuration Parameters

### Load `RunConfig` from YAML for one CLI run

`data-designer create` can load one local `.yaml` or `.yml` file with `--run-config` or `-c`. The YAML root maps directly to `RunConfig` fields; do not add a `run_config:` wrapper.

```yaml
buffer_size: 250
max_in_flight_tasks: 128
display_tui: false
progress_interval: 10.0
request_admission:
  multiplicative_decrease_factor: 0.75
  additive_increase_step: 1
  successes_until_increase: 25
  cooldown_seconds: 2.0
  startup_ramp_seconds: 30.0
```

```bash
data-designer create dataset.yaml --run-config run-config.yaml
# Equivalent short option; the explicit flag overrides display_tui in the YAML.
data-designer create dataset.yaml -c run-config.yaml --no-tui
```

Partial files are valid. The effective settings use this precedence, from lowest to highest:

1. The active `DataDesigner.run_config` baseline (the built-in defaults today)
2. Top-level fields explicitly supplied in the YAML
3. Explicit `--tui` or `--no-tui` flags

Supplying `request_admission` replaces that nested baseline object rather than deep-merging it; omitted nested fields use `RequestAdmissionTuningConfig` defaults. The file is validated through the same `RunConfig` schema as the Python API, including field constraints, unknown-field rejection, normalization, and deprecated-key compatibility.

`DATA_DESIGNER_ASYNC_TRACE=1` forces async tracing on even if the YAML contains `async_trace: false`. Also, `display_tui: true` requests the terminal UI but falls back to periodic log output when the process does not have a TTY.

For resume, use the same `buffer_size` and `preserve_dropped_columns` values as the interrupted run. Generated artifacts record selected resume metadata, but they do not persist the complete effective `RunConfig` or copy the source YAML. Keep the runtime file if you need to reproduce the remaining operational settings.

### `buffer_size` (RunConfig)

Controls how many records are processed per row group.

```python
import data_designer.config as dd
from data_designer.interface import DataDesigner

run_config = dd.RunConfig(buffer_size=2000)

designer = DataDesigner()
designer.set_run_config(run_config)
```

| Value                | Memory Usage | Throughput                 | Error Feedback |
| -------------------- | ------------ | -------------------------- | -------------- |
| **Low** (100-500)    | Lower        | May not saturate inference | Fast           |
| **Default** (1000)   | Moderate     | Good for most cases        | Moderate       |
| **High** (2000-5000) | Higher       | Better for deep pipelines  | Slower         |

**When to increase**: High-capacity inference server, single-model workflows, memory not constrained

**When to decrease**: Memory-constrained environments, development/debugging, complex multi-model pipelines

***

### `max_concurrent_row_groups` (RunConfig)

Sets the fixed hard cap on async row groups that may be active at once. The default is `3`; each row group contains up to `buffer_size` records.

```python
run_config = dd.RunConfig(
    buffer_size=1_000,
    max_concurrent_row_groups=8,
)
designer.set_run_config(run_config)
```

A wider horizon can expose more ready work to high-capacity endpoints, but it retains more row data in memory and may delay checkpoints. It does not adapt during a run or replace `max_parallel_requests`, which remains the per-model request ceiling. Treat this as an expert override: benchmark before and after changing it.

***

## Resuming Interrupted Runs

Long generation jobs can be resumed from checkpoints by passing `resume` to `DataDesigner.create()` or `data-designer create --resume`.

```python
from data_designer.interface import DataDesigner, ResumeMode

designer = DataDesigner()
results = designer.create(
    config_builder,
    num_records=10_000,
    dataset_name="training_data",
    resume=ResumeMode.IF_POSSIBLE,
)
```

Resume modes:

* `ResumeMode.NEVER` (default): always start a fresh generation run. If the dataset directory already exists, Data Designer writes to a timestamped directory.
* `ResumeMode.ALWAYS`: resume the existing dataset directory. Raises if the checkpoint is incompatible or cannot be resumed safely.
* `ResumeMode.IF_POSSIBLE`: resume when the stored config fingerprint matches the current config; otherwise start a fresh timestamped run.

Data Designer does not store the complete effective `RunConfig`. `metadata.json` records selected resume identity values, including `buffer_size`, `preserve_dropped_columns`, the target record count, and dataset-config identity fields; `builder_config.json` stores the dataset builder configuration. Resume scans completed `batch_*.parquet` row groups and reads parquet metadata for the row count actually persisted. That keeps resume crash-safe even if a run was interrupted between writing a row-group parquet and updating metadata, because the filesystem reflects the durable state even when metadata lags by a step.

Resume has a few important invariants:

* `buffer_size` must match the original run.
* `preserve_dropped_columns` must match the original run.
* `num_records` must be at least the original target; you may extend a run by requesting more records.
* Row counts must stay stable within a run. Put filtering, expansion, aggregation, or deduplication at workflow boundaries.
* Once `process_after_generation()` has run, the dataset is considered terminal for resume. Re-running with the same target returns the existing dataset; extending requires a fresh run.
* If a run crashed after every row group was written but before `process_after_generation()` could start, resume runs after-generation on the existing on-disk dataset (the parquet files are still clean) and marks it terminal afterwards. A crash *during* `process_after_generation()` still raises — the parquet files may have been partially rewritten and starting fresh is the only safe option.

The `DatasetCreationResults` returned by a resume invocation reflects the full dataset on disk for anything that reads the artifact directory (`load_dataset`, `count_records`, `load_analysis`, `export`, `push_to_hub`). Per-run observability — `task_traces`, model-usage logs, and telemetry events emitted during the call — is scoped to the resume invocation only; the original run's in-memory traces are not persisted across process boundaries.

Only resume datasets from trusted artifact directories. Resume reads local `metadata.json`, `builder_config.json`, and parquet files to determine checkpoint state.

***

### `max_parallel_requests` (InferenceParams)

Sets the **maximum** concurrent LLM API calls **per model**. This is the ceiling that the AIMD request-admission controller can ramp up to — the actual concurrency at runtime may be lower if the server signals rate limits.

```python
import data_designer.config as dd

model = dd.ModelConfig(
    alias="my-model",
    model="nvidia/nemotron-3-nano-30b-a3b",
    provider="nvidia",
    inference_parameters=dd.ChatCompletionInferenceParams(
        max_parallel_requests=8,
    ),
)
```

**Default**: 4

**When to increase**: Your inference backend has high throughput capacity, you're using a cloud API with generous rate limits, or you're running vLLM/TensorRT-LLM with multiple GPUs. With AIMD, setting an aggressively high value is safer than before — the system will self-correct downward if the server can't keep up, and salvage rounds can reclaim transiently failed rows.

**When to decrease**: You want to cap resource usage to a known safe level, or you want more predictable/debuggable execution.

Finding the optimal value
The right value depends on your inference stack and model. Self-hosted vLLM servers can often handle values as high as 256, 512, or even 1024 depending on your hardware.

With AIMD, a practical approach is to set `max_parallel_requests` to the **upper bound** you're comfortable with and let request admission find the sustainable level automatically. If you see frequent 429 → recovery cycles in the logs, your ceiling is above the server's true capacity but the system is handling it. If you never see any request-admission activity, you may have room to increase the ceiling further.

**Benchmark approach**: Run a small dataset (e.g., 100 records) with increasing `max_parallel_requests` values (4 → 8 → 16 → 32 → ...) and measure generation time. Stop increasing when the runtime stops decreasing—that's when your inference server is saturated.

***

### `non_inference_max_parallel_workers` (RunConfig)

This field is declared for non-LLM worker concurrency, but the production engine does not currently consume it.

```python
run_config = dd.RunConfig(non_inference_max_parallel_workers=8)
designer.set_run_config(run_config)
```

**Default**: 4

Changing or loading this value does not change execution today.

***

### Adaptive Request Admission (RunConfig)

Data Designer uses an AIMD (Additive Increase / Multiplicative Decrease) controller to automatically adjust concurrency per model based on rate-limit feedback from the inference server. The defaults work well for most workloads. Override the recovery behavior through `request_admission` only when you understand the trade-offs.

```python
import data_designer.config as dd
from data_designer.interface import DataDesigner

run_config = dd.RunConfig(
    request_admission=dd.RequestAdmissionTuningConfig(
        multiplicative_decrease_factor=0.75,  # Multiply the limit on a 429
        additive_increase_step=1,             # Add slots after each recovery window
        successes_until_increase=25,          # Successful releases per recovery window
        cooldown_seconds=2.0,                 # Fallback when Retry-After is absent
        startup_ramp_seconds=0.0,              # 0 disables the startup ramp
    ),
)

designer = DataDesigner()
designer.set_run_config(run_config)
```

| Parameter                        | Default | Effect                                                                                                                                                                  |
| -------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `multiplicative_decrease_factor` | 0.75    | How aggressively to cut concurrency on a 429. Lower = more aggressive.                                                                                                  |
| `additive_increase_step`         | 1       | Slots added per recovery step. Higher = faster ramp-up, but riskier.                                                                                                    |
| `successes_until_increase`       | 25      | Successful request releases required before each increase step.                                                                                                         |
| `cooldown_seconds`               | 2.0     | Pause duration after a 429 (used when the server doesn't send `Retry-After`).                                                                                           |
| `startup_ramp_seconds`           | 0.0     | Optional startup ramp duration. When greater than 0, domains start at one concurrent request and linearly climb to the configured ceiling unless a 429 aborts the ramp. |

How it works in practice
When a model endpoint returns HTTP 429, the controller reduces the concurrency limit for that model and pauses briefly. After enough successful releases, it begins ramping back up. If the server rate-limits again, the limit is reduced again and recovery resumes once the server catches up.

You can observe this in the logs — look for messages like `concurrency reduced from X → Y` and `concurrency increased from X → Y`.

***

### Error Handling (RunConfig)

Control retry behavior and early shutdown for failed generations.

```python
run_config = dd.RunConfig(
    max_conversation_restarts=5,           # Full conversation restarts (default: 5)
    max_conversation_correction_steps=0,   # In-conversation corrections (default: 0)
    disable_early_shutdown=False,          # Enable early shutdown (default)
    shutdown_error_rate=0.5,               # Shut down if >50% errors
    shutdown_error_window=10,              # Min tasks before error monitoring
)
designer.set_run_config(run_config)
```

**When to adjust**:

* **Strict schemas**: Increase `max_conversation_restarts` to 7, add `max_conversation_correction_steps=2`
* **Debugging**: Set `disable_early_shutdown=True` to see all errors
* **Simple text**: Reduce `max_conversation_restarts` to 3

***

## Async Engine

The async engine is the execution path. It dispatches work at the cell level rather than the column level, so independent columns overlap in time and per-(provider, model) AIMD pools tune themselves independently. See the [Async All the Way Down](/dev-notes/async-all-the-way-down) dev note for the full architecture.

### Per-model timeouts drive every deadline

The `inference_parameters.timeout` field on a `ModelConfig` sets the per-request HTTP timeout. The same value also drives the sync→async bridge that custom columns use when they call `model.generate()`. There is no separate queue-wait deadline — waits scale with provider speed and AIMD's adaptive concurrency. Slow self-hosted endpoints (e.g. large models on a single GPU) only need this one knob raised:

```python
import data_designer.config as dd

config_builder.add_model_config(
    dd.ModelConfig(
        alias="slow-model",
        model="my/slow-model",
        provider="my-provider",
        inference_parameters=dd.ChatCompletionInferenceParams(
            timeout=600,
        ),
    )
)
```

### Run outcomes

A run can finish with fewer records than requested when non-retryable errors drop rows. Inspect `len(result.load_dataset())` to detect.

If the rate of non-retryable errors crosses `RunConfig.shutdown_error_rate`, generation stops early and raises `DataDesignerEarlyShutdownError` (a subclass of `DataDesignerGenerationError`). Catch it separately when a typed retry path is appropriate:

```python
from data_designer.interface.errors import DataDesignerEarlyShutdownError

try:
    result = dd_instance.create(config_builder, num_records=1000)
except DataDesignerEarlyShutdownError:
    # e.g. retry against a different model alias
    ...
```

## Local OpenTelemetry Metrics

`DataDesigner.create()` starts or reuses a process-wide OpenTelemetry Prometheus endpoint on `http://127.0.0.1:9464/metrics` by default. Use `RunConfig` to disable metrics for an invocation or choose another port:

```python
import data_designer.config as dd
from data_designer.interface import DataDesigner

designer = DataDesigner()
designer.set_run_config(dd.RunConfig(otel_metrics_port=None))

# Or keep metrics enabled on a different loopback port.
designer.set_run_config(dd.RunConfig(otel_metrics_port=9465))
```

Configure a Prometheus server on the same host to scrape the endpoint:

```yaml
scrape_configs:
  - job_name: data-designer
    metrics_path: /metrics
    static_configs:
      - targets: ["127.0.0.1:9464"]
```

The endpoint exposes seven instruments. Prometheus derives throughput and request-completion rates from their counters and histogram counts.

| Instrument                           | Type          | Meaning                                                                                                                                                                                                             |
| ------------------------------------ | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `data_designer.create.duration`      | Histogram     | Create-job duration in seconds, with failure-only `error.type`.                                                                                                                                                     |
| `data_designer.dataset.records`      | Counter       | Checkpointed generated and dropped records, distinguished by `record.result`; derive record throughput from this counter.                                                                                           |
| `data_designer.dataset.progress`     | Gauge         | Fraction of scheduled records processed by the current create job, or across all active jobs when creates overlap. The most recently finished value remains visible until the next job starts.                      |
| `data_designer.scheduler.events`     | Counter       | Job start/completion, non-retryable task errors, worker spawn failures, deferred retries, and task cancellations, identified by `event.name` with bounded `outcome` and failure-only `error.type` where applicable. |
| `data_designer.model.request.active` | UpDownCounter | Model requests currently in progress, identified by `gen_ai.operation.name`, `gen_ai.provider.name`, and `gen_ai.request.model`.                                                                                    |
| `gen_ai.client.operation.duration`   | Histogram     | Completed model-request duration in seconds, identified by `gen_ai.operation.name`, `gen_ai.provider.name`, and `gen_ai.request.model`, with failure-only `error.type`.                                             |
| `data_designer.log.records`          | Counter       | Data Designer log-record counts by normalized `log.severity`.                                                                                                                                                       |

The listener and cumulative metrics remain available until process shutdown so a scrape can collect results after a job finishes. A later enabled job can rebind an idle listener to a different configured port without resetting those metrics. Concurrent jobs that request different ports share the active listener and emit a warning instead of replacing it.

Generated and dropped record counts advance when a row group is durably checkpointed, so `data_designer.dataset.progress` moves in `buffer_size`-sized steps. Use `data_designer.model.request.active` and the `gen_ai.client.operation.duration` histogram count for live request activity between checkpoints.

Setting `otel_metrics_port=None` records nothing for that invocation. If an earlier enabled job already started the process listener, disabling a later job does not stop the listener or remove the earlier cumulative metrics.

The endpoint binds only to loopback and has no authentication. It is intended for a Prometheus server running on the same host, not for remote exposure.

Prometheus pull is metrics-only: `data_designer.log.records` counts safe log metadata, not raw log bodies. Existing raw logs continue to use the configured stdout and file handlers and should be collected from those outputs. The Python Prometheus exporter does not support multiprocessing, so this endpoint is unsupported for multiprocessing-based collection.

## Common Problems

| Problem                            | Symptom                                           | Solution                                                                                                                                                                                                                                     |
| ---------------------------------- | ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Low throughput**                 | Low GPU utilization                               | Increase `max_parallel_requests` and/or `buffer_size`. If request admission has reduced concurrency after earlier 429s (check logs for "concurrency reduced" messages), the server may need more capacity or you can wait for AIMD recovery. |
| **Frequent 429 → recovery cycles** | Logs show repeated concurrency drops and ramp-ups | The `max_parallel_requests` ceiling is above the server's sustained capacity. This is handled automatically, but you can lower the ceiling to reduce the sawtooth or tune `multiplicative_decrease_factor` / `successes_until_increase`.     |
| **Long tail of slow generations**  | Most records fast, few very slow                  | Reduce `max_conversation_restarts`, simplify schemas, improve prompts                                                                                                                                                                        |
| **Multi-model idle periods**       | One model busy, others idle                       | Reduce `buffer_size` for faster cycling, or consolidate models                                                                                                                                                                               |
| **Memory errors**                  | OOM crashes                                       | Reduce `buffer_size` and `max_parallel_requests`                                                                                                                                                                                             |
| **Too many errors**                | Generation fails frequently                       | Check prompts/schemas; adjust `shutdown_error_rate` or disable early shutdown for debugging                                                                                                                                                  |

***

## Tuning Workflow

1. **Start with defaults** for initial development — AIMD handles rate-limit adaptation automatically
2. **Profile your workload**: How many LLM columns? How many records? What models?
3. **Identify bottleneck**: Low GPU util → increase `max_parallel_requests` (AIMD will self-correct if you overshoot). Memory issues → decrease `buffer_size`. Long tails → tune retry settings.
4. **Check request-admission logs**: Look for "concurrency reduced" / "concurrency increased" messages to understand whether rate limits are the bottleneck
5. **Iterate**: Make one change at a time, measure impact before next change

***

## Related Documentation

* [Deployment Options: Library vs. Microservice](/concepts/deployment-options): Choosing between library and microservice
* [Model Configuration](/concepts/models/model-configs): Complete model settings reference
* [Inference Parameters](/concepts/models/inference-parameters): Detailed parameter reference