🏗️ Architecture & Performance

Data Designer is an orchestration framework that coordinates synthetic data generation workflows. It is a client of LLM inference servers—it does not host models itself.

This guide explains the architecture, execution model, and how to tune performance for your specific use case.

Separation of Concerns

┌─────────────────────────────────────┐          ┌─────────────────────────────────────┐
│         Data Designer               │          │       Inference Server(s)           │
│         (Orchestration)             │  HTTP    │       (LLM Hosting)                 │
│                                     │  ─────►  │                                     │
│  • Dataset workflow management      │          │  • Model weights and execution      │
│  • Column dependency resolution     │          │  • GPU allocation and scheduling    │
│  • Batching and parallelism         │          │  • Request queuing                  │
│  • Retry and error handling         │          │  • Token generation                 │
│  • Adaptive concurrency (AIMD)      │          │  • Rate limiting (optional)         │
│  • Data validation and quality      │          │                                     │
└─────────────────────────────────────┘          └─────────────────────────────────────┘
              ▲                                                    ▲
              │                                                    │
        Your workflow                                    Your infrastructure
         configuration                                    (or cloud API)

What Data Designer Does

• Orchestrates the generation workflow across multiple columns
• Resolves dependencies between columns (DAG-based execution)
• Batches work into manageable chunks (buffer_size)
• Parallelizes LLM calls within batches (max_parallel_requests)
• Adapts to rate limits automatically via AIMD concurrency control
• Handles errors with retries and early shutdown logic
• Validates generated data against schemas and constraints

What Data Designer Does NOT Do

• Host models: You must provide LLM endpoints
• Manage GPUs: Your inference server handles GPU allocation
• Scale inference: You must provision sufficient capacity
• Impose rate limits: Your server or API gateway sets rate limits (Data Designer reacts to them automatically)

Execution Model

The async engine processes datasets in row groups, with parallel operations across ready cells and independent columns.

How It Works

Step 1: Split into row groups

Your dataset is divided into row groups of buffer_size records. Row groups checkpoint independently so interrupted runs can resume from durable parquet files.

Step 2: Schedule ready work

Data Designer builds an execution graph from column dependencies. Cells and full-column tasks whose dependencies are satisfied can run concurrently, while order-dependent columns are protected by scheduler locks.

Example workflow:

Row group 1 (100 records)
│
├─► Column 1: category (Sampler)      ──── All 100 values generated
├─► Column 2: prompt (LLM Text)       ──── All 100 values generated
├─► Column 3: response (LLM Text)     ──── All 100 values generated
├─► Column 4: score (Expression)      ──── All 100 values computed
│
└─► Checkpoint row group to disk
    │
    ▼
Row group 2 (100 records)
    ...repeat...

Step 3: Generate cells in parallel

Within each column, cells are processed in parallel up to the configured limit:

Key Concepts

Concurrency Formula

At any moment, the number of concurrent LLM requests is:

1
concurrent_requests = min(
2
    buffer_size,                # Records in an active row group
3
    current_throttle_limit,     # AIMD-managed limit (≤ max_parallel_requests)
4
    remaining_cells_in_column   # Cells left to generate
5
)

max_parallel_requests sets the ceiling. The actual limit (current_throttle_limit) is managed at runtime by an AIMD (Additive Increase / Multiplicative Decrease) controller that reacts to rate-limit signals from the inference server:

• During optional startup ramp: when rampup_seconds is greater than 0, a new throttle domain starts at one concurrent request and increases linearly toward max_parallel_requests over that duration.
• On the first 429 in a burst: the limit is reduced by a configurable factor (default: 25% reduction) and a cooldown is applied. Further 429s from already in-flight requests in the same burst do not reduce the limit again — they release their permits and hold the limit steady.
• After consecutive successes: the limit increases by 1 (by default) until it reaches the ceiling or a stabilized rate-limit threshold.

This means Data Designer automatically finds the right concurrency level for your server without manual tuning.

Example: With buffer_size=100 and max_parallel_requests=32, Data Designer can send up to 32 requests in parallel. If rampup_seconds=30, it starts at one request and climbs linearly toward 32 over 30 seconds. If the server returns 429s, startup ramp stops, concurrency drops automatically (e.g., to 24, then 18), and normal AIMD recovery takes over once the server catches up.

Configuration Parameters

buffer_size (RunConfig)

Controls how many records are processed per row group.

1
import data_designer.config as dd
2
from data_designer.interface import DataDesigner
3
4
run_config = dd.RunConfig(buffer_size=2000)
5
6
designer = DataDesigner()
7
designer.set_run_config(run_config)

When to increase: High-capacity inference server, single-model workflows, memory not constrained

When to decrease: Memory-constrained environments, development/debugging, complex multi-model pipelines

Resuming Interrupted Runs

Long generation jobs can be resumed from checkpoints by passing resume to DataDesigner.create() or data-designer create --resume.

1
from data_designer.interface import DataDesigner, ResumeMode
2
3
designer = DataDesigner()
4
results = designer.create(
5
    config_builder,
6
    num_records=10_000,
7
    dataset_name="training_data",
8
    resume=ResumeMode.IF_POSSIBLE,
9
)

Resume modes:

• ResumeMode.NEVER (default): always start a fresh generation run. If the dataset directory already exists, Data Designer writes to a timestamped directory.
• ResumeMode.ALWAYS: resume the existing dataset directory. Raises if the checkpoint is incompatible or cannot be resumed safely.
• ResumeMode.IF_POSSIBLE: resume when the stored config fingerprint matches the current config; otherwise start a fresh timestamped run.

Data Designer stores the run configuration in metadata.json (buffer_size, target_num_records, config fingerprint) and builder_config.json. Resume scans completed batch_*.parquet row groups and reads parquet metadata for the row count actually persisted. That keeps resume crash-safe even if a run was interrupted between writing a row-group parquet and updating metadata, because the filesystem reflects the durable state even when metadata lags by a step.

Resume has a few important invariants:

• buffer_size must match the original run.
• num_records must be at least the original target; you may extend a run by requesting more records.
• Row counts must stay stable within a run. Put filtering, expansion, aggregation, or deduplication at workflow boundaries.
• Once process_after_generation() has run, the dataset is considered terminal for resume. Re-running with the same target returns the existing dataset; extending requires a fresh run.
• If a run crashed after every row group was written but before process_after_generation() could start, resume runs after-generation on the existing on-disk dataset (the parquet files are still clean) and marks it terminal afterwards. A crash during process_after_generation() still raises — the parquet files may have been partially rewritten and starting fresh is the only safe option.

The DatasetCreationResults returned by a resume invocation reflects the full dataset on disk for anything that reads the artifact directory (load_dataset, count_records, load_analysis, export, push_to_hub). Per-run observability — task_traces, model-usage logs, and telemetry events emitted during the call — is scoped to the resume invocation only; the original run’s in-memory traces are not persisted across process boundaries.

max_parallel_requests (InferenceParams)

Sets the maximum concurrent LLM API calls per model. This is the ceiling that the AIMD throttle controller can ramp up to — the actual concurrency at runtime may be lower if the server signals rate limits.

1
import data_designer.config as dd
2
3
model = dd.ModelConfig(
4
    alias="my-model",
5
    model="nvidia/nemotron-3-nano-30b-a3b",
6
    provider="nvidia",
7
    inference_parameters=dd.ChatCompletionInferenceParams(
8
        max_parallel_requests=8,
9
    ),
10
)

Default: 4

When to increase: Your inference backend has high throughput capacity, you’re using a cloud API with generous rate limits, or you’re running vLLM/TensorRT-LLM with multiple GPUs. With AIMD, setting an aggressively high value is safer than before — the system will self-correct downward if the server can’t keep up, and salvage rounds can reclaim transiently failed rows.

When to decrease: You want to cap resource usage to a known safe level, or you want more predictable/debuggable execution.

non_inference_max_parallel_workers (RunConfig)

Controls thread pool size for non-LLM operations (samplers, expressions, validators).

1
run_config = dd.RunConfig(non_inference_max_parallel_workers=8)
2
designer.set_run_config(run_config)

Default: 4

When to increase: Many CPU-bound columns (complex expressions, heavy sampling)

Adaptive Throttling (RunConfig)

Data Designer uses an AIMD (Additive Increase / Multiplicative Decrease) controller to automatically adjust concurrency per model based on rate-limit feedback from the inference server. The defaults work well for most workloads. Override them via ThrottleConfig only when you understand the trade-offs.

1
import data_designer.config as dd
2
from data_designer.interface import DataDesigner
3
4
run_config = dd.RunConfig(
5
    throttle=dd.ThrottleConfig(
6
        reduce_factor=0.75,       # Multiply limit by this on a 429 (default: 0.75)
7
        additive_increase=1,      # Add this many slots after success_window successes (default: 1)
8
        success_window=25,        # Consecutive successes before increasing (default: 25)
9
        cooldown_seconds=2.0,     # Pause after a 429 when no Retry-After header (default: 2.0)
10
        ceiling_overshoot=0.10,   # Probe 10% above observed server limit (default: 0.10)
11
        rampup_seconds=0.0,       # Optional startup ramp duration; 0 disables it (default: 0.0)
12
    ),
13
)
14
15
designer = DataDesigner()
16
designer.set_run_config(run_config)

Error Handling (RunConfig)

Control retry behavior and early shutdown for failed generations.

1
run_config = dd.RunConfig(
2
    max_conversation_restarts=5,           # Full conversation restarts (default: 5)
3
    max_conversation_correction_steps=0,   # In-conversation corrections (default: 0)
4
    disable_early_shutdown=False,          # Enable early shutdown (default)
5
    shutdown_error_rate=0.5,               # Shut down if >50% errors
6
    shutdown_error_window=10,              # Min tasks before error monitoring
7
)
8
designer.set_run_config(run_config)

When to adjust:

• Strict schemas: Increase max_conversation_restarts to 7, add max_conversation_correction_steps=2
• Debugging: Set disable_early_shutdown=True to see all errors
• Simple text: Reduce max_conversation_restarts to 3

Async Engine

The async engine is the execution path. It dispatches work at the cell level rather than the column level, so independent columns overlap in time and per-(provider, model) AIMD pools tune themselves independently. See the Async All the Way Down dev note for the full architecture.

Per-model timeouts drive every deadline

The inference_parameters.timeout field on a ModelConfig sets the per-request HTTP timeout. The same value also drives the sync→async bridge that custom columns use when they call model.generate(). There is no separate queue-wait deadline — waits scale with provider speed and AIMD’s adaptive concurrency. Slow self-hosted endpoints (e.g. large models on a single GPU) only need this one knob raised:

1
import data_designer.config as dd
2
3
config_builder.add_model_config(
4
    dd.ModelConfig(
5
        alias="slow-model",
6
        model="my/slow-model",
7
        provider="my-provider",
8
        inference_parameters=dd.ChatCompletionInferenceParams(
9
            timeout=600,
10
        ),
11
    )
12
)

Run outcomes

A run can finish with fewer records than requested when non-retryable errors drop rows. Inspect len(result.load_dataset()) to detect.

If the rate of non-retryable errors crosses RunConfig.shutdown_error_rate, generation stops early and raises DataDesignerEarlyShutdownError (a subclass of DataDesignerGenerationError). Catch it separately when a typed retry path is appropriate:

1
from data_designer.interface.errors import DataDesignerEarlyShutdownError
2
3
try:
4
    result = dd_instance.create(config_builder, num_records=1000)
5
except DataDesignerEarlyShutdownError:
6
    # e.g. retry against a different model alias
7
    ...

Common Problems

Tuning Workflow

1. Start with defaults for initial development — AIMD handles rate-limit adaptation automatically
2. Profile your workload: How many LLM columns? How many records? What models?
3. Identify bottleneck: Low GPU util → increase max_parallel_requests (AIMD will self-correct if you overshoot). Memory issues → decrease buffer_size. Long tails → tune retry settings.
4. Check throttle logs: Look for “concurrency reduced” / “concurrency increased” messages to understand whether rate limits are the bottleneck
5. Iterate: Make one change at a time, measure impact before next change

Related Documentation

• Deployment Options: Library vs. Microservice: Choosing between library and microservice
• Model Configuration: Complete model settings reference
• Inference Parameters: Detailed parameter reference