> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/datadesigner/llms.txt.
> For full documentation content, see https://docs.nvidia.com/nemo/datadesigner/llms-full.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/datadesigner/_mcp/server.

# Inference Parameters

Inference parameters control how models generate responses during synthetic data generation. Data Designer provides three types of inference parameters: `ChatCompletionInferenceParams` for text/code/structured generation, `EmbeddingInferenceParams` for embedding generation, and `ImageInferenceParams` for image generation.

## Overview

When you create a `ModelConfig`, you can specify inference parameters to adjust model behavior. These parameters control aspects like randomness (temperature), diversity (top\_p), context size (max\_tokens), and more. Data Designer supports both static values and dynamic distribution-based sampling for certain parameters.

## Chat Completion Inference Parameters

The `ChatCompletionInferenceParams` class controls how models generate text completions (for text, code, and structured data generation). It provides fine-grained control over generation behavior and supports both static values and dynamic distribution-based sampling.

### Fields

| Field                   | Type                      | Required | Description                                                                                                             |
| ----------------------- | ------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------- |
| `temperature`           | `float` or `Distribution` | No       | Controls randomness in generation (0.0 to 2.0). Higher values = more creative/random                                    |
| `top_p`                 | `float` or `Distribution` | No       | Nucleus sampling parameter (0.0 to 1.0). Controls diversity by filtering low-probability tokens                         |
| `max_tokens`            | `int`                     | No       | Maximum number of tokens to generate in the response (≥ 1)                                                              |
| `max_parallel_requests` | `int`                     | No       | Maximum concurrent API requests to this model (default: 4, ≥ 1). See [Concurrency Control](#concurrency-control) below. |
| `timeout`               | `int`                     | No       | API request timeout in seconds (≥ 1)                                                                                    |
| `extra_body`            | `dict[str, Any]`          | No       | Additional parameters to include in the API request body                                                                |

Default Values
If `temperature`, `top_p`, or `max_tokens` are not provided, the model provider's default values will be used. Different providers and models may have different defaults.

Controlling Reasoning Effort for Reasoning Models
For reasoning models like Nemotron 3 Super (`nvidia/nemotron-3-super-120b-a12b`) and GPT-OSS (`gpt-oss-20b`, `gpt-oss-120b`), you can control the reasoning effort using the `extra_body` parameter:

```python
import data_designer.config as dd

# High reasoning effort (more thorough, slower)
inference_parameters = dd.ChatCompletionInferenceParams(
    extra_body={"reasoning_effort": "high"}
)

# Medium reasoning effort (balanced)
inference_parameters = dd.ChatCompletionInferenceParams(
    extra_body={"reasoning_effort": "medium"}
)

# Low reasoning effort (faster, less thorough)
inference_parameters = dd.ChatCompletionInferenceParams(
    extra_body={"reasoning_effort": "low"}
)
```

### Temperature and Top P Guidelines

* **Temperature**:
  * `0.0-0.3`: Highly deterministic, focused outputs (ideal for structured/reasoning tasks)
  * `0.4-0.7`: Balanced creativity and coherence (general purpose)
  * `0.8-1.0`: Creative, diverse outputs (ideal for creative writing)
  * `1.0+`: Highly random and experimental

* **Top P**:
  * `0.1-0.5`: Very focused, only most likely tokens
  * `0.6-0.9`: Balanced diversity
  * `0.95-1.0`: Maximum diversity, including less likely tokens

Adjusting Temperature and Top P Together
When tuning both parameters simultaneously, consider these combinations:

* **For deterministic/structured outputs**: Low temperature (`0.0-0.3`) + moderate-to-high top\_p (`0.8-0.95`)
  * The low temperature ensures focus, while top\_p allows some token diversity
* **For balanced generation**: Moderate temperature (`0.5-0.7`) + high top\_p (`0.9-0.95`)
  * This is a good starting point for most use cases
* **For creative outputs**: Higher temperature (`0.8-1.0`) + high top\_p (`0.95-1.0`)
  * Both parameters work together to maximize diversity

**Avoid**: Setting both very low (overly restrictive) or adjusting both dramatically at once. When experimenting, adjust one parameter at a time to understand its individual effect.

## Distribution-Based Inference Parameters

For `temperature` and `top_p` in `ChatCompletionInferenceParams`, you can specify distributions instead of fixed values. This allows Data Designer to sample different values for each generation request, introducing controlled variability into your synthetic data.

### Uniform Distribution

Samples values uniformly between a low and high bound:

```python
import data_designer.config as dd

inference_params = dd.ChatCompletionInferenceParams(
    temperature=dd.UniformDistribution(
        params=dd.UniformDistributionParams(low=0.7, high=1.0)
    ),
)
```

### Manual Distribution

Samples from a discrete set of values with optional weights:

```python
import data_designer.config as dd

# Equal probability for each value
inference_params = dd.ChatCompletionInferenceParams(
    temperature=dd.ManualDistribution(
        params=dd.ManualDistributionParams(values=[0.5, 0.7, 0.9])
    ),
)

# Weighted probabilities (normalized automatically)
inference_params = dd.ChatCompletionInferenceParams(
    top_p=dd.ManualDistribution(
        params=dd.ManualDistributionParams(
            values=[0.8, 0.9, 0.95],
            weights=[0.2, 0.5, 0.3]  # 20%, 50%, 30% probability
        )
    ),
)
```

## Concurrency Control

The `max_parallel_requests` parameter controls how many concurrent API calls Data Designer makes to a specific model. This directly impacts throughput and should be tuned to match your inference server's capacity.

Performance Tuning
For recommended values by deployment type (NVIDIA API Catalog, vLLM, OpenAI, NIMs) and detailed optimization strategies, see the [Architecture & Performance](/concepts/architecture-and-performance) guide.

## Embedding Inference Parameters

The `EmbeddingInferenceParams` class controls how models generate embeddings. This is used when working with embedding models for tasks like semantic search or similarity analysis.

### Fields

| Field                   | Type                         | Required | Description                                              |
| ----------------------- | ---------------------------- | -------- | -------------------------------------------------------- |
| `encoding_format`       | `Literal["float", "base64"]` | No       | Format of the embedding encoding (default: "float")      |
| `dimensions`            | `int`                        | No       | Number of dimensions for the embedding                   |
| `max_parallel_requests` | `int`                        | No       | Maximum concurrent API requests (default: 4, ≥ 1)        |
| `timeout`               | `int`                        | No       | API request timeout in seconds (≥ 1)                     |
| `extra_body`            | `dict[str, Any]`             | No       | Additional parameters to include in the API request body |

## Image Inference Parameters

The `ImageInferenceParams` class is used for image generation models, including both diffusion models (DALL·E, Stable Diffusion, Imagen) and autoregressive models (Gemini image, GPT image). Unlike text models, image-specific options are passed entirely via `extra_body`, since they vary significantly between providers.

### Fields

| Field                   | Type             | Required | Description                                                      |
| ----------------------- | ---------------- | -------- | ---------------------------------------------------------------- |
| `max_parallel_requests` | `int`            | No       | Maximum concurrent API requests (default: 4, ≥ 1)                |
| `timeout`               | `int`            | No       | API request timeout in seconds (≥ 1)                             |
| `extra_body`            | `dict[str, Any]` | No       | Model-specific image options (size, quality, aspect ratio, etc.) |

### Examples

```python
import data_designer.config as dd

# Autoregressive model (chat completions API, supports image context)
dd.ModelConfig(
    alias="image-model",
    model="black-forest-labs/flux.2-pro",
    provider="openrouter",
    inference_parameters=dd.ImageInferenceParams(
        extra_body={"height": 512, "width": 512}
    ),
)

# Diffusion model (e.g., DALL·E, Stable Diffusion)
dd.ModelConfig(
    alias="dalle",
    model="dall-e-3",
    provider="openai",
    inference_parameters=dd.ImageInferenceParams(
        extra_body={"size": "1024x1024", "quality": "hd"}
    ),
)
```

## See Also

* **[Default Model Settings](/concepts/models/default-model-settings)**: Pre-configured model settings included with Data Designer
* **[Custom Model Settings](/concepts/models/custom-model-settings)**: Learn how to create custom providers and model configurations
* **[Model Configurations](/concepts/models/model-configs)**: Learn about configuring model settings
* **[Model Providers](/concepts/models/model-providers)**: Learn about configuring model providers
* **[Architecture & Performance](/concepts/architecture-and-performance)**: Understanding separation of concerns and optimizing concurrency