data_designer.config.column_configs
data_designer.config.column_configs
NeMo Data Designerdata_designer.config.column_configs
Bases: str, enum.Enum
Strategy for custom column generation.
Initialization:
Initialize self. See help(type(self)) for accurate signature.
Bases: data_designer.config.base.SingleColumnConfig
Configuration for columns generated using built-in samplers.
Sampler columns provide efficient data generation for common data types and distributions. Supported samplers include UUID generation, datetime/timedelta sampling, person generation, category / subcategory sampling, and various statistical distributions (uniform, gaussian, binomial, poisson, scipy).
Parameters:
Type of sampler to use. Available types include: “uuid”, “category”, “subcategory”, “uniform”, “gaussian”, “bernoulli”, “bernoulli_mixture”, “binomial”, “poisson”, “scipy”, “person”, “person_from_faker”, “datetime”, “timedelta”.
Parameters specific to the chosen sampler type. Type varies based on the sampler_type
(e.g., CategorySamplerParams, UniformSamplerParams, PersonSamplerParams).
Optional dictionary for conditional parameters. The dict keys are the conditions that must be met (e.g., “age > 21”) for the conditional parameters to be used. The values of dict are the parameters to use when the condition is met.
Optional type conversion to apply after sampling. For numerical samplers,
must be one of “float”, “int”, or “str”. For datetime and timedelta samplers, accepts
a strftime format string (e.g., "%Y-%m-%d", "%m/%d/%Y %H:%M"). When omitted,
datetime/timedelta columns default to ISO-8601 format (e.g., 2024-01-15T09:30:00).
Inherited Attributes: name (required): Unique name of the column to be generated. drop: If True, generate this column but remove it from the final dataset.
!!! tip “Displaying available samplers and their parameters”
The config builder has an info attribute that can be used to display the
available samplers and their parameters:
Attributes:
Type of sampler to use. Available types include: “uuid”, “category”, “subcategory”, “uniform”, “gaussian”, “bernoulli”, “bernoulli_mixture”, “binomial”, “poisson”, “scipy”, “person”, “person_from_faker”, “datetime”, “timedelta”.
Parameters specific to the chosen sampler type. Type varies based on the sampler_type
(e.g., CategorySamplerParams, UniformSamplerParams, PersonSamplerParams).
Optional dictionary for conditional parameters. The dict keys are the conditions that must be met (e.g., “age > 21”) for the conditional parameters to be used. The values of dict are the parameters to use when the condition is met.
Optional type conversion to apply after sampling. For numerical samplers,
must be one of “float”, “int”, or “str”. For datetime and timedelta samplers, accepts
a strftime format string (e.g., "%Y-%m-%d", "%m/%d/%Y %H:%M"). When omitted,
datetime/timedelta columns default to ISO-8601 format (e.g., 2024-01-15T09:30:00).
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Inject sampler_type into params dict to enable discriminated union resolution.
This allows users to pass params as a simple dict without the sampler_type field, which will be automatically added based on the outer sampler_type field.
Bases: data_designer.config.base.SingleColumnConfig
Configuration for text generation columns using Large Language Models.
LLM text columns generate free-form text content using language models. Prompts support Jinja2 templating to reference values from other columns, enabling context-aware generation. The generated text can optionally include message traces capturing the full conversation history.
Parameters:
Prompt template for text generation. Supports Jinja2 syntax to reference other columns (e.g., “Write a story about {{ character_name }}”). Must be a valid Jinja2 template.
Alias of the model configuration to use for generation. Must match a model alias defined when initializing the DataDesignerConfigBuilder.
Optional system prompt to set model behavior and constraints.
Also supports Jinja2 templating. If provided, must be a valid Jinja2 template.
Do not put any output parsing instructions in the system prompt. Instead,
use the appropriate column type for the output you want to generate - e.g.,
LLMStructuredColumnConfig for structured output, LLMCodeColumnConfig for code.
Optional list of image contexts for multi-modal generation. Enables vision-capable models to generate text based on image inputs.
Optional alias of the tool configuration to use for MCP tool calls. Must match a tool alias defined when initializing the DataDesignerConfigBuilder. When provided, the model may call permitted tools during generation.
Specifies what trace information to capture in a \{column_name\}__trace
column. Options are:
TraceType.NONE (default): No trace is captured.TraceType.LAST_MESSAGE: Only the final assistant message is captured.TraceType.ALL_MESSAGES: Full conversation history (system/user/assistant/tool).If True, creates a \{column_name\}__reasoning_content column
containing only the reasoning_content from the final assistant response. This is
useful for models that expose chain-of-thought reasoning separately from the main
response. Defaults to False.
Inherited Attributes: name (required): Unique name of the column to be generated. drop: If True, generate this column but remove it from the final dataset. Attributes:
Prompt template for text generation. Supports Jinja2 syntax to reference other columns (e.g., “Write a story about {{ character_name }}”). Must be a valid Jinja2 template.
Alias of the model configuration to use for generation. Must match a model alias defined when initializing the DataDesignerConfigBuilder.
Optional system prompt to set model behavior and constraints.
Also supports Jinja2 templating. If provided, must be a valid Jinja2 template.
Do not put any output parsing instructions in the system prompt. Instead,
use the appropriate column type for the output you want to generate - e.g.,
LLMStructuredColumnConfig for structured output, LLMCodeColumnConfig for code.
Optional list of image contexts for multi-modal generation. Enables vision-capable models to generate text based on image inputs.
Optional alias of the tool configuration to use for MCP tool calls. Must match a tool alias defined when initializing the DataDesignerConfigBuilder. When provided, the model may call permitted tools during generation.
Specifies what trace information to capture in a \{column_name\}__trace
column. Options are:
TraceType.NONE (default): No trace is captured.TraceType.LAST_MESSAGE: Only the final assistant message is captured.TraceType.ALL_MESSAGES: Full conversation history (system/user/assistant/tool).If True, creates a \{column_name\}__reasoning_content column
containing only the reasoning_content from the final assistant response. This is
useful for models that expose chain-of-thought reasoning separately from the main
response. Defaults to False.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Get columns referenced in prompt templates and multi-modal context.
Returns:
Any
List of unique column names referenced in Jinja2 templates and multi-modal context configurations.
Returns side-effect columns that may be generated alongside the main column.
Side-effect columns include:
{name}__trace: Generated when with_trace is not TraceType.NONE on the column
config.{name}__reasoning_content: Generated when extract_reasoning_content=True.Returns:
Any
List of side-effect column names.
Validate that prompt and system_prompt are valid Jinja2 templates.
Returns:
typing_extensions.Self
The validated instance.
Raises:
If prompt or system_prompt contains invalid Jinja2 syntax.
Bases: data_designer.config.column_configs.LLMTextColumnConfig
Configuration for code generation columns using Large Language Models.
Extends LLMTextColumnConfig to generate code snippets in specific programming languages or SQL dialects. The generated code is automatically extracted from markdown code blocks for the specified language. Inherits all prompt templating capabilities from LLMTextColumnConfig.
Parameters:
Programming language or SQL dialect for code generation. Supported values include: “python”, “javascript”, “typescript”, “java”, “kotlin”, “go”, “rust”, “ruby”, “scala”, “swift”, “sql:sqlite”, “sql:postgres”, “sql:mysql”, “sql:tsql”, “sql:bigquery”, “sql:ansi”. See CodeLang enum for complete list.
Inherited Attributes:
name (required): Unique name of the column to be generated.
prompt (required): Prompt template for code generation (supports Jinja2).
model_alias (required): Alias of the model configuration to use.
system_prompt: Optional system prompt (supports Jinja2).
multi_modal_context: Optional image contexts for multi-modal generation.
tool_alias: Optional tool configuration alias for MCP tool calls.
with_trace: Specifies what trace information to capture in a {column_name}__trace
column. Options are TraceType.NONE (default), TraceType.LAST_MESSAGE, or
TraceType.ALL_MESSAGES.
extract_reasoning_content: If True, creates a {column_name}__reasoning_content
column containing the reasoning content from the final assistant response.
drop: If True, generate this column but remove it from the final dataset.
Attributes:
Programming language or SQL dialect for code generation. Supported values include: “python”, “javascript”, “typescript”, “java”, “kotlin”, “go”, “rust”, “ruby”, “scala”, “swift”, “sql:sqlite”, “sql:postgres”, “sql:mysql”, “sql:tsql”, “sql:bigquery”, “sql:ansi”. See CodeLang enum for complete list.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Bases: data_designer.config.column_configs.LLMTextColumnConfig
Configuration for structured JSON generation columns using Large Language Models.
Extends LLMTextColumnConfig to generate structured data conforming to a specified schema. Uses JSON schema or Pydantic models to define the expected output structure, enabling type-safe and validated structured output generation. Inherits prompt templating capabilities from LLMTextColumnConfig.
Parameters:
The schema defining the expected output structure. Can be either:
Inherited Attributes:
name (required): Unique name of the column to be generated.
prompt (required): Prompt template for structured generation (supports Jinja2).
model_alias (required): Alias of the model configuration to use.
system_prompt: Optional system prompt (supports Jinja2).
multi_modal_context: Optional image contexts for multi-modal generation.
tool_alias: Optional tool configuration alias for MCP tool calls.
with_trace: Specifies what trace information to capture in a {column_name}__trace
column. Options are TraceType.NONE (default), TraceType.LAST_MESSAGE, or
TraceType.ALL_MESSAGES.
extract_reasoning_content: If True, creates a {column_name}__reasoning_content
column containing the reasoning content from the final assistant response.
drop: If True, generate this column but remove it from the final dataset.
Attributes:
The schema defining the expected output structure. Can be either:
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Convert Pydantic model to JSON schema if needed.
Returns:
typing_extensions.Self
The validated instance with output_format as a JSON schema dict.
Bases: data_designer.config.base.ConfigBase
Configuration for a “score” in an LLM judge evaluation.
Defines a single scoring criterion with its possible values and descriptions. Multiple Score objects can be combined in an LLMJudgeColumnConfig to create multi-dimensional quality assessments.
Parameters:
A clear, concise name for this scoring dimension (e.g., “Relevance”, “Fluency”).
An informative and detailed assessment guide explaining how to evaluate this dimension. Should provide clear criteria for scoring.
Dictionary mapping score values to their descriptions. Keys can be integers (e.g., 1-5 scale) or strings (e.g., “Poor”, “Good”, “Excellent”). Values are descriptions explaining what each score level means.
Attributes:
A clear, concise name for this scoring dimension (e.g., “Relevance”, “Fluency”).
An informative and detailed assessment guide explaining how to evaluate this dimension. Should provide clear criteria for scoring.
Dictionary mapping score values to their descriptions. Keys can be integers (e.g., 1-5 scale) or strings (e.g., “Poor”, “Good”, “Excellent”). Values are descriptions explaining what each score level means.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Bases: data_designer.config.column_configs.LLMTextColumnConfig
Configuration for LLM-as-a-judge quality assessment and scoring columns.
Extends LLMTextColumnConfig to create judge columns that evaluate and score other generated content based on the defined criteria. Useful for quality assessment, preference ranking, and multi-dimensional evaluation of generated data. Inherits prompt templating capabilities from LLMTextColumnConfig.
Parameters:
List of Score objects defining the evaluation dimensions. Each score represents a different aspect to evaluate (e.g., accuracy, relevance, fluency). Must contain at least one score.
Inherited Attributes:
name (required): Unique name of the column to be generated.
prompt (required): Prompt template for the judge evaluation (supports Jinja2).
model_alias (required): Alias of the model configuration to use.
system_prompt: Optional system prompt (supports Jinja2).
multi_modal_context: Optional image contexts for multi-modal generation.
tool_alias: Optional tool configuration alias for MCP tool calls.
with_trace: Specifies what trace information to capture in a {column_name}__trace
column. Options are TraceType.NONE (default), TraceType.LAST_MESSAGE, or
TraceType.ALL_MESSAGES.
extract_reasoning_content: If True, creates a {column_name}__reasoning_content
column containing the reasoning content from the final assistant response.
drop: If True, generate this column but remove it from the final dataset.
Attributes:
List of Score objects defining the evaluation dimensions. Each score represents a different aspect to evaluate (e.g., accuracy, relevance, fluency). Must contain at least one score.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Bases: data_designer.config.base.SingleColumnConfig
Configuration for derived columns using Jinja2 expressions.
Expression columns compute values by evaluating Jinja2 templates that reference other columns. Useful for transformations, concatenations, conditional logic, and derived features without requiring LLM generation. The expression is evaluated row-by-row.
Parameters:
Jinja2 expression to evaluate. Can reference other column values using {{ column_name }} syntax. Supports filters, conditionals, and arithmetic. Must be a valid, non-empty Jinja2 template.
Data type to cast the result to. Must be one of “int”, “float”, “str”, or “bool”. Defaults to “str”. Type conversion is applied after expression evaluation.
Inherited Attributes: name (required): Unique name of the column to be generated. drop: If True, generate this column but remove it from the final dataset. Attributes:
Jinja2 expression to evaluate. Can reference other column values using {{ column_name }} syntax. Supports filters, conditionals, and arithmetic. Must be a valid, non-empty Jinja2 template.
Data type to cast the result to. Must be one of “int”, “float”, “str”, or “bool”. Defaults to “str”. Type conversion is applied after expression evaluation.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Returns the columns referenced in the expression template.
Validate that the expression is a valid, non-empty Jinja2 template.
Returns:
typing_extensions.Self
The validated instance.
Raises:
If expression is empty or contains invalid Jinja2 syntax.
Coerce skip.value to match dtype so skipped and computed rows share a type.
Bases: data_designer.config.base.SingleColumnConfig
Configuration for validation columns that validate existing columns.
Validation columns execute validation logic against specified target columns and return structured results indicating pass/fail status with validation details. Supports multiple validation strategies: code execution (Python/SQL), local callable functions (library only), and remote HTTP endpoints.
Parameters:
List of column names to validate. These columns are passed to the validator for validation. All target columns must exist in the dataset before validation runs.
The type of validator to use. Options:
Parameters specific to the validator type. Type varies by validator:
Number of records to process in each validation batch. Defaults to 10. Larger batches are more efficient but use more memory. Adjust based on validator complexity and available resources.
Inherited Attributes: name (required): Unique name of the column to be generated. drop: If True, generate this column but remove it from the final dataset. Attributes:
List of column names to validate. These columns are passed to the validator for validation. All target columns must exist in the dataset before validation runs.
The type of validator to use. Options:
Parameters specific to the validator type. Type varies by validator:
Number of records to process in each validation batch. Defaults to 10. Larger batches are more efficient but use more memory. Adjust based on validator complexity and available resources.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Returns the columns that need to be validated.
Inject validator_type into validator_params for discriminated union resolution.
Bases: data_designer.config.base.SingleColumnConfig
Configuration for columns sourced from seed datasets.
This config marks columns that come from seed data. It is typically created
automatically when calling with_seed_dataset() on the builder, rather than
being instantiated directly by users.
Inherited Attributes: name (required): Unique name of the column to be generated. drop: If True, generate this column but remove it from the final dataset.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Bases: data_designer.config.base.SingleColumnConfig
Configuration for embedding generation columns.
Embedding columns generate embeddings for text input using a specified model.
Parameters:
The column to generate embeddings for. The column could be a single text string or a list of text strings in stringified JSON format. If it is a list of text strings in stringified JSON format, the embeddings will be generated for each text string.
The model to use for embedding generation.
Inherited Attributes: name (required): Unique name of the column to be generated. drop: If True, generate this column but remove it from the final dataset. Attributes:
The column to generate embeddings for. The column could be a single text string or a list of text strings in stringified JSON format. If it is a list of text strings in stringified JSON format, the embeddings will be generated for each text string.
The model to use for embedding generation.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Bases: data_designer.config.base.SingleColumnConfig
Configuration for image generation columns.
Image columns generate images using either autoregressive or diffusion models. The API used is automatically determined based on the model name:
Parameters:
Prompt template for image generation. Supports Jinja2 templating to reference other columns (e.g., “Generate an image of a {{ character_name }}”). Must be a valid Jinja2 template.
The model to use for image generation.
Optional list of image contexts for multi-modal generation. Enables autoregressive multi-modal models to generate images based on image inputs. Only works with autoregressive models that support image-to-image generation.
Inherited Attributes: name (required): Unique name of the column to be generated. drop: If True, generate this column but remove it from the final dataset. Attributes:
Prompt template for image generation. Supports Jinja2 templating to reference other columns (e.g., “Generate an image of a {{ character_name }}”). Must be a valid Jinja2 template.
The model to use for image generation.
Optional list of image contexts for multi-modal generation. Enables autoregressive multi-modal models to generate images based on image inputs. Only works with autoregressive models that support image-to-image generation.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Get columns referenced in the prompt template and multi-modal context.
Returns:
Any
List of unique column names referenced in Jinja2 templates and multi-modal context configurations.
Validate that prompt is a valid Jinja2 template.
Returns:
typing_extensions.Self
The validated instance.
Raises:
If prompt contains invalid Jinja2 syntax.
Bases: data_designer.config.base.SingleColumnConfig
Configuration for custom user-defined column generators.
Custom columns allow users to provide their own generation logic via a callable function
decorated with @custom_column_generator. Two strategies are supported: cell_by_cell
(default, row-based) and full_column (batch-based with DataFrame access).
Parameters:
A callable decorated with @custom_column_generator.
“cell_by_cell” (row-based) or “full_column” (batch-based).
Optional typed configuration object (Pydantic BaseModel) passed as the second argument to the generator function.
Inherited Attributes: name (required): Unique name of the column to be generated. drop: If True, generate this column but remove it from the final dataset. Attributes:
A callable decorated with @custom_column_generator.
“cell_by_cell” (row-based) or “full_column” (batch-based).
Optional typed configuration object (Pydantic BaseModel) passed as the second argument to the generator function.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Returns the columns required for custom generation (from decorator metadata).
Returns additional columns created by this generator (from decorator metadata).
Returns model aliases for LLM access and health checks (from decorator metadata).
Returns the decorator-declared aliases so the startup health check pings every endpoint.