data_designer.config.analysis.column_statistics
data_designer.config.analysis.column_statistics
NeMo Data Designerdata_designer.config.analysis.column_statistics
ColumnStatisticsT
DEFAULT_COLUMN_STATISTICS_MAP
Bases: str, enum.Enum
Bases: str, enum.Enum
Bases: pydantic.BaseModel, abc.ABC
Abstract base class for all column statistics types.
Serves as a container for computed statistics across different column types in Data-Designer-generated datasets. Subclasses hold column-specific statistical results and provide methods for formatting these results for display in reports.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Creates a formatted dictionary of statistics for display in reports.
Returns:
dict[str, str]
Dictionary mapping display labels to formatted statistic values.
Bases: data_designer.config.analysis.column_statistics.BaseColumnStatistics
Container for general statistics applicable to all column types.
Holds core statistical measures that apply universally across all column types, including null counts, unique values, and data type information. Serves as the base for more specialized column statistics classes that store additional column-specific metrics.
Parameters:
Name of the column being analyzed.
Total number of records in the column.
Number of null/missing values in the column.
Number of distinct values in the column. If a value is not hashable, it is converted to a string.
PyArrow data type of the column as a string.
Simplified human-readable data type label.
Discriminator field, always “general” for this statistics type.
Attributes:
Name of the column being analyzed.
Total number of records in the column.
Number of null/missing values in the column.
Number of distinct values in the column. If a value is not hashable, it is converted to a string.
PyArrow data type of the column as a string.
Simplified human-readable data type label.
Discriminator field, always “general” for this statistics type.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Bases: data_designer.config.analysis.column_statistics.GeneralColumnStatistics
Container for statistics on LLM-generated text columns.
Inherits general statistics plus token usage metrics specific to LLM text generation. Stores both prompt and completion token consumption data.
Parameters:
Mean number of output tokens generated per record.
Median number of output tokens generated per record.
Standard deviation of output tokens per record.
Mean number of input tokens used per record.
Median number of input tokens used per record.
Standard deviation of input tokens per record.
Discriminator field, always “llm-text” for this statistics type.
Attributes:
Mean number of output tokens generated per record.
Median number of output tokens generated per record.
Standard deviation of output tokens per record.
Mean number of input tokens used per record.
Median number of input tokens used per record.
Standard deviation of input tokens per record.
Discriminator field, always “llm-text” for this statistics type.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Bases: data_designer.config.analysis.column_statistics.LLMTextColumnStatistics
Container for statistics on LLM-generated code columns.
Inherits all token usage metrics from LLMTextColumnStatistics. Stores statistics from columns that generate code snippets in specific programming languages.
Parameters:
Discriminator field, always “llm-code” for this statistics type.
Attributes:
Discriminator field, always “llm-code” for this statistics type.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Bases: data_designer.config.analysis.column_statistics.LLMTextColumnStatistics
Container for statistics on LLM-generated structured JSON columns.
Inherits all token usage metrics from LLMTextColumnStatistics. Stores statistics from columns that generate structured data conforming to JSON schemas or Pydantic models.
Parameters:
Discriminator field, always “llm-structured” for this statistics type.
Attributes:
Discriminator field, always “llm-structured” for this statistics type.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Bases: data_designer.config.analysis.column_statistics.LLMTextColumnStatistics
Container for statistics on LLM-as-a-judge quality assessment columns.
Inherits all token usage metrics from LLMTextColumnStatistics. Stores statistics from columns that evaluate and score other generated content based on defined criteria.
Parameters:
Discriminator field, always “llm-judge” for this statistics type.
Attributes:
Discriminator field, always “llm-judge” for this statistics type.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Bases: data_designer.config.analysis.column_statistics.GeneralColumnStatistics
Container for statistics on sampler-generated columns.
Inherits general statistics plus sampler-specific information including the sampler type used and the empirical distribution of generated values. Stores both categorical and numerical distribution results.
Parameters:
Type of sampler used to generate this column (e.g., “uniform”, “category”, “gaussian”, “person”).
Classification of the column’s distribution (categorical, numerical, text, other, or unknown).
Empirical distribution statistics for the generated values. Can be CategoricalDistribution (for discrete values), NumericalDistribution (for continuous values), or MissingValue if distribution could not be computed.
Discriminator field, always “sampler” for this statistics type.
Attributes:
Type of sampler used to generate this column (e.g., “uniform”, “category”, “gaussian”, “person”).
Classification of the column’s distribution (categorical, numerical, text, other, or unknown).
Empirical distribution statistics for the generated values. Can be CategoricalDistribution (for discrete values), NumericalDistribution (for continuous values), or MissingValue if distribution could not be computed.
Discriminator field, always “sampler” for this statistics type.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Bases: data_designer.config.analysis.column_statistics.GeneralColumnStatistics
Container for statistics on columns sourced from seed datasets.
Inherits general statistics and stores statistics computed from columns that originate from existing data provided via the seed dataset functionality.
Parameters:
Discriminator field, always “seed-dataset” for this statistics type.
Attributes:
Discriminator field, always “seed-dataset” for this statistics type.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Bases: data_designer.config.analysis.column_statistics.GeneralColumnStatistics
Container for statistics on expression-based derived columns.
Inherits general statistics and stores statistics computed from columns that are derived from columns that are derived from Jinja2 expressions referencing other column values.
Parameters:
Discriminator field, always “expression” for this statistics type.
Attributes:
Discriminator field, always “expression” for this statistics type.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Bases: data_designer.config.analysis.column_statistics.GeneralColumnStatistics
Container for statistics on validation result columns.
Inherits general statistics plus validation-specific metrics including the count and percentage of records that passed validation. Stores results from validation logic (Python, SQL, local callable, or remote) executed against target columns.
Parameters:
Number of records that passed validation.
Discriminator field, always “validation” for this statistics type.
Attributes:
Number of records that passed validation.
Discriminator field, always “validation” for this statistics type.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Bases: pydantic.BaseModel
Container for categorical distribution histogram data.
Stores the computed frequency distribution of categorical values.
Parameters:
List of unique category values that appear in the data.
List of occurrence counts for each category.
Attributes:
List of unique category values that appear in the data.
List of occurrence counts for each category.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Ensure numerical values are Python objects rather than Numpy types.
Bases: pydantic.BaseModel
Container for computed categorical distribution statistics.
Parameters:
The category value that appears most frequently in the data.
The category value that appears least frequently in the data.
Complete frequency distribution showing all categories and their counts.
Attributes:
The category value that appears most frequently in the data.
The category value that appears least frequently in the data.
Complete frequency distribution showing all categories and their counts.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
Bases: pydantic.BaseModel
Container for computed numerical distribution statistics.
Parameters:
Minimum value in the distribution.
Maximum value in the distribution.
Arithmetic mean (average) of all values.
Standard deviation measuring the spread of values around the mean.
Median value of the distribution.
Attributes:
Minimum value in the distribution.
Maximum value in the distribution.
Arithmetic mean (average) of all values.
Standard deviation measuring the spread of values around the mean.
Median value of the distribution.
Initialization:
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.
self is explicitly positional-only to allow self as a field name.