Tutorials

Providing Images as Context

View as Markdown
Run in Google Colab

🎨 Data Designer Tutorial: Providing Images as Context for Vision-Based Data Generation

📚 What you'll learn

This notebook demonstrates how to provide images as context to generate text descriptions using vision-language models. The same multi_modal_context field can also carry audio or video context when the selected model supports those modalities.

  • ✨ Visual Document Processing: Converting images to chat-ready format for model consumption
  • 🔍 Vision-Language Generation: Using vision models to generate detailed summaries from images
  • 🧩 Media Context Pattern: Understanding how ImageContext, AudioContext, and VideoContext fit into the same configuration field

If this is your first time using Data Designer, we recommend starting with the first notebook in this tutorial series.

📦 Import Data Designer

  • data_designer.config provides access to the configuration API.

  • DataDesigner is the main interface for data generation.

Python
1# Standard library imports
2import base64
3import io
4import os
5import uuid
6 
7# Third-party imports
8import pandas as pd
9import rich
10from datasets import load_dataset
11from IPython.display import display
12from rich.panel import Panel
13 
14# Data Designer imports
15import data_designer.config as dd
16from data_designer.interface import DataDesigner
17

⚙️ Initialize the Data Designer interface

  • DataDesigner is the main object responsible for managing the data generation process.

  • When initialized without arguments, the default model providers are used.

Python
1data_designer = DataDesigner()
2

🏗️ Initialize the Data Designer Config Builder

  • The Data Designer config defines the dataset schema and generation process.

  • The config builder provides an intuitive interface for building this configuration.

  • When initialized without arguments, the default model configurations are used.

Python
1config_builder = dd.DataDesignerConfigBuilder()
2if os.environ.get("DATA_DESIGNER_SKIP_NVIDIA_VISION_HEALTH_CHECK"):
3    for model_config in config_builder.model_configs:
4        if model_config.alias == "nvidia-vision":
5            model_config.skip_health_check = True
6            break
7

🌱 Seed Dataset Creation

In this section, we'll prepare our visual documents as a seed dataset for summarization:

  • Loading Visual Documents: We use a small pets image dataset containing labeled images
  • Image Processing: Convert images to base64 format for vision model consumption
  • Metadata Extraction: Preserve relevant image information (label, etc.)

The seed dataset will be used to generate detailed text descriptions of each image.

Python
1# Dataset processing configuration
2IMG_COUNT = 512  # Number of images to process
3BASE64_IMAGE_HEIGHT = 512  # Standardized height for model input
4 
5# Load the pets dataset (train split, ~23 MB total)
6img_dataset_cfg = {"path": "rokmr/pets", "split": "train"}
7
Python
1def resize_image(image, height: int):
2    """
3    Resize image while maintaining aspect ratio.
4 
5    Args:
6        image: PIL Image object
7        height: Target height in pixels
8 
9    Returns:
10        Resized PIL Image object
11    """
12    original_width, original_height = image.size
13    width = int(original_width * (height / original_height))
14    return image.resize((width, height))
15 
16 
17def convert_image_to_chat_format(record, height: int) -> dict:
18    """
19    Convert PIL image to base64 format for chat template usage.
20 
21    Args:
22        record: Dataset record containing image and metadata
23        height: Target height for image resizing
24 
25    Returns:
26        Updated record with base64_image and uuid fields
27    """
28    image = resize_image(record["image"], height)
29 
30    img_buffer = io.BytesIO()
31    image.save(img_buffer, format="PNG")
32    byte_data = img_buffer.getvalue()
33    base64_encoded_data = base64.b64encode(byte_data)
34    base64_string = base64_encoded_data.decode("utf-8")
35 
36    return record | {"base64_image": base64_string, "uuid": str(uuid.uuid4())}
37
Python
1# Load and process the image dataset
2print("📥 Loading and processing images...")
3 
4img_dataset = load_dataset(**img_dataset_cfg).map(
5    convert_image_to_chat_format, fn_kwargs={"height": BASE64_IMAGE_HEIGHT}
6)
7img_dataset = pd.DataFrame(img_dataset[:IMG_COUNT])
8 
9print(f"✅ Loaded {len(img_dataset)} images with columns: {list(img_dataset.columns)}")
10
Output
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
[17:58:28] [WARNING] Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
📥 Loading and processing images...
README.md: 0.00B [00:00, ?B/s]
dataset_infos.json: 0.00B [00:00, ?B/s]
data/train.zip:   0%|          | 0.00/20.4M [00:00<?, ?B/s]
data/test.zip:   0%|          | 0.00/3.29M [00:00<?, ?B/s]
Generating train split:   0%|          | 0/900 [00:00<?, ? examples/s]
Generating test split:   0%|          | 0/150 [00:00<?, ? examples/s]
Map:   0%|          | 0/900 [00:00<?, ? examples/s]
✅ Loaded 512 images with columns: ['image', 'label', 'base64_image', 'uuid']
Python
1img_dataset.head()
2
Output
image label base64_image uuid
0 <PIL.JpegImagePlugin.JpegImageFile image mode=... 0 iVBORw0KGgoAAAANSUhEUgAAAeQAAAIACAIAAADc8YinAA... 1c12ac17-71d5-4fc2-884a-efd928154da6
1 <PIL.JpegImagePlugin.JpegImageFile image mode=... 0 iVBORw0KGgoAAAANSUhEUgAAAiQAAAIACAIAAAA9rOAHAA... fe1df0c0-0857-4335-8fa2-05a19e16f103
2 <PIL.JpegImagePlugin.JpegImageFile image mode=... 0 iVBORw0KGgoAAAANSUhEUgAAAqoAAAIACAIAAADFYNm1AA... 30748a1e-31bb-42fb-8ea8-6db1ac698825
3 <PIL.JpegImagePlugin.JpegImageFile image mode=... 0 iVBORw0KGgoAAAANSUhEUgAAAwAAAAIACAIAAAC6lJxtAA... c10bdd68-09d7-4abd-92f6-4713817e2677
4 <PIL.PngImagePlugin.PngImageFile image mode=RG... 0 iVBORw0KGgoAAAANSUhEUgAAAqoAAAIACAIAAADFYNm1AA... 01bf9e06-762b-4397-a773-86c6587c9f7a
Python
1# Add the seed dataset containing our processed images
2df_seed = pd.DataFrame(img_dataset)[["uuid", "label", "base64_image"]]
3config_builder.with_seed_dataset(dd.DataFrameSeedSource(df=df_seed))
4
Output
DataDesignerConfigBuilder(
    seed_dataset: df seed
)

🧩 Media context and model capabilities

multi_modal_context accepts media context descriptors such as ImageContext, AudioContext, and VideoContext. Data Designer reads the referenced seed columns and serializes them for the model request, but the selected model still determines which modalities are valid.

This notebook uses image context only because image-capable VLMs are broadly available. Before combining image, audio, and video in one column, choose a model alias backed by an omni or otherwise modality-compatible model, and check that the provider accepts every context type you send.

For base64 seed columns, store the raw base64 payload without a data:<media-type>;base64, prefix and specify the media format on the context object:

media_context = [
    dd.ImageContext(
        column_name="image_base64",
        data_type=dd.ModalityDataType.BASE64,
        image_format=dd.ImageFormat.PNG,
    ),
    dd.AudioContext(
        column_name="audio_base64",
        data_type=dd.ModalityDataType.BASE64,
        audio_format=dd.AudioFormat.MP3,
    ),
    dd.VideoContext(
        column_name="video_base64",
        data_type=dd.ModalityDataType.BASE64,
        video_format=dd.VideoFormat.MP4,
    ),
]

URL-backed media can use data_type=dd.ModalityDataType.URL, subject to the provider's URL support and file-size limits. Local audio/video paths require explicit URL mode and require the model endpoint to have filesystem access to the same paths, typically a colocated vLLM server configured for local media access.

Python
1# Add a column to generate detailed image descriptions
2config_builder.add_column(
3    dd.LLMTextColumnConfig(
4        name="description",
5        model_alias="nvidia-vision",
6        prompt=(
7            "Provide a detailed description of the content in this image in Markdown format. "
8            "Describe the main subject, background, colors, and any notable details."
9        ),
10        multi_modal_context=[dd.ImageContext(column_name="base64_image")],
11    )
12)
13 
14data_designer.validate(config_builder)
15
Output
[18:00:05] [INFO] ✅ Validation passed

🔁 Iteration is key – preview the dataset!

  1. Use the preview method to generate a sample of records quickly.

  2. Inspect the results for quality and format issues.

  3. Adjust column configurations, prompts, or parameters as needed.

  4. Re-run the preview until satisfied.

Python
1preview = data_designer.preview(config_builder, num_records=2)
2
Output
[18:00:05] [INFO] 👁️ Preview generation in progress
[18:00:05] [INFO]   |-- 🔒 Jinja rendering engine: secure
[18:00:05] [INFO] ✅ Validation passed
[18:00:05] [INFO] ⛓️ Sorting column configs into a Directed Acyclic Graph
[18:00:05] [INFO] Skipping model health checks because DATA_DESIGNER_SKIP_MODEL_HEALTH_CHECKS=1
[18:00:05] [INFO] ⚡ Using async task-queue preview
[18:00:05] [INFO] 📝 llm-text model config for column 'description'
[18:00:05] [INFO]   |-- model: 'nvidia/nemotron-3-nano-omni-30b-a3b-reasoning'
[18:00:05] [INFO]   |-- model alias: 'nvidia-vision'
[18:00:05] [INFO]   |-- model provider: 'nvidia'
[18:00:05] [INFO]   |-- inference parameters:
[18:00:05] [INFO]   |  |-- generation_type=chat-completion
[18:00:05] [INFO]   |  |-- max_parallel_requests=4
[18:00:05] [INFO]   |  |-- temperature=0.60
[18:00:05] [INFO]   |  |-- top_p=0.95
[18:00:05] [INFO] ⚡️ Async generation: 1 column(s) (column 'description'), 2 tasks across 1 row group(s)
[18:00:05] [INFO] 🚀 (1/1) Dispatching with 2 records
[18:00:05] [INFO] 🌱 (1/1) Sampling 2 records from seed dataset
[18:00:05] [INFO]   |-- seed dataset size: 512 records
[18:00:05] [INFO]   |-- sampling strategy: ordered
[18:00:23] [INFO] 📊 Progress [17.2s]:
[18:00:23] [INFO]   |-- 🐥 column 'description': 1/2 (50%) 0.1 rec/s
[18:00:26] [INFO] 📊 Progress [20.6s]:
[18:00:26] [INFO]   |-- 🐔 column 'description': 2/2 (100%) 0.1 rec/s
[18:00:26] [INFO] ✅ Async generation complete [20.6s]: 2 ok, 0 failed across 1 column(s)
[18:00:26] [INFO] 📊 Model usage summary:
[18:00:26] [INFO]   |-- model: nvidia/nemotron-3-nano-omni-30b-a3b-reasoning
[18:00:26] [INFO]   |-- tokens: input=658, output=1657, reasoning=949 (estimated), total=2315, tps=112
[18:00:26] [INFO]   |-- reasoning token count estimated with tiktoken
[18:00:26] [INFO]   |-- requests: success=2, failed=0, total=2, rpm=5
[18:00:26] [INFO] 📐 Measuring dataset column statistics:
[18:00:26] [INFO]   |-- 📝 column: 'description'
[18:00:26] [INFO]   |-- 🌱 column: 'uuid'
[18:00:26] [INFO]   |-- 🌱 column: 'label'
[18:00:26] [INFO]   |-- 🌱 column: 'base64_image'
[18:00:26] [INFO] 🏆 Preview complete!
Python
1# Run this cell multiple times to cycle through the 2 preview records.
2preview.display_sample_record()
3
Output
[index: 0]
                                                                                                              
                                                 Seed Columns                                                 
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name         ┃ Value                                                                                       ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ uuid         │ 1c12ac17-71d5-4fc2-884a-efd928154da6                                                        │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────┤
│ label        │ 0                                                                                           │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────┤
│ base64_image │ iVBORw0KGgoAAAANSUhEUgAAAeQAAAIACAIAAADc8YinAAEAAElEQVR4nOy9V5ckuZEmamZwEREpSna1YAv28JLDHT… │
└──────────────┴─────────────────────────────────────────────────────────────────────────────────────────────┘
                                                                                                              
                                                                                                              
                                              Generated Columns                                               
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name        ┃ Value                                                                                        ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ description │ # Close-up Portrait of a Black and White Cat                                                 │
│             │                                                                                              │
│             │ ## Main Subject                                                                              │
│             │ The image features a tight, close-up shot of a domestic cat's face. The cat has a            │
│             │ distinctive **tuxedo coat pattern**, with black fur covering the top of its head, ears, and  │
│             │ sides of its face, while the muzzle, chin, and chest area are white.                         │
│             │                                                                                              │
│             │ ## Facial Features                                                                           │
│             │ - **Eyes:** The cat has large, round eyes with a striking **yellow-green (chartreuse) hue**. │
│             │ The pupils are vertically slit, indicating typical feline vision. They are looking slightly  │
│             │ upward and to the viewer's left, giving an attentive or perhaps curious expression.          │
│             │ - **Nose and Mouth:** It has a small, triangular black nose. The mouth is set in a subtle,   │
│             │ downward curve, creating a **grumpy or serious expression**. Notably, there is a tiny patch  │
│             │ of black fur sticking out from the upper lip, resembling a mustache or a "blep."             │
│             │ - **Whiskers:** Long, prominent white whiskers extend outward from the muzzle, contrasting   │
│             │ sharply against the black fur on the cheeks.                                                 │
│             │                                                                                              │
│             │ ## Background                                                                                │
│             │ The background is out of focus (shallow depth of field), keeping the attention entirely on   │
│             │ the cat.                                                                                     │
│             │ - To the left, there is a plain, light-colored wall (likely white or light grey).            │
│             │ - To the right, a vertical edge is visible, appearing to be a wooden door frame or a piece   │
│             │ of furniture in a warm beige tone.                                                           │
│             │                                                                                              │
│             │ ## Colors                                                                                    │
│             │ The primary colors are the stark contrast of **black and white** in the fur, complemented by │
│             │ the **vibrant yellow-green** of the eyes and the neutral tones of the background.            │
└─────────────┴──────────────────────────────────────────────────────────────────────────────────────────────┘
Python
1# The preview dataset is available as a pandas DataFrame.
2preview.dataset
3
Output
uuid label base64_image description
0 1c12ac17-71d5-4fc2-884a-efd928154da6 0 iVBORw0KGgoAAAANSUhEUgAAAeQAAAIACAIAAADc8YinAA... # Close-up Portrait of a Black and White Cat\n...
1 fe1df0c0-0857-4335-8fa2-05a19e16f103 0 iVBORw0KGgoAAAANSUhEUgAAAiQAAAIACAIAAAA9rOAHAA... ## Main Subject\nThe primary focus of the imag...

📊 Analyze the generated data

  • Data Designer automatically generates a basic statistical analysis of the generated data.

  • This analysis is available via the analysis property of generation result objects.

Python
1# Print the analysis as a table.
2preview.analysis.to_report()
3
Output
──────────────────────────────────────── 🎨 Data Designer Dataset Profile ─────────────────────────────────────────

                                                                                                                   
                                                 Dataset Overview                                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ number of records               ┃ number of columns               ┃ percent complete records                    ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 2                               │ 4                               │ 100.0%                                      │
└─────────────────────────────────┴─────────────────────────────────┴─────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                              🌱 Seed-Dataset Columns                                              
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ column name                      ┃                data type ┃                              number unique values ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ uuid                             │                   string │                                        2 (100.0%) │
├──────────────────────────────────┼──────────────────────────┼───────────────────────────────────────────────────┤
│ label                            │                      int │                                         1 (50.0%) │
├──────────────────────────────────┼──────────────────────────┼───────────────────────────────────────────────────┤
│ base64_image                     │                   string │                                        2 (100.0%) │
└──────────────────────────────────┴──────────────────────────┴───────────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                                📝 LLM-Text Columns                                                
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                  ┃               ┃                              ┃       prompt tokens ┃       completion tokens ┃
┃ column name      ┃     data type ┃         number unique values ┃          per record ┃              per record ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ description      │        string │                   2 (100.0%) │        29.0 +/- 0.0 │           339.5 +/- 3.5 │
└──────────────────┴───────────────┴──────────────────────────────┴─────────────────────┴─────────────────────────┘
                                                                                                                   
                                                                                                                   
╭────────────────────────────────────────────────── Table Notes ──────────────────────────────────────────────────╮
│                                                                                                                 │
│  1. All token statistics are based on a sample of max(1000, len(dataset)) records.                              │
│  2. Tokens are calculated using tiktoken's cl100k_base tokenizer.                                               │
│                                                                                                                 │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
                                                                                                                   
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────

🔎 Visual Inspection

Let's compare the original image with the generated description to validate quality:

Python
1# Compare original image with generated description
2index = 0  # Change this to view different examples
3 
4# Merge preview data with original images for comparison
5comparison_dataset = preview.dataset.merge(pd.DataFrame(img_dataset)[["uuid", "image"]], how="left", on="uuid")
6 
7# Extract the record for display
8record = comparison_dataset.iloc[index]
9 
10print("📄 Original Image:")
11display(resize_image(record.image, BASE64_IMAGE_HEIGHT))
12 
13print("\n📝 Generated Description:")
14rich.print(Panel(record.description, title="Image Description", title_align="left"))
15
Output
📄 Original Image:
Output

📝 Generated Description:
╭─ Image Description ─────────────────────────────────────────────────────────────────────────────────────────────╮
│ # Close-up Portrait of a Black and White Cat                                                                    │
│                                                                                                                 │
│ ## Main Subject                                                                                                 │
│ The image features a tight, close-up shot of a domestic cat's face. The cat has a distinctive **tuxedo coat     │
│ pattern**, with black fur covering the top of its head, ears, and sides of its face, while the muzzle, chin,    │
│ and chest area are white.                                                                                       │
│                                                                                                                 │
│ ## Facial Features                                                                                              │
│ - **Eyes:** The cat has large, round eyes with a striking **yellow-green (chartreuse) hue**. The pupils are     │
│ vertically slit, indicating typical feline vision. They are looking slightly upward and to the viewer's left,   │
│ giving an attentive or perhaps curious expression.                                                              │
│ - **Nose and Mouth:** It has a small, triangular black nose. The mouth is set in a subtle, downward curve,      │
│ creating a **grumpy or serious expression**. Notably, there is a tiny patch of black fur sticking out from the  │
│ upper lip, resembling a mustache or a "blep."                                                                   │
│ - **Whiskers:** Long, prominent white whiskers extend outward from the muzzle, contrasting sharply against the  │
│ black fur on the cheeks.                                                                                        │
│                                                                                                                 │
│ ## Background                                                                                                   │
│ The background is out of focus (shallow depth of field), keeping the attention entirely on the cat.             │
│ - To the left, there is a plain, light-colored wall (likely white or light grey).                               │
│ - To the right, a vertical edge is visible, appearing to be a wooden door frame or a piece of furniture in a    │
│ warm beige tone.                                                                                                │
│                                                                                                                 │
│ ## Colors                                                                                                       │
│ The primary colors are the stark contrast of **black and white** in the fur, complemented by the **vibrant      │
│ yellow-green** of the eyes and the neutral tones of the background.                                             │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

🆙 Scale up!

  • Happy with your preview data?

  • Use the create method to submit larger Data Designer generation jobs.

Python
1results = data_designer.create(config_builder, num_records=10, dataset_name="tutorial-4")
2
Output
[18:00:26] [INFO] OpenTelemetry metrics available at http://127.0.0.1:9464/metrics
[18:00:26] [INFO] 🎨 Creating Data Designer dataset
[18:00:26] [INFO]   |-- 🔒 Jinja rendering engine: secure
[18:00:26] [INFO] ✅ Validation passed
[18:00:26] [INFO] ⛓️ Sorting column configs into a Directed Acyclic Graph
[18:00:26] [INFO] Skipping model health checks because DATA_DESIGNER_SKIP_MODEL_HEALTH_CHECKS=1
[18:00:26] [INFO] ⚡ Using async task-queue builder
[18:00:26] [INFO] 📝 llm-text model config for column 'description'
[18:00:26] [INFO]   |-- model: 'nvidia/nemotron-3-nano-omni-30b-a3b-reasoning'
[18:00:26] [INFO]   |-- model alias: 'nvidia-vision'
[18:00:26] [INFO]   |-- model provider: 'nvidia'
[18:00:26] [INFO]   |-- inference parameters:
[18:00:26] [INFO]   |  |-- generation_type=chat-completion
[18:00:26] [INFO]   |  |-- max_parallel_requests=4
[18:00:26] [INFO]   |  |-- temperature=0.60
[18:00:26] [INFO]   |  |-- top_p=0.95
[18:00:26] [INFO] ⚡️ Async generation: 1 column(s) (column 'description'), 10 tasks across 1 row group(s)
[18:00:26] [INFO] 🚀 (1/1) Dispatching with 10 records
[18:00:26] [INFO] 🌱 (1/1) Sampling 10 records from seed dataset
[18:00:26] [INFO]   |-- seed dataset size: 512 records
[18:00:26] [INFO]   |-- sampling strategy: ordered
[18:00:47] [INFO] 📊 Progress [21.2s]:
[18:00:47] [INFO]   |-- 😴 column 'description': 1/10 (10%) 0.0 rec/s
[18:01:03] [WARNING] Observed retryable model-task error: kind=internal_server, http_status=503; the row task will be deferred.
[18:01:07] [INFO] 📊 Progress [40.4s]:
[18:01:07] [INFO]   |-- 🥱 column 'description': 3/10 (30%) 0.1 rec/s
[18:01:17] [INFO] 📊 Progress [50.7s]:
[18:01:17] [INFO]   |-- 🥱 column 'description': 4/10 (40%) 0.1 rec/s
[18:01:26] [INFO] 📊 Progress [60.3s]:
[18:01:26] [INFO]   |-- 😐 column 'description': 6/10 (60%) 0.1 rec/s
[18:01:40] [INFO] 📊 Progress [73.6s]:
[18:01:40] [INFO]   |-- 😐 column 'description': 7/10 (70%) 0.1 rec/s
[18:01:41] [WARNING] Observed retryable model-task error: kind=timeout; the row task will be deferred.
[18:01:46] [INFO] 📊 Progress [79.5s]:
[18:01:46] [INFO]   |-- 😊 column 'description': 8/10 (80%) 0.1 rec/s
[18:01:46] [INFO] 🔄 (1/1) Salvaging 2 deferred task(s)
[18:02:04] [INFO] 📊 Progress [97.4s]:
[18:02:04] [INFO]   |-- 😊 column 'description': 9/10 (90%) 0.1 rec/s
[18:02:14] [INFO] 📊 Progress [108.1s]:
[18:02:14] [INFO]   |-- 🤩 column 'description': 10/10 (100%) 0.1 rec/s
[18:02:14] [INFO] ✅ Async generation complete [108.1s]: 10 ok, 0 failed across 1 column(s)
[18:02:15] [INFO] 📊 Model usage summary:
[18:02:15] [INFO]   |-- model: nvidia/nemotron-3-nano-omni-30b-a3b-reasoning
[18:02:15] [INFO]   |-- tokens: input=3720, output=9410, reasoning=6360 (estimated), total=13130, tps=121
[18:02:15] [INFO]   |-- reasoning token count estimated with tiktoken
[18:02:15] [INFO]   |-- requests: success=10, failed=2, total=12, rpm=6
[18:02:15] [INFO] 📐 Measuring dataset column statistics:
[18:02:15] [INFO]   |-- 📝 column: 'description'
[18:02:15] [INFO]   |-- 🌱 column: 'uuid'
[18:02:15] [INFO]   |-- 🌱 column: 'label'
[18:02:15] [INFO]   |-- 🌱 column: 'base64_image'
Python
1# Load the generated dataset as a pandas DataFrame.
2dataset = results.load_dataset()
3 
4dataset.head()
5
Output
uuid label base64_image description
0 1c12ac17-71d5-4fc2-884a-efd928154da6 0 iVBORw0KGgoAAAANSUhEUgAAAeQAAAIACAIAAADc8YinAA... # Detailed Description of the Image ## Main S...
1 fe1df0c0-0857-4335-8fa2-05a19e16f103 0 iVBORw0KGgoAAAANSUhEUgAAAiQAAAIACAIAAAA9rOAHAA... # Fluffy Tabby Cat in a Box ## Main Subject T...
2 30748a1e-31bb-42fb-8ea8-6db1ac698825 0 iVBORw0KGgoAAAANSUhEUgAAAqoAAAIACAIAAADFYNm1AA... # Detailed Description **Main Subject:** The ...
3 c10bdd68-09d7-4abd-92f6-4713817e2677 0 iVBORw0KGgoAAAANSUhEUgAAAwAAAAIACAIAAAC6lJxtAA... # Cat in a Green Container **Main Subject:** ...
4 01bf9e06-762b-4397-a773-86c6587c9f7a 0 iVBORw0KGgoAAAANSUhEUgAAAqoAAAIACAIAAADFYNm1AA... **Detailed Description:** **Main Subject:** T...
Python
1# Load the analysis results into memory.
2analysis = results.load_analysis()
3 
4analysis.to_report()
5
Output
──────────────────────────────────────── 🎨 Data Designer Dataset Profile ─────────────────────────────────────────

                                                                                                                   
                                                 Dataset Overview                                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ number of records               ┃ number of columns               ┃ percent complete records                    ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 10                              │ 4                               │ 100.0%                                      │
└─────────────────────────────────┴─────────────────────────────────┴─────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                              🌱 Seed-Dataset Columns                                              
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ column name                      ┃                data type ┃                              number unique values ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ uuid                             │                   string │                                       10 (100.0%) │
├──────────────────────────────────┼──────────────────────────┼───────────────────────────────────────────────────┤
│ label                            │                      int │                                         1 (10.0%) │
├──────────────────────────────────┼──────────────────────────┼───────────────────────────────────────────────────┤
│ base64_image                     │                   string │                                       10 (100.0%) │
└──────────────────────────────────┴──────────────────────────┴───────────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                                📝 LLM-Text Columns                                                
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                  ┃               ┃                              ┃       prompt tokens ┃       completion tokens ┃
┃ column name      ┃     data type ┃         number unique values ┃          per record ┃              per record ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ description      │        string │                  10 (100.0%) │        29.0 +/- 0.0 │          292.5 +/- 70.9 │
└──────────────────┴───────────────┴──────────────────────────────┴─────────────────────┴─────────────────────────┘
                                                                                                                   
                                                                                                                   
╭────────────────────────────────────────────────── Table Notes ──────────────────────────────────────────────────╮
│                                                                                                                 │
│  1. All token statistics are based on a sample of max(1000, len(dataset)) records.                              │
│  2. Tokens are calculated using tiktoken's cl100k_base tokenizer.                                               │
│                                                                                                                 │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
                                                                                                                   
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────

⏭️ Next Steps

Now that you've learned how to use visual context for image summarization in Data Designer, explore more:

  • Experiment with different vision models for specific image types

  • Try different prompt variations to generate specialized descriptions (e.g., technical details, key findings)

  • Combine image, audio, or video context with other column types after confirming your selected model supports those modalities

  • Apply this pattern to other vision tasks like image captioning, OCR validation, or visual question answering

  • Generating images with Data Designer