Tutorials

Seeding with an External Dataset

View as Markdown
Run in Google Colab

🎨 Data Designer Tutorial: Seeding Synthetic Data Generation with an External Dataset

📚 What you'll learn

In this notebook, we will demonstrate how to seed synthetic data generation in Data Designer with an external dataset.

If this is your first time using Data Designer, we recommend starting with the first notebook in this tutorial series.

📦 Import Data Designer

  • data_designer.config provides access to the configuration API.

  • DataDesigner is the main interface for data generation.

Python
1import data_designer.config as dd
2from data_designer.interface import DataDesigner
3

⚙️ Initialize the Data Designer interface

  • DataDesigner is the main object responsible for managing the data generation process.

  • When initialized without arguments, the default model providers are used.

Python
1data_designer = DataDesigner()
2

🎛️ Define model configurations

  • Each ModelConfig defines a model that can be used during the generation process.

  • The "model alias" is used to reference the model in the Data Designer config (as we will see below).

  • The "model provider" is the external service that hosts the model (see the model config docs for more details).

  • By default, we use build.nvidia.com as the model provider.

Python
1# This name is set in the model provider configuration.
2MODEL_PROVIDER = "nvidia"
3 
4# The model ID is from build.nvidia.com.
5MODEL_ID = "nvidia/nemotron-3-nano-30b-a3b"
6 
7# We choose this alias to be descriptive for our use case.
8MODEL_ALIAS = "nemotron-nano-v3"
9 
10model_configs = [
11    dd.ModelConfig(
12        alias=MODEL_ALIAS,
13        model=MODEL_ID,
14        provider=MODEL_PROVIDER,
15        inference_parameters=dd.ChatCompletionInferenceParams(
16            temperature=1.0,
17            top_p=1.0,
18            max_tokens=2048,
19            extra_body={"chat_template_kwargs": {"enable_thinking": False}},
20        ),
21    )
22]
23

🏗️ Initialize the Data Designer Config Builder

  • The Data Designer config defines the dataset schema and generation process.

  • The config builder provides an intuitive interface for building this configuration.

  • The list of model configs is provided to the builder at initialization.

Python
1config_builder = dd.DataDesignerConfigBuilder(model_configs=model_configs)
2

🏥 Prepare a seed dataset

  • For this notebook, we'll create a synthetic dataset of patient notes.

  • We will seed the generation process with a symptom-to-diagnosis dataset.

  • We already have the dataset downloaded in the data directory of this repository.


🌱 Why use a seed dataset?

  • Seed datasets let you steer the generation process by providing context that is specific to your use case.

  • Seed datasets are also an excellent way to inject real-world diversity into your synthetic data.

  • During generation, prompt templates can reference any of the seed dataset fields.

Python
1# Download sample dataset from Github
2import urllib.request
3 
4url = "https://raw.githubusercontent.com/NVIDIA/GenerativeAIExamples/refs/heads/main/nemo/NeMo-Data-Designer/data/gretelai_symptom_to_diagnosis.csv"
5local_filename, _ = urllib.request.urlretrieve(url, "gretelai_symptom_to_diagnosis.csv")
6 
7# Seed datasets are passed as reference objects to the config builder.
8seed_source = dd.LocalFileSeedSource(path=local_filename)
9 
10config_builder.with_seed_dataset(seed_source)
11
Output
DataDesignerConfigBuilder(
    seed_dataset: local seed
)

🎨 Designing our synthetic patient notes dataset

  • The prompt template can reference fields from our seed dataset:
    • {{ diagnosis }} - the medical diagnosis from the seed data
    • {{ patient_summary }} - the symptom description from the seed data
Python
1config_builder.add_column(
2    dd.SamplerColumnConfig(
3        name="patient_sampler",
4        sampler_type=dd.SamplerType.PERSON_FROM_FAKER,
5        params=dd.PersonFromFakerSamplerParams(),
6    )
7)
8 
9config_builder.add_column(
10    dd.SamplerColumnConfig(
11        name="doctor_sampler",
12        sampler_type=dd.SamplerType.PERSON_FROM_FAKER,
13        params=dd.PersonFromFakerSamplerParams(),
14    )
15)
16 
17config_builder.add_column(
18    dd.SamplerColumnConfig(
19        name="patient_id",
20        sampler_type=dd.SamplerType.UUID,
21        params=dd.UUIDSamplerParams(
22            prefix="PT-",
23            short_form=True,
24            uppercase=True,
25        ),
26    )
27)
28 
29config_builder.add_column(dd.ExpressionColumnConfig(name="first_name", expr="{{ patient_sampler.first_name }}"))
30 
31config_builder.add_column(dd.ExpressionColumnConfig(name="last_name", expr="{{ patient_sampler.last_name }}"))
32 
33config_builder.add_column(dd.ExpressionColumnConfig(name="dob", expr="{{ patient_sampler.birth_date }}"))
34 
35config_builder.add_column(
36    dd.SamplerColumnConfig(
37        name="symptom_onset_date",
38        sampler_type=dd.SamplerType.DATETIME,
39        params=dd.DatetimeSamplerParams(start="2024-01-01", end="2024-12-31"),
40    )
41)
42 
43config_builder.add_column(
44    dd.SamplerColumnConfig(
45        name="date_of_visit",
46        sampler_type=dd.SamplerType.TIMEDELTA,
47        params=dd.TimeDeltaSamplerParams(dt_min=1, dt_max=30, reference_column_name="symptom_onset_date"),
48    )
49)
50 
51config_builder.add_column(dd.ExpressionColumnConfig(name="physician", expr="Dr. {{ doctor_sampler.last_name }}"))
52 
53config_builder.add_column(
54    dd.LLMTextColumnConfig(
55        name="physician_notes",
56        prompt="""\
57You are a primary-care physician who just had an appointment with {{ first_name }} {{ last_name }},
58who has been struggling with symptoms from {{ diagnosis }} since {{ symptom_onset_date }}.
59The date of today's visit is {{ date_of_visit }}.
60 
61{{ patient_summary }}
62 
63Write careful notes about your visit with {{ first_name }},
64as Dr. {{ doctor_sampler.first_name }} {{ doctor_sampler.last_name }}.
65 
66Format the notes as a busy doctor might.
67Respond with only the notes, no other text.
68""",
69        model_alias=MODEL_ALIAS,
70    )
71)
72 
73data_designer.validate(config_builder)
74
Output
[20:12:14] [INFO] ✅ Validation passed

🔁 Iteration is key – preview the dataset!

  1. Use the preview method to generate a sample of records quickly.

  2. Inspect the results for quality and format issues.

  3. Adjust column configurations, prompts, or parameters as needed.

  4. Re-run the preview until satisfied.

Python
1preview = data_designer.preview(config_builder, num_records=2)
2
Output
[20:12:14] [INFO] 🕵️ Preview generation in progress
[20:12:14] [INFO]   |-- 🔒 Jinja rendering engine: secure
[20:12:14] [INFO] ✅ Validation passed
[20:12:14] [INFO] ⛓️ Sorting column configs into a Directed Acyclic Graph
[20:12:14] [INFO] 🩺 Running health checks for models...
[20:12:14] [INFO]   |-- 👀 Checking 'nvidia/nemotron-3-nano-30b-a3b' in provider named 'nvidia' for model alias 'nemotron-nano-v3'...
[20:12:15] [INFO]   |-- ✅ Passed!
[20:12:15] [INFO] ⚡ Using async task-queue preview
[20:12:15] [INFO] 📝 llm-text model config for column 'physician_notes'
[20:12:15] [INFO]   |-- model: 'nvidia/nemotron-3-nano-30b-a3b'
[20:12:15] [INFO]   |-- model alias: 'nemotron-nano-v3'
[20:12:15] [INFO]   |-- model provider: 'nvidia'
[20:12:15] [INFO]   |-- inference parameters:
[20:12:15] [INFO]   |  |-- generation_type=chat-completion
[20:12:15] [INFO]   |  |-- max_parallel_requests=4
[20:12:15] [INFO]   |  |-- extra_body={'chat_template_kwargs': {'enable_thinking': False}}
[20:12:15] [INFO]   |  |-- temperature=1.00
[20:12:15] [INFO]   |  |-- top_p=1.00
[20:12:15] [INFO]   |  |-- max_tokens=2048
[20:12:15] [INFO] ⚡️ Async generation: 1 column(s) (column 'physician_notes'), 2 tasks across 1 row group(s)
[20:12:15] [INFO] 🚀 (1/1) Dispatching with 2 records
[20:12:15] [INFO] 🎲 (1/1) Preparing samplers to generate 2 records across 5 columns
[20:12:15] [INFO] 🌱 (1/1) Sampling 2 records from seed dataset
[20:12:15] [INFO]   |-- seed dataset size: 820 records
[20:12:15] [INFO]   |-- sampling strategy: ordered
[20:12:15] [INFO] 🧩 (1/1) Generating column `dob` from expression
[20:12:15] [INFO] 🧩 (1/1) Generating column `first_name` from expression
[20:12:15] [INFO] 🧩 (1/1) Generating column `last_name` from expression
[20:12:15] [INFO] 🧩 (1/1) Generating column `physician` from expression
[20:12:23] [INFO] 📊 Progress [8.2s]:
[20:12:23] [INFO]   |-- 🌗 column 'physician_notes': 1/2 (50%) 0.1 rec/s
[20:12:27] [INFO] 📊 Progress [12.3s]:
[20:12:27] [INFO]   |-- 🌕 column 'physician_notes': 2/2 (100%) 0.2 rec/s
[20:12:27] [INFO] ✅ Async generation complete [12.3s]: 2 ok, 0 failed across 1 column(s)
[20:12:27] [INFO] 📊 Model usage summary:
[20:12:27] [INFO]   |-- model: nvidia/nemotron-3-nano-30b-a3b
[20:12:27] [INFO]   |-- tokens: input=329, output=2526, total=2855, tps=232
[20:12:27] [INFO]   |-- requests: success=2, failed=0, total=2, rpm=9
[20:12:27] [INFO] 📐 Measuring dataset column statistics:
[20:12:27] [INFO]   |-- 🎲 column: 'patient_sampler'
[20:12:27] [INFO]   |-- 🎲 column: 'doctor_sampler'
[20:12:27] [INFO]   |-- 🎲 column: 'patient_id'
[20:12:27] [INFO]   |-- 🧩 column: 'first_name'
[20:12:27] [INFO]   |-- 🧩 column: 'last_name'
[20:12:27] [INFO]   |-- 🧩 column: 'dob'
[20:12:27] [INFO]   |-- 🎲 column: 'symptom_onset_date'
[20:12:27] [INFO]   |-- 🎲 column: 'date_of_visit'
[20:12:27] [INFO]   |-- 🧩 column: 'physician'
[20:12:27] [INFO]   |-- 📝 column: 'physician_notes'
[20:12:27] [INFO] 🏆 Preview complete!
Python
1# Run this cell multiple times to cycle through the 2 preview records.
2preview.display_sample_record()
3
Output
[index: 0]
                                                                                                              
                                                 Seed Columns                                                 
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name            ┃ Value                                                                                    ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ diagnosis       │ cervical spondylosis                                                                     │
├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤
│ patient_summary │ I've been having a lot of pain in my neck and back. I've also been having trouble with   │
│                 │ my balance and coordination. I've been coughing a lot and my limbs feel weak.            │
└─────────────────┴──────────────────────────────────────────────────────────────────────────────────────────┘
                                                                                                              
                                                                                                              
                                              Generated Columns                                               
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name               ┃ Value                                                                                 ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ patient_sampler    │ {                                                                                     │
│                    │     'uuid': '0b64566d-83d0-4e56-83fa-7acffaf336da',                                   │
│                    │     'locale': 'en_US',                                                                │
│                    │     'first_name': 'Julie',                                                            │
│                    │     'last_name': 'Mcconnell',                                                         │
│                    │     'middle_name': None,                                                              │
│                    │     'sex': 'Female',                                                                  │
│                    │     'street_number': '87531',                                                         │
│                    │     'street_name': 'Hood Gardens',                                                    │
│                    │     'city': 'Amandamouth',                                                            │
│                    │     'state': 'Montana',                                                               │
│                    │     'postcode': '26024',                                                              │
│                    │     'age': 73,                                                                        │
│                    │     'birth_date': '1953-03-26',                                                       │
│                    │     'country': 'Nicaragua',                                                           │
│                    │     'marital_status': 'separated',                                                    │
│                    │     'education_level': 'doctorate',                                                   │
│                    │     'unit': '',                                                                       │
│                    │     'occupation': 'Chief of Staff',                                                   │
│                    │     'phone_number': '7899462338',                                                     │
│                    │     'bachelors_field': 'stem_related'                                                 │
│                    │ }                                                                                     │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ doctor_sampler     │ {                                                                                     │
│                    │     'uuid': 'fec70082-9dc6-4cbf-80cd-0fd42a93c8df',                                   │
│                    │     'locale': 'en_US',                                                                │
│                    │     'first_name': 'Victoria',                                                         │
│                    │     'last_name': 'Williams',                                                          │
│                    │     'middle_name': None,                                                              │
│                    │     'sex': 'Female',                                                                  │
│                    │     'street_number': '909',                                                           │
│                    │     'street_name': 'Becker Stravenue',                                                │
│                    │     'city': 'Kellerfort',                                                             │
│                    │     'state': 'Colorado',                                                              │
│                    │     'postcode': '60330',                                                              │
│                    │     'age': 31,                                                                        │
│                    │     'birth_date': '1994-12-07',                                                       │
│                    │     'country': 'China',                                                               │
│                    │     'marital_status': 'never_married',                                                │
│                    │     'education_level': 'bachelors',                                                   │
│                    │     'unit': '',                                                                       │
│                    │     'occupation': 'Engineer, drilling',                                               │
│                    │     'phone_number': '5966603580',                                                     │
│                    │     'bachelors_field': 'stem'                                                         │
│                    │ }                                                                                     │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ patient_id         │ PT-3A051361                                                                           │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ symptom_onset_date │ 2024-06-25T00:00:00                                                                   │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ date_of_visit      │ 2024-06-26T00:00:00                                                                   │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ physician_notes    │ **Dr. Victoria Williams – 2024‑06‑26**                                                │
│                    │ **Patient:** Julie McConnell                                                          │
│                    │ **DOB:** (redacted) – Age 52                                                          │
│                    │ **Visit Type:** Office follow‑up (post‑evaluation for progressive cervical            │
│                    │ spondylosis)                                                                          │
│                    │                                                                                       │
│                    │ ---                                                                                   │
│                    │                                                                                       │
│                    │ ### Chief Complaint                                                                   │
│                    │ - “Neck and back pain getting worse”                                                  │
│                    │ - “Trouble with balance & coordination”                                               │
│                    │ - New cough (productive?)                                                             │
│                    │ - Progressive weakness in limbs                                                       │
│                    │                                                                                       │
│                    │ ### HPI                                                                               │
│                    │ - **Onset:** Symptoms began 06/25/2024 (≈1 day prior to visit).                       │
│                    │ - **Location:** Diffuse neck pain radiating to upper back; mechanical‑type neck       │
│                    │ stiffness.                                                                            │
│                    │ - **Quality:** Constant dull‑achy pain, worsens with neck extension/rotation;         │
│                    │ occasional sharp “shooting” pain down right arm.                                      │
│                    │ - **Severity:** Numeric Rating Scale 7/10 at rest, 9/10 with movement.                │
│                    │ - **Timing:** Daily, progressively worsening over 24 h.                               │
│                    │ - **Associated Symptoms:**                                                            │
│                    │   - New cough (dry, non‑productive) – 2 days history.                                 │
│                    │   - Weakness in bilateral upper & lower extremities; difficulty climbing stairs.      │
│                    │   - Light‑headedness when standing; episodes of near‑syncope.                         │
│                    │   - Impaired coordination – frequent stumbling, difficulty with fine motor tasks      │
│                    │ (buttoning).                                                                          │
│                    │ - **Aggravating Factors:** Head tilt, prolonged standing, coughing episodes.          │
│                    │ - **Relieving Factors:** Rest, mild analgesics, application of heat.                  │
│                    │ - **Prior Episodes:** Chronic intercostal‑cervical radiculopathy secondary to         │
│                    │ cervical spondylosis diagnosed 2022; low‑grade neck pain managed conservatively.      │
│                    │ - **Red‑Flag Features:** New cough, unexplained limb weakness, gait instability –     │
│                    │ prompted urgent evaluation.                                                           │
│                    │                                                                                       │
│                    │ ### PMH                                                                               │
│                    │ - Hypertension (controlled on lisinopril 10 mg daily).                                │
│                    │ - Diabetes mellitus type 2 (metformin 1000 mg BID).                                   │
│                    │ - Osteoarthritis (knees, hips).                                                       │
│                    │ - No prior spinal surgery.                                                            │
│                    │ - No prior hospitalizations.                                                          │
│                    │                                                                                       │
│                    │ ### Medications                                                                       │
│                    │ - Lisinopril 10 mg PO daily                                                           │
│                    │ - Metformin 1000 mg PO BID                                                            │
│                    │ - Acetaminophen 650 mg q6h PRN (pain)                                                 │
│                    │ - Ibuprofen 400 mg q8h PRN (inflammation)                                             │
│                    │ - Vitamin D 2000 IU daily                                                             │
│                    │                                                                                       │
│                    │ ### Allergies                                                                         │
│                    │ - NKDA (no known drug allergies)                                                      │
│                    │                                                                                       │
│                    │ ### Physical Exam – **Neurologic** (focused)                                          │
│                    │ - **Mental Status:** Alert, oriented x3; speech clear.                                │
│                    │ - **Cranial Nerves:** II–XII intact.                                                  │
│                    │ - **Motor:**                                                                          │
│                    │   - Upper extremities: 4/5 strength (right > left), distal weakness noted in hand     │
│                    │ grip.                                                                                 │
│                    │   - Lower extremities: 4/5 strength, impaired tandem gait.                            │
│                    │ - **Sensory:** Decreased pinprick sensation in C5‑C6 dermatomes bilaterally.          │
│                    │ - **Reflexes:** Hyperreflexia (+3) in biceps and triceps; +2 in knees; Babinski       │
│                    │ negative.                                                                             │
│                    │ - **Coordination:** Dysdiadochokinesia in upper and lower limbs.                      │
│                    │ - **Gait:** Wide‑based, unsteady; requires assistance.                                │
│                    │ - **Romberg:** Positive – unable to stand unassisted.                                 │
│                    │ - **Neck:** Limited flexion/extension; tenderness at C4‑C5 spinous process.           │
│                    │                                                                                       │
│                    │ ### NECK CT Scan – **Results (provided by radiology)**                                │
│                    │ - Moderate diffuse disc degeneration at C3‑C5 with circumferential osteophyte         │
│                    │ formation.                                                                            │
│                    │ - Central canal stenosis at C4‑C5 (AP diameter 8 mm) – mild to moderate.              │
│                    │ - No cord signal change; no acute cord compression.                                   │
│                    │ - Cervical cord appears normal caliber; myelopathic changes not evident.              │
│                    │                                                                                       │
│                    │ ### Working Diagnosis                                                                 │
│                    │ 1. **Progressive cervical spondylosis** with evolving myelopathic picture             │
│                    │ (symptomatic spinal canal stenosis).                                                  │
│                    │ 2. **New‑onset dry cough** – likely unrelated (possible post‑viral bronchospasm).     │
│                    │ 3. **Generalized weakness & gait instability** – likely secondary to cervical         │
│                    │ myelopathy.                                                                           │
│                    │                                                                                       │
│                    │ ### Plan                                                                              │
│                    │ | Item | Action |                                                                     │
│                    │ |------|--------|                                                                     │
│                    │ | **Referral** | Immediate neurosurgery consult for surgical decompression evaluation │
│                    │ (anterior cervical discectomy & fusion C4‑C5). |                                      │
│                    │ | **Imaging** | Obtain MRI of cervical spine (full cord assessment) within 48 h. |    │
│                    │ | **Pharmacologic** | Continue NSAID (ibuprofen) PRN; consider short course of oral   │
│                    │ steroids (prednisone 40 mg daily x5 days) pending neurosurgery clearance to reduce    │
│                    │ cord inflammation. |                                                                  │
│                    │ | **Physical Therapy** | Initiate gentle cervical ROM exercises; avoid heavy lifting  │
│                    │ >10 lb. |                                                                             │
│                    │ | **Safety** | Advise fall precautions; use assistive device (cane) for ambulation. | │
│                    │ | **Follow‑up** | Return visit in 1 week or sooner if neurologic status worsens       │
│                    │ (e.g., new weakness, urinary incontinence). |                                         │
│                    │ | **Patient Education** | Discuss signs of cord compromise (numbness, worsening       │
│                    │ weakness, bowel/bladder changes) – call 911 or urgent care if present. |              │
│                    │ | **Support** | Provide hospital referral packet & patient navigator contact. |       │
│                    │ | **Documentation** | Recorded in EMR; copy of imaging report attached. |             │
│                    │                                                                                       │
│                    │ ### Signatures                                                                        │
│                    │ **Dr. Victoria Williams, MD**                                                         │
│                    │ Board‑Certified Family Medicine                                                       │
│                    │ (555) 123‑4567                                                                        │
│                    │ Signature: _______________________                                                    │
│                    │                                                                                       │
│                    │ ---                                                                                   │
│                    │                                                                                       │
│                    │ *End of note.*                                                                        │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ first_name         │ Julie                                                                                 │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ last_name          │ Mcconnell                                                                             │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ dob                │ 1953-03-26                                                                            │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ physician          │ Dr. Williams                                                                          │
└────────────────────┴───────────────────────────────────────────────────────────────────────────────────────┘
Python
1# The preview dataset is available as a pandas DataFrame.
2preview.dataset
3
Output
diagnosis patient_summary patient_sampler doctor_sampler patient_id symptom_onset_date date_of_visit dob last_name physician first_name physician_notes
0 cervical spondylosis I've been having a lot of pain in my neck and ... {'uuid': '0b64566d-83d0-4e56-83fa-7acffaf336da... {'uuid': 'fec70082-9dc6-4cbf-80cd-0fd42a93c8df... PT-3A051361 2024-06-25T00:00:00 2024-06-26T00:00:00 1953-03-26 Mcconnell Dr. Williams Julie **Dr. Victoria Williams – 2024‑06‑26** \n**Pa...
1 impetigo I have a rash on my face that is getting worse... {'uuid': '17f1e779-2cb3-4860-9f0a-0da90bc7a838... {'uuid': '1507d057-e16e-4059-999e-929419ad1149... PT-7BECA1A5 2024-05-19T00:00:00 2024-06-03T00:00:00 1984-11-03 Garcia Dr. Walsh James **SOAP Note – Dr. Elizabeth Walsh, MD** \n**P...

📊 Analyze the generated data

  • Data Designer automatically generates a basic statistical analysis of the generated data.

  • This analysis is available via the analysis property of generation result objects.

Python
1# Print the analysis as a table.
2preview.analysis.to_report()
3
Output
──────────────────────────────────────── 🎨 Data Designer Dataset Profile ─────────────────────────────────────────

                                                                                                                   
                                                 Dataset Overview                                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ number of records               ┃ number of columns               ┃ percent complete records                    ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 2                               │ 10                              │ 100.0%                                      │
└─────────────────────────────────┴─────────────────────────────────┴─────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                                🎲 Sampler Columns                                                 
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ column name                   ┃       data type ┃             number unique values ┃               sampler type ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ patient_sampler               │            dict │                       2 (100.0%) │          person_from_faker │
├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤
│ doctor_sampler                │            dict │                       2 (100.0%) │          person_from_faker │
├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤
│ patient_id                    │          string │                       2 (100.0%) │                       uuid │
├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤
│ symptom_onset_date            │          string │                       2 (100.0%) │                   datetime │
├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤
│ date_of_visit                 │          string │                       2 (100.0%) │                  timedelta │
└───────────────────────────────┴─────────────────┴──────────────────────────────────┴────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                                📝 LLM-Text Columns                                                
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                       ┃               ┃                            ┃     prompt tokens ┃      completion tokens ┃
┃ column name           ┃     data type ┃       number unique values ┃        per record ┃             per record ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩
│ physician_notes       │        string │                 2 (100.0%) │     135.0 +/- 5.0 │       1182.5 +/- 118.1 │
└───────────────────────┴───────────────┴────────────────────────────┴───────────────────┴────────────────────────┘
                                                                                                                   
                                                                                                                   
                                               🧩 Expression Columns                                               
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ column name                    ┃                 data type ┃                               number unique values ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ first_name                     │                    string │                                         2 (100.0%) │
├────────────────────────────────┼───────────────────────────┼────────────────────────────────────────────────────┤
│ last_name                      │                    string │                                         2 (100.0%) │
├────────────────────────────────┼───────────────────────────┼────────────────────────────────────────────────────┤
│ dob                            │                    string │                                         2 (100.0%) │
├────────────────────────────────┼───────────────────────────┼────────────────────────────────────────────────────┤
│ physician                      │                    string │                                         2 (100.0%) │
└────────────────────────────────┴───────────────────────────┴────────────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
╭────────────────────────────────────────────────── Table Notes ──────────────────────────────────────────────────╮
│                                                                                                                 │
│  1. All token statistics are based on a sample of max(1000, len(dataset)) records.                              │
│  2. Tokens are calculated using tiktoken's cl100k_base tokenizer.                                               │
│                                                                                                                 │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
                                                                                                                   
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────

🆙 Scale up!

  • Happy with your preview data?

  • Use the create method to submit larger Data Designer generation jobs.

Python
1results = data_designer.create(config_builder, num_records=10, dataset_name="tutorial-3")
2
Output
[20:12:27] [INFO] OpenTelemetry metrics available at http://127.0.0.1:9464/metrics
[20:12:27] [INFO] 🎨 Creating Data Designer dataset
[20:12:27] [INFO]   |-- 🔒 Jinja rendering engine: secure
[20:12:27] [INFO] ✅ Validation passed
[20:12:27] [INFO] ⛓️ Sorting column configs into a Directed Acyclic Graph
[20:12:27] [INFO] 🩺 Running health checks for models...
[20:12:27] [INFO]   |-- 👀 Checking 'nvidia/nemotron-3-nano-30b-a3b' in provider named 'nvidia' for model alias 'nemotron-nano-v3'...
[20:12:28] [INFO]   |-- ✅ Passed!
[20:12:28] [INFO] ⚡ Using async task-queue builder
[20:12:28] [INFO] 📝 llm-text model config for column 'physician_notes'
[20:12:28] [INFO]   |-- model: 'nvidia/nemotron-3-nano-30b-a3b'
[20:12:28] [INFO]   |-- model alias: 'nemotron-nano-v3'
[20:12:28] [INFO]   |-- model provider: 'nvidia'
[20:12:28] [INFO]   |-- inference parameters:
[20:12:28] [INFO]   |  |-- generation_type=chat-completion
[20:12:28] [INFO]   |  |-- max_parallel_requests=4
[20:12:28] [INFO]   |  |-- extra_body={'chat_template_kwargs': {'enable_thinking': False}}
[20:12:28] [INFO]   |  |-- temperature=1.00
[20:12:28] [INFO]   |  |-- top_p=1.00
[20:12:28] [INFO]   |  |-- max_tokens=2048
[20:12:28] [INFO] ⚡️ Async generation: 1 column(s) (column 'physician_notes'), 10 tasks across 1 row group(s)
[20:12:28] [INFO] 🚀 (1/1) Dispatching with 10 records
[20:12:28] [INFO] 🎲 (1/1) Preparing samplers to generate 10 records across 5 columns
[20:12:28] [INFO] 🌱 (1/1) Sampling 10 records from seed dataset
[20:12:28] [INFO] 🧩 (1/1) Generating column `dob` from expression
[20:12:28] [INFO] 🧩 (1/1) Generating column `first_name` from expression
[20:12:28] [INFO] 🧩 (1/1) Generating column `last_name` from expression
[20:12:28] [INFO] 🧩 (1/1) Generating column `physician` from expression
[20:12:28] [INFO]   |-- seed dataset size: 820 records
[20:12:28] [INFO]   |-- sampling strategy: ordered
[20:12:35] [INFO] 📊 Progress [7.4s]:
[20:12:35] [INFO]   |-- 😴 column 'physician_notes': 2/10 (20%) 0.3 rec/s
[20:12:41] [INFO] 📊 Progress [12.5s]:
[20:12:41] [INFO]   |-- 😐 column 'physician_notes': 5/10 (50%) 0.4 rec/s
[20:12:46] [INFO] 📊 Progress [18.4s]:
[20:12:46] [INFO]   |-- 😐 column 'physician_notes': 7/10 (70%) 0.4 rec/s
[20:12:54] [INFO] 📊 Progress [26.2s]:
[20:12:54] [INFO]   |-- 😊 column 'physician_notes': 9/10 (90%) 0.3 rec/s
[20:12:57] [INFO] 📊 Progress [29.0s]:
[20:12:57] [INFO]   |-- 🤩 column 'physician_notes': 10/10 (100%) 0.3 rec/s
[20:12:57] [INFO] ✅ Async generation complete [29.0s]: 10 ok, 0 failed across 1 column(s)
[20:12:57] [INFO] 📊 Model usage summary:
[20:12:57] [INFO]   |-- model: nvidia/nemotron-3-nano-30b-a3b
[20:12:57] [INFO]   |-- tokens: input=1614, output=8657, total=10271, tps=351
[20:12:57] [INFO]   |-- requests: success=10, failed=0, total=10, rpm=20
[20:12:57] [INFO] 📐 Measuring dataset column statistics:
[20:12:57] [INFO]   |-- 🎲 column: 'patient_sampler'
[20:12:57] [INFO]   |-- 🎲 column: 'doctor_sampler'
[20:12:57] [INFO]   |-- 🎲 column: 'patient_id'
[20:12:57] [INFO]   |-- 🧩 column: 'first_name'
[20:12:57] [INFO]   |-- 🧩 column: 'last_name'
[20:12:57] [INFO]   |-- 🧩 column: 'dob'
[20:12:57] [INFO]   |-- 🎲 column: 'symptom_onset_date'
[20:12:57] [INFO]   |-- 🎲 column: 'date_of_visit'
[20:12:57] [INFO]   |-- 🧩 column: 'physician'
[20:12:57] [INFO]   |-- 📝 column: 'physician_notes'
Python
1# Load the generated dataset as a pandas DataFrame.
2dataset = results.load_dataset()
3 
4dataset.head()
5
Output
patient_sampler doctor_sampler patient_id symptom_onset_date date_of_visit dob first_name last_name physician diagnosis patient_summary physician_notes
0 {'uuid': '7a630f8f-6293-49d3-87a7-efe6a3a609ba... {'uuid': '29bab175-45a7-488b-a3fd-0ef54d64310a... PT-326632E9 2024-03-28T00:00:00 2024-04-25T00:00:00 1975-08-04 Andrea Warren Dr. Martinez cervical spondylosis I've been having a lot of pain in my neck and ... **Patient:** Andrea Warren **DOB:** ___ **...
1 {'uuid': 'b4532e48-2497-402c-b095-a89e659ce221... {'uuid': '9a863cb8-a2b9-4bed-bdbf-062882f43dc8... PT-5D11225C 2024-11-13T00:00:00 2024-11-28T00:00:00 1912-06-18 Dean Murphy Dr. Moran impetigo I have a rash on my face that is getting worse... 2024-11-28 | 09:15 | New Patient Consult | Dr....
2 {'uuid': '26dd0185-c08d-4ad4-8a08-b24f045a4c7a... {'uuid': 'c4f448f6-53e7-4694-9f80-7ede90bd4ae3... PT-514FF6A1 2024-12-09T00:00:00 2024-12-23T00:00:00 1956-08-05 Eric Johnson Dr. Jackson urinary tract infection I have been urinating blood. I sometimes feel ... **EMR Note – Dr. J. Jackson, MD** **Date:** ...
3 {'uuid': 'c7d5ce9a-8a78-4461-8a92-8565160023d0... {'uuid': '6d3f0983-9c80-474d-9899-f055fc94f4f4... PT-CD723205 2024-03-29T00:00:00 2024-04-22T00:00:00 1938-03-10 Jeffrey Williams Dr. Reyes arthritis I have been having trouble with my muscles and... **Visit Note - Dr. Angela Reyes** **Date:** ...
4 {'uuid': 'd605ea67-a2b5-461d-bd8b-05301e5d118b... {'uuid': '8948d6fb-5de5-44f9-b949-e99c98d2ece9... PT-A1FFC123 2024-12-08T00:00:00 2025-01-05T00:00:00 2001-12-18 Cole Serrano Dr. Williams dengue I have been feeling really sick. My body hurts... SOB: 7w F dengue‑like illness, fever 101‑102°,...
Python
1# Load the analysis results into memory.
2analysis = results.load_analysis()
3 
4analysis.to_report()
5
Output
──────────────────────────────────────── 🎨 Data Designer Dataset Profile ─────────────────────────────────────────

                                                                                                                   
                                                 Dataset Overview                                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ number of records               ┃ number of columns               ┃ percent complete records                    ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 10                              │ 10                              │ 100.0%                                      │
└─────────────────────────────────┴─────────────────────────────────┴─────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                                🎲 Sampler Columns                                                 
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ column name                   ┃       data type ┃             number unique values ┃               sampler type ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ patient_sampler               │            dict │                      10 (100.0%) │          person_from_faker │
├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤
│ doctor_sampler                │            dict │                      10 (100.0%) │          person_from_faker │
├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤
│ patient_id                    │          string │                      10 (100.0%) │                       uuid │
├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤
│ symptom_onset_date            │          string │                      10 (100.0%) │                   datetime │
├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤
│ date_of_visit                 │          string │                      10 (100.0%) │                  timedelta │
└───────────────────────────────┴─────────────────┴──────────────────────────────────┴────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                                📝 LLM-Text Columns                                                
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                       ┃               ┃                            ┃     prompt tokens ┃      completion tokens ┃
┃ column name           ┃     data type ┃       number unique values ┃        per record ┃             per record ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩
│ physician_notes       │        string │                10 (100.0%) │     131.5 +/- 5.6 │        663.0 +/- 453.9 │
└───────────────────────┴───────────────┴────────────────────────────┴───────────────────┴────────────────────────┘
                                                                                                                   
                                                                                                                   
                                               🧩 Expression Columns                                               
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ column name                    ┃                 data type ┃                               number unique values ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ first_name                     │                    string │                                        10 (100.0%) │
├────────────────────────────────┼───────────────────────────┼────────────────────────────────────────────────────┤
│ last_name                      │                    string │                                        10 (100.0%) │
├────────────────────────────────┼───────────────────────────┼────────────────────────────────────────────────────┤
│ dob                            │                    string │                                        10 (100.0%) │
├────────────────────────────────┼───────────────────────────┼────────────────────────────────────────────────────┤
│ physician                      │                    string │                                          9 (90.0%) │
└────────────────────────────────┴───────────────────────────┴────────────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
╭────────────────────────────────────────────────── Table Notes ──────────────────────────────────────────────────╮
│                                                                                                                 │
│  1. All token statistics are based on a sample of max(1000, len(dataset)) records.                              │
│  2. Tokens are calculated using tiktoken's cl100k_base tokenizer.                                               │
│                                                                                                                 │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
                                                                                                                   
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────

⏭️ Next Steps

Check out the following notebook to learn more about: