Skip to navigation
Tutorials

Seeding with an External Dataset

View as Markdown
Run in Google Colab

🎨 Data Designer Tutorial: Seeding Synthetic Data Generation with an External Dataset

📚 What you'll learn

In this notebook, we will demonstrate how to seed synthetic data generation in Data Designer with an external dataset.

If this is your first time using Data Designer, we recommend starting with the first notebook in this tutorial series.

📦 Import Data Designer

  • data_designer.config provides access to the configuration API.

  • DataDesigner is the main interface for data generation.

Python
1import data_designer.config as dd
2from data_designer.interface import DataDesigner
3

⚙️ Initialize the Data Designer interface

  • DataDesigner is the main object responsible for managing the data generation process.

  • When initialized without arguments, the default model providers are used.

Python
1data_designer = DataDesigner()
2

🎛️ Define model configurations

  • Each ModelConfig defines a model that can be used during the generation process.

  • The "model alias" is used to reference the model in the Data Designer config (as we will see below).

  • The "model provider" is the external service that hosts the model (see the model config docs for more details).

  • By default, we use build.nvidia.com as the model provider.

Python
1# This name is set in the model provider configuration.
2MODEL_PROVIDER = "nvidia"
3 
4# The model ID is from build.nvidia.com.
5MODEL_ID = "nvidia/nemotron-3.5-lightning-30b-a3b"
6 
7# We choose this alias to be descriptive for our use case.
8MODEL_ALIAS = "nemotron-lightning"
9 
10model_configs = [
11    dd.ModelConfig(
12        alias=MODEL_ALIAS,
13        model=MODEL_ID,
14        provider=MODEL_PROVIDER,
15        inference_parameters=dd.ChatCompletionInferenceParams(
16            temperature=1.0,
17            top_p=0.95,
18            max_tokens=2048,
19            extra_body={"chat_template_kwargs": {"enable_thinking": False}},
20        ),
21    )
22]
23

🏗️ Initialize the Data Designer Config Builder

  • The Data Designer config defines the dataset schema and generation process.

  • The config builder provides an intuitive interface for building this configuration.

  • The list of model configs is provided to the builder at initialization.

Python
1config_builder = dd.DataDesignerConfigBuilder(model_configs=model_configs)
2

🏥 Prepare a seed dataset

  • For this notebook, we'll create a synthetic dataset of patient notes.

  • We will seed the generation process with a symptom-to-diagnosis dataset.

  • The notebook downloads the source CSV before generation.


🌱 Why use a seed dataset?

  • Seed datasets let you steer the generation process by providing context that is specific to your use case.

  • Seed datasets are also an excellent way to inject real-world diversity into your synthetic data.

  • During generation, prompt templates can reference any of the seed dataset fields.

Python
1# Download sample dataset from Github
2import urllib.request
3 
4url = "https://raw.githubusercontent.com/NVIDIA/GenerativeAIExamples/refs/heads/main/nemo/NeMo-Data-Designer/data/gretelai_symptom_to_diagnosis.csv"
5local_filename, _ = urllib.request.urlretrieve(url, "gretelai_symptom_to_diagnosis.csv")
6 
7# Seed datasets are passed as reference objects to the config builder.
8seed_source = dd.LocalFileSeedSource(path=local_filename)
9 
10config_builder.with_seed_dataset(seed_source)
11
Output
DataDesignerConfigBuilder(
    seed_dataset: local seed
)

🎨 Designing our synthetic patient notes dataset

  • The prompt template can reference fields from our seed dataset:
    • {{ diagnosis }} - the medical diagnosis from the seed data
    • {{ patient_summary }} - the symptom description from the seed data
Python
1config_builder.add_column(
2    dd.SamplerColumnConfig(
3        name="patient_sampler",
4        sampler_type=dd.SamplerType.PERSON_FROM_FAKER,
5        params=dd.PersonFromFakerSamplerParams(),
6    )
7)
8 
9config_builder.add_column(
10    dd.SamplerColumnConfig(
11        name="doctor_sampler",
12        sampler_type=dd.SamplerType.PERSON_FROM_FAKER,
13        params=dd.PersonFromFakerSamplerParams(),
14    )
15)
16 
17config_builder.add_column(
18    dd.SamplerColumnConfig(
19        name="patient_id",
20        sampler_type=dd.SamplerType.UUID,
21        params=dd.UUIDSamplerParams(
22            prefix="PT-",
23            short_form=True,
24            uppercase=True,
25        ),
26    )
27)
28 
29config_builder.add_column(dd.ExpressionColumnConfig(name="first_name", expr="{{ patient_sampler.first_name }}"))
30 
31config_builder.add_column(dd.ExpressionColumnConfig(name="last_name", expr="{{ patient_sampler.last_name }}"))
32 
33config_builder.add_column(dd.ExpressionColumnConfig(name="dob", expr="{{ patient_sampler.birth_date }}"))
34 
35config_builder.add_column(
36    dd.SamplerColumnConfig(
37        name="symptom_onset_date",
38        sampler_type=dd.SamplerType.DATETIME,
39        params=dd.DatetimeSamplerParams(start="2024-01-01", end="2024-12-31"),
40    )
41)
42 
43config_builder.add_column(
44    dd.SamplerColumnConfig(
45        name="date_of_visit",
46        sampler_type=dd.SamplerType.TIMEDELTA,
47        params=dd.TimeDeltaSamplerParams(dt_min=1, dt_max=30, reference_column_name="symptom_onset_date"),
48    )
49)
50 
51config_builder.add_column(dd.ExpressionColumnConfig(name="physician", expr="Dr. {{ doctor_sampler.last_name }}"))
52 
53config_builder.add_column(
54    dd.LLMTextColumnConfig(
55        name="physician_notes",
56        prompt="""\
57You are a primary-care physician who just had an appointment with {{ first_name }} {{ last_name }},
58who has been struggling with symptoms from {{ diagnosis }} since {{ symptom_onset_date }}.
59The date of today's visit is {{ date_of_visit }}.
60 
61{{ patient_summary }}
62 
63Write careful notes about your visit with {{ first_name }},
64as Dr. {{ doctor_sampler.first_name }} {{ doctor_sampler.last_name }}.
65 
66Format the notes as a busy doctor might.
67Respond with only the notes, no other text.
68""",
69        model_alias=MODEL_ALIAS,
70    )
71)
72 
73data_designer.validate(config_builder)
74
Output
[16:11:23] [INFO] ✅ Validation passed

🔁 Iteration is key – preview the dataset!

  1. Use the preview method to generate a sample of records quickly.

  2. Inspect the results for quality and format issues.

  3. Adjust column configurations, prompts, or parameters as needed.

  4. Re-run the preview until satisfied.

Python
1preview = data_designer.preview(config_builder, num_records=2)
2
Output
[16:11:23] [INFO] 👁️ Preview generation in progress
[16:11:23] [INFO]   |-- 🔒 Jinja rendering engine: secure
[16:11:23] [INFO] ✅ Validation passed
[16:11:23] [INFO] ⛓️ Sorting column configs into a Directed Acyclic Graph
[16:11:23] [INFO] Skipping model health checks because DATA_DESIGNER_SKIP_MODEL_HEALTH_CHECKS=1
[16:11:23] [INFO] ⚡ Using async task-queue preview
[16:11:23] [INFO] 📝 llm-text model config for column 'physician_notes'
[16:11:23] [INFO]   |-- model: 'nvidia/nemotron-3.5-lightning-30b-a3b'
[16:11:23] [INFO]   |-- model alias: 'nemotron-lightning'
[16:11:23] [INFO]   |-- model provider: 'nvidia'
[16:11:23] [INFO]   |-- inference parameters:
[16:11:23] [INFO]   |  |-- generation_type=chat-completion
[16:11:23] [INFO]   |  |-- max_parallel_requests=4
[16:11:23] [INFO]   |  |-- extra_body={'chat_template_kwargs': {'enable_thinking': False}}
[16:11:23] [INFO]   |  |-- temperature=1.00
[16:11:23] [INFO]   |  |-- top_p=0.95
[16:11:23] [INFO]   |  |-- max_tokens=2048
[16:11:23] [INFO] ⚡️ Async generation: 1 column(s) (column 'physician_notes'), 2 tasks across 1 row group(s)
[16:11:23] [INFO] 🚀 (1/1) Dispatching with 2 records
[16:11:23] [INFO] 🎲 (1/1) Preparing samplers to generate 2 records across 5 columns
[16:11:23] [INFO] 🌱 (1/1) Sampling 2 records from seed dataset
[16:11:23] [INFO]   |-- seed dataset size: 820 records
[16:11:23] [INFO]   |-- sampling strategy: ordered
[16:11:23] [INFO] 🧩 (1/1) Generating column `dob` from expression
[16:11:23] [INFO] 🧩 (1/1) Generating column `first_name` from expression
[16:11:23] [INFO] 🧩 (1/1) Generating column `last_name` from expression
[16:11:23] [INFO] 🧩 (1/1) Generating column `physician` from expression
[16:11:24] [WARNING] Observed retryable model-task error: kind=rate_limit, http_status=429; the row task will be deferred.
[16:11:24] [INFO] 🔄 (1/1) Salvaging 2 deferred task(s)
[16:11:28] [INFO] 🔄 (1/1) Preserving 2 retryable task(s); records will keep retrying.
[16:11:57] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=20; deferred_tasks=2. Run may take longer than expected; salvage will retry these.
[16:12:42] [WARNING] 🔄 (1/1) Preserving 2 retryable task(s) after 10 deferred salvage cycle(s); records will keep retrying.
[16:12:57] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=49; deferred_tasks=1. Run may take longer than expected; salvage will retry these.
[16:13:57] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=78; deferred_tasks=2. Run may take longer than expected; salvage will retry these.
[16:14:06] [WARNING] 🔄 (1/1) Preserving 2 retryable task(s) after 20 deferred salvage cycle(s); records will keep retrying.
[16:14:24] [INFO] 📊 Progress [180.5s]:
[16:14:24] [INFO]   |-- 🌗 column 'physician_notes': 1/2 (50%) 0.0 rec/s
[16:14:24] [INFO] 🔄 (1/1) Preserving 1 retryable task(s); records will keep retrying.
[16:14:57] [WARNING] 🔄 (1/1) Preserving 1 retryable task(s) after 30 deferred salvage cycle(s); records will keep retrying.
[16:14:59] [WARNING] Provider showing degraded performance: 95% of last 20 task outcomes were retryable errors (rate_limit=19, success=1); cumulative outcomes: rate_limit=106, success=1; deferred_tasks=1. Run may take longer than expected; salvage will retry these.
[16:15:12] [INFO] 📊 Progress [228.4s]:
[16:15:12] [INFO]   |-- 🌕 column 'physician_notes': 2/2 (100%) 0.0 rec/s
[16:15:12] [INFO] ✅ Async generation complete [228.4s]: 2 ok, 0 failed across 1 column(s)
[16:15:12] [INFO] 📊 Model usage summary:
[16:15:12] [INFO]   |-- model: nvidia/nemotron-3.5-lightning-30b-a3b
[16:15:12] [INFO]   |-- tokens: input=327, output=1068, reasoning=0, total=1395, tps=6
[16:15:12] [INFO]   |-- requests: success=2, failed=107, total=109, rpm=28
[16:15:12] [INFO] 📐 Measuring dataset column statistics:
[16:15:12] [INFO]   |-- 🎲 column: 'patient_sampler'
[16:15:12] [INFO]   |-- 🎲 column: 'doctor_sampler'
[16:15:12] [INFO]   |-- 🎲 column: 'patient_id'
[16:15:12] [INFO]   |-- 🧩 column: 'first_name'
[16:15:12] [INFO]   |-- 🧩 column: 'last_name'
[16:15:12] [INFO]   |-- 🧩 column: 'dob'
[16:15:12] [INFO]   |-- 🎲 column: 'symptom_onset_date'
[16:15:12] [INFO]   |-- 🎲 column: 'date_of_visit'
[16:15:12] [INFO]   |-- 🧩 column: 'physician'
[16:15:12] [INFO]   |-- 📝 column: 'physician_notes'
[16:15:12] [INFO]   |-- 🌱 column: 'diagnosis'
[16:15:12] [INFO]   |-- 🌱 column: 'patient_summary'
[16:15:12] [INFO] 🙌 Preview complete!
Python
1# Run this cell multiple times to cycle through the 2 preview records.
2preview.display_sample_record()
3
Output
[index: 0]
                                                                                                              
                                                 Seed Columns                                                 
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name            ┃ Value                                                                                    ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ diagnosis       │ cervical spondylosis                                                                     │
├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤
│ patient_summary │ I've been having a lot of pain in my neck and back. I've also been having trouble with   │
│                 │ my balance and coordination. I've been coughing a lot and my limbs feel weak.            │
└─────────────────┴──────────────────────────────────────────────────────────────────────────────────────────┘
                                                                                                              
                                                                                                              
                                              Generated Columns                                               
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name               ┃ Value                                                                                 ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ patient_sampler    │ {                                                                                     │
│                    │     'uuid': '1bbc5b2d-fb4d-49fb-b96e-4fc4360ef6bf',                                   │
│                    │     'locale': 'en_US',                                                                │
│                    │     'first_name': 'Ronald',                                                           │
│                    │     'last_name': 'Dillon',                                                            │
│                    │     'middle_name': None,                                                              │
│                    │     'sex': 'Male',                                                                    │
│                    │     'street_number': '22143',                                                         │
│                    │     'street_name': 'Jensen Lodge',                                                    │
│                    │     'city': 'Washingtonhaven',                                                        │
│                    │     'state': 'Massachusetts',                                                         │
│                    │     'postcode': '44509',                                                              │
│                    │     'age': 78,                                                                        │
│                    │     'birth_date': '1948-07-01',                                                       │
│                    │     'country': 'Mauritania',                                                          │
│                    │     'marital_status': 'separated',                                                    │
│                    │     'education_level': 'doctorate',                                                   │
│                    │     'unit': '',                                                                       │
│                    │     'occupation': 'Psychologist, prison and probation services',                      │
│                    │     'phone_number': '(309)612-1954x50456',                                            │
│                    │     'bachelors_field': 'stem_related'                                                 │
│                    │ }                                                                                     │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ doctor_sampler     │ {                                                                                     │
│                    │     'uuid': 'dfd609b1-bbf0-4b87-a9c1-80f23e6c3aff',                                   │
│                    │     'locale': 'en_US',                                                                │
│                    │     'first_name': 'Robert',                                                           │
│                    │     'last_name': 'Bryant',                                                            │
│                    │     'middle_name': None,                                                              │
│                    │     'sex': 'Male',                                                                    │
│                    │     'street_number': '020',                                                           │
│                    │     'street_name': 'Joseph Avenue',                                                   │
│                    │     'city': 'East Craig',                                                             │
│                    │     'state': 'Connecticut',                                                           │
│                    │     'postcode': '19822',                                                              │
│                    │     'age': 38,                                                                        │
│                    │     'birth_date': '1988-10-29',                                                       │
│                    │     'country': 'French Guiana',                                                       │
│                    │     'marital_status': 'married_present',                                              │
│                    │     'education_level': 'secondary_education',                                         │
│                    │     'unit': '',                                                                       │
│                    │     'occupation': 'Location manager',                                                 │
│                    │     'phone_number': '(344)648-3784x162',                                              │
│                    │     'bachelors_field': 'no_degree'                                                    │
│                    │ }                                                                                     │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ patient_id         │ PT-130757B9                                                                           │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ symptom_onset_date │ 2024-07-24T00:00:00                                                                   │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ date_of_visit      │ 2024-08-22T00:00:00                                                                   │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ physician_notes    │ **DR. ROBERT BRYANT - OFFICE NOTES**                                                  │
│                    │                                                                                       │
│                    │ **Patient:** Ronald Dillon                                                            │
│                    │ **Date:** 2024-08-22                                                                  │
│                    │ **Chief Complaint:** Acute exacerbation of chronic cervical spondylosis; progressive  │
│                    │ imbalance and lower extremity weakness; persistent cough.                             │
│                    │                                                                                       │
│                    │ **HPI:** Patient reports a several-months history of neck and back pain originating   │
│                    │ 07/24/2024. He describes the pain as constant, increased with neck movement, and      │
│                    │ radiating to the shoulders. He now complains of significant trouble with balance and  │
│                    │ coordination, describing a feeling of "instability" when walking. He reports          │
│                    │ progressive weakness in all four limbs, worse in the lower extremities. He also       │
│                    │ reports a persistent, non-productive cough of several weeks' duration. No recent      │
│                    │ trauma. No bowel or bladder dysfunction denied.                                       │
│                    │                                                                                       │
│                    │ **ROS:**                                                                              │
│                    │ *   **Musculoskeletal:** Neck stiffness, limited ROM.                                 │
│                    │ *   **Neurological:** Gait instability, limb weakness (proximal > distal), possible   │
│                    │ myelopathy signs.                                                                     │
│                    │ *   **Respiratory:** Chronic cough, no hemoptysis, no fever/chills.                   │
│                    │ *   **Other:** No recent URI symptoms.                                                │
│                    │                                                                                       │
│                    │ **PE:**                                                                               │
│                    │ *   **Vitals:** BP 138/82, HR 76, RR 16, SpO2 98% RA.                                 │
│                    │ *   **General:** NAD.                                                                 │
│                    │ *   **Neuro:** Cranial nerves grossly intact. Gait: Wide-based, unsteady. Muscle      │
│                    │ strength: 4/5 in bilateral upper extremities; 3/5 in bilateral lower extremities      │
│                    │ (proximal predominance). Deep tendon reflexes: 2+ at knees, 1+ at ankles. Babinski    │
│                    │ sign: Equivocal (right), negative (left). Sensation: Diminished proprioception in     │
│                    │ lower extremities. Romberg's sign: Positive.                                          │
│                    │ *   **Musculoskeletal:** Cervical ROM severely limited due to pain and spasm.         │
│                    │ Tenderness over C5-C6 spinous processes. No lymphadenopathy.                          │
│                    │ *   **Pulmonary:** Clear to auscultation bilaterally.                                 │
│                    │                                                                                       │
│                    │ **ASSESSMENT:**                                                                       │
│                    │ 1.  Cervical spondylosis with myelopathy (C5-C6).                                     │
│                    │ 2.  Severe cervical radiculomyelopathy presenting with gait instability and lower     │
│                    │ extremity weakness.                                                                   │
│                    │ 3.  Chronic cough (etiology unclear; possible coincidental bronchitis vs. referred    │
│                    │ symptoms).                                                                            │
│                    │ 4.  Possible early sarcopenia/neuromuscular deconditioning secondary to chronic pain. │
│                    │                                                                                       │
│                    │ **PLAN:**                                                                             │
│                    │ *   **Urgent MRI:** Cervical spine without and with contrast to evaluate for cord     │
│                    │ compression/signal change.                                                            │
│                    │ *   **EMG/NCS:** To evaluate for radiculopathy vs. myelopathy.                        │
│                    │ *   **Referral:** Physical Therapy (cervical stabilization) once acute instability    │
│                    │ ruled out.                                                                            │
│                    │ *   **Pulmonary:** Chest X-ray to evaluate chronic cough etiology.                    │
│                    │ *   **Follow-up:** Schedule follow-up in 2 weeks pending imaging results.             │
│                    │ *   **Medication:** Prescribed low-dose Methocarbamol for muscle spasm; continue      │
│                    │ NSAID as tolerated. Instruct patient on fall precautions due to gait instability.     │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ first_name         │ Ronald                                                                                │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ last_name          │ Dillon                                                                                │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ dob                │ 1948-07-01                                                                            │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ physician          │ Dr. Bryant                                                                            │
└────────────────────┴───────────────────────────────────────────────────────────────────────────────────────┘
Python
1# The preview dataset is available as a pandas DataFrame.
2preview.dataset
3
Output
diagnosis patient_summary patient_sampler doctor_sampler patient_id symptom_onset_date date_of_visit dob first_name physician last_name physician_notes
0 cervical spondylosis I've been having a lot of pain in my neck and ... {'uuid': '1bbc5b2d-fb4d-49fb-b96e-4fc4360ef6bf... {'uuid': 'dfd609b1-bbf0-4b87-a9c1-80f23e6c3aff... PT-130757B9 2024-07-24T00:00:00 2024-08-22T00:00:00 1948-07-01 Ronald Dr. Bryant Dillon **DR. ROBERT BRYANT - OFFICE NOTES**\n\n**Pati...
1 impetigo I have a rash on my face that is getting worse... {'uuid': '651109f5-be8e-4a7b-ab41-2abc7459844a... {'uuid': 'e05066d8-e97d-4839-9b2a-2b8ce72e6c84... PT-61400B4C 2024-12-22T00:00:00 2024-12-25T00:00:00 1968-11-27 Jessica Dr. Kirk Jones **Patient:** Jessica Jones  \n**Date of Visit:...

📊 Analyze the generated data

  • Data Designer automatically generates a basic statistical analysis of the generated data.

  • This analysis is available via the analysis property of generation result objects.

Python
1# Print the analysis as a table.
2preview.analysis.to_report()
3
Output
──────────────────────────────────────── 🎨 Data Designer Dataset Profile ─────────────────────────────────────────

                                                                                                                   
                                                 Dataset Overview                                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ number of records               ┃ number of columns               ┃ percent complete records                    ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 2                               │ 12                              │ 100.0%                                      │
└─────────────────────────────────┴─────────────────────────────────┴─────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                              🌱 Seed-Dataset Columns                                              
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ column name                          ┃               data type ┃                           number unique values ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ diagnosis                            │                  string │                                     2 (100.0%) │
├──────────────────────────────────────┼─────────────────────────┼────────────────────────────────────────────────┤
│ patient_summary                      │                  string │                                     2 (100.0%) │
└──────────────────────────────────────┴─────────────────────────┴────────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                                🎲 Sampler Columns                                                 
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ column name                   ┃       data type ┃             number unique values ┃               sampler type ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ patient_sampler               │            dict │                       2 (100.0%) │          person_from_faker │
├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤
│ doctor_sampler                │            dict │                       2 (100.0%) │          person_from_faker │
├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤
│ patient_id                    │          string │                       2 (100.0%) │                       uuid │
├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤
│ symptom_onset_date            │          string │                       2 (100.0%) │                   datetime │
├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤
│ date_of_visit                 │          string │                       2 (100.0%) │                  timedelta │
└───────────────────────────────┴─────────────────┴──────────────────────────────────┴────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                                📝 LLM-Text Columns                                                
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                       ┃               ┃                            ┃     prompt tokens ┃      completion tokens ┃
┃ column name           ┃     data type ┃       number unique values ┃        per record ┃             per record ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩
│ physician_notes       │        string │                 2 (100.0%) │     134.0 +/- 4.0 │        528.0 +/- 199.4 │
└───────────────────────┴───────────────┴────────────────────────────┴───────────────────┴────────────────────────┘
                                                                                                                   
                                                                                                                   
                                               🧩 Expression Columns                                               
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ column name                    ┃                 data type ┃                               number unique values ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ first_name                     │                    string │                                         2 (100.0%) │
├────────────────────────────────┼───────────────────────────┼────────────────────────────────────────────────────┤
│ last_name                      │                    string │                                         2 (100.0%) │
├────────────────────────────────┼───────────────────────────┼────────────────────────────────────────────────────┤
│ dob                            │                    string │                                         2 (100.0%) │
├────────────────────────────────┼───────────────────────────┼────────────────────────────────────────────────────┤
│ physician                      │                    string │                                         2 (100.0%) │
└────────────────────────────────┴───────────────────────────┴────────────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
╭────────────────────────────────────────────────── Table Notes ──────────────────────────────────────────────────╮
│                                                                                                                 │
│  1. All token statistics are based on a sample of max(1000, len(dataset)) records.                              │
│  2. Tokens are calculated using tiktoken's cl100k_base tokenizer.                                               │
│                                                                                                                 │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
                                                                                                                   
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────

🆙 Scale up!

  • Happy with your preview data?

  • Use the create method to submit larger Data Designer generation jobs.

Python
1results = data_designer.create(config_builder, num_records=10, dataset_name="tutorial-3")
2
Output
[16:15:12] [INFO] OpenTelemetry metrics available at http://127.0.0.1:9464/metrics
[16:15:12] [INFO] 🎨 Creating Data Designer dataset
[16:15:12] [INFO]   |-- 🔒 Jinja rendering engine: secure
[16:15:12] [INFO] ✅ Validation passed
[16:15:12] [INFO] ⛓️ Sorting column configs into a Directed Acyclic Graph
[16:15:12] [INFO] Skipping model health checks because DATA_DESIGNER_SKIP_MODEL_HEALTH_CHECKS=1
[16:15:12] [INFO] ⚡ Using async task-queue builder
[16:15:12] [INFO] 📝 llm-text model config for column 'physician_notes'
[16:15:12] [INFO]   |-- model: 'nvidia/nemotron-3.5-lightning-30b-a3b'
[16:15:12] [INFO]   |-- model alias: 'nemotron-lightning'
[16:15:12] [INFO]   |-- model provider: 'nvidia'
[16:15:12] [INFO]   |-- inference parameters:
[16:15:12] [INFO]   |  |-- generation_type=chat-completion
[16:15:12] [INFO]   |  |-- max_parallel_requests=4
[16:15:12] [INFO]   |  |-- extra_body={'chat_template_kwargs': {'enable_thinking': False}}
[16:15:12] [INFO]   |  |-- temperature=1.00
[16:15:12] [INFO]   |  |-- top_p=0.95
[16:15:12] [INFO]   |  |-- max_tokens=2048
[16:15:12] [INFO] ⚡️ Async generation: 1 column(s) (column 'physician_notes'), 10 tasks across 1 row group(s)
[16:15:12] [INFO] 🚀 (1/1) Dispatching with 10 records
[16:15:12] [INFO] 🎲 (1/1) Preparing samplers to generate 10 records across 5 columns
[16:15:12] [INFO] 🌱 (1/1) Sampling 10 records from seed dataset
[16:15:12] [INFO]   |-- seed dataset size: 820 records
[16:15:12] [INFO]   |-- sampling strategy: ordered
[16:15:12] [INFO] 🧩 (1/1) Generating column `dob` from expression
[16:15:12] [INFO] 🧩 (1/1) Generating column `first_name` from expression
[16:15:12] [INFO] 🧩 (1/1) Generating column `last_name` from expression
[16:15:12] [INFO] 🧩 (1/1) Generating column `physician` from expression
[16:15:12] [WARNING] Observed retryable model-task error: kind=rate_limit, http_status=429; the row task will be deferred.
[16:15:31] [INFO] 🔄 (1/1) Salvaging 10 deferred task(s)
[16:15:41] [INFO] 📊 Progress [28.5s]:
[16:15:41] [INFO]   |-- 😴 column 'physician_notes': 1/10 (10%) 0.0 rec/s
[16:15:53] [WARNING] Provider showing degraded performance: 95% of last 20 task outcomes were retryable errors (rate_limit=19, success=1); cumulative outcomes: rate_limit=19, success=1; deferred_tasks=9. Run may take longer than expected; salvage will retry these.
[16:16:12] [INFO] 🔄 (1/1) Preserving 9 retryable task(s); records will keep retrying.
[16:16:53] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=48, success=1; deferred_tasks=2. Run may take longer than expected; salvage will retry these.
[16:17:54] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=77, success=1; deferred_tasks=4. Run may take longer than expected; salvage will retry these.
[16:18:54] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=106, success=1; deferred_tasks=6. Run may take longer than expected; salvage will retry these.
[16:19:54] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=135, success=1; deferred_tasks=8. Run may take longer than expected; salvage will retry these.
[16:20:55] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=164, success=1; deferred_tasks=1. Run may take longer than expected; salvage will retry these.
[16:21:15] [INFO] 📊 Progress [363.0s]:
[16:21:15] [INFO]   |-- 😴 column 'physician_notes': 2/10 (20%) 0.0 rec/s
[16:21:17] [INFO] 🔄 (1/1) Preserving 8 retryable task(s); records will keep retrying.
[16:21:50] [WARNING] 🔄 (1/1) Preserving 8 retryable task(s) after 10 deferred salvage cycle(s); records will keep retrying.
[16:21:55] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=189, success=2; deferred_tasks=2. Run may take longer than expected; salvage will retry these.
[16:22:55] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=218, success=2; deferred_tasks=7. Run may take longer than expected; salvage will retry these.
[16:23:15] [INFO] 📊 Progress [482.5s]:
[16:23:15] [INFO]   |-- 🥱 column 'physician_notes': 3/10 (30%) 0.0 rec/s
[16:23:31] [INFO] 🔄 (1/1) Preserving 7 retryable task(s); records will keep retrying.
[16:23:56] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=245, success=3; deferred_tasks=5. Run may take longer than expected; salvage will retry these.
[16:24:56] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=274, success=3; deferred_tasks=6. Run may take longer than expected; salvage will retry these.
[16:25:57] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=303, success=3; deferred_tasks=7. Run may take longer than expected; salvage will retry these.
[16:26:55] [WARNING] 🔄 (1/1) Preserving 7 retryable task(s) after 20 deferred salvage cycle(s); records will keep retrying.
[16:26:57] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=332, success=3; deferred_tasks=1. Run may take longer than expected; salvage will retry these.
[16:27:52] [INFO] 📊 Progress [760.2s]:
[16:27:52] [INFO]   |-- 🥱 column 'physician_notes': 4/10 (40%) 0.0 rec/s
[16:27:54] [INFO] 🔄 (1/1) Preserving 6 retryable task(s); records will keep retrying.
[16:27:59] [WARNING] Provider showing degraded performance: 95% of last 20 task outcomes were retryable errors (rate_limit=19, success=1); cumulative outcomes: rate_limit=360, success=4; deferred_tasks=2. Run may take longer than expected; salvage will retry these.
[16:28:59] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=389, success=4; deferred_tasks=1. Run may take longer than expected; salvage will retry these.
[16:30:39] [WARNING] Provider showing degraded performance: 95% of last 20 task outcomes were retryable errors (rate_limit=19, success=1); cumulative outcomes: rate_limit=415, success=5; deferred_tasks=3. Run may take longer than expected; salvage will retry these.
[16:30:39] [INFO] 📊 Progress [927.3s]:
[16:30:39] [INFO]   |-- 😐 column 'physician_notes': 5/10 (50%) 0.0 rec/s
[16:30:42] [INFO] 🔄 (1/1) Preserving 5 retryable task(s); records will keep retrying.
[16:31:03] [INFO] 📊 Progress [951.1s]:
[16:31:03] [INFO]   |-- 😐 column 'physician_notes': 6/10 (60%) 0.0 rec/s
[16:31:03] [INFO] 🔄 (1/1) Preserving 4 retryable task(s); records will keep retrying.
[16:31:36] [WARNING] 🔄 (1/1) Preserving 4 retryable task(s) after 30 deferred salvage cycle(s); records will keep retrying.
[16:31:41] [WARNING] Provider showing degraded performance: 95% of last 20 task outcomes were retryable errors (rate_limit=19, success=1); cumulative outcomes: rate_limit=444, success=6; deferred_tasks=2. Run may take longer than expected; salvage will retry these.
[16:32:41] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=473, success=6; deferred_tasks=3. Run may take longer than expected; salvage will retry these.
[16:33:41] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=502, success=6; deferred_tasks=4. Run may take longer than expected; salvage will retry these.
[16:34:23] [WARNING] 🔄 (1/1) Preserving 4 retryable task(s) after 40 deferred salvage cycle(s); records will keep retrying.
[16:34:42] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=531, success=6; deferred_tasks=1. Run may take longer than expected; salvage will retry these.
[16:35:42] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=560, success=6; deferred_tasks=2. Run may take longer than expected; salvage will retry these.
[16:36:42] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=589, success=6; deferred_tasks=3. Run may take longer than expected; salvage will retry these.
[16:37:09] [WARNING] 🔄 (1/1) Preserving 4 retryable task(s) after 50 deferred salvage cycle(s); records will keep retrying.
[16:37:43] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=618, success=6; deferred_tasks=4. Run may take longer than expected; salvage will retry these.
[16:38:43] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=647, success=6; deferred_tasks=1. Run may take longer than expected; salvage will retry these.
[16:39:43] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=676, success=6; deferred_tasks=2. Run may take longer than expected; salvage will retry these.
[16:39:55] [INFO] 📊 Progress [1482.6s]:
[16:39:55] [INFO]   |-- 😐 column 'physician_notes': 7/10 (70%) 0.0 rec/s
[16:39:57] [WARNING] 🔄 (1/1) Preserving 3 retryable task(s) after 60 deferred salvage cycle(s); records will keep retrying.
[16:40:45] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=704, success=7; deferred_tasks=2. Run may take longer than expected; salvage will retry these.
[16:41:45] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=733, success=7; deferred_tasks=1. Run may take longer than expected; salvage will retry these.
[16:42:02] [WARNING] 🔄 (1/1) Preserving 3 retryable task(s) after 70 deferred salvage cycle(s); records will keep retrying.
[16:42:45] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=762, success=7; deferred_tasks=3. Run may take longer than expected; salvage will retry these.
[16:43:46] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=791, success=7; deferred_tasks=2. Run may take longer than expected; salvage will retry these.
[16:44:07] [WARNING] 🔄 (1/1) Preserving 3 retryable task(s) after 80 deferred salvage cycle(s); records will keep retrying.
[16:44:46] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=819, success=7; deferred_tasks=3. Run may take longer than expected; salvage will retry these.
[16:45:46] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=848, success=7; deferred_tasks=2. Run may take longer than expected; salvage will retry these.
[16:46:13] [WARNING] 🔄 (1/1) Preserving 3 retryable task(s) after 90 deferred salvage cycle(s); records will keep retrying.
[16:46:47] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=877, success=7; deferred_tasks=1. Run may take longer than expected; salvage will retry these.
[16:47:47] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=906, success=7; deferred_tasks=3. Run may take longer than expected; salvage will retry these.
[16:48:18] [WARNING] 🔄 (1/1) Preserving 3 retryable task(s) after 100 deferred salvage cycle(s); records will keep retrying.
[16:48:47] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=935, success=7; deferred_tasks=2. Run may take longer than expected; salvage will retry these.
[16:49:48] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=964, success=7; deferred_tasks=1. Run may take longer than expected; salvage will retry these.
[16:49:56] [INFO] 📊 Progress [2083.7s]:
[16:49:56] [INFO]   |-- 😊 column 'physician_notes': 8/10 (80%) 0.0 rec/s
[16:49:58] [INFO] 🔄 (1/1) Preserving 2 retryable task(s); records will keep retrying.
[16:50:15] [WARNING] 🔄 (1/1) Preserving 2 retryable task(s) after 110 deferred salvage cycle(s); records will keep retrying.
[16:50:38] [INFO] 📊 Progress [2125.8s]:
[16:50:38] [INFO]   |-- 😊 column 'physician_notes': 9/10 (90%) 0.0 rec/s
[16:50:40] [INFO] 🔄 (1/1) Preserving 1 retryable task(s); records will keep retrying.
[16:50:48] [WARNING] Provider showing degraded performance: 95% of last 20 task outcomes were retryable errors (rate_limit=19, success=1); cumulative outcomes: rate_limit=989, success=9; deferred_tasks=1. Run may take longer than expected; salvage will retry these.
[16:51:09] [WARNING] 🔄 (1/1) Preserving 1 retryable task(s) after 120 deferred salvage cycle(s); records will keep retrying.
[16:51:49] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=1018, success=9; deferred_tasks=1. Run may take longer than expected; salvage will retry these.
[16:51:51] [WARNING] 🔄 (1/1) Preserving 1 retryable task(s) after 130 deferred salvage cycle(s); records will keep retrying.
[16:52:32] [WARNING] 🔄 (1/1) Preserving 1 retryable task(s) after 140 deferred salvage cycle(s); records will keep retrying.
[16:52:49] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=1047, success=9; deferred_tasks=1. Run may take longer than expected; salvage will retry these.
[16:53:14] [WARNING] 🔄 (1/1) Preserving 1 retryable task(s) after 150 deferred salvage cycle(s); records will keep retrying.
[16:53:49] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=1076, success=9; deferred_tasks=1. Run may take longer than expected; salvage will retry these.
[16:53:56] [WARNING] 🔄 (1/1) Preserving 1 retryable task(s) after 160 deferred salvage cycle(s); records will keep retrying.
[16:54:37] [WARNING] 🔄 (1/1) Preserving 1 retryable task(s) after 170 deferred salvage cycle(s); records will keep retrying.
[16:54:50] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=1105, success=9; deferred_tasks=1. Run may take longer than expected; salvage will retry these.
[16:55:19] [WARNING] 🔄 (1/1) Preserving 1 retryable task(s) after 180 deferred salvage cycle(s); records will keep retrying.
[16:55:50] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=1134, success=9; deferred_tasks=1. Run may take longer than expected; salvage will retry these.
[16:56:00] [WARNING] 🔄 (1/1) Preserving 1 retryable task(s) after 190 deferred salvage cycle(s); records will keep retrying.
[16:56:42] [WARNING] 🔄 (1/1) Preserving 1 retryable task(s) after 200 deferred salvage cycle(s); records will keep retrying.
[16:56:50] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=1163, success=9; deferred_tasks=1. Run may take longer than expected; salvage will retry these.
[16:57:24] [WARNING] 🔄 (1/1) Preserving 1 retryable task(s) after 210 deferred salvage cycle(s); records will keep retrying.
[16:57:51] [WARNING] Provider showing degraded performance: 100% of last 20 task outcomes were retryable errors (rate_limit=20); cumulative outcomes: rate_limit=1192, success=9; deferred_tasks=1. Run may take longer than expected; salvage will retry these.
[16:58:05] [WARNING] 🔄 (1/1) Preserving 1 retryable task(s) after 220 deferred salvage cycle(s); records will keep retrying.
[16:58:47] [WARNING] 🔄 (1/1) Preserving 1 retryable task(s) after 230 deferred salvage cycle(s); records will keep retrying.
[16:58:52] [WARNING] Provider showing degraded performance: 95% of last 20 task outcomes were retryable errors (rate_limit=19, success=1); cumulative outcomes: rate_limit=1219, success=10; deferred_tasks=0. Run may take longer than expected; salvage will retry these.
[16:58:52] [INFO] 📊 Progress [2620.3s]:
[16:58:52] [INFO]   |-- 🤩 column 'physician_notes': 10/10 (100%) 0.0 rec/s
[16:58:52] [INFO] ✅ Async generation complete [2620.3s]: 10 ok, 0 failed across 1 column(s)
[16:58:52] [INFO] 📊 Model usage summary:
[16:58:52] [INFO]   |-- model: nvidia/nemotron-3.5-lightning-30b-a3b
[16:58:52] [INFO]   |-- tokens: input=1615, output=4528, reasoning=0, total=6143, tps=2
[16:58:52] [INFO]   |-- requests: success=10, failed=1219, total=1229, rpm=28
[16:58:52] [INFO] 📐 Measuring dataset column statistics:
[16:58:52] [INFO]   |-- 🎲 column: 'patient_sampler'
[16:58:52] [INFO]   |-- 🎲 column: 'doctor_sampler'
[16:58:52] [INFO]   |-- 🎲 column: 'patient_id'
[16:58:52] [INFO]   |-- 🧩 column: 'first_name'
[16:58:52] [INFO]   |-- 🧩 column: 'last_name'
[16:58:52] [INFO]   |-- 🧩 column: 'dob'
[16:58:52] [INFO]   |-- 🎲 column: 'symptom_onset_date'
[16:58:52] [INFO]   |-- 🎲 column: 'date_of_visit'
[16:58:52] [INFO]   |-- 🧩 column: 'physician'
[16:58:52] [INFO]   |-- 📝 column: 'physician_notes'
[16:58:52] [INFO]   |-- 🌱 column: 'diagnosis'
[16:58:52] [INFO]   |-- 🌱 column: 'patient_summary'
Python
1# Load the generated dataset as a pandas DataFrame.
2dataset = results.load_dataset()
3 
4dataset.head()
5
Output
patient_sampler doctor_sampler patient_id symptom_onset_date date_of_visit dob diagnosis patient_summary first_name last_name physician physician_notes
0 {'uuid': '70afdbd5-1ef0-4e48-8697-d2d85cd566d5... {'uuid': 'a43bc73a-e5f2-4fcc-af91-fa68763caa37... PT-C18CFFF2 2024-03-16T00:00:00 2024-03-24T00:00:00 1972-07-19 cervical spondylosis I've been having a lot of pain in my neck and ... Daniel Keith Dr. Gilmore **Patient:** Daniel Keith   **DOB:** [Redacted...
1 {'uuid': '84432e6a-dec1-4be5-ba43-f620100c3bc5... {'uuid': '03ee6c66-3e97-4a07-9f61-3449b2a20ce0... PT-61FAAEBD 2024-12-07T00:00:00 2024-12-30T00:00:00 1930-01-06 impetigo I have a rash on my face that is getting worse... Brianna Lamb Dr. Collins Dr. Susan Collins - Patient Notes Date: 2024-1...
2 {'uuid': 'd2e2cc16-2d76-4f96-b2ed-efca0aab13ca... {'uuid': 'a3e5547d-5593-4cfd-af7e-2ef5d03167f3... PT-0AA60C77 2024-11-20T00:00:00 2024-11-22T00:00:00 1955-08-01 urinary tract infection I have been urinating blood. I sometimes feel ... Ashley Donovan Dr. Smith **Patient:** Ashley Donovan **Date:** 2024-11-...
3 {'uuid': '8e3b3fce-188c-4e97-a52c-55c8427be572... {'uuid': '54babe1a-fa56-4a95-bfc0-f56274a6303a... PT-31D1388A 2024-02-18T00:00:00 2024-03-16T00:00:00 1945-12-23 arthritis I have been having trouble with my muscles and... Jessica Oconnor Dr. Sutton **Patient:** Jessica O'Connor   **Date of Visi...
4 {'uuid': 'a3260d98-0b5b-4b8e-a8e9-2b6eff48015b... {'uuid': '2257a688-5f2f-4a9b-8485-1bf6349b4f8e... PT-62B89D62 2024-03-17T00:00:00 2024-03-31T00:00:00 1986-03-15 dengue I have been feeling really sick. My body hurts... Leslie Frey Dr. Hanson **PATIENT NAME:** Leslie Frey   **DOB:** [Reda...
Python
1# Load the analysis results into memory.
2analysis = results.load_analysis()
3 
4analysis.to_report()
5
Output
──────────────────────────────────────── 🎨 Data Designer Dataset Profile ─────────────────────────────────────────

                                                                                                                   
                                                 Dataset Overview                                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ number of records               ┃ number of columns               ┃ percent complete records                    ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 10                              │ 12                              │ 100.0%                                      │
└─────────────────────────────────┴─────────────────────────────────┴─────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                              🌱 Seed-Dataset Columns                                              
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ column name                          ┃               data type ┃                           number unique values ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ diagnosis                            │                  string │                                      7 (70.0%) │
├──────────────────────────────────────┼─────────────────────────┼────────────────────────────────────────────────┤
│ patient_summary                      │                  string │                                    10 (100.0%) │
└──────────────────────────────────────┴─────────────────────────┴────────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                                🎲 Sampler Columns                                                 
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ column name                   ┃       data type ┃             number unique values ┃               sampler type ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ patient_sampler               │            dict │                      10 (100.0%) │          person_from_faker │
├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤
│ doctor_sampler                │            dict │                      10 (100.0%) │          person_from_faker │
├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤
│ patient_id                    │          string │                      10 (100.0%) │                       uuid │
├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤
│ symptom_onset_date            │          string │                      10 (100.0%) │                   datetime │
├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤
│ date_of_visit                 │          string │                        9 (90.0%) │                  timedelta │
└───────────────────────────────┴─────────────────┴──────────────────────────────────┴────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                                📝 LLM-Text Columns                                                
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                       ┃               ┃                            ┃     prompt tokens ┃      completion tokens ┃
┃ column name           ┃     data type ┃       number unique values ┃        per record ┃             per record ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩
│ physician_notes       │        string │                10 (100.0%) │     130.5 +/- 5.7 │        396.5 +/- 173.6 │
└───────────────────────┴───────────────┴────────────────────────────┴───────────────────┴────────────────────────┘
                                                                                                                   
                                                                                                                   
                                               🧩 Expression Columns                                               
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ column name                    ┃                 data type ┃                               number unique values ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ first_name                     │                    string │                                        10 (100.0%) │
├────────────────────────────────┼───────────────────────────┼────────────────────────────────────────────────────┤
│ last_name                      │                    string │                                        10 (100.0%) │
├────────────────────────────────┼───────────────────────────┼────────────────────────────────────────────────────┤
│ dob                            │                    string │                                        10 (100.0%) │
├────────────────────────────────┼───────────────────────────┼────────────────────────────────────────────────────┤
│ physician                      │                    string │                                          9 (90.0%) │
└────────────────────────────────┴───────────────────────────┴────────────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
╭────────────────────────────────────────────────── Table Notes ──────────────────────────────────────────────────╮
│                                                                                                                 │
│  1. All token statistics are based on a sample of max(1000, len(dataset)) records.                              │
│  2. Tokens are calculated using tiktoken's cl100k_base tokenizer.                                               │
│                                                                                                                 │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
                                                                                                                   
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────

⏭️ Next Steps

Check out the following notebook to learn more about: