NeMo Data DesignerIn this notebook, we will demonstrate how to seed synthetic data generation in Data Designer with an external dataset.
If this is your first time using Data Designer, we recommend starting with the first notebook in this tutorial series.
data_designer.config provides access to the configuration API.
DataDesigner is the main interface for data generation.
DataDesigner is the main object responsible for managing the data generation process.
When initialized without arguments, the default model providers are used.
Each ModelConfig defines a model that can be used during the generation process.
The "model alias" is used to reference the model in the Data Designer config (as we will see below).
The "model provider" is the external service that hosts the model (see the model config docs for more details).
By default, we use build.nvidia.com as the model provider.
The Data Designer config defines the dataset schema and generation process.
The config builder provides an intuitive interface for building this configuration.
The list of model configs is provided to the builder at initialization.
For this notebook, we'll create a synthetic dataset of patient notes.
We will seed the generation process with a symptom-to-diagnosis dataset.
We already have the dataset downloaded in the data directory of this repository.
🌱 Why use a seed dataset?
Seed datasets let you steer the generation process by providing context that is specific to your use case.
Seed datasets are also an excellent way to inject real-world diversity into your synthetic data.
During generation, prompt templates can reference any of the seed dataset fields.
DataDesignerConfigBuilder( seed_dataset: local seed )
{{ diagnosis }} - the medical diagnosis from the seed data{{ patient_summary }} - the symptom description from the seed data[21:16:05] [INFO] ✅ Validation passed
Use the preview method to generate a sample of records quickly.
Inspect the results for quality and format issues.
Adjust column configurations, prompts, or parameters as needed.
Re-run the preview until satisfied.
[21:16:05] [INFO] 🔭 Preview generation in progress
[21:16:05] [INFO] |-- 🔒 Jinja rendering engine: secure
[21:16:05] [INFO] ✅ Validation passed
[21:16:05] [INFO] ⛓️ Sorting column configs into a Directed Acyclic Graph
[21:16:05] [INFO] 🩺 Running health checks for models...
[21:16:05] [INFO] |-- 👀 Checking 'nvidia/nemotron-3-nano-30b-a3b' in provider named 'nvidia' for model alias 'nemotron-nano-v3'...
[21:16:05] [INFO] |-- ✅ Passed!
[21:16:05] [INFO] ⚡ DATA_DESIGNER_ASYNC_ENGINE is enabled - using async task-queue preview
[21:16:05] [INFO] 📝 llm-text model config for column 'physician_notes'
[21:16:05] [INFO] |-- model: 'nvidia/nemotron-3-nano-30b-a3b'
[21:16:05] [INFO] |-- model alias: 'nemotron-nano-v3'
[21:16:05] [INFO] |-- model provider: 'nvidia'
[21:16:05] [INFO] |-- inference parameters:
[21:16:05] [INFO] | |-- generation_type=chat-completion
[21:16:05] [INFO] | |-- max_parallel_requests=4
[21:16:05] [INFO] | |-- extra_body={'chat_template_kwargs': {'enable_thinking': False}}[21:16:05] [INFO] | |-- temperature=1.00
[21:16:05] [INFO] | |-- top_p=1.00
[21:16:05] [INFO] | |-- max_tokens=2048
[21:16:05] [INFO] ⚡️ Async generation: 1 column(s) (physician_notes), 2 tasks across 1 row group(s)
[21:16:05] [INFO] 🚀 (1/1) Dispatching with 2 records
[21:16:05] [INFO] 🎲 (1/1) Preparing samplers to generate 2 records across 5 columns
[21:16:05] [INFO] 🌱 (1/1) Sampling 2 records from seed dataset
[21:16:05] [INFO] |-- seed dataset size: 820 records
[21:16:05] [INFO] |-- sampling strategy: ordered
[21:16:05] [INFO] 🧩 (1/1) Generating column `last_name` from expression
[21:16:05] [INFO] 🧩 (1/1) Generating column `first_name` from expression
[21:16:05] [INFO] 🧩 (1/1) Generating column `dob` from expression
[21:16:05] [INFO] 🧩 (1/1) Generating column `physician` from expression
[21:16:11] [INFO] 📊 Progress [5.4s]:
[21:16:11] [INFO] |-- 🦁 physician_notes: 2/2 (100%) 0.4 rec/s
[21:16:11] [INFO] ✅ Async generation complete [5.4s]: 2 ok, 0 failed across 1 column(s)
[21:16:11] [INFO] 📊 Model usage summary:
[21:16:11] [INFO] |-- model: nvidia/nemotron-3-nano-30b-a3b
[21:16:11] [INFO] |-- tokens: input=330, output=2097, total=2427, tps=447
[21:16:11] [INFO] |-- requests: success=2, failed=0, total=2, rpm=22
[21:16:11] [INFO] 📐 Measuring dataset column statistics:
[21:16:11] [INFO] |-- 🎲 column: 'patient_sampler'
[21:16:11] [INFO] |-- 🎲 column: 'doctor_sampler'
[21:16:11] [INFO] |-- 🎲 column: 'patient_id'
[21:16:11] [INFO] |-- 🧩 column: 'first_name'
[21:16:11] [INFO] |-- 🧩 column: 'last_name'
[21:16:11] [INFO] |-- 🧩 column: 'dob'
[21:16:11] [INFO] |-- 🎲 column: 'symptom_onset_date'
[21:16:11] [INFO] |-- 🎲 column: 'date_of_visit'
[21:16:11] [INFO] |-- 🧩 column: 'physician'
[21:16:11] [INFO] |-- 📝 column: 'physician_notes'
[21:16:11] [INFO] 🥳 Preview complete!
Seed Columns ┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Name ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ diagnosis │ cervical spondylosis │ ├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤ │ patient_summary │ I've been having a lot of pain in my neck and back. I've also been having trouble with │ │ │ my balance and coordination. I've been coughing a lot and my limbs feel weak. │ └─────────────────┴──────────────────────────────────────────────────────────────────────────────────────────┘ Generated Columns ┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Name ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ patient_sampler │ { │ │ │ 'uuid': 'e6cc69aa-eb8c-4d4d-85e5-b4567442f910', │ │ │ 'locale': 'en_US', │ │ │ 'first_name': 'Steven', │ │ │ 'last_name': 'Scott', │ │ │ 'middle_name': None, │ │ │ 'sex': 'Male', │ │ │ 'street_number': '921', │ │ │ 'street_name': 'Natalie Shoals', │ │ │ 'city': 'New Traciebury', │ │ │ 'state': 'New Jersey', │ │ │ 'postcode': '86377', │ │ │ 'age': 101, │ │ │ 'birth_date': '1925-03-31', │ │ │ 'country': 'Belarus', │ │ │ 'marital_status': 'never_married', │ │ │ 'education_level': 'some_college', │ │ │ 'unit': '', │ │ │ 'occupation': 'Waste management officer', │ │ │ 'phone_number': '(600)971-3499x343', │ │ │ 'bachelors_field': 'no_degree' │ │ │ } │ ├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤ │ doctor_sampler │ { │ │ │ 'uuid': '4ff220cc-0ee1-4a8c-ac72-bc0dd5f94aa5', │ │ │ 'locale': 'en_US', │ │ │ 'first_name': 'Nancy', │ │ │ 'last_name': 'Mcneil', │ │ │ 'middle_name': None, │ │ │ 'sex': 'Female', │ │ │ 'street_number': '407', │ │ │ 'street_name': 'Gonzales Cliff', │ │ │ 'city': 'South Rhonda', │ │ │ 'state': 'Colorado', │ │ │ 'postcode': '72426', │ │ │ 'age': 56, │ │ │ 'birth_date': '1970-02-28', │ │ │ 'country': 'Iraq', │ │ │ 'marital_status': 'never_married', │ │ │ 'education_level': 'associates', │ │ │ 'unit': '', │ │ │ 'occupation': 'Psychologist, counselling', │ │ │ 'phone_number': '+1-777-863-7905x1240', │ │ │ 'bachelors_field': 'no_degree' │ │ │ } │ ├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤ │ patient_id │ PT-29834168 │ ├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤ │ symptom_onset_date │ 2024-02-09T00:00:00 │ ├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤ │ date_of_visit │ 2024-03-07T00:00:00 │ ├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤ │ physician_notes │ **Patient:** Steven Scott │ │ │ **DOB:** [Not provided] │ │ │ **Date of Visit:** 2024-03-07 │ │ │ **Chief Complaint:** "Neck and back pain, balance issues, chronic cough, limb │ │ │ weakness." │ │ │ **HPI:** 65M with history of cervical spondylosis (diagnosed 2024-02-09). Reports │ │ │ worsening neck/back pain, subjective imbalance/coordination problems, recurrent cough │ │ │ (non-smoker, no recent URI), and proximal muscle weakness (difficulty rising from │ │ │ chairs, climbing stairs). Symptoms constant, non-traumatic, no radicular symptoms (no │ │ │ arm pain/numbness). Denies bowel/bladder changes, fever, weight loss, or trauma. │ │ │ **PMH:** Controlled HTN, hyperlipidemia. No prior spine surgery. │ │ │ **Medications:** Lisinopril, Atorvastatin. │ │ │ **Allergies:** NKDA. │ │ │ **ROS:** │ │ │ - Neuro: New imbalance, distal weakness (legs more than arms), no myelopathy signs │ │ │ (no gait deterioration, bladder issues). │ │ │ - Musculoskeletal: Chronic neck/back pain, stiffness. │ │ │ - Respiratory: Chronic cough (dry, no hemoptysis). │ │ │ - GI: No nausea/vomiting. │ │ │ - Skin: No rashes. │ │ │ **Physical Exam:** │ │ │ - Vitals: RR 18, T 98.6°F, BP 138/88, P 72, SpO₂ 98% RA. │ │ │ - Neuro: │ │ │ - Strength: 4/5 in proximal leg muscles (quadriceps/hamstrings), 5/5 distal; 4/5 │ │ │ proximal arm, 5/5 distal. │ │ │ - Reflexes: Diminished patellar (knees), biceps normal. │ │ │ - Coordination: Positive Romberg test (unsteadiness with eyes closed). │ │ │ - Sensation: Reduced proprioception in feet (vibration sense). │ │ │ - Musculoskeletal: Cervical/ lumbar tenderness, limited ROM (neck flexion/extension). │ │ │ - No clonus, Babinski negative. │ │ │ - Other: Normal heart/lungs. │ │ │ **Assessment:** │ │ │ - 1. Cervical spondylosis with progressive myelopathic symptoms? │ │ │ - *Rationale:* New neuro deficits (weakness, balance), cervical tenderness, ROM │ │ │ limitation. **Rule out myelopathy** vs. peripheral neuropathy (vibration loss) or │ │ │ vascular cause. │ │ │ - 2. Chronic cough etiology? │ │ │ - *Rationale:* Non-smoker, no URI, but persistent cough warrants evaluation (e.g., │ │ │ GERD, ACEi-related, or less likely respiratory). │ │ │ - 3. Peripheral neuropathy? │ │ │ - *Rationale:* Vibration loss, distal weakness, but pattern more proximal—suggests │ │ │ cervical pathology vs. primary neuropathy. │ │ │ - 4. Secondary concerns: HTN control, medication adherence. │ │ │ **Plan:** │ │ │ - **Neuro:** │ │ │ - Order cervical MRI (cervical spine, T1-T2, sagittal/dynamic flexion-extension). │ │ │ - Consider EMG/NCS if MRI negative (evaluate for neuropathy). │ │ │ - **Urgent referral to Neurology** for myelopathy evaluation (e.g., gait │ │ │ assessment, SSEPs). │ │ │ - **Cough:** │ │ │ - Discontinue ACEi? *No*—patient on lisinopril (no recent cough onset │ │ │ post-initiation). │ │ │ - Trial PPI (e.g., omeprazole) for GERD; refer to PCP/gastro if no improvement. │ │ │ - **Imaging:** │ │ │ - Order X-ray: Cervical spine AP/LAT, lateral views. │ │ │ - Follow-up in 4 weeks to review imaging/neuro referral. │ │ │ - **Symptom Management:** │ │ │ - Avoid neck hyperextension; ergonomic adjustments. │ │ │ - PT referral for cervical stabilization exercises. │ │ │ - **Reassess fall risk** (home safety, footwear). │ │ │ - **Follow-Up:** │ │ │ - Return in 4 weeks for imaging results, neuro consult, and symptom review. │ │ │ - **Do not delay** neurology referral due to myelopathic features. │ │ │ **Provider Notes:** *High suspicion for progressive cervical myelopathy—neuro workup │ │ │ is priority. Avoid attributing cough to spondylosis; investigate independently. │ │ │ Patient anxious about "neurological decline"; explain steps clearly to reduce │ │ │ concern.* │ ├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤ │ first_name │ Steven │ ├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤ │ last_name │ Scott │ ├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤ │ dob │ 1925-03-31 │ ├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤ │ physician │ Dr. Mcneil │ └────────────────────┴───────────────────────────────────────────────────────────────────────────────────────┘
| diagnosis | patient_summary | patient_sampler | doctor_sampler | patient_id | symptom_onset_date | date_of_visit | last_name | first_name | dob | physician | physician_notes | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | cervical spondylosis | I've been having a lot of pain in my neck and ... | {'uuid': 'e6cc69aa-eb8c-4d4d-85e5-b4567442f910... | {'uuid': '4ff220cc-0ee1-4a8c-ac72-bc0dd5f94aa5... | PT-29834168 | 2024-02-09T00:00:00 | 2024-03-07T00:00:00 | Scott | Steven | 1925-03-31 | Dr. Mcneil | **Patient:** Steven Scott \n**DOB:** [Not pro... |
| 1 | impetigo | I have a rash on my face that is getting worse... | {'uuid': 'e85cd49a-9e70-4a8f-806f-bf374146929f... | {'uuid': '9ab39446-0532-4c2d-9f21-882082f4c215... | PT-EA6760F8 | 2024-08-08T00:00:00 | 2024-08-18T00:00:00 | Wright | Andrew | 2000-07-06 | Dr. Villarreal | **Progress Notes - Dr. John Villarreal** \n**... |
Data Designer automatically generates a basic statistical analysis of the generated data.
This analysis is available via the analysis property of generation result objects.
──────────────────────────────────────── 🎨 Data Designer Dataset Profile ───────────────────────────────────────── Dataset Overview ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ number of records ┃ number of columns ┃ percent complete records ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ 2 │ 10 │ 100.0% │ └─────────────────────────────────┴─────────────────────────────────┴─────────────────────────────────────────────┘ 🎲 Sampler Columns ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ column name ┃ data type ┃ number unique values ┃ sampler type ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ patient_sampler │ dict │ 2 (100.0%) │ person_from_faker │ ├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤ │ doctor_sampler │ dict │ 2 (100.0%) │ person_from_faker │ ├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤ │ patient_id │ string │ 2 (100.0%) │ uuid │ ├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤ │ symptom_onset_date │ string │ 2 (100.0%) │ datetime │ ├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤ │ date_of_visit │ string │ 2 (100.0%) │ timedelta │ └───────────────────────────────┴─────────────────┴──────────────────────────────────┴────────────────────────────┘ 📝 LLM-Text Columns ┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ ┃ ┃ ┃ prompt tokens ┃ completion tokens ┃ ┃ column name ┃ data type ┃ number unique values ┃ per record ┃ per record ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩ │ physician_notes │ string │ 2 (100.0%) │ 136.0 +/- 4.0 │ 994.5 +/- 67.2 │ └───────────────────────┴───────────────┴────────────────────────────┴───────────────────┴────────────────────────┘ 🧩 Expression Columns ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ column name ┃ data type ┃ number unique values ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ first_name │ string │ 2 (100.0%) │ ├────────────────────────────────┼───────────────────────────┼────────────────────────────────────────────────────┤ │ last_name │ string │ 2 (100.0%) │ ├────────────────────────────────┼───────────────────────────┼────────────────────────────────────────────────────┤ │ dob │ string │ 2 (100.0%) │ ├────────────────────────────────┼───────────────────────────┼────────────────────────────────────────────────────┤ │ physician │ string │ 2 (100.0%) │ └────────────────────────────────┴───────────────────────────┴────────────────────────────────────────────────────┘ ╭────────────────────────────────────────────────── Table Notes ──────────────────────────────────────────────────╮ │ │ │ 1. All token statistics are based on a sample of max(1000, len(dataset)) records. │ │ 2. Tokens are calculated using tiktoken's cl100k_base tokenizer. │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Happy with your preview data?
Use the create method to submit larger Data Designer generation jobs.
[21:16:11] [INFO] 🎨 Creating Data Designer dataset
[21:16:11] [INFO] |-- 🔒 Jinja rendering engine: secure
[21:16:11] [INFO] ✅ Validation passed
[21:16:11] [INFO] ⛓️ Sorting column configs into a Directed Acyclic Graph
[21:16:11] [INFO] 🩺 Running health checks for models...
[21:16:11] [INFO] |-- 👀 Checking 'nvidia/nemotron-3-nano-30b-a3b' in provider named 'nvidia' for model alias 'nemotron-nano-v3'...
[21:16:11] [INFO] |-- ✅ Passed!
[21:16:11] [INFO] ⚡ DATA_DESIGNER_ASYNC_ENGINE is enabled - using async task-queue builder
[21:16:11] [INFO] 📝 llm-text model config for column 'physician_notes'
[21:16:11] [INFO] |-- model: 'nvidia/nemotron-3-nano-30b-a3b'
[21:16:11] [INFO] |-- model alias: 'nemotron-nano-v3'
[21:16:11] [INFO] |-- model provider: 'nvidia'
[21:16:11] [INFO] |-- inference parameters:
[21:16:11] [INFO] | |-- generation_type=chat-completion
[21:16:11] [INFO] | |-- max_parallel_requests=4
[21:16:11] [INFO] | |-- extra_body={'chat_template_kwargs': {'enable_thinking': False}}[21:16:11] [INFO] | |-- temperature=1.00
[21:16:11] [INFO] | |-- top_p=1.00
[21:16:11] [INFO] | |-- max_tokens=2048
[21:16:11] [INFO] ⚡️ Async generation: 1 column(s) (physician_notes), 10 tasks across 1 row group(s)
[21:16:11] [INFO] 🚀 (1/1) Dispatching with 10 records
[21:16:11] [INFO] 🎲 (1/1) Preparing samplers to generate 10 records across 5 columns
[21:16:11] [INFO] 🧩 (1/1) Generating column `last_name` from expression
[21:16:11] [INFO] 🧩 (1/1) Generating column `first_name` from expression
[21:16:11] [INFO] 🧩 (1/1) Generating column `dob` from expression
[21:16:11] [INFO] 🧩 (1/1) Generating column `physician` from expression
[21:16:11] [INFO] 🌱 (1/1) Sampling 10 records from seed dataset
[21:16:11] [INFO] |-- seed dataset size: 820 records
[21:16:11] [INFO] |-- sampling strategy: ordered
[21:16:18] [INFO] 📊 Progress [6.5s]:
[21:16:18] [INFO] |-- 😸 physician_notes: 5/10 (50%) 0.8 rec/s
[21:16:24] [INFO] 📊 Progress [12.9s]:
[21:16:24] [INFO] |-- 😼 physician_notes: 9/10 (90%) 0.7 rec/s
[21:16:25] [INFO] 📊 Progress [14.0s]:
[21:16:25] [INFO] |-- 🦁 physician_notes: 10/10 (100%) 0.7 rec/s
[21:16:25] [INFO] ✅ Async generation complete [14.0s]: 10 ok, 0 failed across 1 column(s)
[21:16:25] [INFO] 📊 Model usage summary:
[21:16:25] [INFO] |-- model: nvidia/nemotron-3-nano-30b-a3b
[21:16:25] [INFO] |-- tokens: input=1616, output=10000, total=11616, tps=818
[21:16:25] [INFO] |-- requests: success=10, failed=0, total=10, rpm=42
[21:16:25] [INFO] 📐 Measuring dataset column statistics:
[21:16:25] [INFO] |-- 🎲 column: 'patient_sampler'
[21:16:25] [INFO] |-- 🎲 column: 'doctor_sampler'
[21:16:25] [INFO] |-- 🎲 column: 'patient_id'
[21:16:25] [INFO] |-- 🧩 column: 'first_name'
[21:16:25] [INFO] |-- 🧩 column: 'last_name'
[21:16:25] [INFO] |-- 🧩 column: 'dob'
[21:16:25] [INFO] |-- 🎲 column: 'symptom_onset_date'
[21:16:25] [INFO] |-- 🎲 column: 'date_of_visit'
[21:16:25] [INFO] |-- 🧩 column: 'physician'
[21:16:25] [INFO] |-- 📝 column: 'physician_notes'
| patient_sampler | doctor_sampler | patient_id | symptom_onset_date | date_of_visit | last_name | first_name | dob | physician | diagnosis | patient_summary | physician_notes | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | {'age': 109, 'bachelors_field': 'no_degree', '... | {'age': 60, 'bachelors_field': 'no_degree', 'b... | PT-7BFB69D8 | 2024-01-02T00:00:00 | 2024-01-05T00:00:00 | Barnett | Jessica | 1916-12-17 | Dr. Elliott | cervical spondylosis | I've been having a lot of pain in my neck and ... | **2024-01-05T00:00:00 | J. BARNETT, 62F | NEW ... |
| 1 | {'age': 54, 'bachelors_field': 'stem', 'birth_... | {'age': 60, 'bachelors_field': 'business', 'bi... | PT-FDA0BB9B | 2024-02-26T00:00:00 | 2024-03-09T00:00:00 | Brown | Martin | 1971-11-30 | Dr. Williams | impetigo | I have a rash on my face that is getting worse... | **Patient:** Martin Brown **DOB:** 01/12/197... |
| 2 | {'age': 33, 'bachelors_field': 'arts_humanitie... | {'age': 37, 'bachelors_field': 'business', 'bi... | PT-AEF4DE2F | 2024-06-23T00:00:00 | 2024-07-07T00:00:00 | Hicks | Ernest | 1993-02-03 | Dr. Waters | urinary tract infection | I have been urinating blood. I sometimes feel ... | **Visit Summary – Ernest Hicks – 2024‑07‑07** ... |
| 3 | {'age': 54, 'bachelors_field': 'no_degree', 'b... | {'age': 113, 'bachelors_field': 'stem', 'birth... | PT-BF6CE3C1 | 2024-06-03T00:00:00 | 2024-07-01T00:00:00 | Stephenson | Jose | 1972-04-19 | Dr. Rose | arthritis | I have been having trouble with my muscles and... | * 7/1/24 08:15 – Pt presents with 3 wk hx of w... |
| 4 | {'age': 41, 'bachelors_field': 'education', 'b... | {'age': 30, 'bachelors_field': 'business', 'bi... | PT-8A170849 | 2024-07-03T00:00:00 | 2024-07-07T00:00:00 | Smith | Gloria | 1985-04-08 | Dr. Thompson | dengue | I have been feeling really sick. My body hurts... | **Chief Complaint:** "Feeling really sick. Bod... |
──────────────────────────────────────── 🎨 Data Designer Dataset Profile ───────────────────────────────────────── Dataset Overview ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ number of records ┃ number of columns ┃ percent complete records ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ 10 │ 10 │ 100.0% │ └─────────────────────────────────┴─────────────────────────────────┴─────────────────────────────────────────────┘ 🎲 Sampler Columns ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ column name ┃ data type ┃ number unique values ┃ sampler type ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ patient_sampler │ dict │ 10 (100.0%) │ person_from_faker │ ├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤ │ doctor_sampler │ dict │ 10 (100.0%) │ person_from_faker │ ├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤ │ patient_id │ string │ 10 (100.0%) │ uuid │ ├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤ │ symptom_onset_date │ string │ 10 (100.0%) │ datetime │ ├───────────────────────────────┼─────────────────┼──────────────────────────────────┼────────────────────────────┤ │ date_of_visit │ string │ 9 (90.0%) │ timedelta │ └───────────────────────────────┴─────────────────┴──────────────────────────────────┴────────────────────────────┘ 📝 LLM-Text Columns ┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ ┃ ┃ ┃ prompt tokens ┃ completion tokens ┃ ┃ column name ┃ data type ┃ number unique values ┃ per record ┃ per record ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩ │ physician_notes │ string │ 10 (100.0%) │ 132.0 +/- 5.2 │ 1039.5 +/- 310.2 │ └───────────────────────┴───────────────┴────────────────────────────┴───────────────────┴────────────────────────┘ 🧩 Expression Columns ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ column name ┃ data type ┃ number unique values ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ first_name │ string │ 9 (90.0%) │ ├────────────────────────────────┼───────────────────────────┼────────────────────────────────────────────────────┤ │ last_name │ string │ 9 (90.0%) │ ├────────────────────────────────┼───────────────────────────┼────────────────────────────────────────────────────┤ │ dob │ string │ 10 (100.0%) │ ├────────────────────────────────┼───────────────────────────┼────────────────────────────────────────────────────┤ │ physician │ string │ 10 (100.0%) │ └────────────────────────────────┴───────────────────────────┴────────────────────────────────────────────────────┘ ╭────────────────────────────────────────────────── Table Notes ──────────────────────────────────────────────────╮ │ │ │ 1. All token statistics are based on a sample of max(1000, len(dataset)) records. │ │ 2. Tokens are calculated using tiktoken's cl100k_base tokenizer. │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Check out the following notebook to learn more about: