Download this tutorial as a Jupyter notebook

Full SFT Customization#

Learn how to fine-tune all model weights using supervised fine-tuning (SFT) to customize LLM behavior for your specific tasks.

About#

Supervised Fine-Tuning (SFT) customizes model behavior, injects new knowledge, and optimizes performance for specific domains and tasks. Full SFT modifies all model weights during training, providing maximum customization flexibility.

What you can achieve with SFT:

  • 🎯 Specialize for domains: Fine-tune models on legal texts, medical records, or financial data

  • 💡 Inject knowledge: Add new information not present in the base model

  • 📈 Improve accuracy: Optimize for specific tasks like sentiment analysis, summarization, or code generation

SFT vs LoRA: Understanding the Trade-offs#

Full SFT trains all model parameters (for example, all 70 billion weights in Llama 70B):

  • ✅ Maximum model adaptation and knowledge injection

  • ✅ Can fundamentally change model behavior

  • ✅ Best for significant domain shifts or specialized tasks

  • ❌ Requires substantial GPU resources (4-8x more than LoRA)

  • ❌ Produces full model weights (~140GB for Llama 70B)

  • ❌ Longer training time

LoRA trains only ~1% of weights by adding thin matrices to existing weights:

  • ✅ 75-95% less memory required

  • ✅ Faster training (2-4x speedup)

  • ✅ Produces small adapter files (~100-500MB)

  • ✅ Multiple adapters can share one base model

  • ❌ Limited adaptation capability compared to full fine-tuning

When to choose Full SFT:

  • Training small models (1B-8B) where resource cost is manageable

  • Need fundamental behavior changes (for example, medical diagnosis, legal reasoning)

  • Injecting substantial new knowledge not in the base model

When to choose LoRA: Refer to the LoRA tutorial for most use cases, especially with large models (70B+) or limited GPU resources.

Prerequisites#

Before starting this tutorial, ensure you have:

  1. Completed the Quickstart to install and deploy NeMo Platform locally

  2. Installed the Python SDK (included with pip install nemo-platform)

Quick Start#

1. Initialize SDK#

The SDK needs to know your NMP server URL. By default, http://localhost:8080 is used in accordance with the Quickstart guide. If NMP is running at a custom location, you can override the URL by setting the NMP_BASE_URL environment variable:

export NMP_BASE_URL=<YOUR_NMP_BASE_URL>

import json
import os
from nemo_platform import NeMoPlatform, ConflictError

NMP_BASE_URL = os.environ.get("NMP_BASE_URL", "http://localhost:8080")
sdk = NeMoPlatform(
    base_url=NMP_BASE_URL,
    workspace="default"
)

2. Prepare Dataset#

Create your data in JSONL format—one JSON object per line. The platform auto-detects your data format. Supported dataset formats are listed below.

Flexible Data Setup:

  • No validation file? The platform automatically creates a 10% validation split

  • Multiple files? Upload to training/ or validation/ subdirectories—they will be automatically merged

  • Format detection: Your data format is auto-detected at training time

In this tutorial the following dataset directory structure will be used:

my_dataset
`-- training.jsonl
`-- validation.jsonl

Simple Prompt/Completion Format#

The simplest format with input prompt and expected completion:

  • prompt: The input prompt for the model

  • completion: The expected output response

{"prompt": "Write an email to confirm our hotel reservation.", "completion": "Dear Hotel Team, I am writing to confirm our reservation for two guests..."}

Chat Format (for conversational models)#

For multi-turn conversations, use the messages format:

  • messages: List of message objects with role and content fields

  • Roles: system, user, assistant

{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is AI?"}, {"role": "assistant", "content": "AI is..."}]}

Custom Format (specify columns in job)#

You can use custom field names and map them during job creation:

  • Define your own field names

  • Map them to prompt/completion in the job configuration

{"question": "What is 2+2?", "answer": "4"}

3. Create Dataset FileSet and Upload Training Data#

Install huggingface datasets package to download public rajpurkar/squad dataset if it is not installed in your Python environment:

pip install datasets

Download rajpurkar/squad Dataset#

SQuAD (Stanford Question Answering Dataset) is a reading comprehension dataset consisting of questions posed on Wikipedia articles, where the answer is a segment of text from the corresponding passage.

from pathlib import Path
from datasets import load_dataset, DatasetDict
import json

# Load the SQuAD dataset from Hugging Face
print("Loading dataset rajpurkar/squad")
raw_dataset = load_dataset("rajpurkar/squad")
if not isinstance(raw_dataset, DatasetDict):
    raise ValueError("Dataset does not contain expected splits")

print("Loaded dataset")

# Configuration
VALIDATION_PROPORTION = 0.05
SEED = 1234

# For the purpose of this tutorial, we'll use a subset of the dataset
# The larger the datasets, the better the model will perform but longer the training will take
training_size = 3000
validation_size = 300
DATASET_PATH = Path("sft-dataset").absolute()

# Create directory if it doesn't exist
os.makedirs(DATASET_PATH, exist_ok=True)

# Get the train split and create a validation split from it
train_set = raw_dataset.get('train')
split_dataset = train_set.train_test_split(test_size=VALIDATION_PROPORTION, seed=SEED)

# Select subsets for the tutorial
train_ds = split_dataset['train'].select(range(min(training_size, len(split_dataset['train']))))
validation_ds = split_dataset['test'].select(range(min(validation_size, len(split_dataset['test']))))

# Convert SQuAD format to prompt/completion format and save to JSONL
def convert_squad_to_sft_format(example):
    """Convert SQuAD format to prompt/completion format for SFT training."""
    prompt = f"Context: {example['context']} Question: {example['question']} Answer:"
    completion = example["answers"]["text"][0]  # Take the first answer
    return {"prompt": prompt, "completion": completion}

# Save training data
with open(f"{DATASET_PATH}/training.jsonl", "w", encoding="utf-8") as f:
    for example in train_ds:
        converted = convert_squad_to_sft_format(example)
        f.write(json.dumps(converted) + "\n")

# Save validation data
with open(f"{DATASET_PATH}/validation.jsonl", "w", encoding="utf-8") as f:
    for example in validation_ds:
        converted = convert_squad_to_sft_format(example)
        f.write(json.dumps(converted) + "\n")

print(f"Saved training.jsonl with {len(train_ds)} rows")
print(f"Saved validation.jsonl with {len(validation_ds)} rows")

# Show a sample from the training data
print("\nSample from training data:")
with open(f"{DATASET_PATH}/training.jsonl", 'r') as f:
    first_line = f.readline()
    sample = json.loads(first_line)
    print(f"Prompt: {sample['prompt'][:200]}...")
    print(f"Completion: {sample['completion']}")

# Create fileset to store SFT training data
DATASET_NAME = "sft-dataset"

try:
    sdk.files.filesets.create(
        workspace="default",
        name=DATASET_NAME,
        description="SFT training data"
    )
    print(f"Created fileset: {DATASET_NAME}")
except ConflictError:
    print(f"Fileset '{DATASET_NAME}' already exists, continuing...")

# Upload training data files individually to ensure correct structure
sdk.files.upload(
    local_path=DATASET_PATH,  # Local directory with your JSONL files
    remote_path="",
    fileset=DATASET_NAME,
    workspace="default"
)

# Validate training data is uploaded correctly
print("Training data:")
print(json.dumps([f.model_dump() for f in sdk.files.list(fileset=DATASET_NAME, workspace="default").data], indent=2))

4. Secrets Setup#

If you plan to use NGC or HuggingFace models, you will need to configure authentication:

  • NGC models (ngc:// URIs): Requires NGC API key

  • HuggingFace models (hf:// URIs): Requires HF token for gated/private models

Configure these as secrets in your platform. Refer to Managing Secrets for detailed instructions.

Get your credentials to access base models:

Quick Setup Example#

In this tutorial we are going to work with meta-llama/Llama-3.2-1B-Instruct model from HuggingFace. Ensure that you have sufficient permissions to download the model. If you cannot access the files on the meta-llama/Llama-3.2-1B-Instruct Hugging Face page, request access

HuggingFace Authentication:

  • For gated models (Llama, Gemma), you must provide a HuggingFace token via the token_secret parameter

  • Get your token from HuggingFace Settings (requires Read access)

  • Accept the model’s terms on the HuggingFace model page before using it. Example: meta-llama/Llama-3.2-1B-Instruct

  • For public models, you can omit the token_secret parameter when creating a fileset for model in the next step

# Export the HF_TOKEN and NGC_API_KEY environment variables if they are not already set
HF_TOKEN = os.getenv("HF_TOKEN")
NGC_API_KEY = os.getenv("NGC_API_KEY")


def create_or_get_secret(name: str, value: str | None, label: str):
    if not value:
        raise ValueError(f"{label} is not set")
    try:
        secret = sdk.secrets.create(
            name=name,
            workspace="default",
            data=value,
        )
        print(f"Created secret: {name}")
        return secret
    except ConflictError:
        print(f"Secret '{name}' already exists, continuing...")
        return sdk.secrets.retrieve(name=name, workspace="default")


# Create HuggingFace token secret
hf_secret = create_or_get_secret("hf-token", HF_TOKEN, "HF_TOKEN")
print("HF_TOKEN secret:")
print(hf_secret.model_dump_json(indent=2))

# Create NGC API key secret
# Uncomment the line below if you have NGC API Key and want to finetune NGC models
# ngc_api_key = create_or_get_secret("ngc-api-key", NGC_API_KEY, "NGC_API_KEY")

5. Create Base Model FileSet and Model Entity#

Create a fileset pointing to meta-llama/Llama-3.2-1B-Instruct model in HuggingFace that we will train with SFT. Then create a Model Entity that references this fileset. Model downloading will take place at training time.

Note: for public models, you can omit the token_secret parameter when creating a model fileset.

import time

# Create a fileset pointing to the desired HuggingFace model
from nemo_platform.types.files import HuggingfaceStorageConfigParam

HF_REPO_ID = "meta-llama/Llama-3.2-1B-Instruct"
MODEL_NAME = "llama-3-2-1b-base"

# Ensure you have a HuggingFace token secret created
try:
    base_model_fs = sdk.files.filesets.create(
        workspace="default",
        name=MODEL_NAME,
        description="Llama 3.2 1B base model from HuggingFace",
        storage=HuggingfaceStorageConfigParam(
            type="huggingface",
            # repo_id is the full model name from Hugging Face
            repo_id=HF_REPO_ID,
            repo_type="model",
            # we use the secret created in the previous step
            token_secret=hf_secret.name
        )
    )
    print(f"Created base model fileset: {MODEL_NAME}")
except ConflictError:
    print(f"Base model fileset already exists. Skipping creation.")
    base_model_fs = sdk.files.filesets.retrieve(
        workspace="default",
        name=MODEL_NAME,
    )

# Create the Model Entity representation.
try:
    base_model = sdk.models.create(
        workspace="default",
        name=MODEL_NAME,
        fileset=f"default/{MODEL_NAME}",
    )
    print(f"Created Model Entity: {MODEL_NAME}")
except ConflictError:
    print(f"Base model already exists. Updating fileset if different.")
    base_model = sdk.models.update(
        workspace="default",
        name=MODEL_NAME,
        fileset=f"default/{MODEL_NAME}",
    )

print(f"\nBase model fileset: fileset://default/{base_model.name}")
print("Base model fileset files list:")
print(json.dumps([f.model_dump() for f in sdk.files.list(fileset=MODEL_NAME, workspace="default").data], indent=2))

# Wait for ModelSpec to be populated from the checkpoint
print("\nWaiting for ModelSpec to be populated...")
SPEC_TIMEOUT_SECONDS = 120
spec_start = time.time()
while not base_model.spec:
    if time.time() - spec_start > SPEC_TIMEOUT_SECONDS:
        raise TimeoutError(f"ModelSpec not populated within {SPEC_TIMEOUT_SECONDS} seconds")
    time.sleep(2)
    base_model = sdk.models.retrieve(
        workspace="default",
        name=MODEL_NAME,
    )

print(f"ModelSpec populated: {base_model.spec}")

6. Create SFT Finetuning Job#

Create a customization job with an inline target referencing the base model and dataset filesets created in previous steps.

GPU Requirements:

  • 1B models: 1 GPU (24GB+ VRAM)

  • 3B models: 1-2 GPUs

  • 8B models: 2-4 GPUs

  • 70B models: 8+ GPUs

Adjust num_gpus_per_node based on your model size.

import uuid
from nemo_platform.types.customization import (
    CustomizationJobInputParam,
    SftTrainingParam,
    ParallelismParamsParam,
)

job_suffix = uuid.uuid4().hex[:4]

JOB_NAME = f"my-sft-job-{job_suffix}"

job = sdk.customization.jobs.create(
    name=JOB_NAME,
    workspace="default",
    spec=CustomizationJobInputParam(
        model=f"default/{base_model.name}",
        dataset=f"fileset://default/{DATASET_NAME}",
        training=SftTrainingParam(
            type="sft",
            epochs=2,
            batch_size=64,
            learning_rate=0.00005,
            max_seq_length=2048,
            micro_batch_size=1,
            parallelism=ParallelismParamsParam(
                num_gpus_per_node=1,
                num_nodes=1,
                tensor_parallel_size=1,
                pipeline_parallel_size=1,
            ),
        ),
    )
)

print(f"Job ID: {job.name}")
print(f"Output model: {job.spec.output.name}")

7. Track Training Progress#

import time
from IPython.display import clear_output

# Poll job status every 10 seconds until completed
while True:
    status = sdk.customization.jobs.get_status(
        name=job.name,
        workspace="default"
    )

    clear_output(wait=True)
    print(f"Job Status: {status.model_dump_json(indent=2)}")

    # Extract training progress from nested steps structure
    step: int | None = None
    max_steps: int | None = None
    training_phase: str | None = None

    for job_step in status.steps or []:
        if job_step.name == "customization-training-job":
            for task in job_step.tasks or []:
                task_details = task.status_details or {}
                step = task_details.get("step")
                max_steps = task_details.get("max_steps")
                training_phase = task_details.get("phase")
                break
            break

    if step is not None and max_steps is not None:
        progress_pct = (step / max_steps) * 100
        print(f"Training Progress: Step {step}/{max_steps} ({progress_pct:.1f}%)")
        if training_phase:
            print(f"Training Phase: {training_phase}")
    else:
        print("Training step not started yet or progress info not available")

    # Exit loop when job is completed (or failed/cancelled)
    if status.status in ("completed", "failed", "cancelled", "error"):
        print(f"\nJob finished with status: {status.status}")
        break

    time.sleep(10)

Interpreting SFT Training Metrics:

Monitor the relationship between training and validation loss curves:

Scenario

Interpretation

Action

Both decreasing together

Model is learning well

Continue training

Training decreases, validation flat/increasing

Overfitting

Reduce epochs, add data

Both flat/not decreasing

Underfitting

Increase LR, check data

Sudden spikes

Training instability

Lower learning rate

Note: Training metrics measure optimization progress, not final model quality. Always evaluate the deployed model on your specific use case.

8. Deploy Fine-Tuned Model#

After training completes, deploy using the Deployment Management Service:

# Validate model entity exists
model_entity = sdk.models.retrieve(workspace='default', name=job.spec.output.name)
print(model_entity.model_dump_json(indent=2))

from nemo_platform.types.inference import NIMDeploymentParam

# Create deployment config
deploy_suffix = uuid.uuid4().hex[:4]
DEPLOYMENT_CONFIG_NAME = f"sft-model-deployment-cfg-{deploy_suffix}"
DEPLOYMENT_NAME = f"sft-model-deployment-{deploy_suffix}"

deployment_config = sdk.inference.deployment_configs.create(
    workspace="default",
    name=DEPLOYMENT_CONFIG_NAME,
    nim_deployment=NIMDeploymentParam(
        image_name="nvcr.io/nim/nvidia/llm-nim",
        image_tag="1.15.5",
        gpu=1,
        model_name=job.spec.output.name,  # ModelEntity name from training,
        model_namespace="default",  # Workspace where ModelEntity lives
        additional_envs={"NIM_MODEL_PROFILE": "vllm"}
    ),
)

# Deploy model using deployment_config created above
deployment = sdk.inference.deployments.create(
    workspace="default",
    name=DEPLOYMENT_NAME,
    config=deployment_config.name
)


# Check deployment status
deployment_status = sdk.inference.deployments.retrieve(
    name=deployment.name,
    workspace="default"
)

print(f"Deployment name: {deployment.name}")
print(f"Deployment status: {deployment_status.status}")

The deployment service automatically:

  • Downloads model weights from the Files service

  • Provisions storage (PVC) for the weights

  • Configures and starts the NIM container

Multi-GPU Deployment:

For larger models requiring multiple GPUs, configure parallelism with environment variables:

deployment_config = sdk.inference.deployment_configs.create(
    workspace="default",
    name="sft-model-config-multigpu",
    
    nim_deployment={
        "image_name": "nvcr.io/nim/nvidia/llm-nim",
        "image_tag": "1.13.1",
        "gpu": 2,  # Total GPUs
        "additional_envs": {
            "NIM_TENSOR_PARALLEL_SIZE": "2",  # Tensor parallelism
            "NIM_PIPELINE_PARALLEL_SIZE": "1"  # Pipeline parallelism
        }
    }
)

Single-Node Constraint: Model deployments are limited to a single node. The maximum gpu value depends on the total GPUs available on a single node in your cluster. Multi-node deployments are not supported.

GPU Parallelism#

By default, NIM uses all GPUs for tensor parallelism (TP). You can customize this behavior using the NIM_TENSOR_PARALLEL_SIZE and NIM_PIPELINE_PARALLEL_SIZE environment variables.

Strategy

Description

Best For

Tensor Parallel (TP)

Splits model layers across GPUs

Lowest latency

Pipeline Parallel (PP)

Splits model depth across GPUs

Highest throughput

Formula: gpu = NIM_TENSOR_PARALLEL_SIZE × NIM_PIPELINE_PARALLEL_SIZE

Example Configurations#

Default (TP=8, PP=1) — Lowest Latency

"gpu": 8
# NIM automatically sets NIM_TENSOR_PARALLEL_SIZE=8

Balanced (TP=4, PP=2)

"gpu": 8,
"additional_envs": {
    "NIM_TENSOR_PARALLEL_SIZE": "4",
    "NIM_PIPELINE_PARALLEL_SIZE": "2"
}

Throughput Optimized (TP=2, PP=4)

"gpu": 8,
"additional_envs": {
    "NIM_TENSOR_PARALLEL_SIZE": "2",
    "NIM_PIPELINE_PARALLEL_SIZE": "4"
}

Track Deployment Status#

import time
from IPython.display import clear_output

# Poll deployment status every 15 seconds until ready
TIMEOUT_MINUTES = 30
start_time = time.time()
timeout_seconds = TIMEOUT_MINUTES * 60

print(f"Monitoring deployment '{deployment.name}'...")
print(f"Timeout: {TIMEOUT_MINUTES} minutes\n")

while True:
    deployment_status = sdk.inference.deployments.retrieve(
        name=deployment.name,
        workspace="default"
    )

    elapsed = time.time() - start_time
    elapsed_min = int(elapsed // 60)
    elapsed_sec = int(elapsed % 60)

    clear_output(wait=True)
    print(f"Deployment: {deployment.name}")
    print(f"Status: {deployment_status.status}")
    print(f"Elapsed time: {elapsed_min}m {elapsed_sec}s")

    # Check if deployment is ready
    if deployment_status.status == "READY":
        print("\nDeployment is ready!")
        if not sdk.models.wait_for_gateway(deployment.name, workspace="default", timeout=60):
            raise RuntimeError("Inference gateway did not become ready")
        break

    # Check for failure states
    if deployment_status.status in ("FAILED", "ERROR", "TERMINATED", "LOST"):
        raise RuntimeError(f"Deployment failed with status: {deployment_status.status}")

    # Check timeout
    if elapsed > timeout_seconds:
        raise TimeoutError(f"Deployment timeout after {TIMEOUT_MINUTES} minutes")

    time.sleep(15)

9. Evaluate Your Model#

After training, evaluate whether your model meets your requirements:

Quick Manual Evaluation#

# Wait for deployment to be ready, then test
# Test the fine-tuned model with a question answering prompt
context = "The Apollo 11 mission was the first manned mission to land on the Moon. It was launched on July 16, 1969, and Neil Armstrong became the first person to walk on the lunar surface on July 20, 1969. Buzz Aldrin joined him shortly after, while Michael Collins remained in lunar orbit."
question = "Who was the first person to walk on the Moon?"

messages = [
    {"role": "user", "content": f"Based on the following context, answer the question.\n\nContext: {context}\n\nQuestion: {question}"}
]

response = sdk.inference.gateway.provider.post(
    "v1/chat/completions",
    name=deployment.name,
    workspace="default",
    body={
        "model": f"default/{job.spec.output.name}",
        "messages": messages,
        "temperature": 0,
        "max_tokens": 128
    }
)

print("=" * 60)
print("MODEL EVALUATION")
print("=" * 60)
print(f"Question: {question}")
print(f"Expected: Neil Armstrong")
print(f"Model output: {response['choices'][0]['message']['content']}")

Evaluation Best Practices#

Manual Evaluation (Recommended)

  • Test with real-world examples from your use case

  • Compare responses to base model and expected outputs

  • Verify the model exhibits desired behavior changes

  • Check edge cases and error handling

What to look for:

  • ✅ Model follows your desired output format

  • ✅ Applies domain knowledge correctly

  • ✅ Maintains general language capabilities

  • ✅ Avoids unwanted behaviors or biases

  • ❌ Doesn’t hallucinate facts not in training data

  • ❌ Doesn’t produce repetitive or nonsensical outputs

Hyperparameters#

For detailed information on all available hyperparameters, recommended values, and tuning guidance, refer to the Hyperparameter Reference.

Troubleshooting#

Job fails during model download:

  • Verify authentication secrets are configured (refer to Managing Secrets)

  • For gated HuggingFace models (Llama, Gemma), accept the license on the model page

  • Check the model_uri format is correct (fileset://)

  • Ensure you have accepted the model’s terms of service on HuggingFace

  • Check job status and logs: sdk.customization.jobs.retrieve(name=job.name, workspace="default")

Job fails with OOM (Out of Memory) error:

  1. First try: Reduce micro_batch_size from 2 to 1

  2. Still OOM: Reduce batch_size from 4 to 2

  3. Still OOM: Reduce max_seq_length from 2048 to 1024 or 512

  4. Last resort: Increase GPU count and use tensor_parallel_size for model sharding

Loss curves not decreasing (underfitting):

  • Increase training duration: epochs: 5-10 instead of 3

  • Adjust learning rate: Try 1e-5 to 1e-4

  • Add warmup: Set warmup_steps to ~10% of total training steps

  • Check data quality: Verify formatting, remove duplicates, ensure diversity

Training loss decreases but validation loss increases (overfitting):

  • Reduce epochs: Try epochs: 1-2 instead of 5+

  • Lower learning rate: Use 2e-5 or 1e-5

  • Increase dataset size and diversity

  • Verify train/validation split has no data leakage

Model output quality is poor despite good training metrics:

  • Training metrics optimize for loss, not your actual task—evaluate on real use cases

  • Review data quality, format, and diversity—metrics can be misleading with poor data

  • Try a different base model size or architecture

  • Adjust learning rate and batch size

  • Compare to baseline: Test base model to ensure fine-tuning improved performance

Deployment fails:

  • Verify output model exists: sdk.models.retrieve(name=job.spec.output.name, workspace="default")

  • Check deployment logs: sdk.inference.deployments.get_logs(name=deployment.name, workspace="default")

  • Ensure sufficient GPU resources available for model size

  • Verify NIM image tag 1.13.1 is compatible with your model

Next Steps#