LoRA Model Customization | NVIDIA NeMo Helix

Run in Google Colab

Learn how to use NeMo Helix to create a LoRA (Low-Rank Adaptation) customization job using a custom dataset. In this tutorial we use LoRA to fine-tune NVIDIA Nemotron 3.5 Lightning 30B-A3B into a question-answering model with the SQuAD dataset.

LoRA is a parameter-efficient fine-tuning method that requires fewer computational resources than full fine-tuning. If you need full model fine-tuning instead, see the Full SFT Customization Job tutorial.

Time to complete: approximately 45 minutes. Job duration increases with model size and dataset size.

Prerequisites

Before starting this tutorial, ensure you have:

Completed the Quickstart to install and deploy NeMo Helix locally
Installed the Python SDK (PyPI wrapper: pip install "nemo-helix[all]"; source checkout: run make bootstrap from the repository root)
Installed the datasets package for loading SQuAD: pip install datasets
Four NVIDIA Hopper or Blackwell era GPUs with 80 GB+ each and CUDA 13+ (for example, H100 or B200) for training. Nemotron customization is supported on Hopper or Blackwell era GPUs only. The deployment step later reuses two of them.

Quick Start

1. Initialize SDK

The SDK connects to the deployment in your current CLI context, using its base URL, default workspace, and credentials. To connect to another deployment, refer to Connect with CLI Contexts. These examples use the default workspace. If your current CLI context uses a different workspace, run nemo config set --workspace default first.

In environments without a CLI configuration file, such as Google Colab, set NHX_BASE_URL (and NHX_ACCESS_TOKEN for an authenticated deployment) before you create the client. NemoClient.from_config() reads both:

import os
os.environ["NHX_BASE_URL"] = "<YOUR_NHX_BASE_URL>"

import json
import os
import re
import time
import uuid
from pathlib import Path
from IPython.display import clear_output
from nemo_helix_plugin.client.client import NemoClient
from nemo_helix_plugin.client.errors import ConflictError
from nemo_helix_plugin.files.storage_config import HuggingfaceStorageConfig
from nemo_helix_plugin.files.types import CreateFilesetRequest
from nemo_helix_plugin.models.types import CreateModelEntityRequest, UpdateModelEntityRequest
from nemo_helix_plugin.secrets.types import HelixSecretCreateRequest, HelixSecretResponse
from nemo_customizer.sdk import Customization
from filesets import FilesetFileSystem
from nemo_helix_plugin.files.client import FilesClient
from pydantic import SecretStr
from nemo_helix_plugin.schema import SecretRef
def sanitize_name(prefix: str, name: str):
    """Sanitize model_name for deployment/config naming. Compatible with platform naming rules."""
    name = name.split("/")[-1]
    sanitized = re.sub(r"[^a-z0-9@.+_-]", "-", name.lower())
    sanitized = re.sub(r"-+", "-", sanitized).strip("-")
    # 55 leaves room for the deployments-plugin "-weights" suffix (63 - 8).
    return f"{prefix}-{sanitized}"[:55].rstrip("-")
def max_wait_time_checker(seconds: int, job_name: str = ""):
    """Return a check() that raises TimeoutError if called after `seconds` have elapsed."""
    start_time = time.time()
    def check():
        if time.time() - start_time > seconds:
            raise TimeoutError(f"{job_name} took longer than {seconds} seconds")
    return check
client = NemoClient.from_config()
customization = Customization.from_client(client)

2. Prepare Dataset

Create your data in JSONL format (one JSON object per line). For a chat model, use the chat format so training and inference share the model’s chat template.

Dataset structure:

Training files under training/ (or root with training.jsonl)
Validation files under validation/ (or validation.jsonl)

Chat format: Each line is a JSON object with:

messages: A list of user and assistant turns. The final assistant turn is the desired model output.

Example record (single line in .jsonl):

{"messages": [{"role": "user", "content": "Based on the following context, answer the question.\n\nContext: ...\n\nQuestion: What is X?"}, {"role": "assistant", "content": "X is ..."}]}

Download SQuAD and Convert to Chat Format

We use the SQuAD dataset. We convert each example to the chat messages format. The user turn is the same prompt the evaluation step sends, so the model is trained and queried through the same chat template, including the end-of-turn token that stops generation.

from datasets import load_dataset, DatasetDict
print("Loading dataset rajpurkar/squad")
raw_dataset = load_dataset("rajpurkar/squad")
if not isinstance(raw_dataset, DatasetDict):
    raise ValueError("Dataset does not contain expected splits")
print("Loaded dataset")
VALIDATION_PROPORTION = 0.05
SEED = 1234
training_size = 3000
validation_size = 300
DATASET_NAME = "sft-dataset"
DATASET_PATH = Path("sft-dataset").absolute()
os.makedirs(DATASET_PATH, exist_ok=True)
train_set = raw_dataset.get("train")
split_dataset = train_set.train_test_split(test_size=VALIDATION_PROPORTION, seed=SEED)
train_ds = split_dataset["train"].select(range(min(training_size, len(split_dataset["train"]))))
validation_ds = split_dataset["test"].select(range(min(validation_size, len(split_dataset["test"]))))
def convert_squad_to_chat_format(example):
    """Convert SQuAD to a chat `messages` row whose user turn matches the inference prompt."""
    user = f"Based on the following context, answer the question.\n\nContext: {example['context']}\n\nQuestion: {example['question']}"
    return {"messages": [{"role": "user", "content": user}, {"role": "assistant", "content": example["answers"]["text"][0]}]}
with open(f"{DATASET_PATH}/training.jsonl", "w", encoding="utf-8") as f:
    for example in train_ds:
        f.write(json.dumps(convert_squad_to_chat_format(example)) + "\n")
with open(f"{DATASET_PATH}/validation.jsonl", "w", encoding="utf-8") as f:
    for example in validation_ds:
        f.write(json.dumps(convert_squad_to_chat_format(example)) + "\n")
print(f"Saved training.jsonl with {len(train_ds)} rows")
print(f"Saved validation.jsonl with {len(validation_ds)} rows")
with open(f"{DATASET_PATH}/training.jsonl", "r") as f:
    sample = json.loads(f.readline())
print("Sample user turn (first 200 chars):", sample["messages"][0]["content"][:200] + "...")
print("Sample assistant turn:", sample["messages"][1]["content"])

3. Create FileSet and Upload Training Data

Upload the training and validation JSONL files to a FileSet so the customization job can use them.

try:
    client.files.create_fileset(
        body=CreateFilesetRequest(
            name=DATASET_NAME,
            description="SFT training data",
            cache=True,
        ),
    ).data()
    print(f"Created fileset: {DATASET_NAME}")
except ConflictError:
    print(f"Fileset '{DATASET_NAME}' already exists, continuing...")
fs = FilesetFileSystem(client=FilesClient.from_client(client))
fs.put(
    lpath=DATASET_PATH,
    rpath=f"default/{DATASET_NAME}/",
    recursive=True,
)
print("Training data:")
print(client.files.list_files(name=DATASET_NAME).data())

4. Secrets Setup

For Hugging Face models that require authentication, create a secret with your HF token. Get a token from Hugging Face Settings and accept the model terms.

The Nemotron model in this tutorial is public, so the token is optional.

export HF_TOKEN=<your-huggingface-token>

HF_TOKEN = os.getenv("HF_TOKEN")
def create_or_get_secret(name: str, value: str | None, label: str) -> HelixSecretResponse | None:
    if not value:
        print(f"{label} is not set - skipping setting secret")
        return None
    try:
        secret = client.secrets.create_secret(
            body=HelixSecretCreateRequest(name=name, value=SecretStr(value)),
        ).data()
        print(f"Created secret: {name}")
        print(secret.model_dump_json(indent=2))
        return secret
    except ConflictError:
        print(f"Secret '{name}' already exists, continuing...")
        secret = client.secrets.get_secret(name=name).data()
        print(secret.model_dump_json(indent=2))
        return secret
hf_secret = create_or_get_secret("hf-token", HF_TOKEN, "HF_TOKEN")

5. Create Base Model FileSet and Model Entity

Create a fileset pointing to nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 and a Model Entity that references it. Model download happens when the customization job runs.

Train on the BF16 checkpoint. The NVFP4 variant is quantized for inference and cannot be used as a training base.

HF_REPO_ID = "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16"
MODEL_NAME = "nemotron-3-5-lightning-30b-a3b"
try:
    storage = HuggingfaceStorageConfig(
        repo_id=HF_REPO_ID,
        repo_type="model",
        token_secret=SecretRef(hf_secret.name) if hf_secret else None,
    )
    base_model_fs = client.files.create_fileset(
        body=CreateFilesetRequest(
            name=MODEL_NAME,
            description="Nemotron 3.5 Lightning 30B-A3B from Hugging Face",
            storage=storage,
            cache=True,
        ),
    ).data()
except ConflictError:
    base_model_fs = client.files.get_fileset(name=MODEL_NAME).data()
try:
    base_model = client.models.create_model(
        body=CreateModelEntityRequest(
            name=MODEL_NAME,
            fileset=f"default/{MODEL_NAME}",
            trust_remote_code=False,
        ),
    ).data()
except ConflictError:
    client.models.update_model(
        name=MODEL_NAME,
        body=UpdateModelEntityRequest(
            fileset=f"default/{MODEL_NAME}",
            trust_remote_code=False,
        ),
    )
    base_model = client.models.get_model(name=MODEL_NAME).data()
print(f"Base model fileset: fileset://default/{base_model.name}")
print(client.files.list_files(name=MODEL_NAME).data())
time_check = max_wait_time_checker(600, "Model Spec")
while not base_model.spec:
    time_check()
    time.sleep(10)
    base_model = client.models.get_model(name=MODEL_NAME).data()
# Clear verbose linear_layers list for cleaner output
base_model.spec.linear_layers = None
print(f"ModelSpec: {base_model.spec}")

6. Create LoRA Customization Job

Submit to the Automodel backend using AutomodelJobInput with finetuning_type: lora. After training completes, deploy the base model with LoRA support manually (step 8).

The hyperparameters follow the Automodel reference recipes for this architecture, customizer_nemotron_nano_peft.yaml and nemotron_nano_v3_hellaswag_peft.yaml:

Expert parallelism: the 128 experts are sharded across 4 GPUs (expert_parallel_size=4, tensor_parallel_size=1).
LoRA targets: *.out_proj is excluded because the Mamba output projection runs through a fused kernel that LoRA cannot wrap.

from nemo_automodel_plugin.schema import AutomodelJobInput
job_suffix = uuid.uuid4().hex[:4]
JOB_NAME = f"my-sft-job-{job_suffix}"
OUTPUT_NAME = f"lora-adapter-{job_suffix}"
spec = AutomodelJobInput(
    model=f"default/{base_model.name}",
    dataset={"training": f"default/{DATASET_NAME}"},
    training={
        "training_type": "sft",
        "finetuning_type": "lora",
        "lora": {"rank": 16, "alpha": 32, "exclude_modules": ["*.out_proj"]},
        "max_seq_length": 2048,
    },
    schedule={"epochs": 1},
    batch={"global_batch_size": 8, "micro_batch_size": 1},
    optimizer={"learning_rate": 1e-5, "min_learning_rate": 1e-6},
    parallelism={
        "num_gpus_per_node": 4,
        "num_nodes": 1,
        "tensor_parallel_size": 1,
        "pipeline_parallel_size": 1,
        "context_parallel_size": 1,
        "expert_parallel_size": 4,
    },
    output={"name": OUTPUT_NAME},
)
job = customization.automodel.jobs.create(
    spec=spec, workspace="default", name=JOB_NAME
)
print(f"Submitted job: {job.job.name}")
print(f"Output adapter: {OUTPUT_NAME}")

7. Track Training Progress

Poll until the job finishes. Each update shows the current step, how much of that step is done, and the latest loss. Once the backend reports a loss history, the cell charts loss by step.

import time
# Poll the job until it reaches a terminal state
status = None
while True:
    status = client.jobs.get_job_status(name=job.job.name, workspace="default").data()
    print(f"Status: {status.status}")
    if status.status.lower() in ("completed", "failed", "cancelled", "error"):
        break
    time.sleep(15)
assert status.status.lower() == "completed"

8. Validate Output Model and Deployment

With the base model entity and LoRA adapter from training, create a NIM deployment with lora_enabled=True so the adapter is served alongside the base weights. Check the model entity and deployment status.

deploy_suffix = uuid.uuid4().hex[:4]
DEPLOYMENT_CONFIG_NAME = f"lora-deploy-cfg-{deploy_suffix}"
deployment_name = f"lora-deploy-{deploy_suffix}"
from nemo_helix_plugin.models.types import (
    ContainerExecutorConfig,
    CreateModelDeploymentConfigRequest,
    CreateModelDeploymentRequest,
    Engine,
    ModelDeploymentConfigModelSpec,
)
deployment_config = client.models.create_deployment_config(
    body=CreateModelDeploymentConfigRequest(
        name=DEPLOYMENT_CONFIG_NAME,
        engine=Engine.VLLM,
        model_spec=ModelDeploymentConfigModelSpec(
            model_namespace="default",
            model_name=MODEL_NAME,
            lora_enabled=True,
        ),
        executor_config=ContainerExecutorConfig(
            gpu=2,
            image_name="docker.io/vllm/vllm-openai",
            image_tag="v0.22.1",
            additional_args=["--max-lora-rank", "32"],
        ),
    ),
).data()
deployment = client.models.create_deployment(
    body=CreateModelDeploymentRequest(
        name=deployment_name,
        config=deployment_config.name,
    ),
).data()
model_entity = client.models.get_model(name=MODEL_NAME).data()
if model_entity.spec:
    model_entity.spec.linear_layers = None
print(model_entity.model_dump_json(indent=2))
print(f"Deployment status: {deployment.status}")

9. Monitor Deployment Until Ready

Wait for the deployment to reach RUNNING/READY before sending inference requests. Image pull and weight load often take 10–20 minutes, and the status stays PENDING until the server is ready.

TIMEOUT_MINUTES = 30
start_time = time.time()
time_check = max_wait_time_checker(TIMEOUT_MINUTES * 60, "Deployment")
print(f"Monitoring deployment '{deployment_name}'... (timeout {TIMEOUT_MINUTES} min)\n")
while True:
    time.sleep(15)
    time_check()
    deployment_status = client.models.get_deployment(name=deployment_name).data()
    elapsed = time.time() - start_time
    clear_output(wait=True)
    print(f"Deployment: {deployment_name}")
    print(f"Status: {deployment_status.status}")
    print(f"Elapsed: {int(elapsed // 60)}m {int(elapsed % 60)}s")
    if deployment_status.status in ("RUNNING", "READY"):
        print("\nDeployment is ready!")
        if not client.models.wait_for_gateway(deployment_name, workspace="default", timeout=60):
            raise RuntimeError("Inference gateway did not become ready")
        break
    if deployment_status.status in ("FAILED", "ERROR", "TERMINATED"):
        raise RuntimeError(f"Deployment failed with status: {deployment_status.status}")
assert deployment_status.status in ("RUNNING", "READY")

10. Check Model Output

Send a chat completion request to the deployed LoRA model and compare the output to the expected answer.

context = "The Apollo 11 mission was the first manned mission to land on the Moon. It was launched on July 16, 1969, and Neil Armstrong became the first person to walk on the lunar surface on July 20, 1969. Buzz Aldrin joined him shortly after, while Michael Collins remained in lunar orbit."
question = "Who was the first person to walk on the Moon?"
messages = [
    {"role": "user", "content": f"Based on the following context, answer the question.\n\nContext: {context}\n\nQuestion: {question}"}
]
INFERENCE_MODEL_NAME = f"default--{OUTPUT_NAME}"
from nemo_helix_plugin.inference_gateway.client import InferenceGatewayClient
from nemo_helix_plugin.inference_gateway.types import JsonBody
gateway = InferenceGatewayClient.from_client(client)
response = gateway.provider_post(
    name=deployment_name,
    workspace="default",
    trailing_uri="v1/chat/completions",
    body=JsonBody(
        {
            "model": INFERENCE_MODEL_NAME,
            "messages": messages,
            "temperature": 0,
            "max_tokens": 256,
            # Nemotron 3.5 Lightning reasons by default; turn it off for a short answer.
            "chat_template_kwargs": {"enable_thinking": False},
        }
    ),
).data()
print("=" * 60)
print("MODEL INFERENCE")
print("=" * 60)
print(f"Question: {question}")
print(f"Expected: Neil Armstrong")
print(f"Model output: {response['choices'][0]['message']['content']}")

Conclusion

You have started a LoRA customization job, monitored it to completion, and evaluated the fine-tuned model. Use the output.name to access the model for further inference or evaluation.

Next Steps

Monitor training metrics in detail
Evaluate your fine-tuned model using the Evals service
Try Full SFT for other customization options