> This page is for version Latest (default).
> For other versions, use one of these documentation indexes:
> - Latest (default): https://docs.nvidia.com/nemo-helix/latest/llms.txt
> - 0.8.0: https://docs.nvidia.com/nemo-helix/v0.8.0/llms.txt
> - 0.7.1: https://docs.nvidia.com/nemo-helix/v0.7.1/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo-helix/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo-helix/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo-helix/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo-helix/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo-helix/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo-helix/v0.3.0/llms.txt
> - 0.2.0: https://docs.nvidia.com/nemo-helix/v0.2.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-helix/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-helix/_mcp/server.

# LoRA Model Customization

[▶ Run in Google Colab](https://colab.research.google.com/github/NVIDIA-NeMo/nemo-helix/blob/main/docs/customizer/tutorials/lora-customization-job.ipynb)

Learn how to use NeMo Helix to create a LoRA (Low-Rank Adaptation) customization job using a custom dataset. In this tutorial we use LoRA to fine-tune [NVIDIA Nemotron 3.5 Lightning 30B-A3B](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16) into a **question-answering model** with the SQuAD dataset.

LoRA is a parameter-efficient fine-tuning method that requires fewer computational resources than full fine-tuning. If you need full model fine-tuning instead, see the [Full SFT Customization Job](/documentation/customizer-reference/tutorials/sft-customization-job) tutorial.

**Time to complete:** approximately 45 minutes. Job duration increases with model size and dataset size.

## Prerequisites

Before starting this tutorial, ensure you have:

1. **Completed the [Quickstart](/documentation/get-started)** to install and deploy NeMo Helix locally
2. **Installed the Python SDK** (PyPI wrapper: `pip install "nemo-helix[all]"`; source checkout: run `make bootstrap` from the repository root)
3. **Installed the `datasets` package** for loading SQuAD: `pip install datasets`
4. **Four NVIDIA Hopper or Blackwell era GPUs with 80 GB+ each and CUDA 13+** (for example, H100 or B200) for training. Nemotron customization is supported on Hopper or Blackwell era GPUs only. The deployment step later reuses two of them.

## Quick Start

### 1. Initialize SDK

The SDK connects to the deployment in your current CLI context, using its base URL, default workspace, and credentials. To connect to another deployment, refer to [Connect with CLI Contexts](/documentation/reference/cli-reference/connect-to-deployments). These examples use the `default` workspace. If your current CLI context uses a different workspace, run `nemo config set --workspace default` first.

In environments without a CLI configuration file, such as Google Colab, set `NHX_BASE_URL` (and `NHX_ACCESS_TOKEN` for an authenticated deployment) before you create the client. `NemoClient.from_config()` reads both:

```python
import os

os.environ["NHX_BASE_URL"] = "<YOUR_NHX_BASE_URL>"
```

```python
import json
import os
import re
import time
import uuid
from pathlib import Path
from IPython.display import clear_output
from nemo_helix_plugin.client.client import NemoClient
from nemo_helix_plugin.client.errors import ConflictError
from nemo_helix_plugin.files.storage_config import HuggingfaceStorageConfig
from nemo_helix_plugin.files.types import CreateFilesetRequest
from nemo_helix_plugin.models.types import CreateModelEntityRequest, UpdateModelEntityRequest
from nemo_helix_plugin.secrets.types import HelixSecretCreateRequest, HelixSecretResponse
from nemo_customizer.sdk import Customization
from filesets import FilesetFileSystem
from nemo_helix_plugin.files.client import FilesClient
from pydantic import SecretStr
from nemo_helix_plugin.schema import SecretRef


def sanitize_name(prefix: str, name: str):
    """Sanitize model_name for deployment/config naming. Compatible with platform naming rules."""
    name = name.split("/")[-1]
    sanitized = re.sub(r"[^a-z0-9@.+_-]", "-", name.lower())
    sanitized = re.sub(r"-+", "-", sanitized).strip("-")
    # 55 leaves room for the deployments-plugin "-weights" suffix (63 - 8).
    return f"{prefix}-{sanitized}"[:55].rstrip("-")


def max_wait_time_checker(seconds: int, job_name: str = ""):
    """Return a check() that raises TimeoutError if called after `seconds` have elapsed."""
    start_time = time.time()

    def check():
        if time.time() - start_time > seconds:
            raise TimeoutError(f"{job_name} took longer than {seconds} seconds")

    return check


client = NemoClient.from_config()
customization = Customization.from_client(client)

```

### 2. Prepare Dataset

Create your data in JSONL format (one JSON object per line). For a chat model, use the **chat** format so training and inference share the model's chat template.

**Dataset structure:**

* Training files under `training/` (or root with `training.jsonl`)
* Validation files under `validation/` (or `validation.jsonl`)

**Chat format:** Each line is a JSON object with:

* **`messages`**: A list of `user` and `assistant` turns. The final `assistant` turn is the desired model output.

Example record (single line in `.jsonl`):

```json
{"messages": [{"role": "user", "content": "Based on the following context, answer the question.\n\nContext: ...\n\nQuestion: What is X?"}, {"role": "assistant", "content": "X is ..."}]}
```

#### Download SQuAD and Convert to Chat Format

We use the [SQuAD](https://huggingface.co/datasets/rajpurkar/squad) dataset. We convert each example to the chat `messages` format. The user turn is the same prompt the evaluation step sends, so the model is trained and queried through the same chat template, including the end-of-turn token that stops generation.

```python
from datasets import load_dataset, DatasetDict

print("Loading dataset rajpurkar/squad")
raw_dataset = load_dataset("rajpurkar/squad")
if not isinstance(raw_dataset, DatasetDict):
    raise ValueError("Dataset does not contain expected splits")
print("Loaded dataset")

VALIDATION_PROPORTION = 0.05
SEED = 1234
training_size = 3000
validation_size = 300
DATASET_NAME = "sft-dataset"
DATASET_PATH = Path("sft-dataset").absolute()

os.makedirs(DATASET_PATH, exist_ok=True)
train_set = raw_dataset.get("train")
split_dataset = train_set.train_test_split(test_size=VALIDATION_PROPORTION, seed=SEED)
train_ds = split_dataset["train"].select(range(min(training_size, len(split_dataset["train"]))))
validation_ds = split_dataset["test"].select(range(min(validation_size, len(split_dataset["test"]))))

def convert_squad_to_chat_format(example):
    """Convert SQuAD to a chat `messages` row whose user turn matches the inference prompt."""
    user = f"Based on the following context, answer the question.\n\nContext: {example['context']}\n\nQuestion: {example['question']}"
    return {"messages": [{"role": "user", "content": user}, {"role": "assistant", "content": example["answers"]["text"][0]}]}

with open(f"{DATASET_PATH}/training.jsonl", "w", encoding="utf-8") as f:
    for example in train_ds:
        f.write(json.dumps(convert_squad_to_chat_format(example)) + "\n")
with open(f"{DATASET_PATH}/validation.jsonl", "w", encoding="utf-8") as f:
    for example in validation_ds:
        f.write(json.dumps(convert_squad_to_chat_format(example)) + "\n")

print(f"Saved training.jsonl with {len(train_ds)} rows")
print(f"Saved validation.jsonl with {len(validation_ds)} rows")
with open(f"{DATASET_PATH}/training.jsonl", "r") as f:
    sample = json.loads(f.readline())
print("Sample user turn (first 200 chars):", sample["messages"][0]["content"][:200] + "...")
print("Sample assistant turn:", sample["messages"][1]["content"])

```

### 3. Create FileSet and Upload Training Data

Upload the training and validation JSONL files to a FileSet so the customization job can use them.

```python
try:
    client.files.create_fileset(
        body=CreateFilesetRequest(
            name=DATASET_NAME,
            description="SFT training data",
            cache=True,
        ),
    ).data()
    print(f"Created fileset: {DATASET_NAME}")
except ConflictError:
    print(f"Fileset '{DATASET_NAME}' already exists, continuing...")

fs = FilesetFileSystem(client=FilesClient.from_client(client))
fs.put(
    lpath=DATASET_PATH,
    rpath=f"default/{DATASET_NAME}/",
    recursive=True,
)
print("Training data:")
print(client.files.list_files(name=DATASET_NAME).data())

```

### 4. Secrets Setup

For Hugging Face models that require authentication, create a secret with your HF token. Get a token from [Hugging Face Settings](https://huggingface.co/settings/tokens) and accept the model terms.

The Nemotron model in this tutorial is public, so the token is optional.

```sh
export HF_TOKEN=<your-huggingface-token>
```

```python
HF_TOKEN = os.getenv("HF_TOKEN")

def create_or_get_secret(name: str, value: str | None, label: str) -> HelixSecretResponse | None:
    if not value:
        print(f"{label} is not set - skipping setting secret")
        return None
    try:
        secret = client.secrets.create_secret(
            body=HelixSecretCreateRequest(name=name, value=SecretStr(value)),
        ).data()
        print(f"Created secret: {name}")
        print(secret.model_dump_json(indent=2))
        return secret
    except ConflictError:
        print(f"Secret '{name}' already exists, continuing...")
        secret = client.secrets.get_secret(name=name).data()
        print(secret.model_dump_json(indent=2))
        return secret


hf_secret = create_or_get_secret("hf-token", HF_TOKEN, "HF_TOKEN")
```

### 5. Create Base Model FileSet and Model Entity

Create a fileset pointing to [nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16) and a Model Entity that references it. Model download happens when the customization job runs.

Train on the BF16 checkpoint. The [NVFP4 variant](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4) is quantized for inference and cannot be used as a training base.

```python
HF_REPO_ID = "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16"
MODEL_NAME = "nemotron-3-5-lightning-30b-a3b"

try:
    storage = HuggingfaceStorageConfig(
        repo_id=HF_REPO_ID,
        repo_type="model",
        token_secret=SecretRef(hf_secret.name) if hf_secret else None,
    )
    base_model_fs = client.files.create_fileset(
        body=CreateFilesetRequest(
            name=MODEL_NAME,
            description="Nemotron 3.5 Lightning 30B-A3B from Hugging Face",
            storage=storage,
            cache=True,
        ),
    ).data()
except ConflictError:
    base_model_fs = client.files.get_fileset(name=MODEL_NAME).data()

try:
    base_model = client.models.create_model(
        body=CreateModelEntityRequest(
            name=MODEL_NAME,
            fileset=f"default/{MODEL_NAME}",
            trust_remote_code=False,
        ),
    ).data()
except ConflictError:
    client.models.update_model(
        name=MODEL_NAME,
        body=UpdateModelEntityRequest(
            fileset=f"default/{MODEL_NAME}",
            trust_remote_code=False,
        ),
    )
    base_model = client.models.get_model(name=MODEL_NAME).data()

print(f"Base model fileset: fileset://default/{base_model.name}")
print(client.files.list_files(name=MODEL_NAME).data())

time_check = max_wait_time_checker(600, "Model Spec")
while not base_model.spec:
    time_check()
    time.sleep(10)
    base_model = client.models.get_model(name=MODEL_NAME).data()

# Clear verbose linear_layers list for cleaner output
base_model.spec.linear_layers = None
print(f"ModelSpec: {base_model.spec}")

```

### 6. Create LoRA Customization Job

Submit to the **Automodel** backend using `AutomodelJobInput` with `finetuning_type: lora`. After training completes, deploy the base model with LoRA support manually (step 8).

The hyperparameters follow the Automodel reference recipes for this architecture, [`customizer_nemotron_nano_peft.yaml`](https://github.com/NVIDIA-NeMo/Automodel/blob/r0.6.0/examples/llm_finetune/nemotron/customizer_nemotron_nano_peft.yaml) and [`nemotron_nano_v3_hellaswag_peft.yaml`](https://github.com/NVIDIA-NeMo/Automodel/blob/r0.6.0/examples/llm_finetune/nemotron/nemotron_nano_v3_hellaswag_peft.yaml):

* **Expert parallelism:** the 128 experts are sharded across 4 GPUs (`expert_parallel_size=4`, `tensor_parallel_size=1`).
* **LoRA targets:** `*.out_proj` is excluded because the Mamba output projection runs through a fused kernel that LoRA cannot wrap.

```python
from nemo_automodel_plugin.schema import AutomodelJobInput

job_suffix = uuid.uuid4().hex[:4]
JOB_NAME = f"my-sft-job-{job_suffix}"
OUTPUT_NAME = f"lora-adapter-{job_suffix}"

spec = AutomodelJobInput(
    model=f"default/{base_model.name}",
    dataset={"training": f"default/{DATASET_NAME}"},
    training={
        "training_type": "sft",
        "finetuning_type": "lora",
        "lora": {"rank": 16, "alpha": 32, "exclude_modules": ["*.out_proj"]},
        "max_seq_length": 2048,
    },
    schedule={"epochs": 1},
    batch={"global_batch_size": 8, "micro_batch_size": 1},
    optimizer={"learning_rate": 1e-5, "min_learning_rate": 1e-6},
    parallelism={
        "num_gpus_per_node": 4,
        "num_nodes": 1,
        "tensor_parallel_size": 1,
        "pipeline_parallel_size": 1,
        "context_parallel_size": 1,
        "expert_parallel_size": 4,
    },
    output={"name": OUTPUT_NAME},
)

job = customization.automodel.jobs.create(
    spec=spec, workspace="default", name=JOB_NAME
)
print(f"Submitted job: {job.job.name}")
print(f"Output adapter: {OUTPUT_NAME}")

```

### 7. Track Training Progress

Poll until the job finishes. Each update shows the current step, how much of that step is done, and the latest loss. Once the backend reports a loss history, the cell charts loss by step.

```python
import time

# Poll the job until it reaches a terminal state
status = None
while True:
    status = client.jobs.get_job_status(name=job.job.name, workspace="default").data()
    print(f"Status: {status.status}")
    if status.status.lower() in ("completed", "failed", "cancelled", "error"):
        break
    time.sleep(15)

assert status.status.lower() == "completed"

```

### 8. Validate Output Model and Deployment

With the base model entity and LoRA adapter from training, create a NIM deployment with `lora_enabled=True` so the adapter is served alongside the base weights. Check the model entity and deployment status.

```python
deploy_suffix = uuid.uuid4().hex[:4]
DEPLOYMENT_CONFIG_NAME = f"lora-deploy-cfg-{deploy_suffix}"
deployment_name = f"lora-deploy-{deploy_suffix}"

from nemo_helix_plugin.models.types import (
    ContainerExecutorConfig,
    CreateModelDeploymentConfigRequest,
    CreateModelDeploymentRequest,
    Engine,
    ModelDeploymentConfigModelSpec,
)

deployment_config = client.models.create_deployment_config(
    body=CreateModelDeploymentConfigRequest(
        name=DEPLOYMENT_CONFIG_NAME,
        engine=Engine.VLLM,
        model_spec=ModelDeploymentConfigModelSpec(
            model_namespace="default",
            model_name=MODEL_NAME,
            lora_enabled=True,
        ),
        executor_config=ContainerExecutorConfig(
            gpu=2,
            image_name="docker.io/vllm/vllm-openai",
            image_tag="v0.22.1",
            additional_args=["--max-lora-rank", "32"],
        ),
    ),
).data()

deployment = client.models.create_deployment(
    body=CreateModelDeploymentRequest(
        name=deployment_name,
        config=deployment_config.name,
    ),
).data()

model_entity = client.models.get_model(name=MODEL_NAME).data()
if model_entity.spec:
    model_entity.spec.linear_layers = None
print(model_entity.model_dump_json(indent=2))
print(f"Deployment status: {deployment.status}")

```

### 9. Monitor Deployment Until Ready

Wait for the deployment to reach RUNNING/READY before sending inference requests. Image pull and weight load often take 10–20 minutes, and the status stays PENDING until the server is ready.

```python
TIMEOUT_MINUTES = 30
start_time = time.time()
time_check = max_wait_time_checker(TIMEOUT_MINUTES * 60, "Deployment")
print(f"Monitoring deployment '{deployment_name}'... (timeout {TIMEOUT_MINUTES} min)\n")

while True:
    time.sleep(15)
    time_check()
    deployment_status = client.models.get_deployment(name=deployment_name).data()
    elapsed = time.time() - start_time
    clear_output(wait=True)
    print(f"Deployment: {deployment_name}")
    print(f"Status: {deployment_status.status}")
    print(f"Elapsed: {int(elapsed // 60)}m {int(elapsed % 60)}s")
    if deployment_status.status in ("RUNNING", "READY"):
        print("\nDeployment is ready!")
        if not client.models.wait_for_gateway(deployment_name, workspace="default", timeout=60):
            raise RuntimeError("Inference gateway did not become ready")
        break
    if deployment_status.status in ("FAILED", "ERROR", "TERMINATED"):
        raise RuntimeError(f"Deployment failed with status: {deployment_status.status}")

assert deployment_status.status in ("RUNNING", "READY")

```

### 10. Check Model Output

Send a chat completion request to the deployed LoRA model and compare the output to the expected answer.

```python
context = "The Apollo 11 mission was the first manned mission to land on the Moon. It was launched on July 16, 1969, and Neil Armstrong became the first person to walk on the lunar surface on July 20, 1969. Buzz Aldrin joined him shortly after, while Michael Collins remained in lunar orbit."
question = "Who was the first person to walk on the Moon?"
messages = [
    {"role": "user", "content": f"Based on the following context, answer the question.\n\nContext: {context}\n\nQuestion: {question}"}
]
INFERENCE_MODEL_NAME = f"default--{OUTPUT_NAME}"
from nemo_helix_plugin.inference_gateway.client import InferenceGatewayClient
from nemo_helix_plugin.inference_gateway.types import JsonBody

gateway = InferenceGatewayClient.from_client(client)
response = gateway.provider_post(
    name=deployment_name,
    workspace="default",
    trailing_uri="v1/chat/completions",
    body=JsonBody(
        {
            "model": INFERENCE_MODEL_NAME,
            "messages": messages,
            "temperature": 0,
            "max_tokens": 256,
            # Nemotron 3.5 Lightning reasons by default; turn it off for a short answer.
            "chat_template_kwargs": {"enable_thinking": False},
        }
    ),
).data()
print("=" * 60)
print("MODEL INFERENCE")
print("=" * 60)
print(f"Question: {question}")
print(f"Expected: Neil Armstrong")
print(f"Model output: {response['choices'][0]['message']['content']}")

```

## Conclusion

You have started a LoRA customization job, monitored it to completion, and evaluated the fine-tuned model. Use the `output.name` to access the model for further inference or evaluation.

## Next Steps

* [Monitor training metrics](/documentation/customizer-reference/tutorials/metrics) in detail
* [Evaluate your fine-tuned model](/documentation/evaluate-models) using the Evals service
* Try [Full SFT](/documentation/customizer-reference/tutorials/sft-customization-job) for other customization options