> This page is for version 0.10.
> For other versions, use one of these documentation indexes:
> - Latest (default): https://docs.nvidia.com/nemo-helix/latest/llms.txt
> - 0.10: https://docs.nvidia.com/nemo-helix/v0.10/llms.txt
> - 0.9: https://docs.nvidia.com/nemo-helix/v0.9/llms.txt
> - 0.8: https://docs.nvidia.com/nemo-helix/v0.8/llms.txt
> - 0.7: https://docs.nvidia.com/nemo-helix/v0.7/llms.txt
> - 0.6: https://docs.nvidia.com/nemo-helix/v0.6/llms.txt
> - 0.5: https://docs.nvidia.com/nemo-helix/v0.5/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-helix/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-helix/_mcp/server.

# Import and Fine-Tune Private Hugging Face Models

Use this tutorial to learn how to import a private Hugging Face model into NeMo Customizer, fine-tune it with LoRA, and deploy it for inference.

## Prerequisites

#### New to using NeMo Helix?

All platform resources—models, datasets, and more—must belong to a **workspace**. Workspaces provide organizational and authorization boundaries for your work. Within a workspace, you can optionally use **projects** to group related resources.

**If you're new to the platform**, start with the **[Setup guide](/nemo-helix/v0.10/documentation/get-started)** to learn how to deploy and evaluate models, and optimize agents using the platform end-to-end.

**If you're already familiar** with workspaces and how to upload datasets to the platform, you can proceed directly with this tutorial.

The CLI and Python SDK examples in this tutorial use the deployment and workspace in your current CLI context. To connect to a local or remote deployment, see [Connect with CLI Contexts](/nemo-helix/v0.10/documentation/reference/cli-reference/connect-to-deployments).

For more information, see [Workspaces](/nemo-helix/v0.10/documentation/get-started/core-concepts/workspaces) and [Projects](/nemo-helix/v0.10/documentation/get-started/core-concepts/projects).

### Tutorial-Specific Prerequisites

* Completed the [Quickstart](/nemo-helix/v0.10/documentation/get-started) to install and deploy NeMo Helix locally.
* Installed the Python SDK and any tutorial packages you need in your environment.
* A Hugging Face token with access to the private or gated model repository.
* A private or gated Hugging Face model with a compatible architecture. The job settings in this tutorial assume a small dense model (about 2B parameters) on one GPU; for larger or MoE models, [start from a reference recipe](/nemo-helix/v0.10/documentation/customizer-reference/manage-customization-jobs/training-configuration#reference-recipes).
* Sufficient GPU memory for the model and LoRA training job.

> **Note**
>
> This tutorial uses the typed NeMo Helix clients: `client.files`, `client.models`, and the `Customization` resource's `customization.automodel.jobs`.

### Known Issues

> **Warning**
>
> **Conv1D Model Architecture Limitation**: Models that use `Conv1D` layers are not compatible with NeMo Customizer AutoModel LoRA.
>
> **Error signature**: `AttributeError: 'Conv1D' object has no attribute 'config'`
>
> **Affected models include:**
>
> * `microsoft/DialoGPT-*` series
> * `openai-gpt` models
> * Some older `gpt2` variants
> * Other models with Conv1D-based architectures
>
> **Root cause**: These models use Conv1D layers that lack the linear layers expected by NeMo's LoRA transformation utilities.
>
> **Solution**: Use modern transformer architectures instead:
>
> * Nemotron models
> * Other architectures listed in the Model Catalog
>
> For a complete list of tested models, see the [Model Catalog](/nemo-helix/v0.10/documentation/customizer-reference/models/model-catalog).

## Quick Start

### 1. Initialize the SDK

The SDK connects to the deployment in your current CLI context. To connect to another deployment, refer to [Connect with CLI Contexts](/nemo-helix/v0.10/documentation/reference/cli-reference/connect-to-deployments). Export your Hugging Face token:

```sh
export HF_TOKEN=<YOUR_HUGGINGFACE_TOKEN>
```

```python
import json
import os
import time
import uuid
from pathlib import Path

from IPython.display import clear_output
from nemo_automodel_plugin.schema import (
    AutomodelJobInput,
    BatchSpec,
    DatasetSpec,
    OptimizerSpec,
    OutputRequest,
    ParallelismSpec,
    ScheduleSpec,
    TrainingSpec,
)
from nemo_helix_plugin.client.client import NemoClient
from nemo_helix_plugin.client.errors import ConflictError
from nemo_helix_plugin.files.client import FilesClient
from nemo_helix_plugin.files.storage_config import HuggingfaceStorageConfig
from nemo_helix_plugin.files.types import CreateFilesetRequest
from nemo_helix_plugin.models.types import CreateModelEntityRequest, UpdateModelEntityRequest
from nemo_helix_plugin.schema import SecretRef
from nemo_helix_plugin.secrets.types import HelixSecretCreateRequest, HelixSecretResponse
from nemo_customizer.sdk import Customization
from filesets import FilesetFileSystem
from pydantic import SecretStr

def max_wait_time_checker(seconds: int, job_name: str = ""):
    start_time = time.time()

    def check():
        if time.time() - start_time > seconds:
            raise TimeoutError(f"{job_name} took longer than {seconds} seconds")

    return check

client = NemoClient.from_config()
customization = Customization.from_client(client)
```

> **Note**
>
> These examples use the `default` workspace. If your current CLI context uses a different workspace, run `nemo config set --workspace default` first. To check, run `nemo config view`.

### 2. Store the Hugging Face Token

Private and gated Hugging Face repositories require a token. Store it as a NeMo Helix secret and reference that secret from the Hugging Face fileset.

```python
def create_or_get_secret(name: str, value: str | None, label: str) -> HelixSecretResponse:
    if not value:
        raise ValueError(f"{label} is not set")
    try:
        secret = client.secrets.create_secret(
            body=HelixSecretCreateRequest(name=name, value=SecretStr(value)),
        ).data()
        print(f"Created secret: {name}")
        return secret
    except ConflictError:
        print(f"Secret '{name}' already exists, continuing...")
        return client.secrets.get_secret(name=name).data()

hf_secret = create_or_get_secret("hf-token", os.getenv("HF_TOKEN"), "HF_TOKEN")
```

### 3. Create a Model FileSet and Model Entity

Create a Hugging Face-backed fileset for the private model, then register a Model Entity that points to that fileset. Model files are downloaded by the platform when training or deployment needs them.

```python
HF_REPO_ID = "<your-org>/<your-private-model>"  # Private or gated repository your token can read
MODEL_NAME = "my-private-model"
model_storage = HuggingfaceStorageConfig(
    repo_id=HF_REPO_ID,
    repo_type="model",
    token_secret=SecretRef(hf_secret.name),
)

try:
    base_model_fs = client.files.create_fileset(
        body=CreateFilesetRequest(
            name=MODEL_NAME,
            description=f"Private Hugging Face model {HF_REPO_ID}",
            storage=model_storage,
            cache=True,
        ),
    ).data()
    print(f"Created model fileset: {base_model_fs.name}")
except FilesConflictError:
    print(f"Model fileset '{MODEL_NAME}' already exists, refreshing Hugging Face settings...")
    client.files.delete_fileset(name=MODEL_NAME)
    base_model_fs = client.files.create_fileset(
        body=CreateFilesetRequest(
            name=MODEL_NAME,
            description=f"Private Hugging Face model {HF_REPO_ID}",
            storage=model_storage,
            cache=True,
        ),
    ).data()

try:
    base_model = client.models.create_model(
        body=CreateModelEntityRequest(
            name=MODEL_NAME,
            fileset=f"default/{MODEL_NAME}",
            trust_remote_code=False,
        ),
    ).data()
    print(f"Created Model Entity: {base_model.name}")
except ConflictError:
    print(f"Model Entity '{MODEL_NAME}' already exists, updating fileset if needed...")
    base_model = client.models.update_model(
        name=MODEL_NAME,
        body=UpdateModelEntityRequest(
            fileset=f"default/{MODEL_NAME}",
            trust_remote_code=False,
        ),
    ).data()

time_check = max_wait_time_checker(600, "ModelSpec")
while not base_model.spec:
    time_check()
    time.sleep(10)
    base_model = client.models.get_model(name=MODEL_NAME).data()

if base_model.spec:
    base_model.spec.linear_layers = None
    base_model = client.models.update_model(
        name=MODEL_NAME,
        body=UpdateModelEntityRequest(spec=base_model.spec),
    ).data()
print(base_model.model_dump_json(indent=2))
```

### 4. Prepare and Upload Training Data

Create chat-format JSONL files and upload them to a fileset. The file service, not a dataset-specific SDK resource, is the current entry point for training data.

```python
DATASET_NAME = f"{MODEL_NAME}-training-data"
DATASET_PATH = Path(DATASET_NAME).absolute()
DATASET_PATH.mkdir(parents=True, exist_ok=True)

training_data = [
    {
        "messages": [
            {"role": "user", "content": "Hello, how are you?"},
            {"role": "assistant", "content": "I'm doing well. How can I help you today?"},
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "What is machine learning?"},
            {
                "role": "assistant",
                "content": "Machine learning is a way for computers to learn patterns from data.",
            },
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "Can you help me with Python?"},
            {"role": "assistant", "content": "Yes. Share the code or problem you want to work on."},
        ]
    },
]

validation_data = [
    {
        "messages": [
            {"role": "user", "content": "What is artificial intelligence?"},
            {
                "role": "assistant",
                "content": "Artificial intelligence is software that performs tasks associated with human intelligence.",
            },
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "Explain renewable energy."},
            {
                "role": "assistant",
                "content": "Renewable energy comes from naturally replenished sources such as sunlight, wind, and water.",
            },
        ]
    },
]

with open(DATASET_PATH / "training.jsonl", "w", encoding="utf-8") as f:
    for item in training_data:
        f.write(json.dumps(item) + "\n")

with open(DATASET_PATH / "validation.jsonl", "w", encoding="utf-8") as f:
    for item in validation_data:
        f.write(json.dumps(item) + "\n")

try:
    client.files.create_fileset(
        body=CreateFilesetRequest(
            name=DATASET_NAME,
            description="Private Hugging Face model LoRA training data",
            cache=True,
        ),
    ).data()
    print(f"Created dataset fileset: {DATASET_NAME}")
except ConflictError:
    print(f"Dataset fileset '{DATASET_NAME}' already exists, continuing...")

fs = FilesetFileSystem(client=FilesClient.from_client(client))
fs.put(
    lpath=DATASET_PATH,
    rpath=f"default/{DATASET_NAME}/",
    recursive=True,
)
print(client.files.list_files(name=DATASET_NAME).data())
```

### 5. Start a LoRA Customization Job

Submit a LoRA job to the Automodel backend. Reference the model and dataset filesets using `workspace/name`. `deployment_config` deploys the base model with LoRA enabled when training finishes and serves the adapter from it.

```python
job_suffix = uuid.uuid4().hex[:4]
JOB_NAME = f"{MODEL_NAME}-lora-job-{job_suffix}"
OUTPUT_NAME = f"{MODEL_NAME}-lora-{job_suffix}"

spec = AutomodelJobInput(
    model=f"default/{base_model.name}",
    dataset=DatasetSpec(training=f"default/{DATASET_NAME}"),
    training=TrainingSpec(
        training_type="sft",
        finetuning_type="lora",
        max_seq_length=2048,
    ),
    schedule=ScheduleSpec(epochs=1),
    batch=BatchSpec(global_batch_size=64, micro_batch_size=1),
    optimizer=OptimizerSpec(learning_rate=5e-5),
    parallelism=ParallelismSpec(
        num_gpus_per_node=1,
        num_nodes=1,
        tensor_parallel_size=1,
        pipeline_parallel_size=1,
        context_parallel_size=1,
        expert_parallel_size=1,
    ),
    output=OutputRequest(name=OUTPUT_NAME),
    deployment_config={"engine": "vllm", "gpu": 1},
)

job = customization.automodel.jobs.create(
    spec=spec,
    workspace="default",
    name=JOB_NAME,
)

print(f"Submitted job: {job.job.name}")
print(f"Output adapter: {OUTPUT_NAME}")
```

### 6. Track Training Progress

Poll job status until training reaches a terminal state.

```python
time_check = max_wait_time_checker(3600, "Customization Job")

while True:
    time_check()
    status = client.jobs.get_job_status(name=job.job.name, workspace="default").data()
    clear_output(wait=True)
    print(f"Job status: {status.status}")

    step = max_steps = None
    training_phase = None
    for job_step in status.steps or []:
        if job_step.name == "training":
            for task in job_step.tasks or []:
                details = task.status_details if isinstance(task.status_details, dict) else {}
                step = details.get("step")
                max_steps = details.get("max_steps")
                training_phase = details.get("phase")
                break
            break

    if isinstance(step, (int, float)) and isinstance(max_steps, (int, float)) and max_steps:
        print(f"Training: step {step}/{max_steps} ({100 * step / max_steps:.1f}%)")
    if isinstance(training_phase, str):
        print(f"Phase: {training_phase}")

    if status.status in ("completed", "failed", "cancelled", "error"):
        print(f"\nJob finished: {status.status}")
        break

    time.sleep(10)

if status.status != "completed":
    raise RuntimeError(f"Training job finished with status: {status.status}")
```

### 7. Wait for the Base Model Deployment

When training finished, the job deployed the base model with LoRA enabled, and the adapter is served from that deployment. If the base model was already deployed without LoRA, the job created a LoRA-enabled deployment alongside it. The job completes once the deployment is created, not when it is ready, so find the deployment and wait for it.

```python
def find_deployment(model_entity: str, *, lora_enabled: bool = False):
    """Return the newest active deployment that serves a model entity."""
    configs = client.models.list_deployment_configs(
        workspace="default",
        query_params={"filter": json.dumps({"model_entity_id": f"default/{model_entity}"}), "sort": "-created_at"},
    ).items()
    for config in configs:
        deployments = client.models.list_deployments(
            workspace="default",
            query_params={"filter": json.dumps({"config": config.name}), "sort": "-created_at"},
        ).items()
        for deployment in deployments:
            if deployment.status not in ("CREATED", "PENDING", "READY"):
                continue
            # A deployment runs the config version it was created with, not necessarily the latest.
            running = client.models.get_deployment_config_version(
                workspace="default", config=config.name, name=str(deployment.config_version)
            ).data()
            if not lora_enabled or running.model_spec.lora_enabled:
                return deployment
    raise RuntimeError(f"No active deployment serves default/{model_entity}")


deployment = find_deployment(MODEL_NAME, lora_enabled=True)
print(f"Deployment name: {deployment.name}")
print(f"Deployment status: {deployment.status}")
```

```python
if not client.models.wait_for_deployment_status(deployment.name, "READY", workspace="default", timeout=1800):
    raise RuntimeError(f"Deployment {deployment.name} did not become ready")
if not client.models.wait_for_gateway(deployment.name, workspace="default", timeout=60):
    raise RuntimeError("Inference gateway did not become ready")
```

### 8. Test the Deployed Model

Call the inference gateway's OpenAI-compatible route and name each model by its entity ID: `default/<model>` for the base model, and `default/<model>&adapters/default/<adapter>` for the LoRA adapter. The gateway routes the adapter through its base model's deployment.

#### Python SDK

```python
messages = [
    {"role": "user", "content": "Can you summarize what LoRA fine-tuning does?"}
]

from nemo_helix_plugin.inference_gateway.client import InferenceGatewayClient
from nemo_helix_plugin.inference_gateway.types import JsonBody


def chat(model_id: str):
    return InferenceGatewayClient.from_client(client).openai_post(
        workspace="default",
        trailing_uri="v1/chat/completions",
        body=JsonBody(
            {
                "model": model_id,
                "messages": messages,
                "temperature": 0.2,
                "max_tokens": 128,
            }
        ),
    ).data()

BASE_INFERENCE_MODEL_NAME = f"default/{MODEL_NAME}"
INFERENCE_MODEL_NAME = f"default/{MODEL_NAME}&adapters/default/{OUTPUT_NAME}"
base_response = chat(BASE_INFERENCE_MODEL_NAME)
lora_response = chat(INFERENCE_MODEL_NAME)

print("Base model response:")
print(base_response["choices"][0]["message"]["content"])
print("\nLoRA-adapted model response:")
print(lora_response["choices"][0]["message"]["content"])
```

#### cURL

This example uses `NHX_BASE_URL` when it is set and falls back to `http://localhost:8080`. If your CLI context points at a remote deployment, export its URL first. Refer to [When to use `NHX_BASE_URL`](/nemo-helix/v0.10/documentation/reference/cli-reference/connect-to-deployments#when-to-use-nhx-base-url).

```bash
export MODEL_NAME="my-private-model"
export OUTPUT_NAME="<output adapter name from the Python job block>"
export INFERENCE_MODEL_NAME="default/${MODEL_NAME}&adapters/default/${OUTPUT_NAME}"

curl -s "${NHX_BASE_URL:-http://localhost:8080}/apis/inference-gateway/v2/workspaces/default/openai/-/v1/chat/completions" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "'${INFERENCE_MODEL_NAME}'",
    "messages": [
      {"role": "user", "content": "Can you summarize what LoRA fine-tuning does?"}
    ],
    "temperature": 0.2,
    "max_tokens": 128
  }' | jq
```

## Next Steps

Learn how to [check customization job metrics](/nemo-helix/v0.10/documentation/customizer-reference/tutorials/metrics) to monitor training progress and performance for your fine-tuned model.