Import and Fine-Tune Private Hugging Face Models

Use this tutorial to learn how to import a private Hugging Face model into NeMo Customizer, fine-tune it with LoRA, and deploy it for inference.

Prerequisites

New to using NeMo Helix?

All platform resources—models, datasets, and more—must belong to a workspace. Workspaces provide organizational and authorization boundaries for your work. Within a workspace, you can optionally use projects to group related resources.

If you’re new to the platform, start with the Setup guide to learn how to deploy and evaluate models, and optimize agents using the platform end-to-end.

If you’re already familiar with workspaces and how to upload datasets to the platform, you can proceed directly with this tutorial.

The CLI and Python SDK examples in this tutorial use the deployment and workspace in your current CLI context. To connect to a local or remote deployment, see Connect with CLI Contexts.

For more information, see Workspaces and Projects.

Tutorial-Specific Prerequisites

Completed the Quickstart to install and deploy NeMo Helix locally.
Installed the Python SDK and any tutorial packages you need in your environment.
A Hugging Face token with access to the private or gated model repository.
A private or gated Hugging Face model with a compatible architecture. The job settings in this tutorial assume a small dense model (about 2B parameters) on one GPU; for larger or MoE models, start from a reference recipe.
Sufficient GPU memory for the model and LoRA training job.

This tutorial uses the typed NeMo Helix clients: client.files, client.models, and the Customization resource’s customization.automodel.jobs.

Known Issues

Conv1D Model Architecture Limitation: Models that use Conv1D layers are not compatible with NeMo Customizer AutoModel LoRA.

Error signature: AttributeError: 'Conv1D' object has no attribute 'config'

Affected models include:

microsoft/DialoGPT-* series
openai-gpt models
Some older gpt2 variants
Other models with Conv1D-based architectures

Root cause: These models use Conv1D layers that lack the linear layers expected by NeMo’s LoRA transformation utilities.

Solution: Use modern transformer architectures instead:

Nemotron models
Other architectures listed in the Model Catalog

For a complete list of tested models, see the Model Catalog.

Quick Start

1. Initialize the SDK

The SDK connects to the deployment in your current CLI context. To connect to another deployment, refer to Connect with CLI Contexts. Export your Hugging Face token:

export HF_TOKEN=<YOUR_HUGGINGFACE_TOKEN>

import json
import os
import time
import uuid
from pathlib import Path
from IPython.display import clear_output
from nemo_automodel_plugin.schema import (
    AutomodelJobInput,
    BatchSpec,
    DatasetSpec,
    OptimizerSpec,
    OutputRequest,
    ParallelismSpec,
    ScheduleSpec,
    TrainingSpec,
)
from nemo_helix_plugin.client.client import NemoClient
from nemo_helix_plugin.client.errors import ConflictError
from nemo_helix_plugin.files.client import FilesClient
from nemo_helix_plugin.files.storage_config import HuggingfaceStorageConfig
from nemo_helix_plugin.files.types import CreateFilesetRequest
from nemo_helix_plugin.models.types import CreateModelEntityRequest, UpdateModelEntityRequest
from nemo_helix_plugin.schema import SecretRef
from nemo_helix_plugin.secrets.types import HelixSecretCreateRequest, HelixSecretResponse
from nemo_customizer.sdk import Customization
from filesets import FilesetFileSystem
from pydantic import SecretStr
def max_wait_time_checker(seconds: int, job_name: str = ""):
    start_time = time.time()
    def check():
        if time.time() - start_time > seconds:
            raise TimeoutError(f"{job_name} took longer than {seconds} seconds")
    return check
client = NemoClient.from_config()
customization = Customization.from_client(client)

These examples use the default workspace. If your current CLI context uses a different workspace, run nemo config set --workspace default first. To check, run nemo config view.

2. Store the Hugging Face Token

Private and gated Hugging Face repositories require a token. Store it as a NeMo Helix secret and reference that secret from the Hugging Face fileset.

def create_or_get_secret(name: str, value: str | None, label: str) -> HelixSecretResponse:
    if not value:
        raise ValueError(f"{label} is not set")
    try:
        secret = client.secrets.create_secret(
            body=HelixSecretCreateRequest(name=name, value=SecretStr(value)),
        ).data()
        print(f"Created secret: {name}")
        return secret
    except ConflictError:
        print(f"Secret '{name}' already exists, continuing...")
        return client.secrets.get_secret(name=name).data()
hf_secret = create_or_get_secret("hf-token", os.getenv("HF_TOKEN"), "HF_TOKEN")

3. Create a Model FileSet and Model Entity

Create a Hugging Face-backed fileset for the private model, then register a Model Entity that points to that fileset. Model files are downloaded by the platform when training or deployment needs them.

HF_REPO_ID = "<your-org>/<your-private-model>"  # Private or gated repository your token can read
MODEL_NAME = "my-private-model"
model_storage = HuggingfaceStorageConfig(
    repo_id=HF_REPO_ID,
    repo_type="model",
    token_secret=SecretRef(hf_secret.name),
)
try:
    base_model_fs = client.files.create_fileset(
        body=CreateFilesetRequest(
            name=MODEL_NAME,
            description=f"Private Hugging Face model {HF_REPO_ID}",
            storage=model_storage,
            cache=True,
        ),
    ).data()
    print(f"Created model fileset: {base_model_fs.name}")
except FilesConflictError:
    print(f"Model fileset '{MODEL_NAME}' already exists, refreshing Hugging Face settings...")
    client.files.delete_fileset(name=MODEL_NAME)
    base_model_fs = client.files.create_fileset(
        body=CreateFilesetRequest(
            name=MODEL_NAME,
            description=f"Private Hugging Face model {HF_REPO_ID}",
            storage=model_storage,
            cache=True,
        ),
    ).data()
try:
    base_model = client.models.create_model(
        body=CreateModelEntityRequest(
            name=MODEL_NAME,
            fileset=f"default/{MODEL_NAME}",
            trust_remote_code=False,
        ),
    ).data()
    print(f"Created Model Entity: {base_model.name}")
except ConflictError:
    print(f"Model Entity '{MODEL_NAME}' already exists, updating fileset if needed...")
    base_model = client.models.update_model(
        name=MODEL_NAME,
        body=UpdateModelEntityRequest(
            fileset=f"default/{MODEL_NAME}",
            trust_remote_code=False,
        ),
    ).data()
time_check = max_wait_time_checker(600, "ModelSpec")
while not base_model.spec:
    time_check()
    time.sleep(10)
    base_model = client.models.get_model(name=MODEL_NAME).data()
if base_model.spec:
    base_model.spec.linear_layers = None
    base_model = client.models.update_model(
        name=MODEL_NAME,
        body=UpdateModelEntityRequest(spec=base_model.spec),
    ).data()
print(base_model.model_dump_json(indent=2))

4. Prepare and Upload Training Data

Create chat-format JSONL files and upload them to a fileset. The file service, not a dataset-specific SDK resource, is the current entry point for training data.

DATASET_NAME = f"{MODEL_NAME}-training-data"
DATASET_PATH = Path(DATASET_NAME).absolute()
DATASET_PATH.mkdir(parents=True, exist_ok=True)
training_data = [
    {
        "messages": [
            {"role": "user", "content": "Hello, how are you?"},
            {"role": "assistant", "content": "I'm doing well. How can I help you today?"},
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "What is machine learning?"},
            {
                "role": "assistant",
                "content": "Machine learning is a way for computers to learn patterns from data.",
            },
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "Can you help me with Python?"},
            {"role": "assistant", "content": "Yes. Share the code or problem you want to work on."},
        ]
    },
]
validation_data = [
    {
        "messages": [
            {"role": "user", "content": "What is artificial intelligence?"},
            {
                "role": "assistant",
                "content": "Artificial intelligence is software that performs tasks associated with human intelligence.",
            },
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "Explain renewable energy."},
            {
                "role": "assistant",
                "content": "Renewable energy comes from naturally replenished sources such as sunlight, wind, and water.",
            },
        ]
    },
]
with open(DATASET_PATH / "training.jsonl", "w", encoding="utf-8") as f:
    for item in training_data:
        f.write(json.dumps(item) + "\n")
with open(DATASET_PATH / "validation.jsonl", "w", encoding="utf-8") as f:
    for item in validation_data:
        f.write(json.dumps(item) + "\n")
try:
    client.files.create_fileset(
        body=CreateFilesetRequest(
            name=DATASET_NAME,
            description="Private Hugging Face model LoRA training data",
            cache=True,
        ),
    ).data()
    print(f"Created dataset fileset: {DATASET_NAME}")
except ConflictError:
    print(f"Dataset fileset '{DATASET_NAME}' already exists, continuing...")
fs = FilesetFileSystem(client=FilesClient.from_client(client))
fs.put(
    lpath=DATASET_PATH,
    rpath=f"default/{DATASET_NAME}/",
    recursive=True,
)
print(client.files.list_files(name=DATASET_NAME).data())

5. Start a LoRA Customization Job

Submit a LoRA job to the Automodel backend. Reference the model and dataset filesets using workspace/name.

job_suffix = uuid.uuid4().hex[:4]
JOB_NAME = f"{MODEL_NAME}-lora-job-{job_suffix}"
OUTPUT_NAME = f"{MODEL_NAME}-lora-{job_suffix}"
spec = AutomodelJobInput(
    model=f"default/{base_model.name}",
    dataset=DatasetSpec(
        training=f"default/{DATASET_NAME}",
        validation=f"default/{DATASET_NAME}",
    ),
    training=TrainingSpec(
        training_type="sft",
        finetuning_type="lora",
        max_seq_length=2048,
    ),
    schedule=ScheduleSpec(epochs=1),
    batch=BatchSpec(global_batch_size=64, micro_batch_size=1),
    optimizer=OptimizerSpec(learning_rate=5e-5),
    parallelism=ParallelismSpec(
        num_gpus_per_node=1,
        num_nodes=1,
        tensor_parallel_size=1,
        pipeline_parallel_size=1,
        context_parallel_size=1,
        expert_parallel_size=1,
    ),
    output=OutputRequest(name=OUTPUT_NAME),
)
job = customization.automodel.jobs.create(
    spec=spec,
    workspace="default",
    name=JOB_NAME,
)
print(f"Submitted job: {job.job.name}")
print(f"Output adapter: {OUTPUT_NAME}")

6. Track Training Progress

Poll job status until training reaches a terminal state.

time_check = max_wait_time_checker(3600, "Customization Job")
while True:
    time_check()
    status = client.jobs.get_status(name=job.job.name, workspace="default")
    clear_output(wait=True)
    print(f"Job status: {status.status}")
    step = max_steps = None
    training_phase = None
    for job_step in status.steps or []:
        if job_step.name == "training":
            for task in job_step.tasks or []:
                details = task.status_details if isinstance(task.status_details, dict) else {}
                step = details.get("step")
                max_steps = details.get("max_steps")
                training_phase = details.get("phase")
                break
            break
    if isinstance(step, (int, float)) and isinstance(max_steps, (int, float)) and max_steps:
        print(f"Training: step {step}/{max_steps} ({100 * step / max_steps:.1f}%)")
    if isinstance(training_phase, str):
        print(f"Phase: {training_phase}")
    if status.status in ("completed", "failed", "cancelled", "error"):
        print(f"\nJob finished: {status.status}")
        break
    time.sleep(10)
if status.status != "completed":
    raise RuntimeError(f"Training job finished with status: {status.status}")

7. Deploy the Base Model With LoRA Enabled

Create a deployment for the base model with LoRA support enabled. The LoRA adapter from training is served by the same deployment.

deploy_suffix = uuid.uuid4().hex[:4]
DEPLOYMENT_CONFIG_NAME = f"private-hf-lora-cfg-{deploy_suffix}"
DEPLOYMENT_NAME = f"private-hf-lora-{deploy_suffix}"
from nemo_helix_plugin.models.types import (
    ContainerExecutorConfig,
    CreateModelDeploymentConfigRequest,
    CreateModelDeploymentRequest,
    Engine,
    ModelDeploymentConfigModelSpec,
)
deployment_config = client.models.create_deployment_config(
    body=CreateModelDeploymentConfigRequest(
        name=DEPLOYMENT_CONFIG_NAME,
        engine=Engine.VLLM,
        model_spec=ModelDeploymentConfigModelSpec(
            model_namespace="default",
            model_name=MODEL_NAME,
            lora_enabled=True,
        ),
        executor_config=ContainerExecutorConfig(
            gpu=1,
            image_name="docker.io/vllm/vllm-openai",
            image_tag="v0.22.1",
            additional_args=["--max-lora-rank", "32"],
        ),
    ),
).data()
deployment = client.models.create_deployment(
    body=CreateModelDeploymentRequest(
        name=DEPLOYMENT_NAME,
        config=deployment_config.name,
    ),
).data()
print(f"Deployment name: {deployment.name}")
print(f"Deployment status: {deployment.status}")

time_check = max_wait_time_checker(1800, "Deployment")
while True:
    time_check()
    time.sleep(15)
    deployment_status = client.models.get_deployment(name=deployment.name).data()
    clear_output(wait=True)
    print(f"Deployment: {deployment.name}")
    print(f"Status: {deployment_status.status}")
    if deployment_status.status in ("RUNNING", "READY"):
        if not client.models.wait_for_gateway(deployment.name, workspace="default", timeout=60):
            raise RuntimeError("Inference gateway did not become ready")
        break
    if deployment_status.status in ("FAILED", "ERROR", "TERMINATED", "LOST"):
        raise RuntimeError(f"Deployment failed with status: {deployment_status.status}")

8. Test the Deployed Model

Call the inference gateway through the SDK. Use the base model name to test the original model and the output adapter name to test the LoRA-adapted model.

Python SDK

cURL

messages = [
    {"role": "user", "content": "Can you summarize what LoRA fine-tuning does?"}
]
from nemo_helix_plugin.inference_gateway.client import InferenceGatewayClient
from nemo_helix_plugin.inference_gateway.types import JsonBody
def chat(model_id: str):
    return InferenceGatewayClient.from_client(client).provider_post(
        name=deployment.name,
        workspace="default",
        trailing_uri="v1/chat/completions",
        body=JsonBody(
            {
                "model": model_id,
                "messages": messages,
                "temperature": 0.2,
                "max_tokens": 128,
            }
        ),
    ).data()
BASE_INFERENCE_MODEL_NAME = f"default/{MODEL_NAME}"
INFERENCE_MODEL_NAME = f"default--{OUTPUT_NAME}"
base_response = chat(BASE_INFERENCE_MODEL_NAME)
lora_response = chat(INFERENCE_MODEL_NAME)
print("Base model response:")
print(base_response["choices"][0]["message"]["content"])
print("\nLoRA-adapted model response:")
print(lora_response["choices"][0]["message"]["content"])

Next Steps

Learn how to check customization job metrics to monitor training progress and performance for your fine-tuned model.