Import and Fine-Tune Private Hugging Face Models

Use this tutorial to learn how to import a private Hugging Face model into NeMo Customizer, fine-tune it with LoRA, and deploy it for inference.

Prerequisites

New to using NeMo Platform?

All platform resources—models, datasets, and more—must belong to a workspace. Workspaces provide organizational and authorization boundaries for your work. Within a workspace, you can optionally use projects to group related resources.

If you’re new to the platform, start with the Setup guide to learn how to deploy and evaluate models, and optimize agents using the platform end-to-end.

If you’re already familiar with workspaces and how to upload datasets to the platform, you can proceed directly with this tutorial.

For more information, see Workspaces and Projects.

Tutorial-Specific Prerequisites

Completed the Quickstart to install and deploy NeMo Platform locally.
Installed the Python SDK and any tutorial packages you need in your environment.
A Hugging Face token with access to the private or gated model repository.
A Hugging Face model with a compatible architecture. This tutorial uses google/gemma-2-2b-it as an example, but success depends on architectural compatibility.
Sufficient GPU memory for the model and LoRA training job.

This tutorial uses the current NeMo Platform SDK resources: client.files.filesets, client.models, client.customization.automodel.jobs, and client.inference.gateway.

Known Issues

Conv1D Model Architecture Limitation: Models that use Conv1D layers are not compatible with NeMo Customizer AutoModel LoRA.

Error signature: AttributeError: 'Conv1D' object has no attribute 'config'

Affected models include:

microsoft/DialoGPT-* series
openai-gpt models
Some older gpt2 variants
Other models with Conv1D-based architectures

Root cause: These models use Conv1D layers that lack the linear layers expected by NeMo’s LoRA transformation utilities.

Solution: Use modern transformer architectures instead:

Llama models (3.1, 3.2, 3.3 series)
Nemotron models
Phi models
Gemma models (used in this tutorial)

For a complete list of tested models, see the Model Catalog.

Quick Start

1. Initialize the SDK

The SDK needs your NeMo Platform server URL. By default, this tutorial uses http://localhost:8080.

export NMP_BASE_URL=<YOUR_NMP_BASE_URL>
export HF_TOKEN=<YOUR_HUGGINGFACE_TOKEN>

import json
import os
import time
import uuid
from pathlib import Path
from IPython.display import clear_output
from nemo_automodel_plugin.schema import (
    AutomodelJobInput,
    BatchSpec,
    DatasetSpec,
    OptimizerSpec,
    OutputRequest,
    ParallelismSpec,
    ScheduleSpec,
    TrainingSpec,
)
from nemo_platform import ConflictError, NeMoPlatform
from nemo_platform.types.files import HuggingfaceStorageConfigParam
from nemo_platform.types.secrets import PlatformSecretResponse
def max_wait_time_checker(seconds: int, job_name: str = ""):
    start_time = time.time()
    def check():
        if time.time() - start_time > seconds:
            raise TimeoutError(f"{job_name} took longer than {seconds} seconds")
    return check
NMP_BASE_URL = os.environ.get("NMP_BASE_URL", "http://localhost:8080")
client = NeMoPlatform(base_url=NMP_BASE_URL, workspace="default")

2. Store the Hugging Face Token

Private and gated Hugging Face repositories require a token. Store it as a NeMo Platform secret and reference that secret from the Hugging Face fileset.

def create_or_get_secret(name: str, value: str | None, label: str) -> PlatformSecretResponse:
    if not value:
        raise ValueError(f"{label} is not set")
    try:
        secret = client.secrets.create(
            name=name,
            workspace="default",
            value=value,
        )
        print(f"Created secret: {name}")
        return secret
    except ConflictError:
        print(f"Secret '{name}' already exists, continuing...")
        return client.secrets.retrieve(name=name, workspace="default")
hf_secret = create_or_get_secret("hf-token", os.getenv("HF_TOKEN"), "HF_TOKEN")

3. Create a Model FileSet and Model Entity

Create a Hugging Face-backed fileset for the private model, then register a Model Entity that points to that fileset. Model files are downloaded by the platform when training or deployment needs them.

HF_REPO_ID = "google/gemma-2-2b-it"
MODEL_NAME = "gemma-2-2b-it"
model_storage = HuggingfaceStorageConfigParam(
    type="huggingface",
    repo_id=HF_REPO_ID,
    repo_type="model",
    token_secret=hf_secret.name,
)
try:
    base_model_fs = client.files.filesets.create(
        workspace="default",
        name=MODEL_NAME,
        description=f"Private Hugging Face model {HF_REPO_ID}",
        storage=model_storage,
        cache=True,
    )
    print(f"Created model fileset: {base_model_fs.name}")
except ConflictError:
    print(f"Model fileset '{MODEL_NAME}' already exists, refreshing Hugging Face settings...")
    client.files.filesets.delete(workspace="default", name=MODEL_NAME)
    base_model_fs = client.files.filesets.create(
        workspace="default",
        name=MODEL_NAME,
        description=f"Private Hugging Face model {HF_REPO_ID}",
        storage=model_storage,
        cache=True,
    )
try:
    base_model = client.models.create(
        workspace="default",
        name=MODEL_NAME,
        fileset=f"default/{MODEL_NAME}",
        trust_remote_code=False,
    )
    print(f"Created Model Entity: {base_model.name}")
except ConflictError:
    print(f"Model Entity '{MODEL_NAME}' already exists, updating fileset if needed...")
    base_model = client.models.update(
        workspace="default",
        name=MODEL_NAME,
        fileset=f"default/{MODEL_NAME}",
        trust_remote_code=False,
    )
time_check = max_wait_time_checker(600, "ModelSpec")
while not base_model.spec:
    time_check()
    time.sleep(10)
    base_model = client.models.retrieve(workspace="default", name=MODEL_NAME)
if base_model.spec:
    base_model.spec.linear_layers = None
    base_model = client.models.update(
        workspace="default",
        name=MODEL_NAME,
        spec=base_model.spec,
        verbose=True,
    )
print(base_model.model_dump_json(indent=2))

4. Prepare and Upload Training Data

Create chat-format JSONL files and upload them to a fileset. The file service, not a dataset-specific SDK resource, is the current entry point for training data.

DATASET_NAME = f"{MODEL_NAME}-training-data"
DATASET_PATH = Path(DATASET_NAME).absolute()
DATASET_PATH.mkdir(parents=True, exist_ok=True)
training_data = [
    {
        "messages": [
            {"role": "user", "content": "Hello, how are you?"},
            {"role": "assistant", "content": "I'm doing well. How can I help you today?"},
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "What is machine learning?"},
            {
                "role": "assistant",
                "content": "Machine learning is a way for computers to learn patterns from data.",
            },
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "Can you help me with Python?"},
            {"role": "assistant", "content": "Yes. Share the code or problem you want to work on."},
        ]
    },
]
validation_data = [
    {
        "messages": [
            {"role": "user", "content": "What is artificial intelligence?"},
            {
                "role": "assistant",
                "content": "Artificial intelligence is software that performs tasks associated with human intelligence.",
            },
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "Explain renewable energy."},
            {
                "role": "assistant",
                "content": "Renewable energy comes from naturally replenished sources such as sunlight, wind, and water.",
            },
        ]
    },
]
with open(DATASET_PATH / "training.jsonl", "w", encoding="utf-8") as f:
    for item in training_data:
        f.write(json.dumps(item) + "\n")
with open(DATASET_PATH / "validation.jsonl", "w", encoding="utf-8") as f:
    for item in validation_data:
        f.write(json.dumps(item) + "\n")
try:
    client.files.filesets.create(
        workspace="default",
        name=DATASET_NAME,
        description="Private Hugging Face model LoRA training data",
        cache=True,
    )
    print(f"Created dataset fileset: {DATASET_NAME}")
except ConflictError:
    print(f"Dataset fileset '{DATASET_NAME}' already exists, continuing...")
client.files.fsspec.put(
    lpath=DATASET_PATH,
    rpath=f"default/{DATASET_NAME}/",
    recursive=True,
)
print(client.files.list(fileset=DATASET_NAME, workspace="default"))

5. Start a LoRA Customization Job

Submit a LoRA job to the Automodel backend. Reference the model and dataset filesets using workspace/name.

job_suffix = uuid.uuid4().hex[:4]
JOB_NAME = f"{MODEL_NAME}-lora-job-{job_suffix}"
OUTPUT_NAME = f"{MODEL_NAME}-lora-{job_suffix}"
spec = AutomodelJobInput(
    model=f"default/{base_model.name}",
    dataset=DatasetSpec(
        training=f"default/{DATASET_NAME}",
        validation=f"default/{DATASET_NAME}",
    ),
    training=TrainingSpec(
        training_type="sft",
        finetuning_type="lora",
        max_seq_length=2048,
    ),
    schedule=ScheduleSpec(epochs=1),
    batch=BatchSpec(global_batch_size=64, micro_batch_size=1),
    optimizer=OptimizerSpec(learning_rate=5e-5),
    parallelism=ParallelismSpec(
        num_gpus_per_node=1,
        num_nodes=1,
        tensor_parallel_size=1,
        pipeline_parallel_size=1,
        context_parallel_size=1,
        expert_parallel_size=1,
    ),
    output=OutputRequest(name=OUTPUT_NAME),
)
job = client.customization.automodel.jobs.create(
    spec=spec,
    workspace="default",
    name=JOB_NAME,
)
print(f"Submitted job: {job.job.name}")
print(f"Output adapter: {OUTPUT_NAME}")

6. Track Training Progress

Poll job status until training reaches a terminal state.

time_check = max_wait_time_checker(3600, "Customization Job")
while True:
    time_check()
    status = client.jobs.get_status(name=job.job.name, workspace="default")
    clear_output(wait=True)
    print(f"Job status: {status.status}")
    step = max_steps = None
    training_phase = None
    for job_step in status.steps or []:
        if job_step.name == "training":
            for task in job_step.tasks or []:
                details = task.status_details if isinstance(task.status_details, dict) else {}
                step = details.get("step")
                max_steps = details.get("max_steps")
                training_phase = details.get("phase")
                break
            break
    if isinstance(step, (int, float)) and isinstance(max_steps, (int, float)) and max_steps:
        print(f"Training: step {step}/{max_steps} ({100 * step / max_steps:.1f}%)")
    if isinstance(training_phase, str):
        print(f"Phase: {training_phase}")
    if status.status in ("completed", "failed", "cancelled", "error"):
        print(f"\nJob finished: {status.status}")
        break
    time.sleep(10)
if status.status != "completed":
    raise RuntimeError(f"Training job finished with status: {status.status}")

7. Deploy the Base Model With LoRA Enabled

Create a deployment for the base model with LoRA support enabled. The LoRA adapter from training is served by the same deployment.

deploy_suffix = uuid.uuid4().hex[:4]
DEPLOYMENT_CONFIG_NAME = f"private-hf-lora-cfg-{deploy_suffix}"
DEPLOYMENT_NAME = f"private-hf-lora-{deploy_suffix}"
deployment_config = client.inference.deployment_configs.create(
    workspace="default",
    name=DEPLOYMENT_CONFIG_NAME,
    engine="vllm",
    model_spec={
        "model_namespace": "default",
        "model_name": MODEL_NAME,
        "lora_enabled": True,
    },
    executor_config={
        "gpu": 1,
        "image_name": "vllm/vllm-openai",
        "image_tag": "v0.22.1",
        "additional_args": ["--max-lora-rank", "32"],
    },
)
deployment = client.inference.deployments.create(
    workspace="default",
    name=DEPLOYMENT_NAME,
    config=deployment_config.name,
)
print(f"Deployment name: {deployment.name}")
print(f"Deployment status: {deployment.status}")

time_check = max_wait_time_checker(1800, "Deployment")
while True:
    time_check()
    time.sleep(15)
    deployment_status = client.inference.deployments.retrieve(
        name=deployment.name,
        workspace="default",
    )
    clear_output(wait=True)
    print(f"Deployment: {deployment.name}")
    print(f"Status: {deployment_status.status}")
    if deployment_status.status in ("RUNNING", "READY"):
        if not client.models.wait_for_gateway(deployment.name, workspace="default", timeout=60):
            raise RuntimeError("Inference gateway did not become ready")
        break
    if deployment_status.status in ("FAILED", "ERROR", "TERMINATED", "LOST"):
        raise RuntimeError(f"Deployment failed with status: {deployment_status.status}")

8. Test the Deployed Model

Call the inference gateway through the SDK. Use the base model name to test the original model and the output adapter name to test the LoRA-adapted model.

Python SDK

cURL

messages = [
    {"role": "user", "content": "Can you summarize what LoRA fine-tuning does?"}
]
def chat(model_id: str):
    return client.inference.gateway.provider.post(
        "v1/chat/completions",
        name=deployment.name,
        workspace="default",
        body={
            "model": model_id,
            "messages": messages,
            "temperature": 0.2,
            "max_tokens": 128,
        },
    )
BASE_INFERENCE_MODEL_NAME = f"default/{MODEL_NAME}"
INFERENCE_MODEL_NAME = f"default--{OUTPUT_NAME}"
base_response = chat(BASE_INFERENCE_MODEL_NAME)
lora_response = chat(INFERENCE_MODEL_NAME)
print("Base model response:")
print(base_response["choices"][0]["message"]["content"])
print("\nLoRA-adapted model response:")
print(lora_response["choices"][0]["message"]["content"])

Next Steps

Learn how to check customization job metrics to monitor training progress and performance for your fine-tuned model.