> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-platform/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-platform/_mcp/server.

# Import and Fine-Tune Private HuggingFace Models

<a id="ft-tut-private-reg" />

Use this tutorial to learn how to import a private HuggingFace model into NeMo Customizer, fine-tune it with LoRA, and deploy it for inference.

## Prerequisites

All platform resources—models, datasets, and more—must belong to a **workspace**. Workspaces provide organizational and authorization boundaries for your work. Within a workspace, you can optionally use **projects** to group related resources.

**If you're new to the platform**, start with the **[Setup guide](/documentation/get-started)** to learn how to deploy and evaluate models, and optimize agents using the platform end-to-end.

**If you're already familiar** with workspaces and how to upload datasets to the platform, you can proceed directly with this tutorial.

For more information, see [Workspaces](/documentation/get-started/core-concepts/workspaces) and [Projects](/documentation/get-started/core-concepts/projects).

### Tutorial-Specific Prerequisites

* Access to Data Store and Deployment Manager service
* `hf` cli installed on a machine with internet access [(installation instructions)](https://huggingface.co/docs/huggingface_hub/main/en/guides/cli).
* A HuggingFace model with a **compatible architecture**. Not all HuggingFace models are compatible with NeMo Customizer. This tutorial uses `gemma-2-2b-it` as an example, but success depends on architectural compatibility.
* A HuggingFace API token and proper authentication setup.
* Sufficient storage space for the model files (typically 5-50GB depending on model size)
* At least 8GB GPU memory for smaller models, more for larger models

Verify that all required services are running and accessible before proceeding. You can check service health using the health endpoints documented in each service's API specification.

### Known Issues

**Conv1D Model Architecture Limitation**: Models that use `Conv1D` layers are not compatible with NeMo Customizer AutoModel LoRA.

**Error signature**: `AttributeError: 'Conv1D' object has no attribute 'config'`

**Affected models include:**

* `microsoft/DialoGPT-*` series
* `openai-gpt` models
* Some older `gpt2` variants
* Other models with Conv1D-based architectures

**Root cause**: These models use Conv1D layers that lack the linear layers expected by NeMo's LoRA transformation utilities.

**Solution**: Use modern transformer architectures instead:

* ✅ **Recommended**: Llama models (3.1, 3.2, 3.3 series)
* ✅ **Recommended**: Nemotron models
* ✅ **Recommended**: Phi models
* ✅ **Alternative**: Gemma models (used in this tutorial)

For a complete list of tested models, see the [Model Catalog](/documentation/customizer-reference/models/model-catalog).

***

## Download Model From HuggingFace Hub

1. Authenticate to HuggingFace using `hf auth login`.
2. Download the model.

***

## Create Model in Data Store

Next, create a model repository in the NeMo Data Store and upload the downloaded model files.

### Create Namespace and Model Repository

```bash
# Set environment variables - Update these to match your deployment
export NAMESPACE="my-org"
export MODEL_NAME="gemma-2-2b-it"
export MODEL_VERSION="$(date +%Y%m%d-%H%M%S)" # Unique version for this run
export NEMO_BASE_URL="http://nemo.test"
export DATASTORE_URL="http://data-store.test"
export REPO_ID="${NAMESPACE}/${MODEL_NAME}"
export DATASET_NAME="${MODEL_NAME}-training-data"

# Create namespace
curl -X POST "${DATASTORE_URL}/v1/datastore/namespaces" \
-H 'Content-Type: application/json' \
-d '{"namespace": "'${NAMESPACE}'"}'

# Create model repository in datastore
curl -X POST "${DATASTORE_URL}/v1/hf/api/repos/create" \
-H 'Content-Type: application/json' \
-d '{
"organization": "'${NAMESPACE}'",
"name": "'${MODEL_NAME}'",
"type": "model"
}'
```

### Upload Model Files to Data Store

Upload the downloaded model files to the Data Store repository:

## Create Model Entity in Entity Store

After uploading the model files to the Data Store, create a model entity in the Entity Store to register the model with its metadata and specifications for use in customization jobs.

```bash
# Create model entity in Entity Store
curl -X POST "${NEMO_BASE_URL}/v1/models" \
-H 'Content-Type: application/json' \
-d '{
"name": "'${MODEL_NAME}'",
"namespace": "'${NAMESPACE}'",
"description": "Private '${MODEL_NAME}' model imported for customization",
"artifact": {
"files_url": "hf://models/'${REPO_ID}'",
"backend_engine": "hugging_face",
"status": "upload_completed"
},
"spec": {
"num_parameters": 200000000,
"context_size": 1024,
"is_chat": true,
"num_virtual_tokens": -1
},
"peft": {
"finetuning_type": "all_weights"
}
}' | jq .
```

## Deploy the Base Model

Deploy the base model for inference with LoRA adapter support enabled, allowing it to load fine-tuned adapters from customization jobs.

```bash
curl "${NEMO_BASE_URL}/v1/deployment/configs" \
-X POST \
-H 'Content-Type: application/json' \
--data-binary '{
"model": "'${MODEL_NAME}'",
"name": "'${MODEL_NAME}'-deployment-config",
"namespace": "'${NAMESPACE}'",
"nim_deployment": {
"additional_envs": {
"NIM_FT_MODEL": "",
"NIM_GUIDED_DECODING_BACKEND": "outlines",
"NIM_JSONL_LOGGING": "0",
"NIM_MODEL_NAME": "/model-store",
"NIM_PEFT_REFRESH_INTERVAL": "30",
"NIM_PEFT_SOURCE": "http://nemo-entity-store:8000",
"UVICORN_LOG_LEVEL": "DEBUG",
"VLLM_NVEXT_LOG_LEVEL": "DEBUG"
},
"gpu": 1,
"image_name": "nvcr.io/nim/nvidia/llm-nim",
"image_tag": "1.13.1",
"disable_lora_support": false
}
}' | jq .


curl "${NEMO_BASE_URL}/v1/deployment/model-deployments" \
-X POST \
-H 'Content-Type: application/json' \
-d '{
"name": "'${MODEL_NAME}'-deployment",
"namespace": "'${NAMESPACE}'",
"config": "'${NAMESPACE}'/'${MODEL_NAME}'-deployment-config"
}' | jq .

```

## Create Customization Target

Create a customization target that references the uploaded model in the Data Store.

```bash

# Create customization target
curl -X POST \
"${NEMO_BASE_URL}/v1/customization/targets" \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"name": "'${MODEL_NAME}'@v'${MODEL_VERSION}'",
"namespace": "'${NAMESPACE}'",
"description": "Customization target for '${MODEL_NAME}'",
"enabled": true,
"model_uri": "hf://'${NAMESPACE}'/'${MODEL_NAME}'",
"num_parameters": 200000000,
"precision": "bf16-mixed"
}' | jq .
```

Wait for the model to be downloaded and ready:

```bash
# Check target status with comprehensive handling
while true; do
RESPONSE=$(curl -s -X GET \
"${NEMO_BASE_URL}/v1/customization/targets/${NAMESPACE}/${MODEL_NAME}@v${MODEL_VERSION}" \
-H 'accept: application/json')

STATUS=$(echo "$RESPONSE" | jq -r '.status')
echo "Target status: $STATUS"

if [ "$STATUS" = "ready" ]; then
echo "Model is ready for customization!"
break
elif [ "$STATUS" = "failed" ] || [ "$STATUS" = "cancelled" ] || [ "$STATUS" = "unknown" ] || [ "$STATUS" = "delete_failed" ]; then
echo "Model download failed with status: $STATUS"
echo "Contact your administrator for assistance."
break
elif [ "$STATUS" = "created" ] || [ "$STATUS" = "pending" ] || [ "$STATUS" = "downloading" ]; then
echo "Model is still being prepared, waiting..."
elif [ "$STATUS" = "deleted" ] || [ "$STATUS" = "deleting" ]; then
echo "Model is being deleted (status: $STATUS)"
echo "This target cannot be used for customization."
break
else
echo "Unknown status: $STATUS"
fi

sleep 30
done

```

## Create Customization Configuration

Create a configuration for LoRA fine-tuning:

```bash
# Create customization configuration
curl -X POST \
"${NEMO_BASE_URL}/v1/customization/configs" \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"name": "'${MODEL_NAME}'-lora-config@v'${MODEL_VERSION}'",
"namespace": "'${NAMESPACE}'",
"target": "'${NAMESPACE}'/'${MODEL_NAME}'@v'${MODEL_VERSION}'",
"description": "LoRA configuration for '${MODEL_NAME}'",
"training_options": [
{
"training_type": "sft",
"finetuning_type": "lora",
"num_gpus": 1,
"num_nodes": 1,
"tensor_parallel_size": 1,
"pipeline_parallel_size": 1,
"micro_batch_size": 1
}
],
"training_precision": "bf16-mixed",
"max_seq_length": 1024,
"prompt_template": "{prompt} {completion}"
}' | jq .
```

***

## Prepare Training and Validation Datasets

Before starting the customization job, prepare both training and validation datasets. The validation dataset helps track training progress and reduce overfitting.

Create datasets in JSONL format:

```python
import json
# Note: This assumes MODEL_NAME, NAMESPACE, and hf_api are already defined from previous code blocks

# Example training data for Gemma 2 (instruction format)
# Gemma 2 uses standard chat message format
training_data = [
    {
        "messages": [
            {"role": "user", "content": "Hello, how are you?"},
            {
                "role": "assistant",
                "content": "I'm doing well, thank you for asking! How can I assist you today?",
            },
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "What's the weather like?"},
            {
                "role": "assistant",
                "content": "I don't have access to current weather data, but I can help you find weather information or discuss weather patterns.",
            },
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "Tell me a joke"},
            {
                "role": "assistant",
                "content": "Why don't scientists trust atoms? Because they make up everything!",
            },
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "Can you help me with coding?"},
            {
                "role": "assistant",
                "content": "Absolutely! I'd be happy to help you with coding. What programming language or specific problem are you working on?",
            },
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "What is machine learning?"},
            {
                "role": "assistant",
                "content": "Machine learning is a subset of artificial intelligence that enables computers to learn and make decisions from data without being explicitly programmed.",
            },
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "How do I write a function in Python?"},
            {
                "role": "assistant",
                "content": "To write a function in Python, use the 'def' keyword followed by the function name and parameters: def my_function(param): return result",
            },
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "What are the benefits of exercise?"},
            {
                "role": "assistant",
                "content": "Exercise offers numerous benefits including improved cardiovascular health, stronger muscles, better mental health, and increased energy levels.",
            },
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "Explain the solar system"},
            {
                "role": "assistant",
                "content": "The solar system consists of the Sun and all celestial objects that orbit it, including eight planets, moons, asteroids, and comets.",
            },
        ]
    },
]

# Expand training data by repeating examples to create a larger dataset
training_data = training_data * 10  # 80 total examples (8 unique * 10 repetitions)

# Save training data to JSONL file
with open("training_data.jsonl", "w") as f:
    for item in training_data:
        f.write(json.dumps(item) + "\n")

# Create dataset in datastore (similar to model creation)
dataset_name = f"{MODEL_NAME}-training-data"
dataset_repo_id = f"{NAMESPACE}/{dataset_name}"

# Create dataset repository
hf_api.create_repo(dataset_repo_id, repo_type="dataset", exist_ok=True)

# Upload training data
hf_api.upload_file(
    path_or_fileobj="training_data.jsonl",
    path_in_repo="training.jsonl",
    repo_id=dataset_repo_id,
    repo_type="dataset",
    commit_message="Upload training data",
)

validation_data = [
    {
        "messages": [
            {"role": "user", "content": "How do you stay motivated?"},
            {
                "role": "assistant",
                "content": "Staying motivated involves setting clear goals, celebrating small wins, and maintaining a positive mindset even when facing challenges.",
            },
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "What's the capital of France?"},
            {
                "role": "assistant",
                "content": "The capital of France is Paris, a beautiful city known for its art, culture, and iconic landmarks like the Eiffel Tower.",
            },
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "Explain quantum physics"},
            {
                "role": "assistant",
                "content": "Quantum physics is the branch of physics that studies matter and energy at the smallest scales, where particles behave in ways that seem strange compared to our everyday experience.",
            },
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "What is photosynthesis?"},
            {
                "role": "assistant",
                "content": "Photosynthesis is the process by which plants convert sunlight, carbon dioxide, and water into glucose and oxygen, providing energy for the plant.",
            },
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "How does the internet work?"},
            {
                "role": "assistant",
                "content": "The internet works through a global network of interconnected computers that communicate using standardized protocols to share information and resources.",
            },
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "What is artificial intelligence?"},
            {
                "role": "assistant",
                "content": "Artificial intelligence is the simulation of human intelligence in machines that are programmed to think, learn, and solve problems like humans.",
            },
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "Describe the water cycle"},
            {
                "role": "assistant",
                "content": "The water cycle is the continuous movement of water through evaporation, condensation, precipitation, and collection that sustains life on Earth.",
            },
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "What are renewable energy sources?"},
            {
                "role": "assistant",
                "content": "Renewable energy sources include solar, wind, hydroelectric, geothermal, and biomass energy that can be naturally replenished and don't deplete over time.",
            },
        ]
    },
]

# Expand validation data by repeating examples to create a larger dataset
validation_data = validation_data * 10  # 80 total examples (8 unique * 10 repetitions)

# Save validation data to JSONL file
with open("validation_data.jsonl", "w") as f:
    for item in validation_data:
        f.write(json.dumps(item) + "\n")

# Upload validation data
hf_api.upload_file(
    path_or_fileobj="validation_data.jsonl",
    path_in_repo="validation.jsonl",
    repo_id=dataset_repo_id,
    repo_type="dataset",
    commit_message="Upload validation data",
)

# Create dataset entity in Entity Store
dataset = client.datasets.create(
    name=dataset_name, namespace=NAMESPACE, files_url=f"hf://datasets/{dataset_repo_id}"
)
print(f"Created dataset entity: {dataset.namespace}/{dataset.name}")
```

***

## Start Customization Job

Start the LoRA fine-tuning job. The job will create an output artifact with the name specified in `output`, which you'll use later to access your fine-tuned model for inference.

```bash
# Create job and capture job ID
RESPONSE=$(curl -s -X POST \
"${NEMO_BASE_URL}/v1/customization/jobs" \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"name": "'${MODEL_NAME}'-lora-job",
"config": "'${NAMESPACE}'/'${MODEL_NAME}'-lora-config@v'${MODEL_VERSION}'",
"dataset": "'${NAMESPACE}'/'${DATASET_NAME}'",
"output": {"name": "'${NAMESPACE}'/'${MODEL_NAME}'-lora@v'${MODEL_VERSION}'"},
"description": "LoRA fine-tuning job for '${MODEL_NAME}'",
"training": {
"type": "sft",
"peft": {
"type": "lora",
"rank": 16,
"alpha": 32,
"dropout": 0.01
},
"epochs": 3,
"batch_size": 8,
"learning_rate": 5e-5
}
}')

JOB_ID=$(echo "$RESPONSE" | jq -r '.id')
OUTPUT_NAME=$(echo "$RESPONSE" | jq -r '.spec.output.name')
echo "Started job with ID: $JOB_ID"
echo "Output name: $OUTPUT_NAME"
```

Copy the following values from the response:

* `id` (Job ID)
* `spec.output.name`

We'll need them later to monitor the job's status and access the fine-tuned model.

Check job progress:

```bash
# Monitor job status with comprehensive handling
while true; do
RESPONSE=$(curl -s -X GET \
"${NEMO_BASE_URL}/v1/customization/jobs/${JOB_ID}" \
-H 'accept: application/json')

STATUS=$(echo "$RESPONSE" | jq -r '.status')
echo "Job status: $STATUS"

if [ "$STATUS" = "completed" ]; then
echo "Training completed successfully!"
break
elif [ "$STATUS" = "failed" ] || [ "$STATUS" = "cancelled" ]; then
echo "Training finished with status: $STATUS"
if [ "$STATUS" = "failed" ]; then
echo "Check the job logs for error details."
fi
break
elif [ "$STATUS" = "created" ] || [ "$STATUS" = "pending" ]; then
echo "Job is queued and waiting to start..."
elif [ "$STATUS" = "running" ]; then
echo "Training is in progress..."
# Optionally show progress if available
PROGRESS=$(echo "$RESPONSE" | jq -r '.status_details.percentage_done // "N/A"')
if [ "$PROGRESS" != "N/A" ] && [ "$PROGRESS" != "null" ]; then
echo "Progress: ${PROGRESS}%"
fi
elif [ "$STATUS" = "cancelling" ]; then
echo "Job is being cancelled..."
elif [ "$STATUS" = "ready" ] || [ "$STATUS" = "unknown" ]; then
echo "Job finished with status: $STATUS"
break
else
echo "Unknown status: $STATUS"
fi

sleep 60 # Wait 1 minute before checking again
done
```

***

## Test the Deployed Model

After the customization job has been completed, you can use the `output.name` to access the fine-tuned model and evaluate its performance. The base model NIM deployment you created earlier will automatically load the LoRA adapter when you specify the LoRA model ID in your inference requests.

The inference endpoints use the `inference_base_url` configured during client initialization (typically the NIM proxy URL). The base model deployment must be running before you can test inference with LoRA adapters.

If you included a WandB API key, you can view your training results at [wandb.ai](https://wandb.ai/home) under the `nvidia-nemo-customizer` project.

```python
base_model_id = f"{NAMESPACE}/{MODEL_NAME}"

# Option 1: If you still have the job object from creation
# lora_model_id = job.spec.output.name

# Option 2: Construct from job parameters (use this if running in a new session)
lora_model_id = f"{NAMESPACE}/{MODEL_NAME}-lora@v{MODEL_VERSION}"

# First, check if the models are available for inference
# Note: The base model deployment must be running and registered with the NIM proxy
try:
    # List models from the entity store (shows all registered models)
    models_response = client.models.list()
    available_models = models_response.data

    print("Registered models:")
    for model in available_models:
        print(f" - {model.id}")

    # Test base model
    print(f"\nTesting base model: {base_model_id}")
    base_response = client.chat.completions.create(
        model=base_model_id,
        messages=[
            {
                "role": "user",
                "content": "Hello, can you help me?"
            }
        ],
        max_tokens=100,
        temperature=0.7
    )

    print("Base model response:")
    print(base_response.choices[0].message.content)

    # Test LoRA-adapted model (if available)
    print(f"\nTesting LoRA-adapted model: {lora_model_id}")
    lora_response = client.chat.completions.create(
        model=lora_model_id,
        messages=[
            {
                "role": "user",
                "content": "Hello, can you help me?"
            }
        ],
        max_tokens=100,
        temperature=0.7
    )

    print("LoRA-adapted model response:")
    print(lora_response.choices[0].message.content)

except Exception as e:
    print(f"Error testing model: {e}")
```

```bash
# Set the NIM proxy URL
export NIM_PROXY_URL="http://nim.test"

# Option 1: If you captured OUTPUT_MODEL from job creation (from earlier in tutorial)
# export OUTPUT_MODEL="<value from job creation response>"

# Option 2: Construct from environment variables (use this if running in a new session)
export LORA_MODEL_ID="${NAMESPACE}/${MODEL_NAME}-lora@v${MODEL_VERSION}"

# Test model availability
curl -X GET "${NIM_PROXY_URL}/models" | jq

# Test inference against the base model
curl -X POST "${NIM_PROXY_URL}/v1/chat/completions" \
-H 'Content-Type: application/json' \
-d '{
"model": "'${NAMESPACE}'/'${MODEL_NAME}'",
"messages": [
{"role": "user", "content": "Hello! How are you?"},
{"role": "assistant", "content": "Hi! I am quite well, how can I help you today?"},
{"role": "user", "content": "Can you write me a song?"}
],
"top_p": 1,
"n": 1,
"max_tokens": 50,
"frequency_penalty": 1.0
}'

# Test inference against a LoRA-adapted model (after customization completes)
curl -X POST "${NIM_PROXY_URL}/v1/chat/completions" \
-H 'Content-Type: application/json' \
-d '{
"model": "'${LORA_MODEL_ID}'",
"messages": [
{"role": "user", "content": "Hello! How are you?"},
{"role": "assistant", "content": "Hi! I am quite well, how can I help you today?"},
{"role": "user", "content": "Can you write me a song?"}
],
"top_p": 1,
"n": 1,
"max_tokens": 50,
"frequency_penalty": 1.0
}'
```

## Next Steps

Learn how to [check customization job metrics](/documentation/customizer-reference/tutorials/metrics) to monitor the training progress and performance of your fine-tuned model.