> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-platform/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-platform/_mcp/server.
# Import and Fine-Tune Private Hugging Face Models
Use this tutorial to learn how to import a private Hugging Face model into NeMo Customizer, fine-tune it with LoRA, and deploy it for inference.
## Prerequisites
#### New to using NeMo Platform?
All platform resources—models, datasets, and more—must belong to a **workspace**. Workspaces provide organizational and authorization boundaries for your work. Within a workspace, you can optionally use **projects** to group related resources.
**If you're new to the platform**, start with the **[Setup guide](/documentation/get-started)** to learn how to deploy and evaluate models, and optimize agents using the platform end-to-end.
**If you're already familiar** with workspaces and how to upload datasets to the platform, you can proceed directly with this tutorial.
For more information, see [Workspaces](/documentation/get-started/core-concepts/workspaces) and [Projects](/documentation/get-started/core-concepts/projects).
### Tutorial-Specific Prerequisites
* Completed the [Quickstart](/documentation/get-started) to install and deploy NeMo Platform locally.
* Installed the Python SDK and any tutorial packages you need in your environment.
* A Hugging Face token with access to the private or gated model repository.
* A Hugging Face model with a compatible architecture. This tutorial uses `google/gemma-2-2b-it` as an example, but success depends on architectural compatibility.
* Sufficient GPU memory for the model and LoRA training job.
This tutorial uses the current NeMo Platform SDK resources: `client.files.filesets`, `client.models`, `client.customization.automodel.jobs`, and `client.inference.gateway`.
### Known Issues
**Conv1D Model Architecture Limitation**: Models that use `Conv1D` layers are not compatible with NeMo Customizer AutoModel LoRA.
**Error signature**: `AttributeError: 'Conv1D' object has no attribute 'config'`
**Affected models include:**
* `microsoft/DialoGPT-*` series
* `openai-gpt` models
* Some older `gpt2` variants
* Other models with Conv1D-based architectures
**Root cause**: These models use Conv1D layers that lack the linear layers expected by NeMo's LoRA transformation utilities.
**Solution**: Use modern transformer architectures instead:
* Llama models (3.1, 3.2, 3.3 series)
* Nemotron models
* Phi models
* Gemma models (used in this tutorial)
For a complete list of tested models, see the [Model Catalog](/documentation/customizer-reference/models/model-catalog).
## Quick Start
### 1. Initialize the SDK
The SDK needs your NeMo Platform server URL. By default, this tutorial uses `http://localhost:8080`.
```sh
export NMP_BASE_URL=
export HF_TOKEN=
```
```python
import json
import os
import time
import uuid
from pathlib import Path
from IPython.display import clear_output
from nemo_automodel_plugin.schema import (
AutomodelJobInput,
BatchSpec,
DatasetSpec,
OptimizerSpec,
OutputRequest,
ParallelismSpec,
ScheduleSpec,
TrainingSpec,
)
from nemo_platform import ConflictError, NeMoPlatform
from nemo_platform.types.files import HuggingfaceStorageConfigParam
from nemo_platform.types.secrets import PlatformSecretResponse
def max_wait_time_checker(seconds: int, job_name: str = ""):
start_time = time.time()
def check():
if time.time() - start_time > seconds:
raise TimeoutError(f"{job_name} took longer than {seconds} seconds")
return check
NMP_BASE_URL = os.environ.get("NMP_BASE_URL", "http://localhost:8080")
client = NeMoPlatform(base_url=NMP_BASE_URL, workspace="default")
```
### 2. Store the Hugging Face Token
Private and gated Hugging Face repositories require a token. Store it as a NeMo Platform secret and reference that secret from the Hugging Face fileset.
```python
def create_or_get_secret(name: str, value: str | None, label: str) -> PlatformSecretResponse:
if not value:
raise ValueError(f"{label} is not set")
try:
secret = client.secrets.create(
name=name,
workspace="default",
value=value,
)
print(f"Created secret: {name}")
return secret
except ConflictError:
print(f"Secret '{name}' already exists, continuing...")
return client.secrets.retrieve(name=name, workspace="default")
hf_secret = create_or_get_secret("hf-token", os.getenv("HF_TOKEN"), "HF_TOKEN")
```
### 3. Create a Model FileSet and Model Entity
Create a Hugging Face-backed fileset for the private model, then register a Model Entity that points to that fileset. Model files are downloaded by the platform when training or deployment needs them.
```python
HF_REPO_ID = "google/gemma-2-2b-it"
MODEL_NAME = "gemma-2-2b-it"
model_storage = HuggingfaceStorageConfigParam(
type="huggingface",
repo_id=HF_REPO_ID,
repo_type="model",
token_secret=hf_secret.name,
)
try:
base_model_fs = client.files.filesets.create(
workspace="default",
name=MODEL_NAME,
description=f"Private Hugging Face model {HF_REPO_ID}",
storage=model_storage,
cache=True,
)
print(f"Created model fileset: {base_model_fs.name}")
except ConflictError:
print(f"Model fileset '{MODEL_NAME}' already exists, refreshing Hugging Face settings...")
client.files.filesets.delete(workspace="default", name=MODEL_NAME)
base_model_fs = client.files.filesets.create(
workspace="default",
name=MODEL_NAME,
description=f"Private Hugging Face model {HF_REPO_ID}",
storage=model_storage,
cache=True,
)
try:
base_model = client.models.create(
workspace="default",
name=MODEL_NAME,
fileset=f"default/{MODEL_NAME}",
trust_remote_code=False,
)
print(f"Created Model Entity: {base_model.name}")
except ConflictError:
print(f"Model Entity '{MODEL_NAME}' already exists, updating fileset if needed...")
base_model = client.models.update(
workspace="default",
name=MODEL_NAME,
fileset=f"default/{MODEL_NAME}",
trust_remote_code=False,
)
time_check = max_wait_time_checker(600, "ModelSpec")
while not base_model.spec:
time_check()
time.sleep(10)
base_model = client.models.retrieve(workspace="default", name=MODEL_NAME)
if base_model.spec:
base_model.spec.linear_layers = None
base_model = client.models.update(
workspace="default",
name=MODEL_NAME,
spec=base_model.spec,
verbose=True,
)
print(base_model.model_dump_json(indent=2))
```
### 4. Prepare and Upload Training Data
Create chat-format JSONL files and upload them to a fileset. The file service, not a dataset-specific SDK resource, is the current entry point for training data.
```python
DATASET_NAME = f"{MODEL_NAME}-training-data"
DATASET_PATH = Path(DATASET_NAME).absolute()
DATASET_PATH.mkdir(parents=True, exist_ok=True)
training_data = [
{
"messages": [
{"role": "user", "content": "Hello, how are you?"},
{"role": "assistant", "content": "I'm doing well. How can I help you today?"},
]
},
{
"messages": [
{"role": "user", "content": "What is machine learning?"},
{
"role": "assistant",
"content": "Machine learning is a way for computers to learn patterns from data.",
},
]
},
{
"messages": [
{"role": "user", "content": "Can you help me with Python?"},
{"role": "assistant", "content": "Yes. Share the code or problem you want to work on."},
]
},
]
validation_data = [
{
"messages": [
{"role": "user", "content": "What is artificial intelligence?"},
{
"role": "assistant",
"content": "Artificial intelligence is software that performs tasks associated with human intelligence.",
},
]
},
{
"messages": [
{"role": "user", "content": "Explain renewable energy."},
{
"role": "assistant",
"content": "Renewable energy comes from naturally replenished sources such as sunlight, wind, and water.",
},
]
},
]
with open(DATASET_PATH / "training.jsonl", "w", encoding="utf-8") as f:
for item in training_data:
f.write(json.dumps(item) + "\n")
with open(DATASET_PATH / "validation.jsonl", "w", encoding="utf-8") as f:
for item in validation_data:
f.write(json.dumps(item) + "\n")
try:
client.files.filesets.create(
workspace="default",
name=DATASET_NAME,
description="Private Hugging Face model LoRA training data",
cache=True,
)
print(f"Created dataset fileset: {DATASET_NAME}")
except ConflictError:
print(f"Dataset fileset '{DATASET_NAME}' already exists, continuing...")
client.files.fsspec.put(
lpath=DATASET_PATH,
rpath=f"default/{DATASET_NAME}/",
recursive=True,
)
print(client.files.list(fileset=DATASET_NAME, workspace="default"))
```
### 5. Start a LoRA Customization Job
Submit a LoRA job to the Automodel backend. Reference the model and dataset filesets using `workspace/name`.
```python
job_suffix = uuid.uuid4().hex[:4]
JOB_NAME = f"{MODEL_NAME}-lora-job-{job_suffix}"
OUTPUT_NAME = f"{MODEL_NAME}-lora-{job_suffix}"
spec = AutomodelJobInput(
model=f"default/{base_model.name}",
dataset=DatasetSpec(
training=f"default/{DATASET_NAME}",
validation=f"default/{DATASET_NAME}",
),
training=TrainingSpec(
training_type="sft",
finetuning_type="lora",
max_seq_length=2048,
),
schedule=ScheduleSpec(epochs=1),
batch=BatchSpec(global_batch_size=64, micro_batch_size=1),
optimizer=OptimizerSpec(learning_rate=5e-5),
parallelism=ParallelismSpec(
num_gpus_per_node=1,
num_nodes=1,
tensor_parallel_size=1,
pipeline_parallel_size=1,
context_parallel_size=1,
expert_parallel_size=1,
),
output=OutputRequest(name=OUTPUT_NAME),
)
job = client.customization.automodel.jobs.create(
spec=spec,
workspace="default",
name=JOB_NAME,
)
print(f"Submitted job: {job.job.name}")
print(f"Output adapter: {OUTPUT_NAME}")
```
### 6. Track Training Progress
Poll job status until training reaches a terminal state.
```python
time_check = max_wait_time_checker(3600, "Customization Job")
while True:
time_check()
status = client.jobs.get_status(name=job.job.name, workspace="default")
clear_output(wait=True)
print(f"Job status: {status.status}")
step = max_steps = None
training_phase = None
for job_step in status.steps or []:
if job_step.name == "training":
for task in job_step.tasks or []:
details = task.status_details if isinstance(task.status_details, dict) else {}
step = details.get("step")
max_steps = details.get("max_steps")
training_phase = details.get("phase")
break
break
if isinstance(step, (int, float)) and isinstance(max_steps, (int, float)) and max_steps:
print(f"Training: step {step}/{max_steps} ({100 * step / max_steps:.1f}%)")
if isinstance(training_phase, str):
print(f"Phase: {training_phase}")
if status.status in ("completed", "failed", "cancelled", "error"):
print(f"\nJob finished: {status.status}")
break
time.sleep(10)
if status.status != "completed":
raise RuntimeError(f"Training job finished with status: {status.status}")
```
### 7. Deploy the Base Model With LoRA Enabled
Create a deployment for the base model with LoRA support enabled. The LoRA adapter from training is served by the same deployment.
```python
deploy_suffix = uuid.uuid4().hex[:4]
DEPLOYMENT_CONFIG_NAME = f"private-hf-lora-cfg-{deploy_suffix}"
DEPLOYMENT_NAME = f"private-hf-lora-{deploy_suffix}"
deployment_config = client.inference.deployment_configs.create(
workspace="default",
name=DEPLOYMENT_CONFIG_NAME,
engine="vllm",
model_spec={
"model_namespace": "default",
"model_name": MODEL_NAME,
"lora_enabled": True,
},
executor_config={
"gpu": 1,
"image_name": "vllm/vllm-openai",
"image_tag": "v0.22.1",
"additional_args": ["--max-lora-rank", "32"],
},
)
deployment = client.inference.deployments.create(
workspace="default",
name=DEPLOYMENT_NAME,
config=deployment_config.name,
)
print(f"Deployment name: {deployment.name}")
print(f"Deployment status: {deployment.status}")
```
```python
time_check = max_wait_time_checker(1800, "Deployment")
while True:
time_check()
time.sleep(15)
deployment_status = client.inference.deployments.retrieve(
name=deployment.name,
workspace="default",
)
clear_output(wait=True)
print(f"Deployment: {deployment.name}")
print(f"Status: {deployment_status.status}")
if deployment_status.status in ("RUNNING", "READY"):
if not client.models.wait_for_gateway(deployment.name, workspace="default", timeout=60):
raise RuntimeError("Inference gateway did not become ready")
break
if deployment_status.status in ("FAILED", "ERROR", "TERMINATED", "LOST"):
raise RuntimeError(f"Deployment failed with status: {deployment_status.status}")
```
### 8. Test the Deployed Model
Call the inference gateway through the SDK. Use the base model name to test the original model and the output adapter name to test the LoRA-adapted model.
#### Python SDK
```python
messages = [
{"role": "user", "content": "Can you summarize what LoRA fine-tuning does?"}
]
def chat(model_id: str):
return client.inference.gateway.provider.post(
"v1/chat/completions",
name=deployment.name,
workspace="default",
body={
"model": model_id,
"messages": messages,
"temperature": 0.2,
"max_tokens": 128,
},
)
BASE_INFERENCE_MODEL_NAME = f"default/{MODEL_NAME}"
INFERENCE_MODEL_NAME = f"default--{OUTPUT_NAME}"
base_response = chat(BASE_INFERENCE_MODEL_NAME)
lora_response = chat(INFERENCE_MODEL_NAME)
print("Base model response:")
print(base_response["choices"][0]["message"]["content"])
print("\nLoRA-adapted model response:")
print(lora_response["choices"][0]["message"]["content"])
```
#### cURL
```bash
export OUTPUT_NAME="