> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo-platform/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-platform/_mcp/server. # Optimize for Tokens/GPU Throughput [Run in Google Colab](https://colab.research.google.com/github/NVIDIA-NeMo/nemo-platform/blob/main/docs/customizer/tutorials/optimize-throughput.ipynb) ## About Learn how to use the NeMo Platform Customizer to create a [LoRA](/documentation/customizer-reference/customization-concepts#nemo-ms-about-concepts-customization) (Low-Rank Adaptation) customization job optimized for higher tokens/GPU throughput and lower runtime. **In this tutorial, you will:** 1. Fine-tune [meta-llama/Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) on the SQuAD dataset using LoRA, with [sequence packing](/documentation/customizer-reference/customization-concepts#nemo-ms-about-concepts-customization) enabled for one run and disabled for another. 2. Compare training runtime, GPU utilization, and memory allocation between the two runs. 3. Verify that validation loss remains comparable, confirming that sequence packing improves throughput without sacrificing model quality. > **Note:** While this tutorial demonstrates sequence packing with LoRA, the optimization is also available for all\_weights (full) SFT customization jobs. ## Prerequisites Before starting this tutorial, ensure you have: 1. **Completed the [Quickstart](/documentation/get-started)** to install and deploy NeMo Platform locally 2. **Installed the Python SDK** (PyPI wrapper: `pip install "nemo-platform[all]"`; source checkout: run `make bootstrap` from the repository root) 3. **At least one GPU with CUDA 13+** ## Quick Start ### 1. Initialize SDK The SDK needs to know your NeMo Platform server URL. By default, `http://localhost:8080` is used in accordance with the [Quickstart](/documentation/get-started) guide. If NeMo Platform is running at a custom location, you can override the URL by setting the `NMP_BASE_URL` environment variable: ```sh export NMP_BASE_URL= ``` ```python import json import os from nemo_platform import NeMoPlatform, ConflictError NMP_BASE_URL = os.environ.get("NMP_BASE_URL", "http://localhost:8080") client = NeMoPlatform( base_url=NMP_BASE_URL, workspace="default" ) ``` ### 2. Create Dataset FileSet and Upload Training Data Install additional dependencies if they are not installed in your Python environment. The cell below automatically detects your environment and uses: * `uv pip install` if you're in a uv-managed virtual environment * `pip install` otherwise Required packages: * `datasets` - Download the public [rajpurkar/squad](https://huggingface.co/datasets/rajpurkar/squad) dataset * `pandas` - Compare job results in table format * `matplotlib` - Plot live training metrics (loss curves, GPU utilization) * `nvidia-ml-py` - Collect GPU VRAM and compute utilization metrics during training ```sh if command -v uv >/dev/null 2>&1 && [ -n "$VIRTUAL_ENV" ]; then uv pip install datasets pandas matplotlib nvidia-ml-py else pip install datasets pandas matplotlib nvidia-ml-py fi ``` #### Download rajpurkar/squad Dataset SQuAD (Stanford Question Answering Dataset) is a reading comprehension dataset consisting of questions posed on Wikipedia articles, where the answer is a segment of text from the corresponding passage. ```python import json import os from pathlib import Path from datasets import load_dataset, Dataset, DatasetDict # Configuration SEED = 1234 DATASET_NAME = "sft-dataset" # Convert SQuAD format to prompt/completion format and save to JSONL def convert_squad_to_sft_format(example): """Convert SQuAD format to prompt/completion format for SFT training.""" prompt = f"Context: {example['context']} Question: {example['question']} Answer:" completion = example["answers"]["text"][0] # Take the first answer return {"prompt": prompt, "completion": completion} # Load the SQuAD dataset from Hugging Face print("Loading dataset rajpurkar/squad") ds = load_dataset("rajpurkar/squad") if not isinstance(ds, DatasetDict): raise ValueError("Dataset does not contain expected splits") print("Loaded dataset") # For the purpose of this tutorial, we'll use a subset of the dataset # We use a reduced dataset size (3000 training/300 validation samples) to keep tutorial runtime manageable # while still demonstrating the performance benefits of sequence packing. The larger the dataset, # the better the model will perform but the longer the training will take. training_size = 3000 validation_size = 300 DATASET_PATH = Path(DATASET_NAME).absolute() # Get training split and verify it's a Dataset (not IterableDataset) train_dataset = ds["train"] validation_dataset = ds["validation"] assert isinstance(train_dataset, Dataset), "Expected Dataset type" assert isinstance(validation_dataset, Dataset), "Expected Dataset type" # Select subsets and save to JSONL files training_ds = train_dataset.select(range(training_size)) validation_ds = validation_dataset.select(range(validation_size)) # Transform to SFT format (prompt/completion) training_ds = training_ds.map(convert_squad_to_sft_format, remove_columns=training_ds.column_names) validation_ds = validation_ds.map(convert_squad_to_sft_format, remove_columns=validation_ds.column_names) # Create directory if it doesn't exist # Note: This will create a local 'sft-dataset/' directory with training.jsonl and validation.jsonl files os.makedirs(DATASET_PATH, exist_ok=True) # Save subsets to JSONL files training_ds.to_json(f"{DATASET_PATH}/training.jsonl") validation_ds.to_json(f"{DATASET_PATH}/validation.jsonl") print(f"Saved training.jsonl with {len(training_ds)} rows") print(f"Saved validation.jsonl with {len(validation_ds)} rows") ``` ```python # Create fileset to store SFT training data try: client.files.filesets.create( workspace="default", name=DATASET_NAME, description="SFT training data" ) print(f"Created fileset: {DATASET_NAME}") except ConflictError: print(f"Fileset '{DATASET_NAME}' already exists, continuing...") # Upload training data files individually to ensure correct structure client.files.upload( local_path=f"{DATASET_PATH}/", # Trailing slash uploads directory contents to fileset root remote_path="", fileset=DATASET_NAME, workspace="default" ) # Validate training data is uploaded correctly print("Training data:") print(json.dumps([f.model_dump() for f in client.files.list(fileset=DATASET_NAME, workspace="default").data], indent=2)) ``` ### 3. Secrets Setup If you plan to use NGC or Hugging Face models, you will need to configure authentication: * **NGC models** (`ngc://` URIs): Requires NGC API key * **Hugging Face models** (`hf://` URIs): Requires HF token for gated/private models Configure these as secrets in your platform. Refer to [Managing Secrets](/documentation/get-started/core-concepts/manage-secrets) for detailed instructions. Get your credentials to access base models: * [NGC API Key](https://ngc.nvidia.com/) (Setup → Generate API Key) * [Hugging Face Token](https://huggingface.co/settings/tokens) (Create token with Read access) --- #### Quick Setup Example This tutorial uses the [meta-llama/Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct/tree/main) model from Hugging Face. Ensure that you have sufficient permissions to download the model. If you cannot access the files on the [meta-llama/Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct/tree/main) Hugging Face page, request access. **Hugging Face Authentication:** * For gated models (Llama, Gemma), you must provide a Hugging Face token via the `token_secret` parameter * Get your token from [Hugging Face Settings](https://huggingface.co/settings/tokens) (requires Read access) * Accept the model's terms on the Hugging Face model page before using it. Example: [meta-llama/Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct/tree/main) * For public models, you can omit the `token_secret` parameter when creating a fileset for the model in the next step. ```python # Export the HF_TOKEN and NGC_API_KEY environment variables if they are not already set HF_TOKEN = os.getenv("HF_TOKEN") NGC_API_KEY = os.getenv("NGC_API_KEY") def create_or_get_secret(name: str, value: str | None, label: str): if not value: raise ValueError(f"{label} environment variable is not set. Set it and try again.") try: secret = client.secrets.create( name=name, workspace="default", value=value, ) print(f"Created secret: {name}") return secret except ConflictError: print(f"Secret '{name}' already exists, continuing...") return client.secrets.retrieve(name=name, workspace="default") # Create Hugging Face token secret hf_secret = create_or_get_secret("hf-token", HF_TOKEN, "HF_TOKEN") print("HF_TOKEN secret:") print(hf_secret.model_dump_json(indent=2)) # Create NGC API key secret # Uncomment the line below if you have NGC API Key and want to finetune NGC models # ngc_api_key = create_or_get_secret("ngc-api-key", NGC_API_KEY, "NGC_API_KEY") ``` ### 4. Create Base Model FileSet Create a fileset pointing to the [meta-llama/Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct/tree/main) model on Hugging Face. This step creates a pointer to the model on Hugging Face and does not download it. The model is downloaded at job creation time. Note: for public models, you can omit the `token_secret` parameter when creating a model fileset. ```python import time from nemo_platform.types.files import HuggingfaceStorageConfigParam HF_REPO_ID = "meta-llama/Llama-3.2-1B-Instruct" MODEL_NAME = "llama-3-2-1b-base" # Ensure you have a Hugging Face token secret created # Create a fileset pointing to the desired Hugging Face model try: base_model_fs = client.files.filesets.create( workspace="default", name=MODEL_NAME, description="Llama 3.2 1B base model from Hugging Face", storage=HuggingfaceStorageConfigParam( type="huggingface", # repo_id is the full model name from Hugging Face repo_id=HF_REPO_ID, repo_type="model", # we use the secret created in the previous step token_secret=hf_secret.name ) ) print(f"Created base model fileset: {MODEL_NAME}") except ConflictError: print(f"Base model fileset already exists. Skipping creation.") base_model_fs = client.files.filesets.retrieve( workspace="default", name=MODEL_NAME, ) # Create the Model Entity representation. try: base_model = client.models.create( workspace="default", name=MODEL_NAME, fileset=f"default/{MODEL_NAME}", ) print(f"Created Model Entity: {MODEL_NAME}") except ConflictError: print(f"Base model already exists. Updating fileset if different.") base_model = client.models.update( workspace="default", name=MODEL_NAME, fileset=f"default/{MODEL_NAME}", ) print(f"\nBase model fileset: fileset://default/{base_model.name}") print("Base model fileset files list:") print(json.dumps([f.model_dump() for f in client.files.list(fileset=MODEL_NAME, workspace="default").data], indent=2)) # Wait for ModelSpec to be populated from the checkpoint print("\nWaiting for ModelSpec to be populated...") SPEC_TIMEOUT_SECONDS = 120 spec_start = time.time() while not base_model.spec: if time.time() - spec_start > SPEC_TIMEOUT_SECONDS: raise TimeoutError(f"ModelSpec not populated within {SPEC_TIMEOUT_SECONDS} seconds") time.sleep(2) base_model = client.models.retrieve( workspace="default", name=MODEL_NAME, ) print(f"ModelSpec populated: {base_model.spec}") ``` ### 5. Create LoRA Job with Sequence Packing Create a LoRA customization job with **sequence packing** enabled via `AutomodelJobInput` (`batch.sequence_packing=True`). ```python import uuid from nemo_automodel_plugin.schema import AutomodelJobInput SEQUENCE_PACKING_ENABLED = True job_suffix = uuid.uuid4().hex[:4] JOB_NAME = f"packing-job-{job_suffix}" PACK_OUTPUT_NAME = f"packing-out-{job_suffix}" spec = AutomodelJobInput( model=f"default/{base_model.name}", dataset={"training": f"default/{DATASET_NAME}"}, training={ "training_type": "sft", "finetuning_type": "lora", "max_seq_length": 4096, }, schedule={"epochs": 1, "val_check_interval": 0.1}, batch={ "global_batch_size": 64, "micro_batch_size": 1, "sequence_packing": SEQUENCE_PACKING_ENABLED, }, optimizer={"learning_rate": 5e-5}, parallelism={"num_gpus_per_node": 1}, output={"name": PACK_OUTPUT_NAME}, ) job_with_sequence_packing = client.customization.automodel.jobs.create( spec=spec, workspace="default", name=JOB_NAME ) print(f"Submitted job: {job_with_sequence_packing.job.name}") print(f"Output adapter: {PACK_OUTPUT_NAME}") ``` ### 6. Track Fine-Tuning Progress A training job contains multiple steps: * Model and dataset downloading * Fine-tuning where LoRA adapter weights are trained * Creating a fileset entry for the fine-tuned model * Fine-tuned weights uploading The elapsed time printed below reflects progress of the entire job. We compare the time taken by the fine-tuning step for both jobs in the last section of this tutorial. #### Define Helper Functions ```python # Helpers to draw GPU VRAM Utilization and Validation Loss import matplotlib.pyplot as plt try: import pynvml _PYNVML_AVAILABLE = True except ImportError: _PYNVML_AVAILABLE = False print("Note: Install nvidia-ml-py ('pip install nvidia-ml-py' or 'uv pip install nvidia-ml-py') to enable live GPU metrics.") # --------------------------------------------------------------------------- # GPU metrics collection (nvidia-ml-py; import name is pynvml) # --------------------------------------------------------------------------- def _get_gpu_snapshot() -> tuple[list[float], list[float]]: """Return (vram_usage_pcts, compute_util_pcts) for each GPU.""" if not _PYNVML_AVAILABLE: return [], [] pynvml.nvmlInit() try: vram, util = [], [] for i in range(pynvml.nvmlDeviceGetCount()): h = pynvml.nvmlDeviceGetHandleByIndex(i) mem = pynvml.nvmlDeviceGetMemoryInfo(h) rates = pynvml.nvmlDeviceGetUtilizationRates(h) vram.append(int(mem.used) / int(mem.total) * 100) util.append(float(rates.gpu)) return vram, util finally: pynvml.nvmlShutdown() # --------------------------------------------------------------------------- # Dashboard drawing helpers # --------------------------------------------------------------------------- _PALETTE = { "val_loss": "#E74C3C", "train_loss": "#F39C12", "vram": ["#3498DB", "#9B59B6", "#1ABC9C", "#E67E22"], "util": ["#2ECC71", "#E74C3C", "#3498DB", "#F1C40F"], "grid": "#ECECEC", "title": "#2C3E50", "subtitle": "#7F8C8D", "spine": "#CCCCCC", "tick": "#666666", } def _style_axis(ax): """Apply shared cosmetic styling to a subplot axis.""" ax.set_facecolor("white") ax.grid(True, alpha=0.4, color=_PALETTE["grid"], linewidth=0.8) for spine in ("top", "right"): ax.spines[spine].set_visible(False) ax.spines["left"].set_color(_PALETTE["spine"]) ax.spines["bottom"].set_color(_PALETTE["spine"]) ax.tick_params(colors=_PALETTE["tick"], labelsize=9) def _plot_line(ax, xs, ys, color, label, fill=True): """Plot a time series, gracefully skipping None values.""" pts = [(x, y) for x, y in zip(xs, ys) if y is not None] if not pts: return px, py = zip(*pts) ax.plot( px, py, color=color, linewidth=2.2, marker="o", markersize=4, markerfacecolor="white", markeredgewidth=1.8, markeredgecolor=color, label=label, zorder=3, ) if fill: ax.fill_between(px, py, alpha=0.08, color=color) def _plot_gpu_panel(ax, xs, history, colors, fallback_label): """Plot per-GPU time series with area fill.""" if not history or not history[0]: ax.text( 0.5, 0.5, "No GPU data", transform=ax.transAxes, ha="center", va="center", fontsize=11, color="#AAAAAA", ) return n_gpus = max(len(snap) for snap in history) for g in range(n_gpus): vals = [snap[g] if g < len(snap) else 0 for snap in history] c = colors[g % len(colors)] label = f"GPU {g}" if n_gpus > 1 else fallback_label ax.plot(xs[: len(vals)], vals, color=c, linewidth=2, label=label) ax.fill_between(xs[: len(vals)], vals, alpha=0.08, color=c) if n_gpus > 1: ax.legend(fontsize=9, framealpha=0.9, edgecolor="#DDD") def _draw_dashboard( elapsed_mins, val_losses, train_losses, vram_history, util_history, job_name, status_str, step_str, elapsed_str, ): """Render a live 1x3 training dashboard.""" fig, axes = plt.subplots(1, 3, figsize=(20, 5.5)) fig.patch.set_facecolor("#FAFBFC") fig.suptitle( job_name, fontsize=15, fontweight="bold", color=_PALETTE["title"], y=1.10, ) fig.text( 0.5, 1.01, f"{status_str} | {step_str} | {elapsed_str}", ha="center", fontsize=13, color=_PALETTE["subtitle"], ) for ax in axes: _style_axis(ax) # -- Panel 1: Loss curves -- _plot_line(axes[0], elapsed_mins, val_losses, _PALETTE["val_loss"], "Val Loss", fill=True) _plot_line(axes[0], elapsed_mins, train_losses, _PALETTE["train_loss"], "Train Loss", fill=False) axes[0].set_title("Train/Validation Loss", fontsize=13, fontweight="bold", color=_PALETTE["title"], pad=12) axes[0].set_xlabel("Time (min)", fontsize=10, color="#666") axes[0].set_ylabel("Loss", fontsize=10, color="#666") if any(v is not None for v in val_losses + train_losses): axes[0].legend(fontsize=9, framealpha=0.9, edgecolor="#DDD") # -- Panel 2: GPU VRAM usage -- _plot_gpu_panel(axes[1], elapsed_mins, vram_history, _PALETTE["vram"], "VRAM") axes[1].set_title("GPU VRAM Usage", fontsize=13, fontweight="bold", color=_PALETTE["title"], pad=12) axes[1].set_xlabel("Time (min)", fontsize=10, color="#666") axes[1].set_ylabel("Usage (%)", fontsize=10, color="#666") axes[1].set_ylim(-2, 105) # -- Panel 3: GPU utilization -- _plot_gpu_panel(axes[2], elapsed_mins, util_history, _PALETTE["util"], "Utilization") axes[2].set_title("GPU Utilization", fontsize=13, fontweight="bold", color=_PALETTE["title"], pad=12) axes[2].set_xlabel("Time (min)", fontsize=10, color="#666") axes[2].set_ylabel("Utilization (%)", fontsize=10, color="#666") axes[2].set_ylim(-2, 105) plt.tight_layout(rect=[0, 0, 1, 0.98]) plt.show() ``` #### Monitor the Job Until Completion The cell below polls the job status every 10 seconds and renders a live dashboard with validation loss, GPU VRAM usage, and GPU utilization charts. The charts appear empty at first while the model and dataset download; training metrics and GPU activity populate after the fine-tuning step begins. > **Note:** This is additional code. You can also use the Weights & Biases or MLflow integrations. ```python import time from typing import cast from IPython.display import clear_output from nemo_platform.types.shared import PlatformJobStatusResponse # Timeout set to 30 minutes to accommodate typical LoRA training duration for this dataset size. # Actual training time will vary based on hardware, model size, and dataset complexity. TIMEOUT_SECONDS = 30 * 60 # 30 minutes VAL_LOSS_KEY = "val_loss" TRAIN_LOSS_KEY = "train_loss" def get_training_metric( status: PlatformJobStatusResponse, metric_key: str, ) -> float | None: """Return a metric reported by a task in the training step.""" for job_step in status.steps or []: if job_step.name == "training": for task in job_step.tasks or []: value = (task.status_details or {}).get(metric_key) if value is not None: return float(value) return None # --------------------------------------------------------------------------- # Job polling with live dashboard # --------------------------------------------------------------------------- def wait_for_job( workspace: str, job_name: str, timeout: int = TIMEOUT_SECONDS, poll_interval: int = 10, val_loss_key: str = VAL_LOSS_KEY, train_loss_key: str = TRAIN_LOSS_KEY, ) -> PlatformJobStatusResponse: """ Poll job status until completed, failed, cancelled, or timeout. Displays a live dashboard with loss curves and GPU metrics. Args: workspace: The workspace where the job is running. job_name: The name of the job to monitor. timeout: Maximum time to wait in seconds (default: 30 minutes). poll_interval: Time between status checks in seconds (default: 10). Returns: The final job status response. """ start_time = time.time() # Time-series accumulators required for plotting elapsed_mins: list[float] = [] val_losses: list[float | None] = [] train_losses: list[float | None] = [] vram_history: list[list[float]] = [] util_history: list[list[float]] = [] while True: elapsed = time.time() - start_time elapsed_min = elapsed / 60 # Check for timeout if elapsed > timeout: error_message = f"Timeout reached after {elapsed_min:.1f} minutes" print(f"\n{error_message}") print("Job did not complete within the timeout period.") raise Exception(error_message) status = client.jobs.get_status(name=job_name, workspace=workspace) # -- Extract training progress from nested steps structure -- step: int | None = None max_steps: int | None = None training_phase: str | None = None val_loss: float | None = None train_loss: float | None = None current_step_name: str | None = None current_step_phase: str | None = None for job_step in status.steps or []: # Track the current active step name and phase for progress display if job_step.tasks: task = job_step.tasks[0] td = task.status_details or {} phase = cast(str, td.get("phase", "")) # Update current step if it's active or pending (not completed) if job_step.status in ("active", "pending"): current_step_name = job_step.name current_step_phase = phase or "started" if job_step.name == "training": for task in job_step.tasks or []: td = task.status_details or {} step = cast(int, td["step"]) if "step" in td else None max_steps = cast(int, td["max_steps"]) if "max_steps" in td else None training_phase = cast(str, td["phase"]) if "phase" in td else None raw_val_loss = td.get(val_loss_key) val_loss = float(raw_val_loss) if raw_val_loss is not None else None raw_train_loss = td.get(train_loss_key) train_loss = float(raw_train_loss) if raw_train_loss is not None else None break break if val_loss is None: raw_val_loss = (status.status_details or {}).get(val_loss_key) val_loss = float(raw_val_loss) if raw_val_loss is not None else None if train_loss is None: raw_train_loss = (status.status_details or {}).get(train_loss_key) train_loss = float(raw_train_loss) if raw_train_loss is not None else None # -- Collect GPU snapshot -- vram_pcts, util_pcts = _get_gpu_snapshot() # -- Append to accumulators used for the plots -- elapsed_mins.append(elapsed_min) val_losses.append(val_loss) train_losses.append(train_loss) vram_history.append(vram_pcts) util_history.append(util_pcts) # -- Build status strings -- status_str = f"Status: {status.status}" if step is not None and max_steps is not None: pct = step / max_steps * 100 step_str = f"Step {step}/{max_steps} ({pct:.0f}%)" if training_phase: step_str += f" - {training_phase}" else: if current_step_name and current_step_phase: step_str = f"{current_step_name} - {current_step_phase}" elif current_step_name: step_str = f"{current_step_name}" else: step_str = "Waiting for training to start..." elapsed_str = f"Elapsed: {elapsed_min:.1f} min" # -- Redraw dashboard -- clear_output(wait=True) _draw_dashboard( elapsed_mins, val_losses, train_losses, vram_history, util_history, job_name, status_str, step_str, elapsed_str, ) # -- Check terminal conditions -- if status.status.lower() == "completed": # Redraw dashboard one final time with "completed" status status_str = f"Status: {status.status}" if step is not None and max_steps is not None: step_str = f"Step {max_steps}/{max_steps} (100%)" clear_output(wait=True) _draw_dashboard( elapsed_mins, val_losses, train_losses, vram_history, util_history, job_name, status_str, step_str, elapsed_str, ) print(f"\nJob completed in {elapsed_min:.1f} minutes ({elapsed:.0f}s)") return status elif status.status.lower() in ("failed", "cancelled", "error"): print(f"\nJob finished with status: {status.status}") print(f"Total time elapsed: {elapsed_min:.1f} minutes ({elapsed:.0f}s)") # Print error details from the job level if status.error_details: error_msg = status.error_details.get("message", "") if error_msg: print(f"\nError: {error_msg}") # Find and print error details from the failed step/task for job_step in status.steps or []: if job_step.status == "error": print(f"\nFailed step: {job_step.name}") if job_step.error_details: step_error = job_step.error_details.get("message", "") if step_error: print(f"Step error: {step_error}") # Get error_stack from the failed task for task in job_step.tasks or []: if task.status == "error" and hasattr(task, "error_stack") and task.error_stack: print(f"\nError stack trace:\n{task.error_stack}") elif task.status == "error" and task.error_details: task_error = task.error_details.get("message", "") if task_error: print(f"Task error: {task_error}") break raise Exception(f"Job finished with status: {status.status}") time.sleep(poll_interval) # Wait for the job to complete job_with_sequence_packing_status = wait_for_job( workspace="default", job_name=job_with_sequence_packing.job.name, timeout=TIMEOUT_SECONDS, ) packed_val_loss = get_training_metric(job_with_sequence_packing_status, VAL_LOSS_KEY) if packed_val_loss is not None: print(f"Validation loss: {packed_val_loss:.2f}") else: print("Validation loss: not reported in job status") ``` ### 7. Create LoRA Job without Sequence Packing Create a second Automodel LoRA job with `batch.sequence_packing=False` for comparison. ```python import uuid from nemo_automodel_plugin.schema import AutomodelJobInput job_suffix = uuid.uuid4().hex[:4] JOB_NAME = f"no-packing-job-{job_suffix}" NO_PACK_OUTPUT_NAME = f"no-packing-out-{job_suffix}" spec = AutomodelJobInput( model=f"default/{base_model.name}", dataset={"training": f"default/{DATASET_NAME}"}, training={ "training_type": "sft", "finetuning_type": "lora", "max_seq_length": 4096, }, schedule={"epochs": 1, "val_check_interval": 0.1}, batch={ "global_batch_size": 64, "micro_batch_size": 1, "sequence_packing": False, }, optimizer={"learning_rate": 5e-5}, parallelism={"num_gpus_per_node": 1}, output={"name": NO_PACK_OUTPUT_NAME}, ) job_without_sequence_packing = client.customization.automodel.jobs.create( spec=spec, workspace="default", name=JOB_NAME ) print(f"Submitted job: {job_without_sequence_packing.job.name}") print(f"Output adapter: {NO_PACK_OUTPUT_NAME}") ``` ### 8. Track Fine-Tuning Progress for Job without Sequence Packing ```python # Wait for the training step to complete job_without_sequence_packing_status = wait_for_job( workspace="default", job_name=job_without_sequence_packing.job.name, timeout=TIMEOUT_SECONDS ) no_pack_val_loss = get_training_metric(job_without_sequence_packing_status, VAL_LOSS_KEY) if no_pack_val_loss is not None: print(f"Validation loss: {no_pack_val_loss:.2f}") else: print("Validation loss: not reported in job status") ``` ### 9. Compare Results * Time to complete training should be significantly lower for the job that used sequence packing. * The expected validation loss for both jobs should be similar. * Sequence packed version should have a higher GPU utilization and higher GPU Memory Allocation. ```python from nemo_platform.types.jobs import PlatformJobStep import pandas as pd STEP_NAME = "training" def get_elapsed_time(step: PlatformJobStep) -> float: """Calculate elapsed time in seconds from step's created_at to updated_at.""" if step.created_at is None or step.updated_at is None: raise ValueError("Training step timestamps are unavailable") return (step.updated_at - step.created_at).total_seconds() step_with_sequence_packing = client.jobs.steps.retrieve( name=STEP_NAME, workspace="default", job=job_with_sequence_packing.job.name, ) step_without_sequence_packing = client.jobs.steps.retrieve( name=STEP_NAME, workspace="default", job=job_without_sequence_packing.job.name, ) time_to_complete_with_sequence_packing = get_elapsed_time(step_with_sequence_packing) time_to_complete_without_sequence_packing = get_elapsed_time(step_without_sequence_packing) # Display results as a table results_df = pd.DataFrame({ "Seq Packing Enabled": [True, False], "Val Loss": [packed_val_loss, no_pack_val_loss], "Training Step Time, sec": [ time_to_complete_with_sequence_packing, time_to_complete_without_sequence_packing ] }) results_df.style.format({"Val Loss": "{:.2f}", "Training Step Time, sec": "{:.0f}"}).hide(axis='index') ``` #### Examples of Validation Loss The expected validation loss curves should match closely for both jobs. ![Validation loss comparison chart showing similar convergence patterns between sequence-packed and non-packed training runs over training steps](https://fdr-prod-docs-files-public.s3.us-east-1.amazonaws.com/nemo-platform.docs.buildwithfern.com/46aa81bd023f5ac8a05b3afb14e42c178efeae20ebfce990699324767bc51b72/generated/release-versions/v0.3.0/customizer/_images/packed_vs_not_packed_val_loss.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA6KXJSKKNFOCF7G4B%2F20260814%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260814T033800Z&X-Amz-Expires=604800&X-Amz-Signature=8be59f5a27b3dab1ccae6ee4f94d21cca0b71c19f7c00f77102fdde0839eb3de&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject) Sequence packed version should complete significantly faster. ![Runtime comparison chart demonstrating significantly reduced training time for sequence-packed job compared to non-packed baseline](https://fdr-prod-docs-files-public.s3.us-east-1.amazonaws.com/nemo-platform.docs.buildwithfern.com/55e3ed1c7abc734af7f1ae800586c70f0fc8a95bba94153c95a1110c348bd351/generated/release-versions/v0.3.0/customizer/_images/runtime.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA6KXJSKKNFOCF7G4B%2F20260814%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260814T033800Z&X-Amz-Expires=604800&X-Amz-Signature=69b62a6ece934b7a700651f53acae6b6bc5771d11bc32805687f44453517e6a4&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject) #### GPU Utilization Sequence packed version should have a higher GPU utilization. ![GPU utilization chart showing higher and more consistent GPU usage with sequence packing enabled throughout the training process](https://fdr-prod-docs-files-public.s3.us-east-1.amazonaws.com/nemo-platform.docs.buildwithfern.com/58036162ad51e43d8dbee45a70459ec37a10ab06614a4088417de9db6a9c8f64/generated/release-versions/v0.3.0/customizer/_images/gpu_utilization.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA6KXJSKKNFOCF7G4B%2F20260814%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260814T033800Z&X-Amz-Expires=604800&X-Amz-Signature=d0f7a0ea33b708c9b3703bbb062b2ffe82303f0b598e926b70764f4e172887b6&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject) #### GPU Memory Allocation Sequence packed version should have a higher GPU Memory Allocation. ![GPU memory allocation chart illustrating increased memory utilization efficiency with sequence packing enabled](https://fdr-prod-docs-files-public.s3.us-east-1.amazonaws.com/nemo-platform.docs.buildwithfern.com/2dbfd9fb92f561fe1439cde258cec7ec1e776862ca94bf6d7a77ece92d981c5d/generated/release-versions/v0.3.0/customizer/_images/gpu_memory.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA6KXJSKKNFOCF7G4B%2F20260814%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260814T033800Z&X-Amz-Expires=604800&X-Amz-Signature=927c4935b3b0b2bbb09f499077274af8ca294205b98392b68695389ed8a5f598&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject) ## Next Steps * [Monitor customization metrics](/documentation/customizer-reference/tutorials/metrics) for training and validation loss. * [Create a LoRA customization job](/documentation/customizer-reference/tutorials/lora-customization-job). * [Create a Full SFT customization job](/documentation/customizer-reference/tutorials/sft-customization-job).