Skip to navigation
DocumentationFine-tune ModelsManage Customization Jobs

Get Job Status

View as Markdown

Get detailed execution status for a customization job, including step-by-step progress and real-time training metrics.

This endpoint provides granular execution details including:

  • Step-level status: model-and-dataset-download → training → model-upload → model-entity-creation
  • Training progress: step, epoch, max_steps, num_epochs, percentage_done
  • Latest metric values: one field per metric, named <phase>_<metric> — train_loss, train_lr, train_grad_norm, val_loss, and whatever else your backend reports
  • Metric history: metrics, holding each metric as a series of {step, epoch, value} points
  • Progress tracking: downloaded_files, uploaded_bytes, progress_pct

Which metrics appear depends on the backend and the algorithm. See Checking Your Customization Job Metrics for how the names are formed.

To list jobs or get job definitions (model entity, hyperparameters, spec), use List Active Jobs instead.

Prerequisites

Before you can get the status of a customization job, make sure that you have:

To Get the Status of a Customization Job

A submitted customization job runs on the platform’s Jobs service, so you poll its status through that service using the job name returned at submission (for example, automodel-a1b2c3d4e5f6). This works the same way for both the automodel and unsloth backends.

Use the SDK to get detailed job status:

from nemo_helix_plugin.client.client import NemoClient
# Initialize the client
client = NemoClient.from_config()
# Get job status (use the job name returned at submission)
job_name = "automodel-a1b2c3d4e5f6"
status = client.jobs.get_job_status(name=job_name, workspace="default").data()
print(f"Job: {status.name}")
print(f"Status: {status.status}")
# Check step-level status and training progress
for step in status.steps or []:
    print(f"  Step '{step.name}': {step.status}")
    if step.name == "training":
        for task in step.tasks or []:
            task_details = task.status_details or {}
            current_step = task_details.get("step")
            max_steps = task_details.get("max_steps")
            if current_step and max_steps:
                print(f"    Progress: {current_step}/{max_steps}")

These examples use the default workspace. If your current CLI context uses a different workspace, run nemo config set --workspace default first. To check, run nemo config view.

CLI

nemo jobs get-status automodel-a1b2c3d4e5f6 --workspace default

REST API

This example uses NHX_BASE_URL when it is set and falls back to http://localhost:8080. If your CLI context points at a remote deployment, export its URL first, and add an Authorization: Bearer $(nemo auth token) header for authenticated deployments. Refer to When to use NHX_BASE_URL.

curl -X GET \
  "${NHX_BASE_URL:-http://localhost:8080}/apis/jobs/v2/workspaces/default/jobs/automodel-a1b2c3d4e5f6/status" \
  -H 'Accept: application/json' \
  | jq

Active Job (Training in Progress)

{
  "id": "platform-job-2k8i3i1HqJHHPVB5M6Bk9Z",
  "error_details": null,
  "name": "my-sft-job",
  "status": "active",
  "status_details": {},
  "steps": [
    {
      "id": "platform-job-step-4RJTW5wSy4gHJ539EfTUif",
      "error_details": {},
      "name": "model-and-dataset-download",
      "status": "completed",
      "status_details": {
        "message": "Job completed successfully with exit code 0"
      },
      "tasks": [
        {
          "id": "platform-job-task-MBC2vcDEyv6Jw7RtFs5JDd",
          "error_details": {},
          "error_stack": "",
          "name": "task-5969b01a4b5a4bb181ba530ff51dafa6",
          "status": "completed",
          "status_details": {
            "message": "Job completed successfully with exit code 0",
            "phase": "completed",
            "total_filesets": 2,
            "completed_filesets": 1,
            "current_fileset": "default/sft-dataset",
            "fileset": "default/sft-dataset",
            "total_files": 2,
            "total_size": 2984632,
            "downloaded_files": 2,
            "downloaded_bytes": 2984632,
            "current_file": "validation.jsonl",
            "progress_pct": 100
          }
        }
      ]
    },
    {
      "id": "platform-job-step-9kme8ibxDGES4t9TZvLp4X",
      "error_details": {},
      "name": "training",
      "status": "active",
      "status_details": {
        "message": "Job is running"
      },
      "tasks": [
        {
          "id": "platform-job-task-SawMB6ssVZd7NCiruFVJn8",
          "error_details": {},
          "error_stack": "",
          "name": "task-2249da790a574388ba02fcabd06cd338",
          "status": "active",
          "status_details": {
            "message": "Job is running",
            "phase": "training",
            "backend": "automodel",
            "max_steps": 94,
            "num_epochs": 2,
            "step": 8,
            "epoch": 1,
            "percentage_done": 8,
            "train_loss": 2.8918895721435547,
            "train_lr": 4.9101714686276044e-05,
            "train_grad_norm": 26.0,
            "metrics": {
              "train_loss": [
                { "step": 4, "epoch": 1, "value": 3.2087905406951904 },
                { "step": 8, "epoch": 1, "value": 2.8918895721435547 }
              ],
              "val_loss": [],
              "train_lr": [
                { "step": 4, "epoch": 1, "value": 4.9550857343138022e-05 },
                { "step": 8, "epoch": 1, "value": 4.9101714686276044e-05 }
              ],
              "train_grad_norm": [
                { "step": 4, "epoch": 1, "value": 31.5 },
                { "step": 8, "epoch": 1, "value": 26.0 }
              ]
            }
          }
        }
      ]
    }
  ]
}

Completed Job (All Steps Finished)

{
  "id": "platform-job-2k8i3i1HqJHHPVB5M6Bk9Z",
  "error_details": null,
  "name": "my-sft-job",
  "status": "completed",
  "status_details": {},
  "steps": [
    {
      "id": "platform-job-step-4RJTW5wSy4gHJ539EfTUif",
      "error_details": {},
      "name": "model-and-dataset-download",
      "status": "completed",
      "status_details": {
        "message": "Job completed successfully with exit code 0"
      },
      "tasks": [
        {
          "id": "platform-job-task-MBC2vcDEyv6Jw7RtFs5JDd",
          "error_details": {},
          "error_stack": "",
          "name": "task-5969b01a4b5a4bb181ba530ff51dafa6",
          "status": "completed",
          "status_details": {
            "message": "Job completed successfully with exit code 0",
            "phase": "completed",
            "total_filesets": 2,
            "completed_filesets": 1,
            "current_fileset": "default/sft-dataset",
            "fileset": "default/sft-dataset",
            "total_files": 2,
            "total_size": 2984632,
            "downloaded_files": 2,
            "downloaded_bytes": 2984632,
            "current_file": "validation.jsonl",
            "progress_pct": 100
          }
        }
      ]
    },
    {
      "id": "platform-job-step-9kme8ibxDGES4t9TZvLp4X",
      "error_details": {},
      "name": "training",
      "status": "completed",
      "status_details": {
        "message": "Job completed successfully with exit code 0"
      },
      "tasks": [
        {
          "id": "platform-job-task-SawMB6ssVZd7NCiruFVJn8",
          "error_details": {},
          "error_stack": "",
          "name": "task-2249da790a574388ba02fcabd06cd338",
          "status": "completed",
          "status_details": {
            "message": "Job completed successfully with exit code 0",
            "phase": "processing_checkpoint",
            "backend": "automodel",
            "max_steps": 94,
            "num_epochs": 2,
            "step": 94,
            "epoch": 2,
            "percentage_done": 100,
            "train_loss": 0.3437718152999878,
            "train_lr": 5.000000000000001e-07,
            "train_grad_norm": 20.125,
            "val_loss": 0.5527229905128479,
            "checkpoint_path": "/var/run/scratch/job/training/checkpoints",
            "metrics": {
              "train_loss": [
                { "step": 47, "epoch": 1, "value": 1.1204545497894287 },
                { "step": 94, "epoch": 2, "value": 0.3437718152999878 }
              ],
              "val_loss": [
                { "step": 47, "epoch": 1, "value": 0.9182837605476379 },
                { "step": 94, "epoch": 2, "value": 0.5527229905128479 }
              ],
              "train_lr": [
                { "step": 47, "epoch": 1, "value": 2.5e-05 },
                { "step": 94, "epoch": 2, "value": 5.000000000000001e-07 }
              ],
              "train_grad_norm": [
                { "step": 47, "epoch": 1, "value": 24.75 },
                { "step": 94, "epoch": 2, "value": 20.125 }
              ]
            }
          }
        }
      ]
    },
    {
      "id": "platform-job-step-6gkKgfT5AwyamBWNtQFA9t",
      "error_details": {},
      "name": "model-upload",
      "status": "completed",
      "status_details": {
        "message": "Job completed successfully with exit code 0"
      },
      "tasks": [
        {
          "id": "platform-job-task-J2wt8G8X3vkUHZJCEhzYTP",
          "error_details": {},
          "error_stack": "",
          "name": "task-9647899028aa44c493ce1d65a4ec8f7b",
          "status": "completed",
          "status_details": {
            "message": "Job completed successfully with exit code 0",
            "phase": "completed",
            "total_filesets": 1,
            "completed_filesets": 0,
            "current_fileset": "default/customization-b5b20520fe4f",
            "fileset": "default/customization-b5b20520fe4f",
            "total_files": 7,
            "total_size": 2488928480,
            "uploaded_files": 7,
            "uploaded_bytes": 2488928480,
            "current_file": "model-00001-of-00001.safetensors",
            "progress_pct": 100
          }
        }
      ]
    },
    {
      "id": "platform-job-step-9NmC8mrS3VtthjUSo52EiB",
      "error_details": {},
      "name": "model-entity-creation",
      "status": "completed",
      "status_details": {
        "message": "Job completed successfully with exit code 0"
      },
      "tasks": [
        {
          "id": "platform-job-task-8Mbbz6AnZP5fQLu61aVRb",
          "error_details": {},
          "error_stack": "",
          "name": "task-ab80f766085f4203b72a612f38a2c1b0",
          "status": "completed",
          "status_details": {
            "message": "Job completed successfully with exit code 0"
          }
        }
      ]
    }
  ]
}