DocumentationFine-tune ModelsManage Customization Jobs

Get Job Status

View as Markdown

Get detailed execution status for a customization job, including step-by-step progress and real-time training metrics.

This endpoint provides granular execution details including:

  • Step-level status: model-and-dataset-download → training → model-upload → model-entity-creation
  • Training progress: step, epoch, max_steps, num_epochs, percentage_done
  • Latest metric values: one field per metric, named <phase>_<metric> — train_loss, train_lr, train_grad_norm, val_loss, and whatever else your backend reports
  • Metric history: metrics, holding each metric as a series of {step, epoch, value} points
  • Progress tracking: downloaded_files, uploaded_bytes, progress_pct

Which metrics appear depends on the backend and the algorithm. See Checking Your Customization Job Metrics for how the names are formed.

To list jobs or get job definitions (model entity, hyperparameters, spec), use List Active Jobs instead.

Prerequisites

Before you can get the status of a customization job, make sure that you have:

  • Obtained the base URL of your NeMo Platform.
  • Set the NMP_BASE_URL environment variable to your NeMo Platform endpoint
$export NMP_BASE_URL="https://your-nmp-base-url"

To Get the Status of a Customization Job

A submitted customization job runs on the platform’s Jobs service, so you poll its status through that service using the job name returned at submission (for example, automodel-a1b2c3d4e5f6). This works the same way for both the automodel and unsloth backends.

Use the SDK to get detailed job status:

1import os
2from nemo_platform import NeMoPlatform
3
4# Initialize the client
5client = NeMoPlatform(
6    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
7    workspace="default",
8)
9
10# Get job status (use the job name returned at submission)
11job_name = "automodel-a1b2c3d4e5f6"
12status = client.jobs.get_status(name=job_name, workspace="default")
13
14print(f"Job: {status.name}")
15print(f"Status: {status.status}")
16
17# Check step-level status and training progress
18for step in status.steps or []:
19    print(f"  Step '{step.name}': {step.status}")
20    if step.name == "training":
21        for task in step.tasks or []:
22            task_details = task.status_details or {}
23            current_step = task_details.get("step")
24            max_steps = task_details.get("max_steps")
25            if current_step and max_steps:
26                print(f"    Progress: {current_step}/{max_steps}")

CLI

$nemo jobs get-status automodel-a1b2c3d4e5f6 --workspace default

REST API

$curl -X GET \
>  "${NMP_BASE_URL}/apis/jobs/v2/workspaces/default/jobs/automodel-a1b2c3d4e5f6/status" \
>  -H 'Accept: application/json' \
>  | jq

Active Job (Training in Progress)

1{
2  "id": "platform-job-2k8i3i1HqJHHPVB5M6Bk9Z",
3  "error_details": null,
4  "name": "my-sft-job",
5  "status": "active",
6  "status_details": {},
7  "steps": [
8    {
9      "id": "platform-job-step-4RJTW5wSy4gHJ539EfTUif",
10      "error_details": {},
11      "name": "model-and-dataset-download",
12      "status": "completed",
13      "status_details": {
14        "message": "Job completed successfully with exit code 0"
15      },
16      "tasks": [
17        {
18          "id": "platform-job-task-MBC2vcDEyv6Jw7RtFs5JDd",
19          "error_details": {},
20          "error_stack": "",
21          "name": "task-5969b01a4b5a4bb181ba530ff51dafa6",
22          "status": "completed",
23          "status_details": {
24            "message": "Job completed successfully with exit code 0",
25            "phase": "completed",
26            "total_filesets": 2,
27            "completed_filesets": 1,
28            "current_fileset": "default/sft-dataset",
29            "fileset": "default/sft-dataset",
30            "total_files": 2,
31            "total_size": 2984632,
32            "downloaded_files": 2,
33            "downloaded_bytes": 2984632,
34            "current_file": "validation.jsonl",
35            "progress_pct": 100
36          }
37        }
38      ]
39    },
40    {
41      "id": "platform-job-step-9kme8ibxDGES4t9TZvLp4X",
42      "error_details": {},
43      "name": "training",
44      "status": "active",
45      "status_details": {
46        "message": "Job is running"
47      },
48      "tasks": [
49        {
50          "id": "platform-job-task-SawMB6ssVZd7NCiruFVJn8",
51          "error_details": {},
52          "error_stack": "",
53          "name": "task-2249da790a574388ba02fcabd06cd338",
54          "status": "active",
55          "status_details": {
56            "message": "Job is running",
57            "phase": "training",
58            "backend": "automodel",
59            "max_steps": 94,
60            "num_epochs": 2,
61            "step": 8,
62            "epoch": 1,
63            "percentage_done": 8,
64            "train_loss": 2.8918895721435547,
65            "train_lr": 4.9101714686276044e-05,
66            "train_grad_norm": 26.0,
67            "metrics": {
68              "train_loss": [
69                { "step": 4, "epoch": 1, "value": 3.2087905406951904 },
70                { "step": 8, "epoch": 1, "value": 2.8918895721435547 }
71              ],
72              "val_loss": [],
73              "train_lr": [
74                { "step": 4, "epoch": 1, "value": 4.9550857343138022e-05 },
75                { "step": 8, "epoch": 1, "value": 4.9101714686276044e-05 }
76              ],
77              "train_grad_norm": [
78                { "step": 4, "epoch": 1, "value": 31.5 },
79                { "step": 8, "epoch": 1, "value": 26.0 }
80              ]
81            }
82          }
83        }
84      ]
85    }
86  ]
87}

Completed Job (All Steps Finished)

1{
2  "id": "platform-job-2k8i3i1HqJHHPVB5M6Bk9Z",
3  "error_details": null,
4  "name": "my-sft-job",
5  "status": "completed",
6  "status_details": {},
7  "steps": [
8    {
9      "id": "platform-job-step-4RJTW5wSy4gHJ539EfTUif",
10      "error_details": {},
11      "name": "model-and-dataset-download",
12      "status": "completed",
13      "status_details": {
14        "message": "Job completed successfully with exit code 0"
15      },
16      "tasks": [
17        {
18          "id": "platform-job-task-MBC2vcDEyv6Jw7RtFs5JDd",
19          "error_details": {},
20          "error_stack": "",
21          "name": "task-5969b01a4b5a4bb181ba530ff51dafa6",
22          "status": "completed",
23          "status_details": {
24            "message": "Job completed successfully with exit code 0",
25            "phase": "completed",
26            "total_filesets": 2,
27            "completed_filesets": 1,
28            "current_fileset": "default/sft-dataset",
29            "fileset": "default/sft-dataset",
30            "total_files": 2,
31            "total_size": 2984632,
32            "downloaded_files": 2,
33            "downloaded_bytes": 2984632,
34            "current_file": "validation.jsonl",
35            "progress_pct": 100
36          }
37        }
38      ]
39    },
40    {
41      "id": "platform-job-step-9kme8ibxDGES4t9TZvLp4X",
42      "error_details": {},
43      "name": "training",
44      "status": "completed",
45      "status_details": {
46        "message": "Job completed successfully with exit code 0"
47      },
48      "tasks": [
49        {
50          "id": "platform-job-task-SawMB6ssVZd7NCiruFVJn8",
51          "error_details": {},
52          "error_stack": "",
53          "name": "task-2249da790a574388ba02fcabd06cd338",
54          "status": "completed",
55          "status_details": {
56            "message": "Job completed successfully with exit code 0",
57            "phase": "processing_checkpoint",
58            "backend": "automodel",
59            "max_steps": 94,
60            "num_epochs": 2,
61            "step": 94,
62            "epoch": 2,
63            "percentage_done": 100,
64            "train_loss": 0.3437718152999878,
65            "train_lr": 5.000000000000001e-07,
66            "train_grad_norm": 20.125,
67            "val_loss": 0.5527229905128479,
68            "checkpoint_path": "/var/run/scratch/job/training/checkpoints",
69            "metrics": {
70              "train_loss": [
71                { "step": 47, "epoch": 1, "value": 1.1204545497894287 },
72                { "step": 94, "epoch": 2, "value": 0.3437718152999878 }
73              ],
74              "val_loss": [
75                { "step": 47, "epoch": 1, "value": 0.9182837605476379 },
76                { "step": 94, "epoch": 2, "value": 0.5527229905128479 }
77              ],
78              "train_lr": [
79                { "step": 47, "epoch": 1, "value": 2.5e-05 },
80                { "step": 94, "epoch": 2, "value": 5.000000000000001e-07 }
81              ],
82              "train_grad_norm": [
83                { "step": 47, "epoch": 1, "value": 24.75 },
84                { "step": 94, "epoch": 2, "value": 20.125 }
85              ]
86            }
87          }
88        }
89      ]
90    },
91    {
92      "id": "platform-job-step-6gkKgfT5AwyamBWNtQFA9t",
93      "error_details": {},
94      "name": "model-upload",
95      "status": "completed",
96      "status_details": {
97        "message": "Job completed successfully with exit code 0"
98      },
99      "tasks": [
100        {
101          "id": "platform-job-task-J2wt8G8X3vkUHZJCEhzYTP",
102          "error_details": {},
103          "error_stack": "",
104          "name": "task-9647899028aa44c493ce1d65a4ec8f7b",
105          "status": "completed",
106          "status_details": {
107            "message": "Job completed successfully with exit code 0",
108            "phase": "completed",
109            "total_filesets": 1,
110            "completed_filesets": 0,
111            "current_fileset": "default/customization-b5b20520fe4f",
112            "fileset": "default/customization-b5b20520fe4f",
113            "total_files": 7,
114            "total_size": 2488928480,
115            "uploaded_files": 7,
116            "uploaded_bytes": 2488928480,
117            "current_file": "model-00001-of-00001.safetensors",
118            "progress_pct": 100
119          }
120        }
121      ]
122    },
123    {
124      "id": "platform-job-step-9NmC8mrS3VtthjUSo52EiB",
125      "error_details": {},
126      "name": "model-entity-creation",
127      "status": "completed",
128      "status_details": {
129        "message": "Job completed successfully with exit code 0"
130      },
131      "tasks": [
132        {
133          "id": "platform-job-task-8Mbbz6AnZP5fQLu61aVRb",
134          "error_details": {},
135          "error_stack": "",
136          "name": "task-ab80f766085f4203b72a612f38a2c1b0",
137          "status": "completed",
138          "status_details": {
139            "message": "Job completed successfully with exit code 0"
140          }
141        }
142      ]
143    }
144  ]
145}