Skip to navigation

Async jobs

View as MarkdownOpen in Claude

Some RPCs return before the work finishes. They return a job ID, and you poll a status RPC until the job reaches a final state. This page explains how to tell which RPCs behave this way, how to poll them, and what the job states and errors mean.

Execution modes

The RPC catalog lists an execution mode for every RPC.

ModeWhat the response carriesWhat you do next
SynchronousThe full resultRead the response. No job exists.
Asynchronous, one jobOne job_idPoll the RPC in the Poll with column.
Asynchronous, parent and child jobsA NodeBatchResponse whose job_id is the parent job, plus a result for each node that RMS refusedPoll the parent job. RMS creates one child job for each admitted node.
Synchronous, with background workThe full result of the foreground stepRead the response. The RPC page describes the work that continues.

The response shape alone doesn’t tell you the mode, because synchronous batch RPCs also return a NodeBatchResponse. Rely on the catalog, or read the Execution row on the RPC page.

Poll a job

RMS provides four status RPCs. Use the one that the catalog names for your RPC.

RPCUse it forIf the job doesn’t exist
GetJobStatusAny job that RMS tracks. Set include_child_job_states to also return each child.gRPC error NOT_FOUND
GetFirmwareJobStatusFirmware update jobs and their parent batch jobsRETURN_CODE_FAILURE in the response
GetSwitchSystemImageJobStatusSwitch system image update jobsRETURN_CODE_FAILURE in the response
GetConfigureSwitchCertificateJobStatusSwitch certificate configuration jobsRETURN_CODE_FAILURE in the response

GetJobStatus returns a list of JobStatus messages. When you query a parent job, the parent comes first in the list. The other status RPCs return the single job as an aggregate of the parent and its children, and carry their own state and error enums.

Job states

A job moves through the states of JobExecutionState:

  1. JOB_EXECUTION_STATE_QUEUED: RMS created the job and it hasn’t started.
  2. JOB_EXECUTION_STATE_RUNNING: The job is executing.
  3. JOB_EXECUTION_STATE_COMPLETED or JOB_EXECUTION_STATE_FAILED: The job is final. RMS never changes a final state.

Stop polling when you see a final state. Read state_description for a human-readable progress message while the job runs.

Poll in a loop

This Bash example polls a job with GetJobStatus every five seconds until it finishes. It needs jq, and it assumes you defined the variables in Getting started. grpcurl prints protobuf JSON with lower camel case field names, such as jobStates, so the jq expression reads those names and accepts the snake case names as a fallback.

set -o pipefail
job_id="<job_id>"
while true; do
state=$(grpcurl $RMS_GRPCURL_FLAGS -d "{\"job_id\": \"$job_id\"}" \
"$RMS_ADDR" rack_manager.RackManager/GetJobStatus \
| jq -r '(.jobStates // .job_states)[0] | (.executionState // .execution_state)') \
|| { echo "Polling failed" >&2; break; }
echo "$state"
case "$state" in
JOB_EXECUTION_STATE_COMPLETED|JOB_EXECUTION_STATE_FAILED) break ;;
esac
sleep 5
done

After the loop, call the status RPC once more and read result_json and, for a failed job, error_code and error_message.

Read the result

For a job that reaches JOB_EXECUTION_STATE_COMPLETED, result_json holds a JSON document with detailed results. Its schema depends on the workflow that created the job, so the RPC page describes it where it’s verified.

For a failed job, error_message is always set, and error_code is a JobError value. Common values include:

JobError valueMeaning
JOB_ERROR_INVALID_ARGUMENTThe arguments that started the job aren’t valid.
JOB_ERROR_CLIENT_ERRORRMS couldn’t communicate with the device over HTTP or Redfish.
JOB_ERROR_TIMEOUTAn operation didn’t complete in time.
JOB_ERROR_UNAUTHENTICATEDThe credentials couldn’t authenticate.
JOB_ERROR_TARGET_NOT_FOUNDThe target component wasn’t found on the node.
JOB_ERROR_FILE_NOT_FOUNDThe firmware file doesn’t exist.
JOB_ERROR_UPDATE_IN_PROGRESSThe node already has an active job.

The firmware status RPC reports the equivalent FirmwareUpdateError enum in its error_code field.

Parent and child jobs

A batch RPC creates a parent job and then one child job for each node that it admits. The job_id in the response identifies the parent. RMS derives the parent state from its children:

  • The parent completes when all children complete without a failure.
  • The parent fails when all children are final and at least one failed. Its result_json lists the failed children.
  • Otherwise the parent keeps running, and its state_description shows progress.

Poll the parent. To see each node, call GetJobStatus with include_child_job_states set to true. Each child has a parent_job_id, and the parent lists its children in child_job_ids.

Node exclusion

RMS allows one active job for each node at a time. It refuses a new job for a node while another job for that node is still running or queued, and it reports JOB_ERROR_UPDATE_IN_PROGRESS. RMS doesn’t queue the request, so retry after the active job ends.

In a batch RPC, a refused node doesn’t fail the request. The node appears in node_results with the reason, and RMS still starts every node that it admitted. RMS also treats a node that appears twice in one request as refused the second time. For the full rules, refer to the Operations overview.

Job retention

RMS tracks jobs in memory. It keeps a final job for the interval set by terminal_job_ttl_seconds, which is 24 hours by default, and then removes it. Poll to completion and save any result that you need before that time. A job that RMS no longer tracks looks the same as a job ID that never existed. For details, refer to the Operations overview.