Flow Operation Rules | NVIDIA Infra Controller

Operation rules define how Flow executes power control, firmware, ingestion, and bring-up operations. Each rule specifies a sequence of steps that determine component ordering, parallelism, verification, and retry behavior.

Concepts

Rules and Operations

Each rule is bound to a single operation (e.g., power_on, power_off). When an operation is triggered, Flow resolves the applicable rule for the target rack and executes it.

Steps and Stages

A rule contains steps. Each step targets one component type and belongs to a numbered stage. Execution proceeds stage by stage in ascending order. Within a stage, all steps run in parallel.

Stage 1: [powershelf step]
Stage 2: [nvswitch step]
Stage 3: [compute step]        ← stages are sequential
         [nvswitch step]      ← steps within a stage are parallel

Action Sequences

Each step defines three optional action lists:

pre_operation — runs before the main operation (e.g., Sleep to settle)
main_operation — the primary work (PowerControl, FirmwareControl, or a verification action when used as the entire step’s purpose)
post_operation — runs after the main operation (e.g., verify status)

All three phases execute inside a single child workflow per component. The step-level timeout applies to the entire sequence (pre + main + post).

Rule Schema

Top-level fields

Rules are submitted as JSON via the --rule-file flag or embedded in a YAML batch file.

1 {
2   "version": "v1",
3   "steps": [ ... ]
4 }

Field	Type	Required	Description
`version`	string	no	Schema version. Defaults to `"v1"`
`steps`	array	yes	One or more step definitions

Step fields

Field	Type	Required	Description
`component_type`	string	yes	Component this step targets: `"compute"`, `"nvswitch"`, `"powershelf"`
`stage`	integer	yes	Execution order. Steps with the same stage run in parallel. Must be ≥ 1
`max_parallel`	integer	yes	Max concurrent components. `0` = unlimited, `1` = sequential
`timeout`	duration	no	Timeout for the entire child workflow (pre + main + post). E.g. `"10m"`
`retry`	object	no	Retry policy for the child workflow
`pre_operation`	array	no	Actions to run before `main_operation`
`main_operation`	object	yes	The primary action
`post_operation`	array	no	Actions to run after `main_operation`

Retry policy fields

Field	Type	Required	Description
`max_attempts`	integer	yes	Total attempts including the first. Must be ≥ 1
`initial_interval`	duration	yes	Wait before first retry. E.g. `"5s"`
`backoff_coefficient`	float	yes	Multiplier for each subsequent interval. Must be ≥ 1.0
`max_interval`	duration	no	Cap on retry interval. E.g. `"1m"`

Duration format

All duration fields accept Go duration strings: "5s", "30s", "2m", "1m30s", "10m", "1h".

Actions Reference

PowerControl

Executes a power operation (on/off/restart) for the component.

When used from the power workflow, the operation is inherited from the task context — no parameters required. When used cross-workflow (e.g., firmware power recycle, bring-up), specify operation explicitly:

1 # Within a power workflow (inherits from task):
2 main_operation:
3   name: PowerControl
4 
5 # Cross-workflow usage (explicit operation):
6 main_operation:
7   name: PowerControl
8   parameters:
9     operation: "force_power_off"

Field	Required	Description
`timeout`	no	Overrides step timeout for this action only

Parameter	Required	Description
`operation`	no*	Power operation code. Required when used outside a power workflow. Valid values: `power_on`, `force_power_on`, `power_off`, `force_power_off`, `restart`, `force_restart`, `warm_reset`, `cold_reset`

FirmwareControl

Starts a firmware update and polls for completion (async start + poll pattern). Calls FirmwareControl to initiate, then repeatedly calls GetFirmwareStatus until all components complete or the poll timeout expires.

1 main_operation:
2   name: FirmwareControl
3   parameters:
4     poll_interval: 2m    # time between status polls (default: 2m)
5     poll_timeout: 30m    # max wait for completion (default: 30m)

Field	Required	Description
`timeout`	no	Overrides step timeout for this action only

Parameter	Required	Description
`poll_interval`	no	Time between status polls (default `2m`)
`poll_timeout`	no	Max time to wait for completion (default `30m`)

BringUpControl

Opens the power-on gate for target components. No parameters required.

1 main_operation:
2   name: BringUpControl

WaitBringUp

Polls GetBringUpStatus until all components reach the MachineCreated state (ingestion complete). Used as a post-operation after BringUpControl.

1 post_operation:
2   - name: WaitBringUp
3     timeout: 15m
4     poll_interval: 30s

Field	Required	Description
`timeout`	yes	Max time to wait for bring-up
`poll_interval`	yes	Time between polls

VerifyPowerStatus

Polls the component until its power status matches expected_status. Typically used in post_operation to confirm the result of PowerControl.

1 {
2   "name": "VerifyPowerStatus",
3   "timeout": "30s",
4   "poll_interval": "5s",
5   "parameters": {
6     "expected_status": "on"
7   }
8 }

Field	Required	Description
`timeout`	yes	Maximum time to wait for status to match
`poll_interval`	yes	How often to check status
`expected_status` (param)	yes	`"on"` or `"off"`

When used as main_operation, the step performs only verification (no power command is sent). This is the pattern for forceful operation final-verification stages.

VerifyReachability

Polls until all components of the specified types in the rack become reachable over the network. Used after powering on a powershelf to confirm downstream components have booted, or before bring-up to wait for all PMCs.

By default, a component type is considered reachable when the GetPowerStatus API call succeeds. With require_all: true, every individual component within the type must respond (i.e., the returned status map must contain all target component IDs).

1 # Basic reachability (API call succeeds):
2 - name: VerifyReachability
3   timeout: 3m
4   poll_interval: 10s
5   parameters:
6     component_types: ["compute", "nvswitch"]
7 
8 # Strict mode (every individual component must respond):
9 - name: VerifyReachability
10   timeout: 10m
11   poll_interval: 30s
12   parameters:
13     component_types: ["powershelf"]
14     require_all: true

Field	Required	Description
`timeout`	yes	Maximum time to wait
`poll_interval`	yes	How often to probe

Parameter	Required	Description
`component_types`	yes	Array of component type strings to check
`require_all`	no	When `true`, every individual component must respond (default `false`)

Sleep

Pauses execution for a fixed duration. Implemented as a durable workflow timer (survives worker restarts). Useful for hardware settle time.

1 {
2   "name": "Sleep",
3   "parameters": {
4     "duration": "30s"
5   }
6 }

Field	Required	Description
`duration` (param)	yes	How long to sleep. E.g. `"30s"`, `"2m"`

GetPowerStatus

Queries the current power status of components and returns a status map.

1 {
2   "name": "GetPowerStatus",
3   "timeout": "30s"
4 }

Field	Required	Description
`timeout`	yes	Maximum time for the query

InjectExpectation

Registers expected component configurations with their respective component manager services. Used as the initial ingestion stage in full bring-up rules, and as the sole action in ingestion-only rules.

The IngestRack gRPC API is a convenience wrapper that triggers a BringUp workflow with an ingestion-only rule (all component types run InjectExpectation in parallel).

1 main_operation:
2   name: InjectExpectation

No parameters or timeout overrides required — the action uses the step-level timeout and the component Target to call the InjectExpectation activity.

Examples

Graceful power on

Powers components in dependency order (powershelf → nvswitch → compute) and verifies status at each stage before proceeding.

1 {
2   "version": "v1",
3   "steps": [
4     {
5       "component_type": "powershelf",
6       "stage": 1,
7       "max_parallel": 1,
8       "timeout": "10m",
9       "retry": {
10         "max_attempts": 3,
11         "initial_interval": "5s",
12         "backoff_coefficient": 2.0,
13         "max_interval": "1m"
14       },
15       "main_operation": { "name": "PowerControl" },
16       "post_operation": [
17         {
18           "name": "VerifyPowerStatus",
19           "timeout": "30s",
20           "poll_interval": "5s",
21           "parameters": { "expected_status": "on" }
22         },
23         {
24           "name": "VerifyReachability",
25           "timeout": "3m",
26           "poll_interval": "10s",
27           "parameters": { "component_types": ["compute", "nvswitch"] }
28         },
29         {
30           "name": "Sleep",
31           "parameters": { "duration": "30s" }
32         }
33       ]
34     },
35     {
36       "component_type": "nvswitch",
37       "stage": 2,
38       "max_parallel": 4,
39       "timeout": "15m",
40       "retry": {
41         "max_attempts": 3,
42         "initial_interval": "5s",
43         "backoff_coefficient": 2.0,
44         "max_interval": "1m"
45       },
46       "main_operation": { "name": "PowerControl" },
47       "post_operation": [
48         {
49           "name": "VerifyPowerStatus",
50           "timeout": "30s",
51           "poll_interval": "5s",
52           "parameters": { "expected_status": "on" }
53         }
54       ]
55     },
56     {
57       "component_type": "compute",
58       "stage": 3,
59       "max_parallel": 8,
60       "timeout": "20m",
61       "retry": {
62         "max_attempts": 3,
63         "initial_interval": "5s",
64         "backoff_coefficient": 2.0,
65         "max_interval": "1m"
66       },
67       "main_operation": { "name": "PowerControl" },
68       "post_operation": [
69         {
70           "name": "VerifyPowerStatus",
71           "timeout": "30s",
72           "poll_interval": "5s",
73           "parameters": { "expected_status": "on" }
74         }
75       ]
76     }
77   ]
78 }

Graceful power off

Reverse dependency order (compute → nvswitch → powershelf). A Sleep in the powershelf pre_operation allows downstream components to finish shutting down before cutting power.

1 {
2   "version": "v1",
3   "steps": [
4     {
5       "component_type": "compute",
6       "stage": 1,
7       "max_parallel": 8,
8       "timeout": "20m",
9       "retry": {
10         "max_attempts": 3,
11         "initial_interval": "5s",
12         "backoff_coefficient": 2.0
13       },
14       "main_operation": { "name": "PowerControl" },
15       "post_operation": [
16         {
17           "name": "VerifyPowerStatus",
18           "timeout": "30s",
19           "poll_interval": "5s",
20           "parameters": { "expected_status": "off" }
21         }
22       ]
23     },
24     {
25       "component_type": "nvswitch",
26       "stage": 2,
27       "max_parallel": 4,
28       "timeout": "15m",
29       "retry": {
30         "max_attempts": 3,
31         "initial_interval": "5s",
32         "backoff_coefficient": 2.0
33       },
34       "main_operation": { "name": "PowerControl" },
35       "post_operation": [
36         {
37           "name": "VerifyPowerStatus",
38           "timeout": "30s",
39           "poll_interval": "5s",
40           "parameters": { "expected_status": "off" }
41         }
42       ]
43     },
44     {
45       "component_type": "powershelf",
46       "stage": 3,
47       "max_parallel": 1,
48       "timeout": "10m",
49       "retry": {
50         "max_attempts": 3,
51         "initial_interval": "5s",
52         "backoff_coefficient": 2.0
53       },
54       "pre_operation": [
55         {
56           "name": "Sleep",
57           "parameters": { "duration": "30s" }
58         }
59       ],
60       "main_operation": { "name": "PowerControl" },
61       "post_operation": [
62         {
63           "name": "VerifyPowerStatus",
64           "timeout": "30s",
65           "poll_interval": "5s",
66           "parameters": { "expected_status": "off" }
67         }
68       ]
69     }
70   ]
71 }

Forceful power on

Skips per-stage verification for maximum speed. All power commands are issued first; a dedicated final stage (4) verifies all component types simultaneously.

1 {
2   "version": "v1",
3   "steps": [
4     {
5       "component_type": "powershelf",
6       "stage": 1,
7       "max_parallel": 0,
8       "timeout": "10m",
9       "retry": {
10         "max_attempts": 3,
11         "initial_interval": "1s",
12         "backoff_coefficient": 2.0
13       },
14       "main_operation": { "name": "PowerControl" },
15       "post_operation": [
16         { "name": "Sleep", "parameters": { "duration": "5s" } }
17       ]
18     },
19     {
20       "component_type": "nvswitch",
21       "stage": 2,
22       "max_parallel": 0,
23       "timeout": "15m",
24       "retry": {
25         "max_attempts": 3,
26         "initial_interval": "1s",
27         "backoff_coefficient": 2.0
28       },
29       "main_operation": { "name": "PowerControl" },
30       "post_operation": [
31         { "name": "Sleep", "parameters": { "duration": "5s" } }
32       ]
33     },
34     {
35       "component_type": "compute",
36       "stage": 3,
37       "max_parallel": 0,
38       "timeout": "20m",
39       "retry": {
40         "max_attempts": 3,
41         "initial_interval": "1s",
42         "backoff_coefficient": 2.0
43       },
44       "main_operation": { "name": "PowerControl" },
45       "post_operation": [
46         { "name": "Sleep", "parameters": { "duration": "5s" } }
47       ]
48     },
49     {
50       "component_type": "powershelf",
51       "stage": 4,
52       "max_parallel": 0,
53       "timeout": "2m",
54       "retry": {
55         "max_attempts": 2,
56         "initial_interval": "5s",
57         "backoff_coefficient": 1.5
58       },
59       "main_operation": {
60         "name": "VerifyPowerStatus",
61         "timeout": "1m",
62         "poll_interval": "5s",
63         "parameters": { "expected_status": "on" }
64       }
65     },
66     {
67       "component_type": "nvswitch",
68       "stage": 4,
69       "max_parallel": 0,
70       "timeout": "2m",
71       "retry": {
72         "max_attempts": 2,
73         "initial_interval": "5s",
74         "backoff_coefficient": 1.5
75       },
76       "main_operation": {
77         "name": "VerifyPowerStatus",
78         "timeout": "1m",
79         "poll_interval": "5s",
80         "parameters": { "expected_status": "on" }
81       }
82     },
83     {
84       "component_type": "compute",
85       "stage": 4,
86       "max_parallel": 0,
87       "timeout": "2m",
88       "retry": {
89         "max_attempts": 2,
90         "initial_interval": "5s",
91         "backoff_coefficient": 1.5
92       },
93       "main_operation": {
94         "name": "VerifyPowerStatus",
95         "timeout": "1m",
96         "poll_interval": "5s",
97         "parameters": { "expected_status": "on" }
98       }
99     }
100   ]
101 }

Ingestion only

Registers components with their respective component manager services without performing power or firmware operations. All component types are ingested in parallel in a single stage. The IngestRack gRPC API triggers this rule automatically.

1 {
2   "version": "v1",
3   "steps": [
4     {
5       "component_type": "powershelf",
6       "stage": 1,
7       "max_parallel": 0,
8       "timeout": "10m",
9       "main_operation": { "name": "InjectExpectation" }
10     },
11     {
12       "component_type": "compute",
13       "stage": 1,
14       "max_parallel": 0,
15       "timeout": "10m",
16       "main_operation": { "name": "InjectExpectation" }
17     },
18     {
19       "component_type": "nvswitch",
20       "stage": 1,
21       "max_parallel": 0,
22       "timeout": "10m",
23       "main_operation": { "name": "InjectExpectation" }
24     }
25   ]
26 }

Full bring-up (with ingestion)

A complete bring-up sequence that first ingests components, then powers on powershelves, and finally brings up compute nodes. The BringUpRack gRPC API uses this as its default rule.

1 {
2   "version": "v1",
3   "steps": [
4     {
5       "component_type": "powershelf",
6       "stage": 1,
7       "max_parallel": 0,
8       "timeout": "10m",
9       "main_operation": { "name": "InjectExpectation" }
10     },
11     {
12       "component_type": "compute",
13       "stage": 1,
14       "max_parallel": 0,
15       "timeout": "10m",
16       "main_operation": { "name": "InjectExpectation" }
17     },
18     {
19       "component_type": "nvswitch",
20       "stage": 1,
21       "max_parallel": 0,
22       "timeout": "10m",
23       "main_operation": { "name": "InjectExpectation" }
24     },
25     {
26       "component_type": "powershelf",
27       "stage": 2,
28       "max_parallel": 0,
29       "timeout": "20m",
30       "pre_operation": [
31         {
32           "name": "VerifyReachability",
33           "timeout": "10m",
34           "poll_interval": "30s",
35           "parameters": { "component_types": ["powershelf"], "require_all": true }
36         }
37       ],
38       "main_operation": {
39         "name": "PowerControl",
40         "parameters": { "operation": "power_on" }
41       },
42       "post_operation": [
43         { "name": "VerifyPowerStatus", "timeout": "5m", "poll_interval": "10s", "parameters": { "expected_status": "on" } }
44       ]
45     },
46     {
47       "component_type": "compute",
48       "stage": 3,
49       "max_parallel": 0,
50       "timeout": "30m",
51       "pre_operation": [
52         { "name": "BringUpControl" },
53         { "name": "WaitBringUp", "timeout": "15m", "poll_interval": "30s" },
54         { "name": "VerifyPowerStatus", "timeout": "10m", "poll_interval": "15s", "parameters": { "expected_status": "on" } }
55       ],
56       "main_operation": {
57         "name": "PowerControl",
58         "parameters": { "operation": "force_restart" }
59       },
60       "post_operation": [
61         { "name": "Sleep", "parameters": { "duration": "30s" } },
62         { "name": "VerifyPowerStatus", "timeout": "10m", "poll_interval": "15s", "parameters": { "expected_status": "on" } }
63       ]
64     }
65   ]
66 }

Temporal Execution Model

This section describes what actually happens inside the Temporal executor when a rule is applied to a task.

1. Rule resolution

Before any workflow starts, the task manager resolves the applicable rule for the operation and rack using this priority order:

1. Rack-specific association  (rack_rule_associations table for this rack)
2. Global default rule        (is_default = true for this operation)
3. Hardcoded fallback         (built into the binary)

The resolved RuleDefinition is embedded in the ExecutionInfo passed to the parent workflow. The workflow never queries the database.

2. Parent workflow — sequential stages

PowerControl (or FirmwareControl) is the parent Temporal workflow. It does pure orchestration:

for each stage in ascending stage number:
    executeGenericStageParallel(stage.steps)   ← waits before advancing

If any stage fails, the workflow stops and the task is marked failed. Stages do not roll back.

The parent workflow has no retry policy of its own (MaxAttempts = 1). Retries are configured at the child workflow (step) level.

The parent workflow’s execution timeout is auto-calculated from the sum of each stage’s maximum step timeout, plus a 10 % overhead buffer. You do not set it directly.

3. Stage execution — parallel child workflows

Within a stage, each step spawns one GenericComponentStepWorkflow child workflow. All child workflows in the stage are launched simultaneously and the parent waits for all of them to finish before advancing.

Stage N:
  ┌─────────────────────────┐  ┌─────────────────────────┐
  │ child: powershelf step  │  │ child: nvswitch step   │  ← in parallel
  └─────────────────────────┘  └─────────────────────────┘
           both must complete before Stage N+1 begins

Steps whose component type is not present in the rack are silently skipped.

The child workflow’s WorkflowExecutionTimeout is set to the step’s timeout field (defaulting to 30 minutes if unset). This timeout covers the entire pre + main + post sequence.

4. Child workflow — action sequence

GenericComponentStepWorkflow runs the three action phases in order:

pre_operation actions  (sequential)
       ↓
main_operation action
       ↓
post_operation actions (sequential)

The step’s retry policy applies to the entire child workflow. If any action fails and retries are exhausted, the child workflow fails, which fails the stage, which fails the parent workflow.

Activity options (timeout, retry) for individual Temporal activities are derived from the step configuration via buildActivityOptions. If the step has no retry block, a default of 3 attempts with 1 s initial interval and 2× backoff is used.

5. Action dispatch

Each action name is looked up in a registry that maps it to an executor function:

Action	Executor	Temporal primitive
`Sleep`	`executeSleepAction`	`workflow.Sleep()` — durable timer
`PowerControl`	`executePowerControlAction`	`workflow.ExecuteActivity("PowerControl")`
`FirmwareControl`	`executeFirmwareControlAction`	`FirmwareControl` + poll `GetFirmwareStatus`
`GetPowerStatus`	`executeGetPowerStatusAction`	`workflow.ExecuteActivity("GetPowerStatus")`
`VerifyPowerStatus`	`executeVerifyPowerStatusAction`	polling loop (see below)
`VerifyReachability`	`executeVerifyReachabilityAction`	polling loop (see below)
`BringUpControl`	`executeBringUpControlAction`	`workflow.ExecuteActivity("BringUpControl")`
`WaitBringUp`	`executeWaitBringUpAction`	polling loop on `GetBringUpStatus`

Sleep is a workflow timer, not an activity. It survives worker restarts and does not consume an activity slot.

6. Polling actions

VerifyPowerStatus and VerifyReachability are implemented as workflow-level polling loops — not as single activities. Each iteration calls GetPowerStatus (a Temporal activity) and then workflow.Sleep for poll_interval before trying again. The loop exits when the condition is satisfied or timeout is exceeded.

loop:
    call GetPowerStatus activity
    if condition met → return success
    if elapsed > timeout → return error
    workflow.Sleep(poll_interval)

Because the sleep inside the loop is a durable workflow timer, the worker can restart mid-poll without losing progress.

VerifyReachability uses the same pattern but checks multiple component types, remembering which types have already become reachable across iterations.

7. Cross-component verification

VerifyReachability needs to check component types other than the one the current step targets (e.g., a powershelf step checking whether compute and nvswitch are reachable). The parent workflow passes the full map[ComponentType]Target to every child workflow, which in turn passes it to every action executor. VerifyReachability uses this map to probe the correct targets regardless of the step’s own component type.

Execution flow diagram

gRPC request
    │
    ▼
Task Manager ── resolves rule ──► RuleDefinition embedded in ExecutionInfo
    │
    ▼
PowerControl workflow (parent)
    │
    ├─ Stage 1 ──────────────────────────────────────────┐
    │   ├─ child: GenericComponentStepWorkflow (powershelf)│
    │   │     pre_operation → main_operation → post_op    │ parallel
    │   └─ child: GenericComponentStepWorkflow (nvswitch) │
    │         pre_operation → main_operation → post_op    │
    │                                                    ◄┘ (wait all)
    │
    ├─ Stage 2 ──────────────────────────────────────────┐
    │   └─ child: GenericComponentStepWorkflow (compute)  │
    │         pre_operation → main_operation → post_op   ◄┘
    │
    ▼
UpdateTaskStatus (completed / failed)

CLI Usage

Create a single rule

$ flow rule create \
>   --name "Graceful Power On" \
>   --description "Power-on with verification" \
>   --operation-type power_control \
>   --operation power_on \
>   --rule-file my-rule.json \
>   --is-default

Load rules from a YAML batch file

$ # Create (skip rules that already exist by name)
$ flow rule create --from-yaml examples/operation-rules-example.yaml
$ 
$ # Create or overwrite existing rules
$ flow rule create --from-yaml examples/operation-rules-example.yaml --overwrite
$ 
$ # Validate without writing to the database
$ flow rule create --from-yaml examples/operation-rules-example.yaml --dry-run

Manage rules

$ # List all rules
$ flow rule list
$ 
$ # Set a rule as the default for its operation
$ flow rule set-default --id <rule-id>
$ 
$ # Associate a rule with a specific rack
$ flow rule associate --rack-id R1 --rule-id <rule-id>

YAML batch file format

1 version: v1
2 
3 rules:
4   - name: "My Power On Rule"
5     description: "..."
6     operation_type: power_control
7     operation: power_on
8     steps:
9       - component_type: powershelf
10         stage: 1
11         max_parallel: 1
12         timeout: 10m
13         main_operation:
14           name: PowerControl
15         post_operation:
16           - name: VerifyPowerStatus
17             timeout: 30s
18             poll_interval: 5s
19             parameters:
20               expected_status: "on"

Reference YAML

The file examples/operation-rules-example.yaml is the canonical loadable reference covering all built-in rule patterns. Load it with:

$ flow rule create --from-yaml examples/operation-rules-example.yaml

Table of Contents