Batch Configuration Deploy

View as Markdown

The Batch Deploy workflow applies one shared configuration diff to a small group of network devices behind a single human approval gate. It is an internal child workflow invoked by Multi-Device Deploy: Multi-Deploy discovers deploy-enabled devices, collects their diffs, groups devices with identical diffs, splits oversized groups into batches, and starts one Batch Deploy child workflow per batch. Batch Deploy is not exposed through the Workflow API or as a standalone UI workflow.

Prerequisites

Before approving a Batch Deploy child invocation, confirm the following:

  • Every device in the batch is in Nautobot and deploy-enabled with a current intended configuration in the Config Store. Multi-Deploy enforces the deploy-enabled safeguard during discovery and loads each intended configuration before starting Batch Deploy. See Hosting Options for why deploy-disabled management switches must not be changed.
  • All devices in the batch are expected to apply the same diff. The approval display shows a single diff; if devices in the batch actually have different running configurations, only those matching the approved diff will apply cleanly and the rest will fail at apply time.
  • Approval to change traffic-bearing devices. The first stage is a hard approval gate and will wait indefinitely for a human reviewer.

Running the workflow

Batch Deploy is not exposed as a standalone form in the Config Manager UI today. It runs as a child of Multi-Deploy, which collects diffs across a role, groups devices with identical diffs, splits oversized groups into batches, and spawns one Batch Deploy child workflow per batch. Multi-Deploy posts a deep link on its execute_batches display for every batch it spawns; click that link to land directly on the child workflow’s status page.

The operator’s job on a Batch Deploy run is to review and approve (or reject) the diff at the first stage. The workflow blocks at that stage until a human approves or rejects.

Execution stages

The workflow runs three stages in order. Only the first stage requires human approval; the remaining stages run automatically based on the approval outcome.

  1. review_shared_diff — Hard approval gate. Requires one approver.

    The stage display renders a single shared diff with the full list of devices that will receive it. The reviewer should confirm:

    • The diff content is what was expected (no surprise lines from drift or template bugs).

    • Every device in the listed batch is one the reviewer is authorized to change.

    • The change is safe to apply in parallel across all devices in the batch.

      The stage waits indefinitely (state PENDING_APPROVAL) until a reviewer clicks Approve or Reject in the Config Manager UI. On approval, the workflow records the approver and continues. On rejection, the workflow records the rejecter, marks apply_configurations and perform_backups as unreachable, and exits with approved=false and an empty success/failure list. No device is touched on a rejection.

  2. apply_configurations — Apply the approved diff to every device in parallel.

    Each device’s apply runs as a separate apply_approved_configuration activity, all started concurrently with asyncio.gather. Each activity has a 5-minute start-to-close timeout and retries up to 3 attempts on transient errors. Two error types are non-retryable: ConfigSyntaxException (the rendered config is malformed) and DiffChangedException (the device’s running config drifted between the diff capture and apply — re-run from Multi-Deploy to get a fresh diff).

    Apply failures on individual devices do not stop the stage. Successful devices are listed under successful_devices; failures are reported under failed_devices as a {device_name: error_string} map. The stage display summarizes the success and failure counts.

  3. perform_backups — Capture a fresh backup for every successful device.

    For each device in successful_devices, the workflow starts the Configuration Backup workflow as a child workflow with trigger=WORKFLOW, attaching the device’s intended-config commit SHA so the new backup is correlated with the configuration version that was just applied. All backup children are started immediately and awaited concurrently with a 10-minute run timeout.

    The stage displays a direct link to every backup child as soon as it starts and updates each link as the child succeeds or fails. It also records each terminal result by device name, including whether the backup succeeded, whether it changed the stored configuration, any error, and the child workflow ID. A failed backup is included in failed_backups and the per-device result, but it does not fail the deployment batch — the device is configured and in service; only the post-deploy backup is missing.

If review_shared_diff rejects, the workflow archives the result and exits without running the two subsequent stages.

Verifying outcomes

Confirm the run succeeded by checking all of the following:

  • All three stages green on the Config Manager run page (or review_shared_diff green and the others marked unreachable, if the diff was rejected).
  • successful_devices in the workflow result lists every device the operator expected to receive the change.
  • failed_devices is empty. Any entry is a {device_name: error} pair; investigate the error and re-run with a corrected batch input.
  • The backups summary is complete. Its total count should equal the number of successfully configured devices, and successful + failed should equal total. The results map contains the success state, changed state, error, and child workflow ID for every device backup.
  • A fresh backup exists for every result with success=true. Open its link in the perform_backups stage to inspect the child run. For a failed result, use the same link and its recorded error to correct the underlying problem.
  • Parent workflow link (when invoked by Multi-Deploy) shows the configuration and backup success/failure counts in the parent’s execute_batches display.

Common issues

Stage is stuck on “Waiting for approval”.

review_shared_diff blocks until a human approves or rejects. There is no timeout. Open the Config Manager workflow page, review the diff and device list, and click Approve or Reject. If the batch was spawned by Multi-Deploy, the parent’s execute_batches display has a direct link to this child.

Some devices in the batch failed to apply.

The stage continues past individual failures. The most common per-device errors are:

  • ConfigSyntaxException — the rendered intended config does not parse on the device. Re-render via the Render Service and check the template; do not retry the batch until the rendered config is fixed.
  • DiffChangedException — the device’s running config changed between when the diff was captured and when the apply ran. Re-run Multi-Deploy so a fresh diff is captured, re-approved, and re-applied.
  • Connectivity / timeout — the device became unreachable mid-apply. With commit_confirm=True the device will auto-roll back on its own. Investigate the device and re-run.

DiffChangedException on every device.

Something is rewriting devices’ running configurations out of band (a parallel manual change, another deploy workflow, or a configuration agent on the device). Resolve the source of the drift before retrying.

Backup workflow failed but apply succeeded.

The device is configured and in service; only the post-deploy backup is missing. Inspect the device’s entry in backups.results for the error and child workflow ID, then re-run the Configuration Backup workflow against the device once the underlying issue is resolved.

  • Multi-Device Deploy — the parent workflow that normally spawns Batch Deploy, including role-based device discovery and diff grouping.
  • Configuration Deploy — single-device deploy for one-off changes.
  • Configuration Backup — the child workflow triggered for every successfully applied device.
  • Render Service — produces the intended configurations Batch Deploy applies.
  • Config Store — where intended and backup configurations are versioned.
  • Hosting Options — explains the deploy-disabled safeguard and why it matters for batch operations.