User Guides

New Site Bringup

View as Markdown

This guide walks operators through bringing up a new site managed by Config Manager — from initial power-up through ZTP, cable validation, and the troubleshooting steps to apply when something fails.

The guide assumes Nautobot has already been populated with the site’s device data (serial numbers, MAC addresses, IP addresses, cabling, and config contexts).

The Config Manager, Nautobot, Kea, relay, and packet-flow checks in this guide apply across supported network operating systems. Device commands and file paths vary by vendor. Cumulus Linux-specific material is explicitly labeled, with its main command and recovery reference grouped in Cumulus Linux device checks and recovery.

The workflow and service steps apply to every DCIM provider. The data-entry examples on this page use the bundled Nautobot provider. For another provider, model the corresponding device identity, IP, platform, intended firmware, and topology data according to that provider’s documentation.

Prerequisites

Before starting bringup, confirm the following are in place:

  • Hosting — A Config Manager cell instance has been deployed for this site. See Hosting Options.
  • Network topology — The site’s management network satisfies the requirements in Network Topology Requirements, and a clear ZTP provisioning order has been documented.
  • Firewall / security group rules — All required ports are open between the device network and Config Manager. See Firewall Ports.
  • Nautobot data — Devices are registered in Nautobot with serial number, MAC addresses, primary IPv4, platform, and any required config contexts. See DHCP Modeling in Nautobot.
  • Racking and cabling complete — All switches are physically racked and cabled per the site design, with serial numbers and MAC addresses captured in the inventory system.
  • OS images uploaded to the ZTP server — Every Cumulus / NVOS / MLNX-OS version the site’s switches will ZTP onto must be present on the ZTP server before the switches power on. See Upload Images to the ZTP Server.

Cumulus Linux version requirement: Cumulus ZTP requires the switch to arrive on Cumulus 5.x. If a switch arrives on an older release, perform an OS upgrade first; ZTP and DHCP will then proceed as expected.

Bringup sequence

Follow these steps in order. Each step assumes the previous one has completed successfully.

  1. Power up all network switches. All switches can be powered up at once. Each switch will boot and attempt DHCP. Provisioning will naturally fan out from Config Manager along the documented ZTP provisioning order as each upstream layer comes online.

    Do not assume a switch is still retrying because its console says that ZTP is in progress. After fixing cabling or relay connectivity, verify that DHCP or the NOS provisioning log has a recent attempt (/var/log/autoprovision on Cumulus Linux). If its attempts stopped while the path was broken, power-cycle the switch to restart DHCP and ZTP.

  2. ZTP device configurations from Config Manager. Each switch boots with factory defaults, requests an IP via DHCP, and downloads its boot script and rendered configuration from the Config Manager ZTP service. See Network ZTP for how this works under the hood.

    When ZTP completes successfully, the device’s status in Nautobot transitions to Provisioned. If a switch does not reach Provisioned within a reasonable window, work through the Monitoring DHCP and ZTP and Common errors sections below. Where the underlying issue is physical (miscabled, unpowered, factory reset required), the on-site operations team will need to investigate.

  3. Run the Cable Validation workflow. Once switches have reached Provisioned, run the cable validation workflow to confirm cabling matches the design. See the Cable Validation Guide for how to run the workflow and interpret the report.

    Incorrect cabling typically surfaces as a MAC mismatch or a down link. The report identifies the device and port to investigate. Devices that do not respond require the on-site operations team to confirm power and connectivity.

  4. Run the hardware error check workflow. This collects hardware diagnostic command output from each switch into a per-device file for review by the on-site operations team.

Monitoring DHCP and ZTP

When a device fails to provision, the first step is to look at the DHCP and ZTP logs to determine where in the chain it stalled.

Config Manager streams DHCP (Kea), ZTP, and render-service logs to whatever observability stack the deployment uses. Filtering by MAC address, IP address, or device UUID is the fastest way to isolate a single device’s activity.

Start from the Config Manager overview dashboard for the site and deployment under investigation:

  1. Open the DHCP logs panel and switch to the log explorer before filtering DHCP traffic. The dashboard panel itself is useful for a quick health check, but the explorer is better for following a single device and widening the time range.
  2. Open the ZTP logs panel and switch to the log explorer before filtering by device UUID.
  3. Use the DHCP lease activity dashboard in Config Manager to inspect active leases, reservations, and pools, or to clear a stale lease. This dashboard shows current Kea state; the log explorer shows the packet history.

Do not use a generic application-log panel to decide that DHCP traffic is absent. Select the Kea container logs and confirm that the dashboard is scoped to the correct cluster and Config Manager namespace.

Deployments without an aggregated log backend can read the same logs directly with kubectl logs. Each section below shows both the aggregated-query form and the kubectl equivalent.

Follow one device through the provisioning path

Use this sequence to avoid mixing results from nearby devices:

CheckEvidenceConclusion
Device sends a discoverA console-side packet capture shows DHCPDISCOVERThe device is trying; this does not prove that Kea received it.
Kea receives the discoverKea logs contain DHCP4_PACKET_RECEIVED for the same transaction ID, MAC, client ID, or relay addressThe relay and routing path to Config Manager works.
Kea sends an offerKea logs contain DHCP4_PACKET_SEND and DHCPOFFER for the same transactionKea selected a subnet and an address. Confirm that the return path reaches the device.
Device installs the leaseThe expected interface has a dynamic address and remaining lease lifetimeDHCP completed. Move to ZTP; repeated DHCP-client restart messages are not the root cause.
Device downloads a boot scriptThe NOS provisioning log contains a recent script-start entry. On Cumulus Linux, this is /var/log/autoprovision with ztp starting script.DHCP supplied usable ZTP options and the device started the script.
ZTP serves the deviceZTP service logs contain the Nautobot device UUIDFollow the last endpoint and HTTP status to find the failed stage.
Provisioning completesZTP logs show the device calling the provisioned endpoint and Nautobot shows ProvisionedZTP completed. If the intended config is still wrong, inspect render logs and the current rendered configuration.

Record the device name, device UUID, serial number, relevant MAC, expected subnet, expected leased address, and DHCP transaction ID before searching. A client ID in Kea logs is commonly a leading type byte followed by the serial number encoded as colon-separated hexadecimal; decode it before attributing a lease to a device.

Confirm the expected DHCP model

The interface used to provision determines what should exist in Kea:

Provisioning pathNautobot modelBest search keys
Device management interface such as eth0The containing prefix has dhcp-subnet; the device IP has dhcp-reserve and is assigned to the correct interface. The reservation matches a MAC or client ID.Management MAC, serial/client ID, reserved IP
Front-panel point-to-point management linkThe /31 prefix has dhcp-subnet; the device-side address has dhcp-pool. The resulting pool contains one address and is not tied to a client ID./31 network address, relay/source address, pool IP

Do not add dhcp-reserve to a pool-of-one address to bind it to a switch. On a front-panel /31, physical topology and the relay path select the pool. A switch on the wrong link can therefore consume another switch’s lease even when the Nautobot pool data is correct.

Use the DHCP lease activity dashboard to confirm that the expected subnet and either its reservation or pool are present. If they are absent after the configuration refresh interval, correct the Nautobot tags and assignments first. If they are present, continue with packet and lease troubleshooting instead of repeatedly toggling tags.

For a lower-level check, use the DHCP API’s /config endpoint from the deployment’s API documentation and search the returned Kea configuration for the exact subnet. A subnet that is missing entirely points to the prefix or dhcp-subnet tag. A subnet with no expected pool or reservation points to the IP assignment or tag. If the complete subnet data is present, move on to active leases and packet flow.

Filtering DHCP logs

In your log explorer, filter the Kea container logs in the Config Manager namespace and search for the device’s eth0 MAC address. A typical aggregated-log query:

{cluster="<cluster>", k8s_namespace_name="<config-manager-namespace>", k8s_container_name="kea"} |= `b0:cf:0e:61:2c:c8`

Or directly via kubectl:

$kubectl -n <config-manager-namespace> logs \
>  -l app.kubernetes.io/component=network-dhcp -c kea \
>  --tail=-1 | grep -i 'b0:cf:0e:61:2c:c8'

Query options:

  • By MAC address (lowercase) — for devices that provision over the management interface, the eth0 MAC is the right identifier.

  • By uplink network address — for management-network switches that provision via Front Panel Ports (FPP) using a /31 DHCP pool of size 1, the MAC is not known ahead of time. Instead query by the /31 prefix; for example, an uplink of 10.0.0.1/31 is searchable as 10.0.0.0/31.

  • By serial number (hex; Cumulus Linux 5.10+) — these switches send the serial number as DHCP client ID. Convert the serial to colon-separated hex and query for it:

    $python3 -c "import sys; print(':'.join(f'{ord(c):02x}' for c in sys.argv[1]))" "MT2519J002PD"
    $# 4d:54:32:35:31:39:4a:30:30:32:50:44

Keep the DHCP transaction ID (tid) in the result when following a discover through an offer and acknowledgement. A MAC or IP search can return packets from different devices, especially after cabling changes. Compare the decoded client ID, observed interface MAC, expected device serial, and transaction ID before concluding which device holds a lease.

For a front-panel /31, Kea may log the relay-side or peer address rather than the one-address pool value you expect the device to receive. Search for both addresses in the /31 and the complete prefix.

ZTP logs by device ID

Filter the ZTP service logs by the Nautobot device UUID:

{cluster="<cluster>", k8s_namespace_name="<config-manager-namespace>", k8s_container_name=~"ztp.*"} |= `<device-uuid>`

Or directly via kubectl:

$kubectl -n <config-manager-namespace> logs \
>  -l app.kubernetes.io/component=network-ztp --all-containers \
>  --tail=-1 | grep '<device-uuid>'

A successfully provisioned device always ends with a POST to the provisioned endpoint. If you do not see this log, ZTP did not complete — see Common errors below.

Render logs

Filter the render-service logs by namespace and device UUID:

{cluster="<cluster>", k8s_namespace_name="<config-manager-namespace>", k8s_container_name=~"render-service-api|render-service-render-service-device-consumer|render-service-template-consumer"} |= `<device-uuid>`

Or directly via kubectl:

$kubectl -n <config-manager-namespace> logs \
>  -l app.kubernetes.io/component=render-service --all-containers \
>  --tail=-1 | grep '<device-uuid>'

Renders are triggered by Nautobot change events or by template version changes, and rendered configs are written to the Config Store. You can navigate to a device’s current render from the Config Manager Device Status tab in Nautobot.

Common errors and troubleshooting

Fast triage

Work from the first missing signal rather than resetting the device immediately:

Last confirmed signalInvestigate next
No discover on the deviceDevice interface state, DHCP client state, and factory/ZTP state
Discover on device, nothing in Kea logsVLAN, relay configuration, routing, firewall rules, and whether the selected log dashboard targets the right Config Manager instance
Discover in Kea, no offerGenerated subnet/reservation/pool data, Kea allocation messages, and active leases
Offer in Kea, no address on deviceReturn routing, relay behavior, packet loss, and device interface selection
Address on device, no NOS provisioning logDHCP ZTP options and HTTP/HTTPS reachability to the ZTP service
Recent NOS provisioning failureThe final command and HTTP status in the vendor-specific provisioning log, then the matching ZTP service log by device UUID
ZTP succeeds, intended config absentRender status, rendered configuration, and render-service logs

DHCP errors

No pools were available for the address allocation (or the device received a dynamic IP from a pool instead of a static reservation)

This means no static reservation is tied to the device’s MAC or client-id. Walk through the following checks in Nautobot:

  • Confirm ZTP is enabled. On the device’s Config Manager Info tab, verify the ztp_enabled flag is set. If it was just enabled, allow up to 10 minutes for the DHCP config to refresh.
  • Confirm device status is one of Provisioning, Provisioned, or Active — set it to Provisioning if not.
  • Confirm the device has a Primary IPv4 and Platform set — without these the reservation will not be generated.
  • For Cumulus pre-5.10 devices (which cannot send serial as DHCP client ID): the eth0 MAC address must be populated in Nautobot for non-management devices. Management-network switches provision via the /31 uplink IP using a pool-of-size-one, so neither MAC nor client ID is required. In DHCP logs, pre-5.10 instances appear with cid=[no info].

See DHCP Modeling in Nautobot for the full data model behind reservations and pools.

Failed to select a subnet for incoming packet

This occurs for /31 allocations on the management network when ZTP is not enabled on the device. Enable ZTP and wait up to 10 minutes.

No DHCP requests incoming for a specific MAC

  • The device may not be cabled to the upstream router. Check interface status on the upstream port.
  • The MAC supplied by the vendor or on-site operations team may be incorrect. Check LLDP data on the upstream switch: if the serial matches but the MAC does not, update the MAC in Nautobot.

If the device provisions over a front-panel /31, searching only for its management MAC is insufficient. Search the Kea logs for the observed front-panel MAC, both addresses in the /31, and the serial/client ID. Confirm the actual interface MAC on the device rather than relying only on inventory data.

No DHCP requests incoming for any device

Likely a firewall, security group, or routing issue. Confirm that the device subnets (IPMI/management/OOB) are permitted UDP 67/68 to and from Config Manager at every layer between the device and Config Manager. See Firewall Ports.

To isolate where DHCP traffic is being lost, capture on UDP 67/68 at each hop along the relay chain. On Cumulus (and most Linux devices) tcpdump is the quickest tool:

$# On the device itself — confirm the switch is actually sending DHCP discovers on eth0
$sudo tcpdump -i eth0 -nn -vv 'port 67 or port 68'
$
$# On an upstream relay agent — confirm requests arrive and are relayed toward Config Manager
$sudo tcpdump -i any -nn 'port 67 or port 68'

You should see a BOOTP/DHCP Request from the device’s MAC, followed by a relayed copy whose source becomes the relay agent’s IP and whose destination is the Config Manager DHCP VIP. If the discover appears on the device but never reaches the relay (or never leaves the relay), the gap identifies which hop is dropping the traffic.

If the device capture shows discovers but the correctly scoped Kea log explorer has no matching MAC, client ID, or relay address, the request did not reach Kea. Investigate the network path; changing Nautobot DHCP tags cannot repair an absent packet.

Kea receives a discover but does not offer an address

  1. In the DHCP lease activity dashboard, confirm the exact subnet and expected reservation or pool exist.
  2. For an eth0 reservation, confirm the reservation identifier matches the request’s MAC or decoded client ID.
  3. For a front-panel /31, confirm there is a one-address pool. The pool is not expected to contain a client ID.
  4. Inspect active leases for the expected address. After miscabling, another interface or device may still hold the lease for the configured valid lifetime.
  5. Search historical Kea logs for both addresses in the subnet, the offered address, and DHCP4_LEASE_ALLOC. Decode the client ID and compare it with the serial on the physical device.
  6. If the lease holder is stale and the cabling is now correct, clear that address from the DHCP lease activity dashboard. Otherwise wait for expiry. Power-cycle or restart ZTP on the intended device afterward if it stopped retrying.

Clearing a lease before correcting the cabling lets the wrong device acquire it again. Confirm LLDP and the local interface MAC first.

For local reproduction of DHCP config generation issues, run the DHCP config-gen tooling against a copy of the Nautobot data — see the DHCP service README for the procedure.

ZTP errors

HTTP 500 on any ZTP call

A transient backend issue, most often a Nautobot timeout. Engage the platform team to investigate.

HTTP 400 on POST /v1/device/<device-uuid>/validate_serial

The device’s reported serial number does not match the serial Nautobot expects for this configuration. This usually indicates miscabling or incorrect inventory data: the device may be trying to claim a /31 that belongs to another device, or the recorded serial may contain an error. Serial comparison is case-insensitive, so letter case alone is not a mismatch. Compare the complete serial reported by the device with Nautobot and the physical chassis label, then check LLDP on the upstream switch to identify the actual neighbor. For example, on Cumulus:

$nv show interface <port> lldp neighbor

On the device, use nv show platform to read the serial and interface commands such as ip link show <interface> to read the MAC. Do not infer identity from a DHCP pool address alone.

HTTP 403 unauthorized

The source IP of the request does not match any IP Nautobot has for the device. Either Nautobot data is wrong, the DHCP-generated pool is wrong, or — more commonly — the device is miscabled and is sending from an IP that belongs to a different device. Check Nautobot IPs, the rendered DHCP config, and LLDP on the upstream switch.

Cumulus Linux: stuck requesting startup.yaml and not progressing

The rendered configuration is failing to apply or has broken connectivity back to Config Manager. Access the switch via console or upstream SSH and inspect /var/log/autoprovision.

Cumulus Linux: console reports that ZTP is in progress for hours

The banner or nv show system ztp state says that ZTP is enabled or incomplete; it does not prove that attempts are still occurring. Check the newest timestamp in /var/log/autoprovision and the ZTP service logs for the device UUID.

  • If /var/log/autoprovision does not exist, return to DHCP troubleshooting. The device has not received and started a boot script.
  • If the file exists, read the final command and error rather than only the final line. A recent ztp starting script entry proves that DHCP and boot-script delivery worked.
  • If the last attempt predates a cabling, routing, data, or firewall fix, restart ZTP using the supported device command or power-cycle the switch. On-site operators should power-cycle affected devices after correcting a provisioning-path cabling fault when attempts have stopped.
  • Messages such as Restarting dhclients, Execution lock acquired, and Execution lock released only describe ZTP service mechanics. They do not show that Kea offered a lease or explain why provisioning stalled.

Device received a DHCP lease but no ZTP traffic follows

  • Confirm TCP 443 (or 80, depending on configuration) is open from the device subnets to Config Manager at every firewall and security-group layer. See Firewall Ports.

  • Routing between the device and Config Manager may be broken. Access via console or upstream SSH to investigate.

  • The device may be on Cumulus pre-5.x, which cannot ZTP. Upgrade and retry.

    From a Cumulus Linux shell, you can verify that the ZTP URL is reachable by fetching the boot script directly. On a non-management switch, run the request in the management VRF so it leaves the switch over eth0:

    $sudo ip vrf exec mgmt curl -v http://<ztp-server>/v1/device/<device-uuid>/boot-script

    A 200 response (or a 403, which still confirms the network path is open) means HTTP reachability is fine and the problem is elsewhere. A connection refused, timeout, or no route to host indicates a firewall, security-group, or routing issue between the device and Config Manager.

Render errors

No render occurs after a Nautobot change

Some Nautobot changes (for example, site-prefix updates) do not trigger a render. To force one, either call the render service API directly or make a no-op change (e.g. add whitespace) to the device’s config context in Nautobot.

Render failure

The most common causes are:

  • The device is missing the platform field, or no intended-firmware is set in its config context. Populate both.
  • A template bug or filter bug. Run a local render against the template repo with --debug to identify the failing line.

After committing a template change, follow the template repo’s release process and bump the template package version in the render service to pick up the new release. Deployment of the render service is then handled through the cluster’s GitOps workflow (e.g. ArgoCD).

Device swaps

When two devices end up swapped in the inventory system — for example, racked in each other’s positions — it is usually faster to swap them in Nautobot and let Config Manager swap their configurations than to ask the on-site operations team to re-rack or re-cable.

To perform the swap:

  1. Ensure the inventory-system owner renames the devices in their system to match the new physical positions.

  2. In Nautobot, swap the serial number, asset tag, and eth0 MAC address between the two device records.

    Do not rename the devices in Nautobot — renaming forces you to rebuild all of the cabling relationships.

  3. On the console for both switches, reset the DHCP configuration. The following commands are for Cumulus Linux; use the vendor’s equivalent procedure for another NOS:

    $nv unset interface eth0
    $nv set interface eth0 ip address dhcp
    $nv config apply
    $nv config save

  4. Reset ZTP to its original state and reboot. The documented reset command applies to Cumulus Linux 5.10 and later:

    $sudo ztp -R
    $sudo reboot

After reboot, each switch will ZTP again and pick up the configuration that now corresponds to its physical position.

Cumulus Linux device checks and recovery

The commands and paths in this section apply to Cumulus Linux. For another NOS, use the vendor’s commands for inspecting platform identity, DHCP state, provisioning state, and provisioning logs. Keep using the vendor-neutral Kea, ZTP service, render service, and DHCP lease activity checks from the earlier sections.

Inspect device identity, DHCP, and ZTP

From the console, collect the following before changing state:

$# Chassis serial number and platform identity
$nv show platform
$
$# Neighbor identity and physical cabling
$nv show interface <port> lldp
$
$# Address, MAC, and remaining DHCP lease lifetime
$ip address show <interface>
$
$# ZTP state; this does not prove that attempts are current
$nv show system ztp
$
$# Actual ZTP script attempts and failures
$sudo tail -100 /var/log/autoprovision

Match the observed serial, interface MAC, neighbor, address, and log timestamps to the same Nautobot device before taking recovery action.

Rerunning ZTP on a live device

The Cumulus user account can occasionally be dropped during ZTP — for example, if ZTP was triggered via ztp -r (which does not pass through DHCP) and the user-creation step failed. This leaves the switch with no sudo-capable account.

On Cumulus 5.11+, you can re-run the ZTP boot script without sudo using:

$nv action run system ztp

This re-executes the boot script — including the cumulus-user creation steps — and restores the cumulus user with the latest password from the secrets store. If the device is already on the correct firmware version, this will not trigger a reboot.

Options for logging into a stuck device

If a device has not attempted ZTP yet, log in with the factory default credentials (cumulus / cumulus) and set the password to the current site root password from your secrets store.

If a device has already attempted ZTP, it will have either the cumulus user or the breakglass user installed. Try the site root password against cumulus first, then fall back to breakglass.

The final step of ZTP is to remove the breakglass user. If you log in as breakglass and stay logged in, ZTP will loop indefinitely waiting to remove the account. Once you have diagnosed the issue, you must log out of breakglass to allow ZTP to complete.

You can reach a stuck device by either of the following paths:

Console

  1. Look up the device’s Console Port in Nautobot to find the attached console server.
  2. Reach the console server via its OOB VIP (typically accessed through the OOB firewall).
  3. Authenticate using the console server credentials from your secrets store.

If a device directly upstream of the stuck switch has already provisioned, you can SSH to that upstream device and then SSH across the /31 link to the stuck switch.

  1. Identify the uplink interfaces and IPs in Nautobot for the stuck device.

  2. SSH to one of the upstream device’s loopback IPs.

  3. Drop into the appropriate VRF on the upstream device:

    $ip vrf exec <vrf> bash

    For management-network devices this is the default VRF; for non-management devices, choose the VRF that carries the eth0 management connection.

  4. SSH to the uplink IP on the stuck device’s interface.