Switch Certificate Configuration (ConfigureCertificate)

This document describes how the switch state controller configures switch TLS certificates during the Configuring phase. The handler delegates device operations to Component Manager (CM), which in turn calls Rack Manager Service (RMS) asynchronously and polls job status until completion.

Goals

• Install or rotate the switch NVOS certificate as part of initial switch bring-up, before NVOS admin credentials are stored (RotateOsPassword).
• Keep RMS-specific protobuf and job semantics behind the CM NvSwitchManager abstraction so the state handler stays backend-agnostic (RMS, NSM, mock).
• Persist the async job ID in controller state so restarts can resume polling.

Placement in the Switch FSM

ConfigureCertificate is a sub-state of SwitchControllerState::Configuring, before RotateOsPassword, FetchInfo, and Validating.

Transient CM or transport failures during Start or WaitForComplete return StateHandlerError and leave the switch in the same sub-state for retry on the next handler iteration. Only a terminal RMS job status of Failed (or missing component manager while polling) transitions to Error.

Sub-states (ConfigureCertificateState)

Job status values use ConfigureSwitchCertificateState: Started, InProgress, Completed, Failed.

Domain name (domain_name) and mTLS service selection

Switch state controller

The switch state handler passes:

• domain_name = None for both bring-up and maintenance reconfiguration. RMS receives an unset domain field; rack association is enforced separately when deciding whether certificate configuration can run.
• services from SwitchStateHandlerServices.switch_mtls_services, sourced from [switch_state_controller].switch_mtls_services in site config. When omitted or empty, the default list contains all four service values. The switch state handler excludes scale_up_fabric_manager and scale_up_fabric_telemetry when the switch is not designated primary. If no services remain, bring-up advances without an RMS certificate job and switch certificate maintenance returns to Ready.

Direct ComponentConfigureSwitchCertificate RPC

ComponentConfigureSwitchCertificate uses the same [switch_state_controller].switch_mtls_services setting when the RPC runs directly because switch state-controller routing is disabled or bypass_state_controller is true. The direct path forwards the request’s domain_name to Component Manager.

When switch state-controller routing is enabled and bypass_state_controller is false, the RPC queues a ReconfigureCertificate maintenance operation. The switch state handler then uses the state-controller behavior described above.

Service list configuration

[switch_state_controller].switch_mtls_services controls certificate bindings for both switch state-controller and direct RPC operations. Rack ConfigureNmxCluster uses the effective service list for primary nmx-telemetry and only explicitly listed gNMI for the rack-wide batch. A non-empty list replaces the switch/direct default. Omission or an empty list uses all four values for switch/direct operations and primary nmx-telemetry, but not rack-wide gNMI. The switch state controller omits primary-only cluster applications on non-primary switches.

[rack_state_controller].nmx_cluster_switch_mtls_services is deprecated. The field remains accepted and ignored. Before RMS V2, rack ConfigureNmxCluster always binds nvue_api on every switch and also binds the explicitly configured telemetry interface. RMS V2 then selects the primary, binds NMX-C to the current NVUE material, and reconciles the fabric. NICo follows V2 with a primary-only scale_up_fabric_telemetry request when selected by the effective service list. Both NICo certificate requests pass domain_name = None.

The complete rack skip, retry, restart, polling, success, and error transition contract is defined under ConfigureNmxCluster sub-states.

Cluster application bindings can enable cluster state on the target switch. The target switch build must support each selected binding.

Component Manager API

CM exposes single-switch submission for the switch handler, batch submission for rack maintenance, and a shared status method:

The single-switch SwitchEndpoint is built from:

• Switch BMC MAC and BMC IP (required)
• Associated NVOS machine interface MAC and IP (both required; matches power-control validation in maintenance.rs)
• NVOS admin credentials from the credential vault (SwitchNvosAdmin); endpoint resolution failures during Start transition to Error (they do not return StateHandlerError).

The rack-batch SwitchCertificateEndpoint contains the switch BMC MAC as a persisted identity key plus the NVOS MAC, IP, hostname, and admin credentials. Certificate preparation does not require BMC IP, credentials, or connectivity. The subsequent V2 phase keeps its existing full-endpoint requirements.

Backend matrix

Rack-wide batch submission is implemented by the RMS backend. Other backends return Unsupported through the default implementation.

RMS integration

Identity resolution (RMS backend only)

Before calling RMS, RmsBackend:

1. Looks up switch.id and switch.rack_id via find_rms_identities_by_macs.
2. Builds rms::NodeInfo from the SwitchEndpoint and resolved identity.
3. Passes optional domain, services, and device info to RMS.

If the switch has no rack_id in the database, identity resolution fails and CM returns an internal error (the state handler normally skips earlier when switch.rack_id is unset during bring-up).

RMS RPCs

RMS job states are mapped in map_rms_configure_switch_certificate_job_state.

Sequence diagrams

Happy path (RMS backend)

One state-controller iteration runs Start; a later iteration runs WaitForComplete until RMS reports completion.

Skip path (no rack association)

Error path (job failed)

Maintenance reconfiguration (ReconfigureCertificate)

Operator maintenance can reinstall switch certificates without leaving Ready permanently. The flow reuses certificate.rs with ConfigureSwitchCertificateMode::Reconfigure:

Ready
  -> Maintenance { operation: ReconfigureCertificate, configure_certificate: Start }
  -> Maintenance { ..., configure_certificate: WaitForComplete { job_id } }
  -> Ready (success, maintenance request cleared)
  -> Error (failure)

Differences from bring-up:

Entry point: switch_maintenance_requested.operation = ReconfigureCertificate from Ready or Error. See Switch State Diagram.

Persistence

Controller state is stored in switches.controller_state (JSON). Example after job submission:

{
  "state": "configuring",
  "config_state": {
    "ConfigureCertificate": {
      "configure_certificate": {
        "WaitForComplete": {
          "job_id": "stub-switch-cert-job"
        }
      }
    }
  }
}

The job ID is only in controller state (unlike rack firmware upgrade, which also stores a separate firmware_upgrade_job row). This is sufficient for a single-switch, single-job certificate operation.

Implementation map

Testing

Integration tests cover:

• Skip when rack_id or component manager is absent → RotateOsPassword
• Start → WaitForComplete with mock CM
• WaitForComplete → RotateOsPassword on success
• WaitForComplete → Error on failed job status
• ConfigureCertificate (completed or skipped) → RotateOsPassword → FetchInfo → Validating
• Maintenance ReconfigureCertificate success and failure paths

Run with DATABASE_URL set (sqlx test harness), filter: cargo test -p carbide-api-core configure_certificate.

Future work

1. Decide whether domain should be set explicitly (for example to rack_id) once the RMS certificate catalog contract is finalized.
2. Decide whether NSM backend should support certificate configuration or remain explicitly unsupported.