machine-a-tron Build & Deployment Guide

machine-a-tron is a bare-metal simulator for NICo testing. It hosts mock DPUs and servers via Redfish BMC, allowing end-to-end NICo flows without real hardware. This guide documents everything needed to build the container image and deploy it on a cluster.

Overview

machine-a-tron runs in Override Mode: site-explorer redirects all Redfish traffic to the mock BMC server running inside the pod. It is only suitable for simulation-only clusters (no real hardware).

Quick path: setup-machine-a-tron.sh

For a running NICo Core site, the fastest and most reliable way to deploy is the end-to-end script, which performs every step in this guide (namespace, pull secret, CA/Vault secret refresh, BMC credential seeding, bmc_proxy configuration, DHCP-pool sizing, cert reissue, deploy, and verification) and is idempotent:

$
export KUBECONFIG=/path/to/kubeconfig
$
export REGISTRY_PULL_SECRET=<NVIDIA_API_KEY>   # only if the pull secret is absent
$
helm-prereqs/setup-machine-a-tron.sh           # add -y for non-interactive

Quick start with 4500-host simulated fleet

Build and push the image to the registry of your choice, then run the setup script in scale mode. Nothing registry-specific is committed — the image location, tag, and pull credentials all come from the environment, exactly like setup.sh:

$
export KUBECONFIG=/path/to/site/kubeconfig
$
export NICO_IMAGE_REGISTRY=<registry>/<repo>     # e.g. registry.example.com/nico
$
export MAT_IMAGE_TAG=<tag>                       # tag you built and pushed (see §2)
$
export REGISTRY_PULL_SECRET=<api-key>            # omit if the pull secret already exists
$
$
MAT_MODE=scale HOST_COUNT=4500 helm-prereqs/setup-machine-a-tron.sh -y

That is the whole procedure. The script exits after its bounded verification window while ingestion continues in-cluster; progress is visible with:

$
kubectl exec -n postgres <patroni-primary> -- su postgres -c \
>
  "psql -d nico_system_nico -tAc \"SELECT
>
     (SELECT count(*) FROM explored_endpoints) || ' explored / ' ||
>
     (SELECT count(*) FROM machines) || ' machines';\""

What to expect

The following is measured on a 3-node dev cluster, with dev-sized Postgres.

4500 hosts × 2 DPUs = 13,500 BMC endpoints → 13,500 machines.

The pipeline is autonomous once the script completes — it has run through multi-hour client connectivity outages without intervention. Occasional nico-api restarts under peak ingestion load are absorbed by the pipeline (machines resume within a cycle). Re-running the script is always safe (idempotent) and re-registers any expected machines that arrived late.

The rest of this document explains what that script does and why, and is the reference for manual deployment or debugging. The script’s header comments enumerate the non-obvious failure modes it guards against.

Prerequisites

• Docker with buildx and a linux/arm64 builder available (see below)
• A container registry you can push to and the cluster can pull from (referenced via NICO_IMAGE_REGISTRY, same convention as setup.sh)
• A cluster where machine-a-tron can reach nico-api.nico-system.svc.cluster.local:1079
• The nico-machine-a-tron Helm chart (helm/charts/nico-machine-a-tron)
• The image pull secret machine-a-tron-pull in the nico-mat namespace (created automatically by the setup script from REGISTRY_PULL_SECRET)

Building the Container Image

Why cross-compilation is required

machine-a-tron must run on x86_64 cluster nodes. The Rust dependency aws-lc-sys contains hand-written x86_64 assembly (s2n-bignum). Compiling under QEMU emulation causes a SIGSEGV in the assembler (bignum_madd_n25519.S). We therefore use true cross-compilation: a native linux/arm64 Rust compiler targeting x86_64-unknown-linux-gnu.

The carbide-rpc crate runs a protobuf build script that requires both protoc and the protobuf well-known types (libprotobuf-dev on Debian). libredfish is a Git dependency so git must also be present in the build stage.

Build command

Run from the repository root:

$
# Same convention as setup.sh: your registry/repository prefix, no scheme.
$
REGISTRY=${NICO_IMAGE_REGISTRY:?export NICO_IMAGE_REGISTRY=<registry>/<repo>}
$
COMMIT=$(git rev-parse --short HEAD)
$
TAG="${COMMIT}-amd64"
$
$
docker buildx build \
>
  --platform linux/amd64 \
>
  -f crates/machine-a-tron/Dockerfile \
>
  --push \
>
  -t "${REGISTRY}/machine-a-tron:${TAG}" \
>
  .

Registry authentication

$
docker login "${NICO_IMAGE_REGISTRY%%/*}" \
>
  -u "${REGISTRY_PULL_USERNAME:-\$oauthtoken}" \
>
  -p "${REGISTRY_PULL_SECRET}"

Some registries use a fixed username with API-key auth — set REGISTRY_PULL_USERNAME accordingly (default: $oauthtoken).

Cluster Prerequisites

Namespaces and secrets

$
# Create the deployment namespace
$
kubectl create namespace nico-mat --dry-run=client -o yaml | kubectl apply -f -
$
$
# Image pull secret (replace <API_KEY> with your NVIDIA API key)
$
kubectl create secret docker-registry machine-a-tron-pull \
>
  -n nico-mat \
>
  --docker-server="${NICO_IMAGE_REGISTRY%%/*}" \
>
  --docker-username="${REGISTRY_PULL_USERNAME:-\$oauthtoken}" \
>
  --docker-password="${REGISTRY_PULL_SECRET}" \
>
  --dry-run=client -o yaml | kubectl apply -f -
$
$
# Copy nico-roots CA secret from nico-system
$
kubectl get secret nico-roots -n nico-system -o json \
>
  | jq 'del(.metadata.namespace,.metadata.resourceVersion,.metadata.uid,.metadata.creationTimestamp)' \
>
  | kubectl apply -n nico-mat -f -

Vault secrets

machine-a-tron’s service account needs the Vault secrets present in its namespace (even though they are optional — their absence produces confusing log noise):

$
for secret in nico-vault-approle-tokens nico-vault-token vault-cluster-info; do
$
  kubectl get secret "$secret" -n nico-system -o json \
>
    | jq 'del(.metadata.namespace,.metadata.resourceVersion,.metadata.uid,.metadata.creationTimestamp)' \
>
    | kubectl apply -n nico-mat -f -
$
done

$
kubectl get secret nico-roots -n nico-system -o json \
>
  | jq 'del(.metadata.namespace,.metadata.resourceVersion,.metadata.uid,.metadata.creationTimestamp,.metadata.ownerReferences)' \
>
  | kubectl apply -n nico-mat -f -
$
kubectl delete secret nico-machine-a-tron-certificate -n nico-mat --ignore-not-found

BMC credentials in Vault (required for site-explorer)

site-explorer’s check_preconditions requires three site-default Vault credentials before it will explore any endpoint: machines/bmc/site/root, machines/all_dpus/site_default/uefi-metadata-items/auth, and machines/all_hosts/site_default/uefi-metadata-items/auth. The two UEFI paths are created by the nico-prereqs kvSeeds but with empty passwords, which fails the check (vault does not have a valid password entry) — they must be re-seeded with any non-empty password. machines/bmc/site/root is not seeded at all; without it every run aborts with MissingCredentials.

Beyond the preconditions, the credential rotation flow requires this exact chain (all handled by setup-machine-a-tron.sh Phase 4):

site-explorer logs into each BMC with its factory default, rotates the password to the site root value, then proceeds — using the wrong factory password (or a site root equal to factory) yields 401 Unauthorized, which latches a self-perpetuating AvoidLockout (NICO-SITEEXPLORER-144) until an operator clears it (nico-admin-cli site-explorer refresh <bmc-ip>).

Deploying the Helm Chart

Site values file

Copy helm-prereqs/values/machine-a-tron.yaml and fill in the site-specific values:

SPIFFE URI override

The cert-manager Certificate resource auto-generates a SPIFFE URI based on the deployment namespace: spiffe://nico.local/nico-mat/sa/nico-machine-a-tron.

nico-api’s spiffe_service_base_paths only includes /nico-system/sa/ (and two others), so this URI is not recognized. The result is that machine-a-tron’s principal is only TrustedCertificate — not SpiffeServiceIdentifier("machine-a-tron") — and every gRPC call beyond Version returns HTTP 403.

The values file already includes the fix:

1
certificate:
2
  uris:
3
    - "spiffe://nico.local/nico-system/sa/machine-a-tron"

This overrides the auto-generated URI so nico-api can correctly identify and authorize machine-a-tron as the Machineatron RBAC principal.

Deploy

$
helm upgrade --install nico-machine-a-tron \
>
  helm/charts/nico-machine-a-tron \
>
  -n nico-mat \
>
  --create-namespace \
>
  -f helm-prereqs/values/machine-a-tron.yaml

Configuring Override Mode

Configure nico-core’s site-explorer to redirect all Redfish traffic to the mock. Add this to the nico-core site config under [site_explorer]:

1
[site_explorer]
2
bmc_proxy = "nico-machine-a-tron-bmc-mock.nico-mat.svc.cluster.local:1266"

Use the cross-namespace FQDN. site-explorer runs inside nico-api in nico-system; a bare service name resolves against that namespace and fails (“connection refused” on every Redfish call) because the mock’s Service lives in nico-mat.

This setting does not survive a nico-core helm upgrade — the ConfigMap is chart-owned, so an upgrade silently reverts it. Re-run setup-machine-a-tron.sh (idempotent) after any nico-core upgrade.

Field name matters. The config field is bmc_proxy — a single "host:port" string (crates/site-explorer/src/config.rs). The older override_target_ip / override_target_port fields are deprecated, and override_target_host was never a valid field at all (earlier revisions of this guide were wrong — a value under that key is silently ignored).

Setting bmc_proxy at launch also makes allow_changing_bmc_proxy default to true. That is what allows the chart value machineATron.configureBmcProxyHost to work: when set, machine-a-tron calls nico-api’s set_dynamic_config to set bmc_proxy at runtime, but that call is rejected with PermissionDenied unless allow_changing_bmc_proxy is true. The two mechanisms are complementary — the values file ships configureBmcProxyHost: "nico-machine-a-tron-bmc-mock.nico-mat.svc.cluster.local" (FQDN, same cross-namespace requirement), and the nico-core bmc_proxy setting both enables that path and covers the case where the runtime call has not happened yet.

Apply via helm upgrade of the nico-core chart, or patch the configmap and restart nico-api (site-explorer runs in-process in nico-api):

$
kubectl rollout restart deployment/nico-api -n nico-system

Why machines get created (expected_machines)

site-explorer’s MachineCreator refuses to create a managed host unless a matching expected_machines row exists (by BMC MAC) — otherwise it logs Refusing to create managed host, expected machines entry not found. machine-a-tron auto-registers these when machineATron.registerExpectedMachines: true (the default in the values file). DHCP discovery alone is not sufficient.

Multi-pod simulation with per-BMC ClusterIP services (bmcServices)

The chart can shard the simulated fleet across several machine-a-tron pods, each with a dedicated ClusterIP range where every simulated BMC gets its own Service (pods.<name>.cidr + bmcServices.enabled: true). NICo then dials each BMC IP directly — no bmc_proxy. A validated two-pod example lives at helm-prereqs/values/machine-a-tron-multipod.yaml (2 pods × 5 wiwynn_gb200_nvl hosts × 2 DPUs → 30 machines, 15 per pod).

Everything single-pod mode needs still applies (namespaces, CA copy, Vault seeds, SPIFFE URI). Multi-pod adds the following requirements, all hit in practice:

1. Kubernetes 1.29+ for the ServiceCIDR object. On older clusters set bmcServices.serviceCIDR.create: false and pick pod CIDRs inside the apiserver’s --service-cluster-ip-range — a static clusterIP outside that range is rejected (“the provided IP is not in the valid range”). Check the chosen sub-ranges are free of existing ClusterIPs first.
2. Image with the Host-header routing fallback (PR #3190). Without a bmc_proxy the Redfish client sends no Forwarded header; older bmc-mock builds then 404 every request with no router configured. Leave site_explorer.bmc_proxy unset in this mode.
3. Disjoint MAC pools per pod. All machine-a-tron instances derive MACs from the same default pools, so the pod that leases second is rejected on every DHCP with Network segment mismatch for existing MAC address and simulates nothing. Until the chart grows a per-pod MAC knob, supply full per-pod TOML overrides via configFiles.matConfigs.<pod> setting distinct mac_address_pool / hw_mac_address_ranges bases (see the example values file).
4. One NICo network segment per pod CIDR. network_prefixes allows one IPv4 prefix per segment, so each pod CIDR needs its own cloned underlay segment (same technique as the scale-mode segment fallback), gateway .1, num_reserved 1 — BMC Service IPs start at .2, matching the DHCP allocator.
5. Hardware-type specifics. Vendors libredfish does not recognize (e.g. wiwynn_gb200_nvl reports WIWYNN) resolve to unknown, so seed the host factory credential at machines/all_hosts/factory_default/bmc-metadata-items/unknown. GB-class hosts (NvidiaGBx00 flow) additionally require expected_machines rows with a working BMC credential before exploration completes (NICO-SITEEXPLORER-141 Missing credential expected_machine); on nico-api builds without the Machineatron AddExpectedMachine RBAC grant the auto-registration is 403’d and the rows must be inserted directly with the pinned password (hostBmcPassword), not the factory default. DPU BMCs explore without expected rows.

Verifying Startup

Check that machine-a-tron passes the initial API calls:

$
kubectl logs -n nico-mat deployment/nico-machine-a-tron | grep -E "firmware|Got desired|Error:"

Expected: Got desired firmware versions from the server: [...]

Check nico-api for denied requests (should be empty after the SPIFFE fix):

$
kubectl logs -n nico-system deployment/nico-api | grep "Request denied.*machine-a-tron"

DHCP Address Space

machine-a-tron allocates one OOB IP per BMC interface (1 per host + 1 per DPU). With 5 hosts and 2 DPUs each that is 15 IPs (5 + 5×2). Ensure the OOB DHCP prefix in nico-core is large enough. Note the usable count is smaller than the raw CIDR: a /28 (16 addresses) yields only ~10–13 usable after subtracting network, broadcast, gateway, and any reserved addresses — on dev6 a /28 yielded 10, so hostCount: 3 × 2 DPUs = 9 fit but 5 × 2 = 15 did not. Use at least a /27 for the default counts, or reduce hostCount / dpuPerHostCount. Symptom of overflow: No IP addresses left in prefix ... and machines stuck in BmcInit.

If the prefix is exhausted from previous runs, do one of the following:

• Force-delete old machine records via the admin CLI
• Reprovision the site (which clears the database) and redeploy from scratch

1
INSERT INTO machine_interfaces_deletion (id) VALUES (1) ON CONFLICT DO NOTHING;

The machine_dhcp_records view inner-joins the singleton control row machine_interfaces_deletion (id=1); if that row is deleted (easy to do by accident when clearing related tables) the view returns zero rows and DiscoverDhcp fails for every BMC with Database Error: no rows returned by a query that expected to return at least one row. If you hit that, restore the row:

Non-Obvious Fixes