Tenant Management

Operator guide for tenant creation, resource allocation, and instance provisioning on NICo, using nicocli (REST API CLI) as the primary tool.

This is a Day 1 (Configuration) activity in NICo’s lifecycle model — the phase after hardware has been discovered, validated, and ingested (Day 0). Day 1 is when operators configure tenant boundaries, define resource allocations, and provision instances so tenants can consume bare-metal infrastructure.

The primary tool throughout this guide is nicocli, the CLI client that wraps the NICo REST API. Every REST endpoint is available as a CLI command, and nicocli handles authentication, token refresh, and multi-environment configuration automatically. The nico-admin-cli (which talks to the Core gRPC API) is referenced only where an operation has no REST API equivalent.

Before You Start

This guide assumes you have completed the Quick Start Guide, which covers NICo deployment, site creation, and host discovery (Day Zero). You should already have:

• A running NICo deployment with healthy REST API, database, Temporal workflow engine, and at least one site controller.
• At least one site registered and in Registered status, with machines discovered and available for allocation.
• nicocli installed (make nico-cli from the rest-api/ directory of the infra-controller repo) and reachable on $PATH.

If you plan to enable SPIFFE JWT-SVID machine identity, complete Day 0 Machine Identity before provisioning instances, then configure per-org identity after tenants exist. Refer to Machine Identity.

For nicocli mechanics and conventions (flag ordering, api.name selection, --data vs flag forms, output formats, pagination, --debug), see the nicocli reference guide. The examples in this guide assume you’ve read it.

Roles Required

NICo’s authorization model has three roles, all managed in the upstream identity provider (any OIDC-compatible IdP, e.g. Keycloak):

A single user can hold roles in multiple orgs simultaneously. On dev/service-account orgs, one user typically holds both Provider Admin and Tenant Admin in the same org.

These roles are not settable through NICo. They are assigned in the identity provider and read from each request’s token, which makes them a Day 0 concern that has to be in place before any of the Day 1 steps below will work. Authentication and Authorization is the authoritative reference for that configuration, covering both the issuers and keycloak modes, the four claim-mapping recipes, and the deployment surfaces where each value belongs.

Authentication

The Quick Start covers nicocli login end-to-end for Keycloak-backed deployments (the default for setup.sh installs). For other identity providers and for the static-token / token-command flows useful in automation, see the nicocli reference guide.

Verifying Connectivity

Confirm that nicocli can reach the API and your credentials are valid:

nicocli site list
nicocli user get

nicocli user get returns your identity as NICo sees it.

Creating a Tenant

How Tenant Creation Works

NICo uses a lazy creation model for tenants. There is no explicit “create tenant” API call. Instead, a tenant record is automatically created the first time a Tenant Admin retrieves the current tenant for their organization:

nicocli tenant current

If no tenant exists for the configured org, NICo creates one and returns it. If a tenant already exists, the same command returns the existing record. The operation is idempotent.

Each tenant maps one-to-one to an organization in the configured identity provider. The tenant’s identity (org name, display name) is derived from the IdP’s org metadata — there are no separate fields to supply.

In TUI mode:

nicocli tui
> tenant current

Required Conditions

• The authenticated user must be a member of the organization specified in the nicocli config (api.org).
• The user must hold the Tenant Admin role within that org.

If either condition is not met, the API returns HTTP 403. NICo trusts whatever the IdP says in the token’s claims, so getting these conditions met is an IdP administration task. It is not done through nicocli or the NICo API. The Quick Start Guide walks through the bundled Keycloak reference implementation (a dev Keycloak deployed by setup.sh with a pre-loaded realm), which is the simplest path for first-time setup. For production, point NICo at any OIDC-compatible IdP (Keycloak, Okta, Auth0, your existing enterprise IdP) by configuring the issuers block in nico-rest-api’s config. Refer to Authentication and Authorization for the deployment-side wiring. Its “Provider with Multiple Tenant IdPs” example is the recipe for one provider org serving tenants that each authenticate through their own identity provider.

For Keycloak deployments, Tenant Management with Keycloak covers the realm-side steps in full: the realm role naming convention NICo reads as org membership, creating the Tenant’s identity, and granting the privileged Tenant capability.

Worked Example

$ nicocli tenant current
{
  "capabilities": {
    "targetedInstanceCreation": true
  },
  "created": "2026-04-23T01:11:07.452525Z",
  "id": "<tenant-uuid>",
  "org": "acme-corp",
  "orgDisplayName": "Acme Corporation",
  "updated": "2026-04-28T18:20:10.438572Z"
}

Verifying the Tenant

Check tenant health with the stats endpoint:

nicocli tenant stats

Example response for a tenant in active use:

{
  "instance": {"error": 0, "pending": 0, "ready": 6, "registering": 0, "terminating": 0, "total": 6, "updating": 0},
  "subnet": {"deleting": 0, "error": 0, "pending": 0, "provisioning": 0, "ready": 0, "total": 0},
  "tenantAccount": {"error": 0, "invited": 0, "pending": 0, "ready": 1, "total": 1},
  "vpc": {"deleting": 0, "error": 0, "pending": 0, "provisioning": 0, "ready": 2, "total": 2}
}

A freshly created tenant shows all zeroes. In TUI mode: tenant stats. Status keys in the response are alphabetical, not lifecycle order.

What Happens Behind the Scenes

When GET /v2/org/{org}/nico/tenant/current is called and no tenant exists, the API:

1. Validates org membership and the Tenant Admin role.
2. Creates a tenant record in the database with name, org, and display name pulled from the IdP claims in the token.
3. Links any pre-existing tenant account invitations referencing this org.
4. Returns the new tenant as JSON with HTTP 200.

Subsequent calls return the existing tenant. If the org’s display name has changed in the IdP, NICo silently updates it on the next call.

Establishing a Tenant Account

A tenant alone cannot consume infrastructure — it needs a tenant account that links it to an infrastructure provider. This is a provider-side operation requiring the Provider Admin role.

Creating the Link

nicocli tui
> tenant-account create

The TUI prompts for the infrastructure provider ID and the tenant org name. Non-interactive (using individual flags):

nicocli tenant-account create --infrastructure-provider-id <provider-uuid> --tenant-org acme-corp

The tenant account starts in Invited status. Listing tenant accounts requires a filter flag — the bare nicocli tenant-account list returns HTTP 400:

nicocli tenant-account list --tenant-id <tenant-uuid>
nicocli tenant-account list --infrastructure-provider-id <provider-uuid>

Example tenant account detail (Ready, with active allocations):

{
  "accountNumber": "<auto-generated>",
  "allocationCount": 4,
  "id": "<account-uuid>",
  "infrastructureProviderId": "<provider-uuid>",
  "infrastructureProviderOrg": "acme-infra",
  "status": "Ready",
  "subscriptionId": null,
  "subscriptionTier": null,
  "tenantContact": null,
  "tenantId": "<tenant-uuid>",
  "tenantOrg": "acme-corp"
}

accountNumber is auto-generated. allocationCount is a live count of allocations under this account. tenantContact records the user who accepted the invitation (null until accepted, or null when the same org is both provider and tenant).

Accepting the Invitation (Tenant Side)

The tenant admin accepts the invitation, which moves the account to Ready.

Accepting is a tenant call, so nicocli has to act as the tenant. Two things decide who it acts as: the token it sends, and the org in the request path. Both have to name the tenant org.

Give the tenant its own config file, and set api.org to the tenant org in it. Pass that file with --config to nicocli login and to every tenant command. The request then carries the tenant’s token, which can act only in orgs it holds a role for. If it holds no role for the org in api.org, the request fails with 403 Requested organization not found in token claims. If it holds roles in several orgs, api.org picks which one it acts as.

Don’t log in to the provider’s config file. Login overwrites the provider’s auth.oidc credentials there. A provider token in auth.token also outranks auth.oidc.token. In that case the tenant’s login changes nothing that is sent.

On Keycloak deployments, Pointing nicocli at the Tenant Org has the commands.

To accept, send an empty PATCH body. Flags go before the account ID:

nicocli --config ~/.nico/<tenant-org>.yaml tenant-account update --data '{}' <account-id>

Or use the TUI:

nicocli --config ~/.nico/<tenant-org>.yaml tui
> tenant-account update

Only an account in Invited status can be accepted. An account that is already Ready returns this error:

$ nicocli --config ~/.nico/<tenant-org>.yaml tenant-account update --data '{}' <account-id>
Error: API error 400: Tenant Account status is not Invited

Granting Targeted Instance Creation

targetedInstanceCreation is the privileged tenant capability. A tenant that has it can:

• Create an instance against a specific machine ID, or narrow placement with a machine label selector.
• Set isRepairTenant: true when releasing an instance, which is what the repair tenant workflow requires.
• Read a set of otherwise provider-only resources, including machines, machine health, SKUs, racks, trays, and expected machines, at the sites of a provider it holds a Ready tenant account with, wherever the capability is effective. A site override that disables it also removes these reads at that site.
• Receive alternative VPC routing profiles from tenant/current/routing-profile.

A Provider Admin configures it per tenant account with the siteCapabilities field. This is the only supported way to grant it, and it applies to regular tenants as well as service-account orgs:

nicocli tenant-account update \
  --data '{"siteCapabilities":[{"siteIds":[],"targetedInstanceCreation":true}]}' \
  <account-id>

Payload rules

• Exactly one entry must have an empty or omitted siteIds. That entry sets the account-level default.
• Any further entry lists site IDs that override the default. Each site must already be associated with the tenant, which happens when the tenant’s first allocation at that site is created.
• No site ID may appear in more than one entry.
• targetedInstanceCreation is required on every entry.
• The request must not also carry tenantContactId, which is the invitation-acceptance field. Sending both returns HTTP 400.

Updates use replace semantics. A per-site override whose site ID is omitted from a later payload is cleared. To leave the capability on everywhere except one site, send both entries together:

{
  "siteCapabilities": [
    {"siteIds": [], "targetedInstanceCreation": true},
    {"siteIds": ["<site-uuid>"], "targetedInstanceCreation": false}
  ]
}

How the effective value resolves

Three inputs decide whether the capability is in force for a tenant at a given site:

1. The tenant account must be Ready. Setting the capability on an Invited account grants nothing until the tenant accepts.
2. The account-level default applies where no site override exists.
3. A site override, when present, wins over the account default.

Because the account is per provider, a tenant with accounts at two providers can be privileged at one and not the other.

Reading the current value

nicocli tenant-account list returns siteCapabilities and is the source of truth:

nicocli tenant-account list --tenant-id <tenant-uuid>

Instance Types

Instance types define hardware classes — they map a named category (like “GB200-NVL72” or “DGX-H100”) to a set of physical machines with specific GPU, CPU, and network configurations. Before a tenant can create compute allocations, the instance types must exist and machines must be associated with them.

How Instance Types Map to Hardware

Each instance type record includes:

• A name identifying the hardware class (e.g., GB200-NVL72)
• GPU count and type metadata
• Machine associations linking specific physical machines to the type

The site administrator defines instance types during Day Zero. NICo validates that machines associated with an instance type have compatible hardware before accepting the association.

Viewing Instance Types

nicocli instance-type list --output table

The TUI provides richer detail:

nicocli tui
> instance-type list
> instance-type get

The detail view for an instance type includes an allocationStats section showing how many machines are assigned, allocated, and available (maxAllocatable). This tells you the upper bound for a new allocation’s constraint value.

Creating Instance Types (Provider Admin, gRPC Only)

Instance type creation is a gRPC operation via nico-admin-cli:

nico-admin-cli -a <core-api-url> instance-type create --name "GB200-NVL72" --gpu-count 72
nico-admin-cli -a <core-api-url> instance-type associate --instance-type-id <id> --machine-id <machine-id>

See the Quick Start Guide, Step 7 for nico-admin-cli access patterns. The REST API exposes instance type CRUD endpoints, but machine association management is currently gRPC-only.

Assigning Resources with Allocations

An allocation grants a tenant access to infrastructure at a specific site. Without at least one allocation, a tenant cannot create instances, VPCs, VPC Prefixes, or Subnets. Allocations are provider-side operations requiring the Provider Admin role.

Allocation Model

Each allocation ties together an infrastructure provider, a tenant, and a site, with exactly one allocation constraint that specifies:

The only constraint type currently supported is Reserved, which guarantees the specified capacity. The API validator also accepts OnDemand and Preemptible, but these are not implemented end-to-end.

An allocation starts in Pending status, transitions to Registered once processed, and can move to Error or Deleting. The allocation name must be unique per tenant per site.

Creating a Compute Allocation

Using the TUI (recommended):

nicocli tui
> allocation create

The TUI prompts in this order:

1. Site — select from discovered sites
2. Allocation name — a unique name for this allocation
3. Description — optional
4. Tenant — select from tenant accounts, your own org (shown as “self”), or enter a UUID manually
5. Resource type — InstanceType or IPBlock
6. Instance Type — scoped to the selected site so you only see valid types
7. Constraint type — Reserved
8. Constraint value — machine count (e.g., 8)

Non-interactive:

nicocli allocation create --data-file - <<'EOF'
{
  "name": "acme-gpu-pool",
  "description": "GB200 allocation for Acme Corp",
  "tenantId": "<tenant-uuid>",
  "siteId": "<site-uuid>",
  "allocationConstraints": [
    {
      "resourceType": "InstanceType",
      "resourceTypeId": "<instance-type-uuid>",
      "constraintType": "Reserved",
      "constraintValue": 8
    }
  ]
}
EOF

The system validates that enough machines of the specified instance type are available before accepting the allocation.

Creating a Network Allocation

nicocli allocation create --data-file - <<'EOF'
{
  "name": "acme-network",
  "tenantId": "<tenant-uuid>",
  "siteId": "<site-uuid>",
  "allocationConstraints": [
    {
      "resourceType": "IPBlock",
      "resourceTypeId": "<ip-block-uuid>",
      "constraintType": "Reserved",
      "constraintValue": 24
    }
  ]
}
EOF

A constraintValue of 24 allocates a /24 sub-block (256 addresses). The value must be between 1 and 32 and must be greater than or equal to the parent block’s prefix length.

Listing and Inspecting Allocations

nicocli allocation list --output table
nicocli allocation get <allocation-id>

allocation list supports rich filter flags (verified via --help): --site-id, --tenant-id, --infrastructure-provider-id, --resource-type (InstanceType or IPBlock), --resource-type-id, --status, --constraint-type, --constraint-value. The --query flag is a free-text search over name/description/status, NOT a key-value filter — use the dedicated flags instead.

nicocli allocation list --resource-type InstanceType
nicocli allocation list --site-id <site-uuid> --tenant-id <tenant-uuid>
nicocli allocation list --status Registered --output table

TUI equivalents: allocation list, allocation get. The list-table output prints a one-line pagination summary on stderr (e.g. Page 1/18 (5 items, 88 total). Use --all to fetch everything.); --all follows pagination for you.

Example allocation detail (IPBlock allocation, /28 reservation against a /16 pool):

{
  "allocationConstraints": [{
    "ResourceTypeID": "<ip-block-uuid>",
    "allocationId": "<allocation-uuid>",
    "constraintType": "Reserved",
    "constraintValue": 28,
    "derivedResourceId": "<derived-block-uuid>",
    "id": "<constraint-uuid>",
    "ipBlock": {
      "id": "<ip-block-uuid>",
      "name": "site-ipv4-pool",
      "prefix": "10.99.0.0",
      "prefixLength": 16,
      "routingType": "Public",
      "status": "Ready"
    },
    "resourceType": "IPBlock"
  }],
  "id": "<allocation-uuid>",
  "infrastructureProviderId": "<provider-uuid>",
  "name": "acme-network",
  "siteId": "<site-uuid>",
  "status": "Registered",
  "tenantId": "<tenant-uuid>"
}

derivedResourceId is the ID of the sub-resource carved from the parent (the actual sub-block for IP allocations). ResourceTypeID is mixed-case in the response — field name is what the JSON API returns.

Modifying Allocation Constraints

Adjust an existing constraint value (e.g., increase a machine quota). Note the flag-first ordering with two positionals:

nicocli allocation constraint update --constraint-value 12 <allocation-id> <constraint-id>

Or with --data:

nicocli allocation constraint update --data '{"constraintValue": 12}' <allocation-id> <constraint-id>

The system validates:

• Increases: Enough machines must be available to support the new total.
• Decreases: The new total cannot fall below the tenant’s active instance count for that instance type. If the tenant has 6 running instances and you reduce to 4, the request is rejected.

Deleting an Allocation

nicocli allocation delete <allocation-id>

Deletion is blocked if the tenant has active instances, VPC Prefixes, or Subnets consuming resources from the allocation. Terminate dependent resources first.

Multiple Allocations per Tenant

A tenant can have multiple allocations at the same site with different resource types or instance types. The aggregate compute quota for a given instance type is the sum of all allocation constraints for that type across the tenant’s allocations at that site.

Allocation Workflow Summary

1. Provision the tenant — nicocli tenant current
2. Establish a tenant account — Provider admin links provider to tenant org
3. Discover available resources — List sites, instance types, and IP blocks
4. Create compute allocation(s) — One per instance type the tenant needs
5. Create network allocation(s) — One per IP block the tenant needs
6. Verify — List allocations and confirm they reach Registered status

Creating VPCs and Tenant Networks

After allocations are in place, the tenant can create VPCs and their tenant network resources within the allocated boundaries. The VPC’s networkVirtualizationType determines the resource: FNN VPCs use VPC Prefixes, while ETHERNET_VIRTUALIZER VPCs use IPv4 Subnets.

VPC Creation

A VPC is the logical network container for tenant workloads. It defines the tenant boundary for networking and provides the parent context for VPC Prefixes or Subnets and instances.

nicocli vpc create --help shows these flags:

Realistic non-interactive form:

nicocli vpc create \
  --name acme-prod \
  --site-id <site-uuid> \
  --routing-profile internal \
  --network-virtualization-type FNN

FNN is the supported target for production deployments with DPUs. See VPC Routing Profiles for the support boundary and Flat VPCs for Zero-DPU and NIC-Mode Hosts for the FLAT workflow.

TUI flow (prompts in order):

nicocli tui
> vpc create

1. Site — select the site
2. VPC name — unique name
3. Description — optional

The specialized TUI does not prompt for networkVirtualizationType; it uses the Site default. Use the non-interactive command when the VPC type must be selected explicitly.

Verify the VPC reaches Ready status:

nicocli vpc list --output table

Routing profiles govern which VPCs can exchange routes with which others. The REST API accepts external, internal, and privileged-internal; the underlying gRPC API supports any profile defined under fnn.routing_profiles in the API server config. For details, see VPC Routing Profiles. For the full networking architecture (VRFs, VNI pools, BGP, deny prefixes), see VPC Network Virtualization.

VPC Prefix Creation for FNN

An FNN VPC uses a VPC Prefix carved from a tenant IP Block. Create the VPC Prefix after the parent VPC reaches Ready:

nicocli vpc-prefix create \
  --name acme-prefix-1 \
  --vpc-id <fnn-vpc-uuid> \
  --ip-block-id <ip-block-uuid> \
  --prefix-length 28

For the supported IPv4 creation path, --prefix-length accepts 8 through 31 and must be greater than the selected parent IP Block’s prefix length. REST support for creating IPv6 FNN VPC Prefixes is tracked by #5407.

IPv4 Subnet Creation for ETHERNET_VIRTUALIZER

A Subnet is an IPv4 range within an ETHERNET_VIRTUALIZER VPC, carved from a Ready tenant IPv4 IP Block at the same Site. Subnets do not support IPv6; FNN VPCs use VPC Prefixes instead.

nicocli subnet create --help shows these flags. Note the flag name --ipv4block-id, with no separator between v4 and block:

Non-interactive:

nicocli subnet create \
  --name acme-subnet-1 \
  --vpc-id <vpc-uuid> \
  --ipv4block-id <ip-block-uuid> \
  --prefix-length 28

Or via --data (the JSON body uses camelCase even though the flag is single-word):

nicocli subnet create --data '{"name": "acme-subnet-1", "vpcId": "<vpc-uuid>", "ipv4BlockId": "<ip-block-uuid>", "prefixLength": 28}'

TUI flow:

nicocli tui
> subnet create

1. Ready Ethernet virtualizer VPC — select the parent VPC
2. Subnet name — unique name
3. Description — optional
4. IPv4 prefix length — 8 through 30
5. Tenant IPv4 Block — select a Ready tenant IPv4 IP Block at the VPC’s Site

Verify:

nicocli subnet list --output table

Launching an Instance

An instance in NICo is a bare-metal machine assigned to a tenant within a VPC. Creating an instance claims a machine from the tenant’s compute allocation, associates it with a VPC, and triggers the provisioning workflow (OS installation, network configuration, security lockdown).

nicocli instance create --help shows these flags:

interfaces[] and sshKeyGroupIds[] are array-typed and must go through --data / --data-file.

Non-interactive form for an FNN VPC (with one interface and one SSH key group):

nicocli instance create --data-file - <<'EOF'
{
  "name": "acme-worker-01",
  "tenantId": "<tenant-uuid>",
  "vpcId": "<vpc-uuid>",
  "instanceTypeId": "<instance-type-uuid>",
  "interfaces": [
    {"vpcPrefixId": "<vpc-prefix-uuid>"}
  ],
  "sshKeyGroupIds": ["<ssh-key-group-uuid>"]
}
EOF

If you want to target a specific machine instead, replace instanceTypeId with machineId. Machine targeting requires the tenant’s account with the site’s provider to be Ready and to have the capability enabled for that site. Refer to Granting Targeted Instance Creation.

TUI flow:

nicocli tui
> instance create

1. VPC — select the target VPC
2. Machine — select from machines in Ready state at the VPC’s site
3. Instance name — unique name
4. Operating system — optional
5. Network resource — select a VPC Prefix for FNN or a Subnet for ETHERNET_VIRTUALIZER (loops, can add more). FLAT VPCs attach the network automatically and skip this prompt.
6. SSH key groups — optional, attaches SSH keys for serial-console access

For an image-based operating system, review Image-Based Operating Systems before making the definition available to tenants. Its disk selector applies to every eligible target and the selected whole disk is overwritten during installation.

Verifying an Instance Is Running

After creation, the instance goes through these states:

Pending -> Ready -> BootCompleted          (initial provisioning, OS up + phone-home)
Pending -> Ready                            (initial provisioning, no phone-home)
Ready -> Configuring -> Ready               (in-place reconfigure: NSG, SSH keys, OS, iPXE, user-data)
Ready -> Rebooting -> Ready -> BootCompleted (during reboot)
Error                                       (if provisioning fails)

Two states are easy to miss:

• BootCompleted — follows Ready once the OS phones home. Instances without phone-home enabled stop at Ready.
• Configuring — the instance transitions through this state when its config is being updated in place (e.g. attaching a new Network Security Group, rotating SSH key groups, swapping the OS image, or pushing new user-data). Configuring does NOT trigger a reboot on its own; the instance returns to Ready once the change is applied. A reboot caused by --trigger-reboot=true goes through Rebooting, not Configuring.

Monitor progress:

nicocli instance list --output table
nicocli instance get <instance-id>
nicocli instance status-history <instance-id>

The instance detail response is rich — it includes interfaces[] with assigned IP addresses and VPC prefix info, ipxeScript showing the live boot script, serialConsoleUrl for console access, full machine and SKU metadata, and any active deprecations[] warnings. The API uses inline deprecations[] arrays to flag fields scheduled for removal — watch for these in your responses.

Phone-home

phoneHomeEnabled (--phone-home-enabled) controls whether NICo waits for the booted operating system to call back — to “phone home” — before it reports the instance as ready. It is a property of the operating-system definition (operating-system create / operating-system update) and can also be set per instance (instance create / instance update).

What it does. When phone-home is enabled, NICo does not consider the instance fully provisioned until the booted OS contacts NICo’s metadata service from inside the guest. The instance is held in a provisioning state — the transition to Ready is gated — until that callback arrives. When it is disabled, NICo reports the instance ready as soon as provisioning and config sync finish, without waiting for any signal from the OS.

What NICo injects. When phone-home is enabled, NICo edits your userData. You do not add the callback yourself. NICo parses your cloud-init YAML, removes the existing phone_home blocks that it can edit, and inserts one that sends a POST request to the site’s metadata endpoint:

#cloud-config
# ... your first-boot setup (SSH keys, passwords, packages, ...) ...
phone_home:
  post: all
  url: http://169.254.169.254/latest/meta-data/phone_home

The injected url is the site-configured phone-home endpoint in site.phoneHomeUrl. The default is http://169.254.169.254/latest/meta-data/phone_home. The platform operator controls this site-wide value; it is not a per-instance transport selection. If you supply no userData when enabling phone-home, NICo generates a minimal #cloud-config that contains only the block above.

For direct access to the embedded agent’s default listener, set the URL to http://169.254.169.254:7777/latest/meta-data/phone_home, or http://[fd00:76b9::254]:7777/latest/meta-data/phone_home when IPv6 metadata is enabled. Standalone FMDS defaults to port 80; both modes support listener overrides. A site mixing deployment modes must provide one URL reachable by all affected instances, accounting for any deployment-specific proxy or port forwarding. NICo does not choose the URL’s address family or port separately for each DPU.

When you disable phone-home, the removal behavior depends on whether the request supplies userData. If it does, NICo removes only a phone_home block that reports to the site endpoint and leaves blocks that report elsewhere unchanged. If the request omits userData, NICo edits the stored blob. If NICo stored that blob with phone-home enabled, it removes every phone_home block. A stored block can contain a URL from before a change to site.phoneHomeUrl.

Which user data NICo can edit. When phone-home is enabled, the userData you provide must be one of these formats:

• A #cloud-config mapping.
• A #cloud-config-archive sequence.

The API rejects these inputs:

• A #! script.
• A ## template: jinja document, because NICo cannot render the template back to text.
• Another cloud-init format, including #cloud-boothook or #cloud-config-jsonp.
• More than one YAML document.
• Text that is not YAML.

NICo treats user data without a format marker as #cloud-config, including user data with only a comment above the keys. Enabling phone-home writes the marker at the top so that cloud-init reads the document. When you disable phone-home for user data that NICo cannot edit, NICo leaves the document unchanged instead of rejecting the request. Any phone_home block in that document remains.

Where the block lands. In a #cloud-config document, NICo adds the block at the top level. If the document contains an autoinstall section, NICo adds the block under autoinstall.user-data. This placement makes the callback come from the installed system instead of the installer.

In a #cloud-config-archive, NICo adds the block to the part that installs a target system, again under autoinstall.user-data. Cloud-init determines a part’s format from its type without reading the content. If type is absent, cloud-init uses the content’s first line. If neither value identifies a format, cloud-init treats the part as cloud-config. The first-line rule above therefore applies to the document you send, not to the parts inside it.

NICo leaves a part whose content is a ## template: jinja document unchanged because it cannot render the template back to text. If no part installs a target system, NICo appends a #cloud-config part that contains only the block. NICo does the same when the installation part is a template that it leaves unchanged. In both cases, NICo leaves your parts unchanged:

#cloud-config-archive
- type: text/x-shellscript
  content: |
    #!/bin/sh
    # ... your part, untouched ...
- type: text/cloud-config
  content: |
    #cloud-config
    phone_home:
      post: all
      url: http://169.254.169.254/latest/meta-data/phone_home

cloud-init runs the phone_home module in its final stage, after the rest of your configuration has been applied, so the callback fires only after your setup has completed. When NICo receives the POST, it records the contact and releases the readiness gate, allowing the instance to become Ready.

When to use it. Enable phone-home when “ready” must mean “the guest has finished its own first-boot setup” — for example, to delay readiness until cloud-init has applied SSH keys and passwords, so that automation watching for Ready does not connect before the host is fully configured. Leave phone-home disabled when NICo finishing provisioning is a sufficient ready signal and you do not need a callback from inside the guest.

Notes and caveats.

• Phone-home only gates status reporting. It does not change how the host is provisioned or booted — the reboot into the provisioned OS happens identically whether or not phone-home is enabled.
• If phone-home is enabled but the booted OS never runs cloud-init, or the guest cannot reach the metadata endpoint, the callback never arrives and the instance stays in its provisioning state, never reporting ready.
• The callback uses the operator-configured site.phoneHomeUrl, which defaults to the DPU-local IPv4 metadata endpoint. Enabling IPv6 metadata transport does not change this setting. Before changing the site-wide URL, the platform operator must confirm that every affected phone-home-enabled guest can reach the selected address and deployment-specific port. Refer to IPv6 Metadata Transport. Changing the site setting does not rewrite existing stored user data automatically.
• Rebooting the instance with a one-time custom iPXE override (instance update --reboot-with-custom-ipxe=true) re-arms the gate when phone-home is enabled: NICo clears the recorded contact, so the OS must phone home again before the instance is reported ready.

Batch Instance Creation

For creating multiple identical instances at once, use instance batch-create. Unlike create, batch-create takes a single shared spec plus a count — it provisions N instances with auto-generated names from the same instance type, tenant, and VPC. nicocli instance batch-create --help shows:

Non-interactive form:

nicocli instance batch-create \
  --tenant-id <tenant-uuid> \
  --vpc-id <vpc-uuid> \
  --instance-type-id <instance-type-uuid> \
  --name-prefix acme-worker \
  --count 4 \
  --topology-optimized=true

interfaces[] and sshKeyGroupIds[] are still array-typed and must go through --data / --data-file if you need to attach them at create time.

For batches where each instance needs a different machine ID, OS, or interface set, call instance create in a loop instead — batch-create only handles the homogeneous case.

Instance Power Management

NICo provides instance-level power management through the REST API. These operations send commands to the underlying BMC via Redfish.

Rebooting an Instance

nicocli instance update --help exposes individual flags for every common operation — prefer them over --data:

Reboot:

nicocli instance update --trigger-reboot=true <instance-id>

Reboot with re-provisioning iPXE and pending updates:

nicocli instance update --trigger-reboot=true --reboot-with-custom-ipxe=true --apply-updates-on-reboot=true <instance-id>

TUI:

nicocli tui
> instance reboot

The TUI prompts for instance, custom-iPXE flag, apply-updates flag, and a confirmation.

Renaming or Updating an Instance

nicocli instance update --name acme-worker-01-renamed <instance-id>
nicocli instance update --description "production worker" <instance-id>

sshKeyGroupIds[] is an array, so changes go through the body:

nicocli instance update --data '{"sshKeyGroupIds": ["<group-uuid-1>", "<group-uuid-2>"]}' <instance-id>

Deleting (Terminating) an Instance

nicocli instance delete <instance-id>

In TUI mode, instance delete prompts for confirmation before proceeding. Deletion triggers the full sanitization workflow: secure erase of NVMe storage, GPU and system memory wipe, TPM reset, re-attestation, and network isolation teardown. The machine returns to the available pool once sanitization completes.

Machine-Level Emergency Operations (gRPC Only)

If a machine is stuck or unresponsive, operators can use nico-admin-cli for recovery. Choose the operation that matches the problem:

• To restart the machine, refer to the machine reboot reference.
• To remove its NICo records and restart discovery, follow Force Deleting and Rebuilding NICo Hosts. Force deletion does not wipe the tenant operating system or its data. The playbook explains deletion consent and the optional wait for DPU confirmation.

Tenant Lifecycle Operations

Viewing the Current Tenant

nicocli tenant current

For provider admins needing visibility across tenants, list tenant accounts (a filter flag is required):

nicocli tenant-account list --infrastructure-provider-id <provider-uuid> --output table

Monitoring Tenant Health

nicocli tenant stats

Non-zero error counts warrant investigation:

nicocli instance list --status error --output table

Provider admins can get cross-tenant compute allocation stats at a site. instance-type-stats is a sub-resource of tenant, with a stats leaf action — the full command has three tokens:

nicocli tenant instance-type-stats stats --site-id <site-uuid>

A bare nicocli tenant instance-type-stats --site-id <id> returns flag provided but not defined: -site-id — the trailing stats is required.

Disabling a Tenant

NICo has no first-class “disable” operation. Options:

• Revoke identity provider roles: Remove TENANT_ADMIN from all users. Existing resources remain but cannot be managed.
• Remove allocations: Delete all allocations. Existing instances continue running but no new ones can be created.
• Delete the tenant account: Sever the provider-tenant relationship entirely (requires all allocations deleted first).

Tenant Teardown Sequence

There is no DELETE /tenant endpoint — tenant records are permanent. For peerings with another tenant, arrange deletion with the Site’s Provider Admin.

To fully decommission:

1. Terminate all instances — delete every instance; each must reach Terminated status.
2. Delete all tenant network resources — remove VPC Prefixes or Subnets from every VPC, according to its networkVirtualizationType.
3. Delete all VPC peerings — follow the peering deletion guide and wait for each connection to disappear.
4. Delete all VPCs — remove the tenant’s VPCs.
5. Delete all allocations — provider admin removes compute and network allocations.
6. Delete the tenant account — provider admin severs the link.
7. Revoke identity provider access — remove roles and optionally org membership.

After this sequence, the tenant record still exists but is inert.

This teardown is destructive and irreversible at the resource level. Terminated instances cannot be recovered. Always confirm with the tenant team before beginning.

End-to-End Walkthrough

This section ties together the full Day One workflow. The TUI flow is the recommended path for first-time operators — it scopes lookups (instance types to sites, VPC prefixes to VPCs, etc.) and you can’t easily get the order wrong.

Step 1: Provision the Tenant (Tenant Admin)

nicocli tenant current

Idempotent — creates the tenant lazily on first call.

Step 2: Establish Tenant Account (Provider Admin)

nicocli tenant-account create --infrastructure-provider-id <provider-uuid> --tenant-org <tenant-org>

Or via TUI: tenant-account create.

Step 3: Accept Tenant Account (Tenant Admin)

# Find the invitation
nicocli tenant-account list --tenant-id <tenant-uuid>
# Accept it
nicocli tenant-account update --data '{}' <account-id>

Step 4: Create Compute Allocation (Provider Admin)

Use the TUI for the first one — it filters instance types by the selected site and validates capacity:

nicocli tui
> allocation create
# Site -> name -> tenant -> InstanceType -> select type -> Reserved -> machine count

Step 5: Create Network Allocation (Provider Admin)

nicocli tui
> allocation create
# Same site -> name -> tenant -> IPBlock -> select block -> Reserved -> prefix length

Step 6: Verify Allocations

nicocli allocation list --output table --tenant-id <tenant-uuid>

All allocations should show Registered status.

Step 7: Create an FNN VPC (Tenant Admin)

nicocli vpc create --name <n> --site-id <site-uuid> --routing-profile internal --network-virtualization-type FNN

Step 8: Create a VPC Prefix (Tenant Admin)

nicocli vpc-prefix create --name <n> --vpc-id <vpc-uuid> --ip-block-id <ip-block-uuid> --prefix-length 28

Step 9: Launch an Instance (Tenant Admin)

The first instance is easiest via TUI because interfaces[] is array-typed:

nicocli tui
> instance create
# VPC -> Machine -> name -> OS (optional) -> VPC prefix -> SSH key groups (optional)

For automation, use --data-file — see the Launching an Instance section above.

Step 10: Verify

nicocli tenant stats
nicocli instance list --output table
nicocli instance status-history <instance-id>

The instance should reach Ready (or BootCompleted if phoneHomeEnabled: true). Tenant stats should now show non-zero counts for instance and vpc.

Troubleshooting

Common Issues

Debugging with nicocli

Use --debug on any command to see the full HTTP request and response. The token is redacted in the log; the path-rewriting from nico to whatever api.name is set to is visible. Real output:

$ nicocli --debug tenant current
time=... msg="API request: GET http://<api>/v2/org/<org>/<api-name>/tenant/current"
time=... msg="Request headers: {\"Accept\":[\"application/json\"],\"Authorization\":[\"Bearer <redacted>\"]}"
time=... msg="API response: ... -> 200 OK"
time=... msg="Response body: {...}"

Version mismatch is normal

The CLI version and the API server version are independent. CLI is generated from the OpenAPI spec at the time of build; the server reports its own image version (visible in audit response apiVersion field). Different versions are expected — the wire protocol is stable enough that mismatches rarely matter.

Audit log

The list view is intentionally lightweight — each entry has id, endpoint, method, statusCode, userId, clientIP, apiVersion, and timestamp. To see the full request, including the request body and the resolved user object, fetch a single entry:

nicocli audit list --output json --page-size 5      # find entries of interest
nicocli audit get <audit-id>                         # fetch full detail

audit get adds body, queryParams, extraData, durationMs, statusMessage, and the resolved user (with email and name when the caller was a human, blanks when the caller was a service account). The table form of audit list only shows id — always use --output json (or audit get) for anything more than ID discovery.

Using the TUI for Exploration

The TUI is the recommended tool for exploratory work. It handles config selection, authentication, and provides tab-complete interactive commands:

nicocli tui

The TUI discovers all config*.yaml files in ~/.nico/ and lets you pick an environment at startup. Commands like allocation create scope lookups to the relevant site automatically, reducing the chance of selecting a resource from the wrong site.

Quick Reference

Flag-first ordering — always put flags before positional args.

Related Documentation

• Authentication and Authorization, Day 0 auth configuration: issuers and keycloak modes, claim mappings, validation rules
• Tenant Management with Keycloak, realm-side steps for onboarding a tenant on Keycloak deployments
• Network Isolation, per-plane tenant isolation (Ethernet, InfiniBand, NVLink)
• Templated iPXE Operating Systems, reusable template-based boot definitions and Site synchronization
• Organization & Permissions, IdP-managed roles and user setup
• Quick Start Guide, NICo deployment and Day Zero walkthrough
• VPC Routing Profiles, profile configuration and behavior
• VPC Network Virtualization, full networking architecture
• VPC Peering, connecting VPCs and waiting for peering deletion
• NVLink Partitioning, NVLink logical partition management
• Machine Reboot Playbook, emergency BMC reboot procedures
• Force Delete Playbook, removing stuck machines
• Day 0/1/2 Lifecycle, NICo lifecycle model overview