Plan your installation

Preview: NVKE is available as a preview. APIs, configuration and behavior can change between releases, and we do not recommend NVKE for production workloads. See Preview status and known limitations.

This page describes what you need before you install NVIDIA Kubernetes Engine (NVKE) on your own hardware, which settings cannot change later, and which values to collect. Read it before every other installation page.

Before you begin

• You have read What is NVKE.
• You have checked your hardware and versions against Supported platforms and versions.
• You have read Preview status and known limitations. Two limitations shape the plan: the management cluster does not upgrade, scale or replace its own nodes, and NVIDIA Infra Controller (NICo) is the default infrastructure provider, with LaunchLayer available only once a deployment enables it.

What you end up with

A management cluster with an odd number of nodes, each a control-plane node that also runs the NVKE management plane. The management plane requests machines from NICo through CAPNICo, the Cluster API infrastructure provider for NICo, and forms workload clusters on them for your customers. You get there in seven steps:

What you need before you start

Machines and storage

• Management cluster nodes. An odd number of machines, such as one, three or five, each with a TPM 2.0 device. Your provisioning system boots each from the node image with cloud-init user data. The node image uses DHCP, so reserve each node’s address. Each node has more memory than the kubelet reserves for the system, 16 GiB by default.
• A storage provisioner for the management cluster’s default StorageClass. The bootstrap installs none, and the management plane’s database claims a 40 GiB volume.
• Workload cluster machines enrolled in NICo, which provisions them from your node image.

NICo

• A site with machines enrolled.
• A virtual private cloud (VPC) in that site, with VPC prefixes or subnets for the nodes.
• One or more instance types, each with an allocation for your organization. An instance type with no allocation provisions nothing.
• An SSH key group, if you want SSH access to nodes.
• The NICo API endpoint, your organization ID, an API token and the API name your NICo deployment serves.

Network

• A Kubernetes API virtual IP address (VIP) for the management cluster when it has more than one node, and for each workload cluster with more than one control-plane node. Each VIP is an unused, routable IPv4 address outside the cluster’s pod and service networks. A single-node management cluster uses its node’s IP instead.
• Border Gateway Protocol (BGP) peering for each VIP. Every control-plane node advertises the VIP from local autonomous system (AS) 65100 to the node’s default gateway, with peer AS 4294967000. The local AS is fixed. The peer AS has two exceptions, both for workload clusters: a control plane that uses VPC prefix IDs reads its peer AS from NICo’s metadata service on each node, and setting KUBE_VIP_FMDS_ENDPOINT in the management plane’s env makes every workload cluster read it from that URL. Your network must accept the route and carry it to the cluster’s API callers.
• Pod and service networks for each cluster that do not overlap each other or your node network.
• Outbound access from the bootstrap host and the management cluster to the registries and chart repositories listed in Run NVKE end to end on one machine.
• A boot-artifact HTTP(S) endpoint reachable from the machines’ provisioning network before the installed OS starts. For QCOW2 imaging, it serves the EFI imager and QCOW2; for an ISO install, it serves the kernel, initrd and ISO. See Serve the boot artifacts over HTTP(S) for hosting, an S3 example and download checks.

Ports

All connections are TCP; traffic between the nodes of one cluster is not listed. The management address is the management cluster’s API VIP, or the node IP of a single-node management cluster.

Identity and trust

• A SPIFFE trust domain, such as mgmt.example.com.
• Your own SPIRE certificate authority (CA): a certificate and key as files. The certificate goes into every management node’s user data; both files go on the bootstrap host. Management nodes built from the node image cannot attest without it. The install also makes it the upstream authority of the management cluster’s SPIRE server, which workload cluster nodes trust through the bundle NVKE puts in their user data.

Registries

• A container registry you control, only if you supply the controller or profile content image from one, with an account that can push to it. If pulling needs credentials, the management cluster needs them too.
• CAPNICo’s published image is public on ghcr.io, so it needs no credentials.

Hosts and tools

Identity provider

API authentication is off by default: any caller that reaches port 30080 can call every endpoint. To secure the API, you need an OpenID Connect (OIDC) identity provider that:

• Publishes an OIDC discovery document over HTTPS, with a signing-key endpoint the management cluster can reach.
• Signs tokens with an asymmetric algorithm, such as RS256 or ES256.
• Puts group membership in a token claim named groups.
• Has an OIDC client for nvkectl that needs no client secret.

NVKE admits callers by group but does no authorization: every admitted caller can call every endpoint.

Decide before you install

Fill in your values

Each placeholder has the same name on the page that uses it. Keep credentials in your secret store, not in this list.

Management cluster

Registries and images

NICo

Management plane and identity provider

Each workload cluster

The workload cluster pages name each cluster <WORKLOAD_CLUSTER>. Create a workload cluster reuses <K8S_VERSION> and <API_VIP>; record them separately from the management cluster’s.

Verify

Your plan is complete when every row in Fill in your values that applies to you has a value, every connection in Ports is open, and you have decided each setting in Decide before you install.

Next steps

• Build and host node OS images.