VNI Resource Pools

This page explains how VNI resource pools work in NICo, which named pools exist, how they are configured, how to size them correctly, and how to inspect and extend them at runtime.

This page is intended for operations engineers who are deploying or growing a site and need to configure the API server’s resource pools correctly.

Related documentation

• docs/manuals/vpc/vpc_routing_profiles.md — how the internal flag on a routing profile determines which VNI pool is used
• docs/getting-started/prerequisites/network.md — site-wide networking prerequisites, including general VNI and ASN allocation guidance
• docs/manuals/networking/ip_resource_pools.md — IP resource pool configuration
• docs/manuals/vpc/vpc_network_virtualization.md — end-to-end VPC network virtualization overview that ties VNI pools, IP pools, and routing profiles together

What VNIs are used for

Each VPC is assigned one VNI from a resource pool at creation time. That VNI serves two purposes.

VXLAN tunnel identifier. The VNI identifies the L3VPN VXLAN tunnel associated with the VPC’s overlay network. All traffic within the VPC is encapsulated using this identifier.

Native BGP EVPN route-target. The VPC’s VNI is used to construct the VPC’s native route-target in the form <datacenter_asn>:<vpc_vni>. This route-target governs which routes are imported and exported for the VPC through BGP EVPN. This is the mechanism by which routing profiles and route-target configuration take effect.

Because the VNI feeds directly into the route-target, VNI ranges must be coordinated with the network team before a site goes live. VNI ranges for internal and external VPCs must be distinct from each other and from any other pool on the site.

A supported FNN VPC can change to a routing profile that uses the other VNI pool. Core selects a destination VNI and retains the previous allocation until the operator explicitly releases it after verifying convergence. Refer to Changing a VPC Routing Profile for prerequisites and the procedure. Core blocks VPC deletion while an inactive allocation remains retained.

Named pools

VPC pools

These two pools cover the VPCs. The selection between them is made automatically at VPC creation time based on whether the resolved routing profile has internal = true or internal = false. See docs/manuals/vpc/vpc_routing_profiles.md for how the routing profile is resolved.

Pool selection is automatic: if the resolved routing profile has internal = true, the VNI is allocated from vpc-vni; otherwise it is allocated from external-vpc-vni.

The external-vpc-vni pool is optional. If it is absent from the configuration and no external VPCs are created, the API server will start normally. However, if any VPC creation request resolves to an external routing profile and the pool is not defined, the request will fail with a resource-exhausted error.

Additional pools

The following pools are used for other allocation types. They are not covered in detail on this page but are listed here for completeness.

Pool configuration

Pools are defined in the API server TOML configuration file under [pools.<name>].

Schema

[pools.<name>]
type = "integer"          # required; must be "integer" for VNI pools
ranges = [
  { start = "<value>", end = "<value>" },
  { start = "<value>", end = "<value>", auto_assign = false },
]

Each pool entry accepts the following fields.

type — required. For VNI pools, this must be "integer". The ipv4 pool type is used for IP address pools and is not relevant here.

ranges — a list of range objects. Each range has:

• start — the first integer value in the range (inclusive).
• end — the last integer value in the range (exclusive). The value at end is not included in the pool.
• auto_assign — optional boolean, defaults to true. When true, values in this range are eligible for automatic allocation. When false, values are reserved for explicit requests only. During a VPC routing-profile change, an exact VNI request can select a free entry with either auto_assign setting. Automatic selection uses only automatically assignable entries unless the VPC already owns a retained destination allocation, which it reuses.

Multiple ranges may be provided for a single pool. The API server treats them as a single logical pool. Ranges within a pool must not overlap, and a pool’s ranges must not overlap with the ranges of any other pool.

Example configuration

The following is a representative configuration for a site that uses both internal and external VPCs.

[pools.vpc-vni]
type = "integer"
ranges = [{ start = "2024500", end = "2025500" }]
[pools.external-vpc-vni]
type = "integer"
ranges = [{ start = "3024500", end = "3025500" }]

This example defines 1,000 VNIs in each pool. Adjust the ranges to match your site’s planned VPC counts plus appropriate headroom.

For development and test environments, much smaller ranges are sufficient:

[pools.vpc-vni]
type = "integer"
ranges = [{ start = "2024500", end = "2024550" }]

Sizing pools

Each VPC normally consumes one active VNI. A VPC with an unreleased routing-profile transition consumes two allocations, one in each pool. Size each pool for its active VPCs, retained allocations from VPCs that changed to the other pool, and headroom for ordinary creation and planned changes.

Use the following approach to determine the required pool size for each pool.

1. Estimate the maximum number of simultaneously active VPCs of each type (internal or external).
2. Add capacity for retained transition allocations and planned changes, then headroom for burst creation.
3. Coordinate the resulting ranges with the network team. VNI values map directly to BGP route-targets, so the network team must configure import and export policies that reference the same ranges you define in the pool.

The Network Prerequisites describe the baseline of one VNI per expected VPC. Retained transition allocations require additional capacity in these pools.

Pool exhaustion. When a pool is exhausted, VPC creation requests that would draw from that pool fail immediately with a resource-exhausted error. No partial allocations occur. Recover by growing the pool or freeing allocations through normal VPC deletion or verified inactive-VNI release, then retry creation. Do not release an unverified inactive VNI to make space.

The API server enforces a maximum pool size of 250,000 values per pool. VNI pools are 24-bit integers, so the theoretical maximum VNI value is 16,777,215, but the enforced maximum pool size is 250,000 values per pool regardless of the value range.

Startup behavior

At startup, the API server reads the [pools.*] entries from the configuration file and writes or updates the pool definitions in the database.

The behavior when a pool already exists in the database is additive: existing entries are not removed or changed, and new values from the configuration are inserted. This means:

• Adding a new range to an existing pool is safe. The new values are added to the database on the next restart.
• Removing a range from the configuration does not remove values from the database. Previously defined values remain allocated or available in the pool.
• Shrinking a range (moving start up or end down) has no effect on values already in the database.

The practical consequence is that pools can only grow, never shrink, through configuration changes alone. To reduce a pool, a manual database operation would be required. This is intentional: it prevents accidental deallocation of VNIs that may be in active use.

When listen_only = true is set in the configuration, the API server skips registration of the configured pool inventory at startup. It uses existing pools in the database, assuming another instance has already populated them. Pool changes in this mode must be applied using the nico-admin-cli resource-pool grow command described below.

Runtime operations

Listing Pools

Run the following command to inspect resource pools:

nico-admin-cli resource-pool list

In v2.4, nico-api fails to start if a pool selects a gRPC backend. Integrated pools report known counts. The remote statuses below describe the inspection API for internal testing; gRPC lifecycle integration is tracked separately.

The pool inspection API lists pools that have local inventory or a saved record of which backend they use, ordered by NICo pool name. When connected to an older server without inspection support, the CLI falls back to local inventory only. That view labels pools Integrated and Known, omits saved backend names, and cannot show a pool with no inventory rows.

The resource pool table contains the following columns:

Use Status to interpret the counts:

• Known: NICo obtained valid counts. An empty pool has Size and Used equal to 0. There is no percentage because total capacity is zero.
• Unknown: The Stats request returned Unimplemented. The backend might not support statistics, or the endpoint might not serve the expected RPC. Both count cells display Unknown.
• Unavailable: NICo could not read or validate the statistics, for example because no accepted pool definition exists or the provider cannot be reached. Both count cells display Unavailable.

Unknown and unavailable counts have no percentage because total capacity is unknown. When a diagnostic is available, the command writes it to standard error as <pool-name>: <message>. A failed statistics observation for one pool does not prevent other rows from being displayed or make the command exit unsuccessfully. A failure of the listing request itself does make the command fail.

Min and Max are blank for an empty Integrated pool or a remote pool. Remote allocation ranges do not describe observed inventory bounds. Remote counts apply to the pool definition the provider accepted. Size and Used include both automatic and manual allocation values: a pool can still have free manual values after its automatic values are used up.

This excerpt from a captured test run shows a pool named local with 2 of its 10 values allocated:

+-------------+---------------+-------------+-----+-----+-------------+-------------+-------------+
| Name        | Backend       | Remote Pool | Min | Max | Size        | Used        | Status      |
+=============+===============+=============+=====+=====+=============+=============+=============+
| local       | Integrated    | local       | 1   | 10  | 10          | 2 (20%)     | Known       |
+-------------+---------------+-------------+-----+-----+-------------+-------------+-------------+

Monitor Used when Status is Known, and grow the required allocation range before it is exhausted. VPC creation fails when the selected pool has no eligible free value. If no pools are returned, the command prints No resource pools defined and exits unsuccessfully.

Growing a pool

To add capacity to an existing pool at runtime without restarting the API server:

nico-admin-cli resource-pool grow -f <toml-file>

The argument to -f is the path to a TOML file containing the updated pool definition.

Constraints

• Ranges can only be extended, not reduced. Attempting to remove values that are already in the database has no effect. The grow operation is strictly additive.
• New ranges must not overlap existing pool ranges. Adding a range whose values are already present in the pool for a different pool name will result in a conflict error.
• The pool name must already exist in the database. The grow operation adds values to an existing pool; it cannot create a new pool name that was never defined in the configuration.

Example

To add 500 more VNIs to the internal VPC pool, create a TOML file containing the new range:

[pools.vpc-vni]
type = "integer"
ranges = [{ start = "2025500", end = "2026000" }]

Then run:

nico-admin-cli resource-pool grow -f grow-vpc-vni.toml

The server will insert values 2025500 through 2025999 into the vpc-vni pool. The previously defined range (2024500–2025499) is not affected.

After the grow operation completes, run nico-admin-cli resource-pool list to confirm the new size is reflected in the output.

Network team coordination required. Before growing a VNI pool, confirm with the network team that the new VNI range is covered by their route-target import and export policies. VNI values that fall outside the network’s configured policy range will result in VPCs that have no external connectivity even though they are created successfully by the API.

Relationship to routing profiles

The routing profile assigned to a VPC determines which pool its VNI is drawn from. Specifically, the internal field of the resolved routing profile controls pool selection.

• If internal = true, the VNI is allocated from vpc-vni.
• If internal = false, the VNI is allocated from external-vpc-vni.

This means that the VNI pool ranges implicitly define the route-target range for each profile class. Internal VPCs will have native route-targets in the form <asn>:<value-from-vpc-vni>. External VPCs will have native route-targets in the form <asn>:<value-from-external-vpc-vni>. The network team must configure their EVPN policies accordingly.

For supported FNN VPCs, Changing a VPC Routing Profile updates the active profile and VNI without deleting the VPC. The old VNI remains allocated until the operator verifies every affected consumer and explicitly releases it. Retention permits conditional reuse of that VNI; it does not guarantee that a reverse change passes current authorization and configuration checks.

For full details on how routing profiles are configured and resolved, see docs/manuals/vpc/vpc_routing_profiles.md.

Simplifying network team coordination

A VPC’s native route-target (<datacenter_asn>:<vpc_vni>) is unique per VPC. Network devices that need to import routes from multiple VPCs must either track each VNI individually or import a covering aggregate. Either approach requires ongoing coordination as VPCs are created and destroyed.

route_targets_on_exports in the routing profile avoids this. Route-targets listed there are applied to every route the VPC advertises, in addition to the native VNI-based tag. Because all VPCs sharing the same routing profile carry the same additional export tags, the network team can configure a single import policy that covers all VPCs in that profile class — regardless of how many VPCs exist or what their individual VNIs are. When a new VPC is created under the same profile, its routes are immediately visible to the existing network policy without any reconfiguration.

A conventional deployment adds a shared export tag to the INTERNAL profile (for example, { asn = <datacenter_asn>, vni = 50200 }) and configures network devices to import :50200. All internal VPC routes then become reachable from the fabric under that single import, without tracking individual VNIs.

See VPC Routing Profiles for the route_targets_on_exports field reference.

Troubleshooting

VPC creation fails with a resource-exhausted error

The selected pool has no eligible automatic VNI available. The VNI pool selected depends on the routing profile of the VPC being created.

1. Run nico-admin-cli resource-pool list and check Status. For Known counts, Used equal to Size confirms there are no free values, including when both are zero. A lower Used count does not rule out automatic allocation exhaustion: the remaining values may be manual-only. Check that the range being grown allows automatic allocation (auto_assign = true, the default).
2. Coordinate a new range with the network team.
3. Run nico-admin-cli resource-pool grow -f <file> with the new range.
4. Retry the VPC creation.

If the pool appears to have free capacity but creation is still failing, verify that the routing profile is being resolved as expected. If the VPC’s routing profile resolves to internal = false but only the vpc-vni pool was grown, the growth will not help.

The external-vpc-vni pool is missing or full at startup

The external-vpc-vni pool is optional. If the site has no external VPCs, the pool does not need to be defined. However, if the API server is started without this pool and a request arrives for an external VPC, the request will fail.

If you need to add this pool after the site is already running, add the definition to the configuration file and restart the server, or use nico-admin-cli resource-pool grow -f <file> to populate the pool without a restart.

Pool size appears smaller than expected after a config change

Reducing or removing a range in the configuration file has no effect at startup. Only new values are inserted. Verify the effective pool state with nico-admin-cli resource-pool list, not the configuration file alone.