Skip to navigation
Operations (Day 2)Managing VPCs

VPC Peering

View as Markdown

VPC peering connects two VPCs so their instances can communicate in both directions. This page explains how to manage peering with nico-admin-cli and delete connections through REST.

Virtualized-to-Flat Routing Prerequisite

An active Virtualized-to-Flat peering imports the Flat VPC’s native route target into the FNN VRF. It does not create or advertise a route for the Flat VPC’s prefixes.

Prepare routing before any of these operations:

  • Upgrading an agent that serves an active Virtualized-to-Flat peering to a release that uses null-route isolation.
  • Creating a new Virtualized-to-Flat peering on an agent that uses null-route isolation.
  • Activating a stored Virtualized-to-Flat peering by enabling vpc_peering_policy_on_existing.

The prerequisite applies when both of these conditions hold:

  • vpc_isolation_behavior is mutual_isolation, and an affected Flat prefix is contained by an effective site_fabric_null_routes prefix.
  • The tenant VRF reaches that Flat prefix only through a less-specific underlay route, such as a leaked default route.

Before creating, activating, or upgrading the affected peering, prepare a route for every affected Flat prefix by using one of these methods:

  • Have the fabric advertise the Flat route through EVPN with the Flat VPC’s native route target, <datacenter_asn>:<flat_vni>. The FNN peering import admits that route into the FNN VRF after the peering exists and its activation policy permits the import.
  • Ensure the Flat route is present in the underlay/default VRF and explicitly admit it through the tenant VPC’s effective accepted_leaks_from_underlay list.

The imported route must be at least as specific as the containing FNN blackhole. A leaked default route is insufficient when the blackhole is more specific than /0, because longest-prefix selection chooses the blackhole before administrative distance is compared. Do not combine an effective 0.0.0.0/0 or ::/0 null route with leak_default_route_from_underlay = true for the same address family. Core accepts this unsupported combination, but the leaked default has a better administrative distance than the equal-prefix distance-250 blackhole and overrides it.

VPC Peering Commands

The nico-admin-cli vpc-peering command provides three main operations:

nico-admin-cli vpc-peering <COMMAND>
Commands:
  create  Create VPC peering connection
  show    Show list of VPC peering connections
  delete  Delete VPC peering connection

Creating VPC Peering Connections

To create a new VPC peering connection between two VPCs:

nico-admin-cli vpc-peering create <VPC1_ID> <VPC2_ID>

Example:

nico-admin-cli vpc-peering create e65a9d69-39d2-4872-a53e-e5cb87c84e75 366de82e-1113-40dd-830a-a15711d54ef1

Notes:

  • The operator should confirm with both VPC owners (VPC tenant org) that they approve the peering before creating the connection
  • The VPC IDs can be provided in any order
  • The system will automatically enforce canonical ordering (smaller ID becomes vpc1_id)
  • If a peering connection already exists between the two VPCs, including one being deleted, the command returns an error. Wait for deletion to finish before creating the same pair again.
  • Both VPCs must exist before creating the peering connection
  • For a Virtualized-to-Flat peering, follow the routing preparation and verification steps

Listing VPC Peering Connections

To view VPC peering connections, you can either show all connections or filter by a specific VPC:

Show all peering connections:

nico-admin-cli vpc-peering show

Show peering connections for a specific VPC:

nico-admin-cli vpc-peering show --vpc-id <VPC_ID>

Example:

# Show all peering connections
nico-admin-cli vpc-peering show
# Show peering connections for a specific VPC
nico-admin-cli vpc-peering show --vpc-id 550e8400-e29b-41d4-a716-446655440000

The table shows Id, VPC1 ID (smaller UUID), and VPC2 ID (larger UUID). It does not show state or creation time. Connections stay in the list until their deletion finishes.

Deleting VPC Peering Connections

Deleting a peering disconnects the two VPCs. It takes time because each affected DPU must stop allowing that connection before nico-api removes the peering.

Use vpc-peering show to find the connection ID, then request deletion:

nico-admin-cli vpc-peering delete --id <PEERING_CONNECTION_ID>

Example:

nico-admin-cli vpc-peering delete --id 123e4567-e89b-12d3-a456-426614174000

Use the connection ID, not either VPC ID. A successful command starts deletion; it does not mean deletion has finished. Wait until the connection disappears from vpc-peering show. Until then, you cannot delete either VPC or create another peering between the same pair.

You can safely repeat the request while deletion is pending. It does not restart the wait or send another network update. After deletion finishes, another request returns NotFound.

If the command returns FailedPrecondition with a host ID and asks you to retry, run it again. A host changed while the request was running, so nico-api left the peering and its network settings unchanged.

For the exact RPC states and FNN routing changes, refer to the configuration reference.

Deleting Through REST

Send DELETE /v2/org/{org}/nico/vpc-peering/{id}, then poll GET /v2/org/{org}/nico/vpc-peering/{id}. Replace {org} with your organization name and {id} with the peering connection ID. Your role must have the TENANT_ADMIN or PROVIDER_ADMIN suffix.

Tenant Admins can delete peerings between their own VPCs. Only the Site’s Provider Admin can delete a peering between different tenants. Provider Admins cannot use this endpoint to delete peerings within one tenant.

An accepted DELETE returns HTTP 202 with {"message":"Deletion request was accepted"}. The connection stays Deleting until Site inventory confirms removal. This can take longer than the Core deletion itself. Wait for GET to return 404 before deleting a VPC or recreating the peering.

If a host changes during the request, REST returns HTTP 412 and leaves the connection’s status unchanged. Retry the DELETE.

When Deletion Is Waiting

An offline DPU can block deletion indefinitely. Bring it back online and make sure it can apply network updates. Later network changes can also extend the wait, because the DPU must confirm its latest configuration.

To find the host holding up deletion, search the nico-api logs. This example searches the last hour across the selected pods:

kubectl logs -n <namespace> -l app.kubernetes.io/name=nico-api \
  --all-containers=true --since=1h --tail=-1 --prefix=true \
  | grep -F 'Waiting for VPC peering removal acknowledgements'

The selector uses the chart’s default name. If you set nameOverride, use that value instead of nico-api. For logs shipped to Loki or another log service, search for the same message in your site’s API logs. Each matching entry includes:

  • vpc_peering_id: the connection being deleted.
  • host_machine_id: the first host still waiting.
  • network_config_version: the version its DPUs must apply.

The wait reason is also in vpc_peerings.controller_state_outcome in the Core database, but is not shown in the CLI table.

Do not force delete a Machine or Instance just to clear this wait. Removing database records does not prove that its DPU stopped allowing the connection.

When first upgrading to this deletion behavior, follow the API stop-and-drain procedure. Old and new API versions must not run together during that upgrade.