DOCA Documentation v2.9.1
DOCA SDK 2.9.1

NVIDIA DOCA PSP Gateway Application Guide

This document describes the usage of the NVIDIA DOCA PSP Gateway sample application on top of an NVIDIA® BlueField® networking platform or NVIDIA® ConnectX® SmartNIC.

Note

DOCA PSP Gateway is supported at alpha level.

Note

DOCA PSP Gateway is supported only on BlueField-3 or ConnectX-7 and later.

The DOCA PSP Gateway application leverages the BlueField or ConnectX hardware capability for fully offloaded secure network communication using the PSP security protocol. The application demonstrates how to exchange keys between application instances and insert rules controlling PSP encryption and decryption using the DOCA Flow library.

Warning

The application exchanges keys using an unencrypted gRPC channel. If your environment requires the protection of encryption keys, you must modify the application to create the gRPC channel using the applicable certificates.

Info

The PSP Gateway application supports only the PSP tunnel protocol. The PSP transport protocol is not supported by the application in this release, although it is supported by the underlying DOCA Flow library.

Info

The PSP Gateway application supports only IPv4 inner and IPv6 outer headers. Other combinations are not supported by the application in the current release, although they are supported by the underlying DOCA Flow library.

The application can be configured to establish out-bound PSP tunnel connections via individual command-line arguments, or via a text file configured via a command-line argument. The connections are established on-demand by default, but can also be configured to connect at startup.

The DOCA PSP Gateway is designed to run with three ports:

  • A secure (encrypted) uplink netdev (i.e., p0)

  • An unsecured (plaintext) netdev representor (VF or SF)

  • An out-of-bound (OOB) management port, used to communicate with peer instances using standard sockets

Whether the DOCA PSP Gateway is deployed to a BlueField or a ConnectX device, the functionality is the same. The Out of Bounds (OOB) network device carries PSP parameters between peers, the Uplink port carries secure (encrypted) traffic, and the VF carries the unencrypted traffic.

image-2024-5-17_9-34-57-version-1-modificationdate-1724675741923-api-v2.png

When the application is deployed to a DPU, the operation of the PSP encryption protocol is entirely transparent to the Host. All the resources required to manage the PSP connections are physically located on the DPU.

image-2024-5-6_15-33-24-version-1-modificationdate-1724675742283-api-v2.png

When the application is deployed to the host, the operation of the PSP encryption protocol is the responsibility of the host, and resources are allocated from the host. However, the operation of the PSP encryption protocol is entirely transparent to any virtual machines and containers attached to the VF network devices.

The creation of PSP tunnel connections requires two-way communication between peers. Each "sender" must request a unique security parameters index (SPI) and encryption key from the intended "receiver". The receiver derives sequential SPIs and encryption keys using the hardware resources inside the BlueField or ConnectX device, which manages a secret pair of master keys to produce the SPIs and encryption keys.

One key architectural benefit of PSP over similar protocols (e.g., IPsec) is that the receiver does not incur any additional resource utilization whenever it creates a new SPI and encryption key. This is because the decryption key associated with the SPI is computed on the fly, based on the SPI and master key, for each received packet. This lack of requirement for additional context memory for each additional decryption rule is partly responsible for the ability of the PSP protocol to scale to many thousands of peers.

Startup vs. On-Demand Tunnel Creation

The default mode of operation is on-demand tunnel creation. That is, when a packet is received from the unsecured port for which the flow pipeline does not have an encryption rule, the packet misses to RSS, where the CPU must decide how to handle the packet. If the destination IP address in the packet belongs to a known peer’s virtual network, the CPU uses gRPC on the OOB network connection to attempt a key exchange with the peer. If the key exchange is successful and a new encryption flow is created successfully, then the packet is then resubmitted to the pipeline, where it is encrypted and sent just as any of the following packets having the same destination IP address.

The following diagram illustrates this sequence (the "Slow Path"), for Virtual Machine V1 which intends to send a packet to Virtual Machine V2. In this case, V1 is hosted on physical host H1 and V2 on physical host H2. The first packet sent (1) results in a miss (2), so the packet is retained (3) while the keys are exchanged in both directions (4-8). Then the pipeline is updated (9) and the original packet is resubmitted (10). From there, the packet follows the same logic as the fast path, below.

image-2024-4-23_14-21-16-version-1-modificationdate-1724675742470-api-v2.png

Once the tunnel is established, and packets received from the VF (1) match a rule (2) and are encrypted and sent (3-4) without any intervention from the CPU ("Fast Path").

image-2024-4-23_14-24-13-version-1-modificationdate-1724675742687-api-v2.png

In the case of on-startup tunnel creation, the application's main thread repeatedly attempts to perform the key exchange for each of the peers specified on the command line until the list is completed. Each peer is connected only once and, if a connection to one peer fails, the loop continues onto the next peer and retries the failed connection after all the others have been attempted.

Sampling

The PSP gateway application supports the sample-at-receiver (S) bit in the PSP header. If sampling is enabled, then packets marked with the S bit are mirrored to the RSS queues and logged to the console. In addition, on transmit, the S bit of a random subset of packets (1 out of 2^N for command-line parameter N) is set to 1, and those packets are mirrored to RSS.

To avoid out-of-sequence indication in ROCE traffic, the --maintain-order flag can be used. When used, the application creates two mirror resources for egress sampling:

  • One for when the sample bit is on – mirror is to RSS

  • One for when the sample bit is off – mirror with fwd type drop

Note

Sampling packets on transmit is currently supported only following encryption. Sampling of egress packets before encryption will be supported in a future release.


Pipelines

Host-to-Network Flows

Traffic sent from the local, unsecured port (host-to-net) without sampling enabled travels through the pipeline as shown in the diagrams that follow. Note that the Ingress Root Pipe is the first destination for packets arriving from either the VF or the secured uplink port. However, the Egress ACL pipe is the first destination for packets sent via tx_burst on the PF (in the switch model's expert mode).

The Empty Pipe is a vestigial transition from the Default Domain, in which the Ingress Root Pipe is created, to the Secure Egress Domain, where the Egress ACL pipe performs encryption.

Note

This pipe may be removed in a future release.

image-2024-4-23_14-50-36-version-1-modificationdate-1724675744490-api-v2.png

If sampling is enabled, the host-to-net pipeline is modified as shown in the following:

image-2024-4-23_15-12-1-version-1-modificationdate-1724675744703-api-v2.png

Here, an Egress Sampling Pipe is added between the Egress ACL Pipe and the Secured Port. It performs a match of the random metadata, masked according to command-line parameters, and then:

  • On match, the following actions occur:

    1. Packet modifications:

      1. The S bit in the PSP header is set to true.

      2. The pkt_meta field is set to a sentinel value to indicate to CPU software why the packet was sent to RSS.

    2. The original packet is forwarded to RSS.

    3. The mirror action forwards the packet to the secured port.

  • On miss, the following actions occur:

    1. No packet modifications are made.

    2. The packet is forwarded to a vestigial pipe which can then forward the packet to the wire.

      Info

      A fwd_miss cannot target a port.

      Note

      This pipe may be removed in a future release.

Network-to-Host Flows

When a packet arrives from the secured port, the following flows are executed.

image-2024-4-23_15-30-41-version-1-modificationdate-1724675744930-api-v2.png

As before, the Ingress Root Pipe is the first destination and, here, the secured port ID as well as IPv6 outer L3 type are matched for. Matching packets flow to the decryption pipe, which matches the outer UDP port number against 1000, the constant specified in the PSP specification. On match, the packet is decrypted, but not yet de-capped. Then the Ingress ACL pipe checks the following:

  • PSP_Syndrome – did the packet decrypt correctly and pass its ICV check?

  • PSP SPI and inner IP src address – was this packet encrypted with the key associated with the given source?

If the packet passes the syndrome and ACL check, it is forwarded to the VF. Otherwise, the Syndrome Stats pipe counts the occurrences of the different bits in the PSP Syndrome word.

When sampling is enabled, the Ingress Sampling Pipe is inserted before the ACL. Unlike the Egress Sampling Pipe, no randomness is involved; the match criteria is the sample-on-receive flag in the PSP header. On a match, the incoming packet are mirrored to RSS with pkt_meta indicating the reason for forwarding the packet to RSS. On match or miss, the next pipe is the Ingress ACL Pipe.

image-2024-4-23_15-41-2-version-1-modificationdate-1724675745313-api-v2.png

DOCA Libraries

This application leverages the following DOCA libraries:

Refer to their respective programming guide for more information.

Info

Please refer to the NVIDIA DOCA Installation Guide for Linux for details on how to install BlueField-related software.

The installation of DOCA's reference applications contains the sources of the applications, alongside the matching compilation instructions. This allows for compiling the applications "as-is" and provides the ability to modify the sources, then compile a new version of the application.

Tip

For more information about the applications as well as development and compilation tips, refer to the DOCA Applications page.

The sources of the application can be found under the application's directory: /opt/mellanox/doca/applications/psp_gateway/.

Prerequisites

  • The application relies on the json-c open source, requiring the following to be installed:

    • Ubuntu/Debian:

      Copy
      Copied!
                  

      $ sudo apt install libjson-c-dev

    • CentOS/RHEL:

      Copy
      Copied!
                  

      $ sudo yum install json-c-devel

  • Installing the gRPC open source

Compiling All Applications

All DOCA applications are defined under a single meson project. So, by default, the compilation includes all of them.

To build all the applications together, run:

Copy
Copied!
            

cd /opt/mellanox/doca/applications/ meson /tmp/build ninja -C /tmp/build

Info

doca_psp_gateway is created under /tmp/build/psp_gateway/.


Compiling Only the Current Application

To directly build only the PSP Gateway application:

Copy
Copied!
            

cd /opt/mellanox/doca/applications/ meson /tmp/build -Denable_all_applications=false -Denable_psp_gateway=true ninja -C /tmp/build

Info

doca_psp_gateway is created under /tmp/build/psp_gateway/.

Alternatively, users can set the desired flags in the meson_options.txt file instead of providing them in the compilation command line:

  1. Edit the following flags in /opt/mellanox/doca/applications/meson_options.txt:

    • Set enable_all_applications to false

    • Set enable_psp_gateway to true

  2. Run the following compilation commands :

    Copy
    Copied!
                

    cd /opt/mellanox/doca/applications/ meson /tmp/build ninja -C /tmp/build

    Info

    doca_psp_gateway is created under /tmp/build/psp_gateway/.

Troubleshooting

Refer to the NVIDIA DOCA Troubleshooting Guide for any issue encountered with the compilation of the application .

Prerequisites

The PSP gateway application is based on DOCA Flow. Therefore, the user is required to allocate huge pages:

Copy
Copied!
            

$ echo '2048' | sudo tee -a /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages $ sudo mkdir /mnt/huge $ sudo mount -t hugetlbfs -o pagesize=2M nodev /mnt/huge


Application Execution

The PSP Gateway application is provided in source form. Therefore, a compilation is required before the application can be executed.

  1. Application usage instructions:

    Copy
    Copied!
                

    Usage: doca_psp_gateway [DOCA Flags] [Program Flags]   DOCA Flags: -h, --help Print a help synopsis -v, --version Print program version information -l, --log-level Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE> --sdk-log-level Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE> -j, --json <path> Parse all command flags from an input json file   Program Flags: -p, --pci-addr PCI BDF of the device in BB:DD.F format (required) -r, --repr Device representor list in vf[x-y]pf[x-y] format (required) -m, --core-mask EAL Core Mask --decap-dmac Mac_dst addr of the decapped packets -d, --vf-name Name of the virtual function device / unsecured port -n, --nexthop-dmac Next-hop mac_dst addr of the encapped packets --cookie Enable use of PSP virtualization cookies -a, --disable-ingress-acl Allows any ingress packet that successfully decrypts --sample-rate Sets the log2 sample rate: 0: disabled, 1: 50%, ... 16: 1.5e-3% -x, --max-tunnels Specify the max number of PSP tunnels -o, --crypt-offset Specify the PSP crypt offset --psp-version Specify the PSP version for outgoing connections (0 or 1) -z, --static-tunnels Create tunnels at startup -k, --debug-keys Enable debug keys --stat-print Enable printing statistics --perf-print Enable printing performance metrics (key-gen, insertion, all) --show-rss-rx-packets Show RSS rx packets --outer-ip-type Outer IP type --inner-ip-type Inner IP type -c, --config Path to the JSON file with application configuration --maintain-order Maintain original packet ordering

  2. This usage printout can be printed to the command line using the -h (or --help) options:

    Copy
    Copied!
                

    ./doca_psp_gateway -h

    Info

    For additional information, refer to section "Command Line Flags".

  3. CLI example for running the application on the BlueField or host:

    Copy
    Copied!
                

    ./doca_psp_gateway -p 03:00.0 -r vf0pf0 -d ens2f0v0 -c /tmp/psp_gw_config.json

    • The PCIe address identifier ( -p flag) should match the addresses of the desired PCIe device

    • The -d flag indicates the vf that is used to extract MAC address and local IP. It should match the MAC address of the virtual function specified by the -r argument.

    • The -c flag is the path to json configuration file that includes all info about tunnels.

  4. The application also supports a JSON-based deployment mode, in which all command-line arguments are provided through a JSON file:

    Copy
    Copied!
                

    ./doca_psp_gateway --json [json_file]

    For example:

    Copy
    Copied!
                

    ./doca_psp_gateway --json psp_gateway_params.json

    Note

    Before execution, ensure that the used JSON file contains the correct configuration parameters, and especially the PCIe addresses necessary for the deployment.

Command Line Flags

Flag Type

Short Flag

Long Flag/JSON Key

Description

JSON Content

General flags

h

help

Prints a help synopsis

N/A

v

version

Prints program version information

N/A

l

log-level

Set the log level for the application:

  • DISABLE=10

  • CRITICAL=20

  • ERROR=30

  • WARNING=40

  • INFO=50

  • DEBUG=60

  • TRACE=70 ( requires compilation with TRACE log level support )

Copy
Copied!
            

"log-level": 60

N/A

sdk-log-level

Sets the log level for the program:

  • DISABLE=10

  • CRITICAL=20

  • ERROR=30

  • WARNING=40

  • INFO=50

  • DEBUG=60

  • TRACE=70

Copy
Copied!
            

"sdk-log-level": 40

j

json

Parse all command flags from an input JSON file

N/A

Program flags

p

pci-addr

PCIe BDF of the device in BB:DD.F format

Copy
Copied!
            

"p""03:00.0"

r

repr

Device representor list in vf[x-y]pf[x-y] format

Copy
Copied!
            

"r": "vf0pf0"

m

core-mask

EAL core mask

Copy
Copied!
            

"m""0xf"

N/A

decap-dmac

mac_dst address of the decapped packets (cannot be used with vf-name)

Copy
Copied!
            

"dcap-dmac": "11:22:33:44:55::66"

N/A

vf-name

Name of the VF device/unsecured port (Host only. Automatically detects MAC/VIP. Requires IP address bound to VF.)

Copy
Copied!
            

"vf-name": "ens2f0v0"

n

nexthop-dmac

Next-hop mac_dst address of the encapped packets

Copy
Copied!
            

"nexthop-dmac""77:88:99:aa:bb:cc"

N/A

cookie

Enable use of PSP virtualization cookies

Copy
Copied!
            

"cookie"true

a

disable-ingress-acl

Allows any ingress packet that successfully decrypts

Copy
Copied!
            

"disable-ingress-acl"true

N/A

sample-rate

Sets the log2 sample rate:

  • 0 – disabled,

  • 1 – 50%, ...

  • 16 – 1.5e-3%

Copy
Copied!
            

"sample-rate": 16

x

max-tunnels

Specify the max number of PSP tunnels

Copy
Copied!
            

"max-tunnels": 4096

o

crypt-offset

Specify the PSP crypt offset

Copy
Copied!
            

"crypt-offset": 7

N/A

psp-version

Specify the PSP version for outgoing connections (0 or 1)

Copy
Copied!
            

"psp-version": 0

z

static-tunnels

Create tunnels at startup using the given local IP address

Copy
Copied!
            

"static-tunnels""192.168.1.99"

k

debug-keys

Enable debug keys

Copy
Copied!
            

"debug-keys"true

N/A

stat-print

Print statistics regarding active sessions

Copy
Copied!
            

"stat-print": true

N/A

perf-print

Print performance metrics of key generation of key-SPI pairs (key-gen) or insertion rate (insertion) or both (all)

Copy
Copied!
            

"perf-print": "all"

N/A

show-rss-rx-packets

Print RX packets that got to RSS queues (i.e., to software)

Copy
Copied!
            

"show-rss-rx-packets": false

N/A

outer-ip-type

Outer IP tunnel type (ipv4 or ipv6). When not specified, default is ipv6.

Copy
Copied!
            

"outer-ip-type": "ipv4"

N/A

inner-ip-type

Inner IP tunnel type (ipv4 or ipv6). When not specified, the default is ipv4.

Copy
Copied!
            

"inner-ip-type": "ipv6"

c

config

Path to JSON file that includes tunnel information

Info

See section "Config File JSON Format" for specifics on the JSON file format.

Copy
Copied!
            

"config": "/tmp/psp_gw_config.json"

N/A

maintain-order

Force maintaining original packet ordering

Copy
Copied!
            

"maintain-order": true

Refer to DOCA Arg Parser for more information regarding the supported flags and execution modes.

Config File JSON Format

The JSON configuration file maps an OOB network address (gRPC server) to a list of virtual IP addresses behind that physical address. Each "peer" in the list contains information about a remote gRPC server to which the pairs of local-vip and remote-vip are mapped to.

The following is a template of the JSON file structure:

Json config

Copy
Copied!
            

{ "config": { "local-grpc-address": <local gRPC server ip address to listen for connections on, possible with port> }, "peers": [ /* list of peers (differentiated by remote gRPC server) */ { "remote-grpc-address": <remote gRPC server address to connect to for connections of given sessions> , "sessions": [ /* list of sessions allowed with peer (differentiated by local vip) */ { "local-vip": <local virtual ip>, /* if not present, take the value from vf interface's IP ("--vf-name" flag) */ "remote-vips": [ <ip address, possible with prefix mask>, /* [<more ip addresses>...] */ ] }, /* [<more sessions>... ] */ ] }, /* [<more peers>...] */ ] }

When a packet from the VF does not match any existing flows, this table defines the physical peer which should provide the tunnel to the given (virtual) src-ip, dst-ip 2-tuple.

Example JSON:

Json config

Copy
Copied!
            

{ "config": { "local-grpc-address": "10.237.113.143:3000" }, "peers": [ { "remote-grpc-address": "10.237.113.144", "sessions": [ { "local-vip": "6.6.6.5", "remote-vips": [ "8.8.8.0/24", "7.7.7.7" ] }, { "local-vip": "6.6.6.4", "remote-vips": [ "9.9.9.9" ] } ] }, { "remote-grpc-address": "10.237.1.1", "sessions": [ { // "local vip" not present - default value from vf interface's IP (Must run application with "--vf-name" param) "remote-vips": [ "11.11.11.11/24" ] } ] } ] }


Troubleshooting

Refer to the NVIDIA DOCA Troubleshooting Guide for any issue encountered with the installation or execution of the DOCA applications .

  1. Main loop code flow

    1. Initialize the logger facility.

      1. The standard logger and the SDK logger are created, and the SDK logger default log level is selected.

        Copy
        Copied!
                    

        doca_log_backend_create_standard(); doca_log_backend_create_with_file_sdk(stdout, &sdk_log); doca_log_backend_set_sdk_level(sdk_log, DOCA_LOG_LEVEL_WARNING);

      2. The signal handler is connected to enable a clean shutdown.

        Copy
        Copied!
                    

        signal(SIGINT, signal_handler); signal(SIGTERM, signal_handler);

    2. Parse application arguments. The main function invokes psp_gw_argp_exec(), which initializes the arg parser resources and registers DOCA general parameters, and then registers the PSP application-specific parameters. Then the parser is invoked.

      Copy
      Copied!
                  

      doca_argp_init(); psp_gw_register_params(); doca_argp_start();

    3. Parse input configuration JSON file. This will include all information about peers and their sessions.

      Copy
      Copied!
                  

      psp_gw_parse_config_file();

    4. DPDK initialization. Call rte_eal_init() to initialize EAL resources with the provided EAL flags for not probing the ports (-a00:0.0).

      Copy
      Copied!
                  

      rte_eal_init(n_eal_args, (char **)eal_args);

    5. Initialize devices and ports.

      1. Open DOCA devices with input PCIe addresses/interface names.

      2. Probe DPDK port from each opened device.

        Copy
        Copied!
                    

        open_doca_device_with_pci(...); // not part of doca_flow; see doca/samples/common.c doca_dpdk_port_probe(...);

      3. The MAC and IP addresses of the PF are queried and logged.

        Copy
        Copied!
                    

        rte_eth_macaddr_get(...); doca_devinfo_get_ipv6_addr(...); DOCA_LOG_INFO("Port %d: Detected PF mac addr: %s, IPv6 addr: %s, total ports: %d", ...);

    6. Initialize and start DPDK ports. Initialize DPDK ports, including mempool allocation. No hairpin queues are created.

      Copy
      Copied!
                  

      dpdk_queues_and_ports_init(); // not part of doca_flow; see doca/applications/common/dpdk_utils.c

    7. Initialize DOCA Flow objects used by the PSP Gateway application. The DOCA Flow library is initialized with the string "switch,hws,isolated,expert", because it is desirable for the application to act as an intermediary between the uplink physical port and some number of VF representors (switch mode), and hws (hardware steering mode) and isolated mode are mandatory for switch mode. The optional expert flag prevents DOCA Flow from automating certain packet operations and gives more control to the application, as described in the DOCA Flow page.

      Copy
      Copied!
                  

      PSP_GatewayFlows psp_flows(&pf_dev, vf_port_id, &app_config); psp_flows.init();

      1. Initialize DOCA Flow library.

      2. Start the ports.

      3. Allocate shared resources (PSP crypto objects and Mirror actions).

      4. Create the ingress and egress pipes.

    8. Create the gRPC service.

      Copy
      Copied!
                  

      PSP_GatewayImpl psp_svc(&app_config, &psp_flows);

    9. Launch the L-Core threads to handle RSS packets.

      Copy
      Copied!
                  

      rte_eal_remote_launch(lcore_pkt_proc_func, &lcore_params, lcore_id);

    10. Launch the gRPC service.

      1. This implementation uses InsecureServerCredentials. Update as needed.

        Copy
        Copied!
                    

        grpc::ServerBuilder builder; builder.AddListeningPort(server_address, grpc::InsecureServerCredentials()); builder.RegisterService(&psp_svc); auto server_instance = builder.BuildAndStart();

    11. Wait for traffic. If configured to connect at startup, process the list of remaining connections. Then display the flow pipe counters.

      Copy
      Copied!
                  

      while (!force_quit) { psp_svc.try_connect(remotes_to_connect, local_vf_addr); ... psp_flows.show_static_flow_counts(); psp_svc.show_flow_counts(); }

      • Wait in a loop until the user terminates the program.

    12. PSP Gateway cleanup:

      1. Destroy DPDK ports and queues.

        Copy
        Copied!
                    

        dpdk_queues_and_ports_fini();

      2. DPDK finish.

        Copy
        Copied!
                    

        dpdk_fini();

        Calls rte_eal_destroy() to destroy initialized EAL resources.

      3. Arg parser destroy.

        Copy
        Copied!
                    

        doca_argp_destroy()

  2. Miss-packet code flow.

    1. The L-Core launch routine from the main loop pointed to the lcore_pkt_proc_func routine.

    2. The force_quit flag is polled to respond to the signal handler.

      Copy
      Copied!
                  

      while (!*params->force_quit) { ... }

    3. The rte_eth_rx_burst function polls the PF queue for received packets.

      Copy
      Copied!
                  

      nb_rx_packets = rte_eth_rx_burst(port_id, queue_id, rx_packets, MAX_RX_BURST_SIZE);

    4. Inside handle_packet(), the packet metadata is inspected to detect whether this packet is sampled on ingress, sampled on egress, or a miss packet.

      Copy
      Copied!
                  

      uint32_t pkt_meta = rte_flow_dynf_metadata_get(packet);

      1. Sampled packets are simply logged using the rte_pktmbuf_dump function.

    5. Miss packets are passed to the handle_miss_packet method of the gRPC service. This method handles cases where an application attached to the VF wishes to send a packet to another virtual address, but a PSP tunnel must first be established by exchanging SPI and key information between hosts.

    6. The service acts as a gRPC client, and the appropriate server is looked up from the config->net_config.hosts vector, which is comprised of hosts passed via the -t tunnels arguments or the -f tunnels file argument.

    7. Once the client connection exists, the request_tunnel_to_host method takes care of invoking the the RequestTunnelParams operation defined in the schema.

      • Optionally, this function generates a corresponding set of tunnel parameters appropriate for the server host to send traffic back via generate_tunnel_params().

        Copy
        Copied!
                    

        doca_flow_crypto_psp_spi_key_bulk_generate(bulk_key_gen); doca_flow_crypto_psp_spi_key_bulk_get(bulk_key_gen, 0, &spi, key); doca_flow_crypto_psp_spi_key_wipe(bulk_key_gen, 0);

    8. The RPC operation is invoked, and if successful, create_tunnel_flow is called to create the egress flow:

      Copy
      Copied!
                  

      status = stub->RequestTunnelParams(&context, request, &response);

    9. The create_tunnel_flow method translates the resulting Protobuf objects to application-specific data structures and passes them to the add_encrypt_entry method of the flows object. Here, the PSP SPI and key are programmed into an available crypto_id index as follows.

      Note

      SPI and crypto_id are two independent concepts:

      • The SPI value in the PSP packet header indicates to the receiver which key was used by the sender to encrypt the data. Each receiver computes an SPI and key to provide to a sender. Since each receiver is responsible for tracking its next SPI, multiple receivers may provide the same SPI to a sender, so one sender may send the same SPI to multiple different peers. This is allowed, as each of the receiving peers has its own decryption key to handle that SPI.

      • The crypto_id acts as an index into the bucket of PSP keys allocated by DOCA Flow. The doca_flow_shared_resource_cfg() function writes a given PSP encryption key to a given slot in the bucket of keys in NIC memory. These slots can be overwritten as needed by the application.

      • There is no explicit association between crypto_id and SPI. The doca_flow_shared_resource_cfg() function writes a key at the slot provided by the crypto_id argument, then the flow pipe entry actions.crypto.crypto_id references this key, and actions.crypto_encap.encap_data includes a PSP header with the desired SPI.

      Copy
      Copied!
                  

      struct doca_flow_shared_resource_cfg res_cfg = {}; res_cfg.domain = DOCA_FLOW_PIPE_DOMAIN_SECURE_EGRESS; res_cfg.psp_cfg.key_cfg.key_type = DOCA_FLOW_CRYPTO_KEY_256; res_cfg.psp_cfg.key_cfg.key = (uint32_t *)encrypt_key; doca_flow_shared_resource_cfg(DOCA_FLOW_SHARED_RESOURCE_PSP, session->crypto_id, &res_cfg);

    10. A flow pipe entry which references the newly programmed PSP encryption key (via its index crypto.crypto_id) must be inserted. Additionally, this pipe entry must specify all the outer Ethernet, IP, UDP, and PSP header fields to insert.

      Copy
      Copied!
                  

      format_encap_data(session, actions.crypto_encap.encap_data); actions.crypto.action_type = DOCA_FLOW_CRYPTO_ACTION_ENCRYPT; actions.crypto.resource_type = DOCA_FLOW_CRYPTO_RESOURCE_PSP; actions.crypto.crypto_id = session->crypto_id; ...   doca_flow_pipe_add_entry(pipe_queue, pipe, match, actions, mon, fwd, flags, &status, entry); ...   doca_flow_entries_process(port, 0, DEFAULT_TIMEOUT_US, num_of_entries);

    11. The original packet received via rte_ethdev_rx_burst is sent back through the newly updated pipelines via rte_ethdev_tx_burst. Since the port_id argument is that of the PF, and since DOCA Flow has been initialized in expert mode, the packet is transferred to the root of the egress domain (the "empty pipe" before egress_acl_pipe).

      Copy
      Copied!
                  

      nsent = rte_eth_tx_burst(port_id, queue_id, &packet, 1);

  3. Tunnel parameter request handling

    1. The gRPC service provided by the PSP Gateway implements the RequestTunnelParams operation referenced above. A client uses this operation to request an SPI and key to encrypt traffic to send to the server's NIC device. The request indicates the virtual remote address for which the tunnel will be created.

    2. This operation begins by generating a new SPI and key inside generate_tunnel_params() as described previously.

    3. The operation creates an ACL entry permitting the new SPI and the remote virtual address using the add_ingress_acl_entry method of the Flows object.

      Copy
      Copied!
                  

      doca_flow_match match = {}; match.parser_meta.psp_syndrome = 0; match.tun.type = DOCA_FLOW_TUN_PSP; match.tun.psp.spi = RTE_BE32(session->spi_ingress); match.inner.l3_type = DOCA_FLOW_L3_TYPE_IP4; match.inner.ip4.src_ip = session->src_vip; ...   doca_flow_pipe_add_entry(pipe_queue, pipe, match, actions, mon, fwd, flags, &status, entry); ...   doca_flow_entries_process(port, 0, DEFAULT_TIMEOUT_US, num_of_entries);

    4. If the request included parameters for traffic in the reverse direction (traffic to encrypt and send to the client), these parameters are translated and passed to the Flows object by calling create_tunnel_flow described above.

References

© Copyright 2024, NVIDIA. Last updated on Dec 6, 2024.