NVIDIA BlueField-3 SNAP for NVMe and Virtio-blk v4.3.0
1.0

Appendix – SNAP Memory Consumption

This appendix explains how SNAP consumes memory and how to manage memory allocation.

The user must allocate the DPA hugepages memory according to the section "Step 1: Allocate Hugepages". It is possible to use use a portion of the DPU memory allocation in the SNAP container as described in section "Adjusting YAML Configuration". This configuration includes the following minimum and maximum values:

  • The minimum allocation which the SNAP container consumes:

    Copy
    Copied!
                

    resources: requests: memory: "4Gi"

  • The maximum allocation that the SNAP container is allowed to consume:

    Copy
    Copied!
                

    resources: limits:    hugepages-2Mi: "4Gi"

Hugepage memory is used by the following:

  • SPDK mem-size global variable which controls the SPDK hugepages consumption (configurable in SPDK, 1GB by default)

  • SNAP SNAP_MEMPOOL_SIZE_MB – used with non-ZC mode as IO buffers staging buffers on the Arm. By default, the SNAP mempool consumes 1G from the SPDK mem-size hugepages allocation. SNAP mempool may be configured using the SNAP_MEMPOOL_SIZE_MB global variable (minimum is 64 MB).

    Warning

    If the value assigned is too low, with non-ZC, a performance degradation could be seen.

  • SNAP and SPDK internal usage – 1G should be used by default. This may be reduced depending on the overall scale (i.e., VFs/num queues/QD).

  • XLIO buffers – allocated only when NVMeTCP XLIO is enabled.

The following is the limit of the container memory allowed to be used by the SNAP container:

Copy
Copied!
            

resources: limits: memory: "6Gi"

Note

This includes the hugepages limit (in this example, additional 2G of non-hugepages memory).

The SNAP container also consumes DPU SHMEM memory when NVMe recovery is used (described in section "NVMe Recovery"). In addition, the following resources are used:

Copy
Copied!
            

limits: memory:

© Copyright 2023, NVIDIA. Last updated on Dec 13, 2023.