Priority Flow Control (PFC)

NVIDIA Onyx User Manual v3.10.4006

Priority Flow Control (PFC) provides an enhancement to the existing pause mechanism in Ethernet. The current Ethernet pause option stops all traffic on a link. PFC creates eight separate virtual links on the physical link and allows any of these links to be paused and restarted independently, enabling the network to create a no-drop class of service for an individual virtual link.

PFC offers the following features:

  • Provides per-priority enabling or disabling of flow control

  • Transmits PFC-PAUSE frames when the receive threshold for a particular traffic class is reached

  • Provides the management capability for an administrator to configure the flow control properties on each port of the switch

  • Keeps flow control disabled for all priorities on all ports by default

  • Allows an administrator to enable or disable flow control per port and per priority level

  • Supports flow control only on physical ports, not on logical interfaces such as tunnels or interfaces defined by sharing a physical port in multiple virtual switch contexts

  • Uses the configured threshold values to set up the queue buffer spaces accordingly in the data-path

  • Provides hardware abstraction layer call-outs for the following:

  • Enabling or disabling of flow control on each port for each priority

  • Configuring the queue depth for each priority on each port

  • Provides trace logs for execution upon error conditions and for any event notifications from the hardware or data-path. These trace logs are a useful aid in troubleshooting.

  • Allows the administrator to configure the minimum and maximum threshold values for flow control. These configurations are applied globally on all ports and priorities.

Priority Based Flow Control (PFC) provides an enhancement to the existing pause flow control mechanism as described in 802.1x.

To enable PFC globally,

Copy
Copied!
            

switch (config) # dcb priority-flow-control enable This action might cause traffic loss while shutting down a port with priority-flow-control mode on Type 'yes' to confirm enable pfc globally: yes

To enable PFC per priority:

  1. Enable PFC globally on the switch.

    Copy
    Copied!
                

    switch (config) # dcb priority-flow-control enable This action might cause traffic loss while shutting down a port with priority-flow-control mode on Type 'yes' to confirm enable pfc globally: yes

  2. Choose the priority you want to enable.

    Copy
    Copied!
                

    switch (config) # dcb priority-flow-control priority 5 enable

To enable PFC per interface, do the following.

  1. Enable PFC globally on the switch.

    Copy
    Copied!
                

    switch (config) # dcb priority-flow-control enable

  2. Choose the priority you want to enable.

    Copy
    Copied!
                

    switch (config) # dcb priority-flow-control 5 enable

  3. Change to Interface mode.

    Copy
    Copied!
                

    switch (config) # interface ethernet 1/1 switch (config interface ethernet 1/1) #

  4. Enable PFC for the specific interface.

    Copy
    Copied!
                

    switch (config interface ethernet 1/1) # dcb priority-flow-control mode on

When working with lossless traffic, the receiving side sends a pause frame (Xoff) to the transmitting side before the buffer is filled. When the buffer empties, the receiving side sends an un-pause frame (Xon) to the transmitting side.

The user has to set the buffer usage Xoff and Xon thresholds. The thresholds depend on network parameters (bandwidth, link latency, MTU) and the allocated size for the region.

Xon_Xoff_Configuration-version-1-modificationdate-1709536825860-api-v2.png

Warning

When working with global flow control mode only, a single PG shall be used and Xoff and Xon shall be set on this PG. When working with priority flow control, Xoff and Xon shall be set on each lossless PG.

Warning

See the “Shared Buffers” page for more information on flow control.

Lossless networks with PFC enabled provide strong packet delivery guarantees. However, lossless networks introduce a new fault scenario where a queue of an end-port (e.g. the port of a host connected to the network) may not be able to receive any traffic from the network and keeps sending pause frames towards the switch. Since lossless switch paths do not drop packets but decline receiving more packets when their buffers fill up, if the end-port queue is stuck for a long time, the buffers fill up not only for the target switch, but also on all switches with problematic port queues in the traffic forwarding path. This leads to endless PFC pause frames, also called a PFC storm, being observed on all switch ports along the path to the traffic source.

PFC watchdog prevents congestion from spreading in such a case. When switches detect this situation on any TC queue, all the packets in the queue are flushed and new packets destined to the same queue are dropped as well until PFC storming is relieved.

For lossless networks with global flow control configured, we will face the same issue of global pause storm. To resolve this, global-flow-control-watchdog mode is supported.

For more information about this feature and its potential applications, please refer to the following community post:

dcb priority-flow-control enable

dcb priority-flow-control enable [force]

disable dcb priority-flow-control [force]

no dcb priority-flow-control enable [force]

Enables PFC globally on the switch. It is also possible to assign specific interface behavior in dcb priority-flow-control mode.

The disable form of the command globally disables PFC on the switch when RoCE mode is set to lossless/semi-lossless.

The no form of the command sets global PFC to the default value. See “Default” section below.

Syntax Description

force

Forces operation

Default

PFC is generally disabled. See “RoCE Parameters” for specific RoCE modes in which the default is enabled

Configuration Mode

config

History

3.1.0000

3.3.0000

Updated example

3.8.2100

  • Updated example

  • Added "disable dcb priority-flow-control" command

  • Changes the function of the no form of the command

3.9.0500

Updated the description of the "disable" form of the command and added a note

Example

switch (config)# no roce

switch (config)# no dcb priority-flow-control enable force

switch (config)# show dcb priority-flow-control

PFC: disabled

switch (config)# dcb priority-flow-control enable

This action might cause traffic loss while shutting down a port with priority-flow-control mode on

Type 'yes' to confirm enable pfc globally: yes

switch (config)# show dcb priority-flow-control

PFC: enabled

switch (config)# roce semi-lossless

switch (config)# show dcb priority-flow-control

PFC: enabled

switch (config)# disable dcb priority-flow-control force

switch (config)# show dcb priority-flow-control

PFC: disabled

switch (config)# no dcb priority-flow-control enable force

switch (config)# show dcb priority-flow-control

PFC: enabled

Related Commands

show dcb priority-flow-control

dcb priority-flow-control mode

Notes

  • This command asks the user to approve traffic loss because some interfaces with DCB mode activated might get shut down.

  • The disable command is valid only for roce lossless/semi-lossless modes. For explicitly disabling PFC on other scenarios, please set the interface PFC mode to 'off' for all required ports.


dcb priority-flow-control priority

dcb priority-flow-control priority <prio> enable

no dcb priority-flow-control priority <prio> enable

Enables PFC per priority on the switch.

The no form of the command disables PFC per priority on the switch.

Syntax Description

prio

0-7

Default

PFC is disabled for all priorities.

Configuration Mode

config

History

3.1.0000

3.9.0500

Added note

Example

switch (config)# dcb priority-flow-control priority 0 enable

Related Commands

show dcb priority-flow-control

Notes

When RoCE mode is set to lossless/semi-lossless, the no form of the command is not applicable. For explicitly disabling PFC, set the interface PFC mode to 'off' for all required ports.


dcb priority-flow-control mode

dcb priority-flow-control mode <mode> [force]

no dcb priority-flow-control mode [force]

Changes PFC mode per interface.

The no form of the command disables PFC per interface.

Syntax Description

force

Configures the PFC admin mode as on or auto with no confirmation needed if the port is admin enabled

mode

The interface PFC mode. Possible values:

  • on – enables PFC per interface

  • off – disables PFC per interface

  • auto – set PFC mode for the interface to be controlled with traffic pool configuration

Default

auto – PFC mode is established by traffic pool configuration (not a directly configurable mode)

Configuration Mode

config interface ethernet

config interface port-channel

config interface mlag-port-channel

History

3.1.0000

3.3.4500

Added MPO configuration mode

3.6.6000

Added “force” parameter

3.6.6102

Added “mode” parameter

3.6.7100

Updated “mode” parameter description

Example

switch (config interface ethernet 1/1) # dcb priority-flow-control mode on

Related Commands

show dcb priority-flow-control

Notes

  • For the “force” parameter, the no form of the command disables priority-flow-control without the preceding confirmation prompt

  • For mode value “auto”, if a lossless traffic pool is configured, PFC is enabled for this port. Otherwise, PFC is disabled.

pfc-wd

pfc-wd

no pfc-wd

Enables PFC watchdog on interface.

The no form of the command disables PFC watchdog on interface.

Syntax Description

N/A

Default

Disabled

Configuration Mode

config interface ethernet

config interface port-channel

config interface mlag-port-channel

History

3.6.6000

Example

switch (config interface ethernet 1/1) # pfc-wd

Related Commands

show interface pfc-wd

Notes

When a user enables both "flowcontrol receive on" and "pfc-wd" on specific port, global-flow-control-watchdog

mode is activated. If only "pfc-wd" is enabled, then the PFC-watchdog mode is activated.


show dcb priority-flow-control

show dcb priority-flow-control [interface <type> <inf>] [detail]

Displays DCB priority flow control configuration and status.

Syntax Description

type

  • ethernet

  • port-channel

inf

The interface number

detail

Adds details information to the show output

Default

N/A

Configuration Mode

Any command mode

History

3.1.0000

Example

switch (config) # show dcb priority-flow-control

PFC enabled

Priority Enabled List : 0

Priority Disabled List : 1 2 3 4 5 6 7

TC Lossless

--- ----------

0 N

1 Y

2 Y

3 N

Interface PFC admin PFC oper

------------ -------------- -------------

1/1 On Enabled

1/2 Disabled Disabled

1/3 Disabled Disabled

1/4 Disabled Disabled

...

Related Commands

Notes


show dcb priority-flow-control interface mlag-port-channel

show dcb priority-flow-control interface mlag-port-channel <inf> [detail]

Displays DCB priority flow control configuration and status for MPO interfaces.

Syntax Description

inf

The interface number.

detail

Adds details information to the show output.

Default

N/A

Configuration Mode

Any command mode

History

3.1.0000

3.6.6000

Updated example

Example

switch (config) # show dcb priority-flow-control interface mlag-port-channel 1 detail

PFC: disabled

Priority Enabled List:

Priority Disabled List: 0 1 2 3 4 5 6 7

PFC Port Mpo1 Information:

Port Mode : On

Operational state : Off

No Remote Entry is Present

Related Commands

Notes


show interface pfc-wd

show interface <type> <id> pfc-wd

Displays PFC watchdog information.

Syntax Description

type

Interface type:

  • ethernet

  • port-channel

  • mlag-port-channel

id

Interface ID

Default

N/A

Configuration Mode

Any command mode

History

3.6.6000

3.8.1300

Updated example

Example

switch (config) # show interfaces ethernet 1/1 pfc-wd
Interface ethernet 1/1:
   PFC-WD admin : enable
   PFC-WD mode : global / per-priority / n/a
   Traffic Class 0 state: OK
   Traffic Class 1 state: OK
   Traffic Class 2 state: OK
   Traffic Class 3 state: OK
   Traffic Class 4 state: OK
   Traffic Class 5 state: OK
   Traffic Class 6 state: OK
   Traffic Class 7 state: OK
switch (config) # 

Related Commands

pfc-wd

Notes

When PFC-watchdog mode is activated, display "per-priority" in "PFC-WD mode". While global flow control watchdog activated, display "global". Otherwise, display "n/a".


© Copyright 2023, NVIDIA. Last updated on Mar 5, 2024.