Priority Flow Control (PFC) provides an enhancement to the existing pause mechanism in Ethernet. The current Ethernet pause option stops all traffic on a link. PFC creates eight separate virtual links on the physical link and allows any of these links to be paused and restarted independently, enabling the network to create a no-drop class of service for an individual virtual link.
PFC offers the following features:
- Provides per-priority enabling or disabling of flow control
- Transmits PFC-PAUSE frames when the receive threshold for a particular traffic class is reached
- Provides the management capability for an administrator to configure the flow control properties on each port of the switch
- Keeps flow control disabled for all priorities on all ports by default
- Allows an administrator to enable or disable flow control per port and per priority level
- Supports flow control only on physical ports, not on logical interfaces such as tunnels or interfaces defined by sharing a physical port in multiple virtual switch contexts
- Uses the configured threshold values to set up the queue buffer spaces accordingly in the data-path
- Provides hardware abstraction layer call-outs for the following:
- Enabling or disabling of flow control on each port for each priority
- Configuring the queue depth for each priority on each port
- Provides trace logs for execution upon error conditions and for any event notifications from the hardware or data-path. These trace logs are a useful aid in troubleshooting.
- Allows the administrator to configure the minimum and maximum threshold values for flow control. These configurations are applied globally on all ports and priorities.
Priority Based Flow Control (PFC) provides an enhancement to the existing pause flow control mechanism as described in 802.1x.
To enable PFC globally,
switch (config) # dcb priority-flow-control enable This action might cause traffic loss while shutting down a port with priority-flow-control mode on Type 'yes' to confirm enable pfc globally: yes
To enable PFC per priority:
Enable PFC globally on the switch.
switch (config) # dcb priority-flow-control enable This action might cause traffic loss while shutting down a port with priority-flow-control mode on Type 'yes' to confirm enable pfc globally: yes
Choose the priority you want to enable.
switch (config) # dcb priority-flow-control priority 5 enable
To enable PFC per interface, do the following.
Enable PFC globally on the switch.
switch (config) # dcb priority-flow-control enable
Choose the priority you want to enable.
switch (config) # dcb priority-flow-control 5 enable
Change to Interface mode.
switch (config) # interface ethernet 1/1 switch (config interface ethernet 1/1) #
Enable PFC for the specific interface.
switch (config interface ethernet 1/1) # dcb priority-flow-control mode on
When working with lossless traffic, the receiving side sends a pause frame (Xoff) to the transmitting side before the buffer is filled. When the buffer empties, the receiving side sends an un-pause frame (Xon) to the transmitting side.
Flow Control Threshold Configuration
The user has to set the buffer usage Xoff and Xon thresholds. The thresholds depend on network parameters (bandwidth, link latency, MTU) and the allocated size for the region.
When working with global flow control mode only, a single PG shall be used and Xoff and Xon shall be set on this PG. When working with priority flow control, Xoff and Xon shall be set on each lossless PG.
PFC Watchdog
Lossless networks with PFC enabled provide strong packet delivery guarantees. However, lossless networks introduce a new fault scenario where a queue of an end-port (e.g. the port of a host connected to the network) may not be able to receive any traffic from the network and keeps sending pause frames towards the switch. Since lossless switch paths do not drop packets but decline receiving more packets when their buffers fill up, if the end-port queue is stuck for a long time, the buffers fill up not only for the target switch, but also on all switches with problematic port queues in the traffic forwarding path. This leads to endless PFC pause frames, also called a PFC storm, being observed on all switch ports along the path to the traffic source.
PFC watchdog prevents congestion from spreading in such a case. When switches detect this situation on any TC queue, all the packets in the queue are flushed and new packets destined to the same queue are dropped as well until PFC storming is relieved.
For lossless networks with global flow control configured, we will face the same issue of global pause storm. To resolve this, global-flow-control-watchdog mode is supported.
Additional Reading and Use Cases
For more information about this feature and its potential applications, please refer to the following community post:
PFC Commands
dcb priority-flow-control enable
dcb priority-flow-control enable [force] Enables PFC globally on the switch. It is also possible to assign specific interface behavior in dcb priority-flow-control mode. | ||
Syntax Description | force | Forces operation |
Default | PFC is generally disabled. See “RoCE Parameters” for specific RoCE modes in which the default is enabled | |
Configuration Mode | config | |
History | 3.1.0000 | |
3.3.0000 | Updated example | |
3.8.2100 |
| |
3.9.0500 | Updated the description of the "disable" form of the command and added a note | |
Example | switch (config)# no roce | |
Related Commands | show dcb priority-flow-control | |
Notes |
|
dcb priority-flow-control priority
dcb priority-flow-control priority <prio> enable Enables PFC per priority on the switch. | ||
Syntax Description | prio | 0-7 |
Default | PFC is disabled for all priorities. | |
Configuration Mode | config | |
History | 3.1.0000 | |
3.9.0500 | Added note | |
Example | switch (config)# dcb priority-flow-control priority 0 enable | |
Related Commands | show dcb priority-flow-control | |
Notes | When RoCE mode is set to lossless/semi-lossless, the no form of the command is not applicable. For explicitly disabling PFC, set the interface PFC mode to 'off' for all required ports. |
dcb priority-flow-control mode
dcb priority-flow-control mode <mode> [force] Changes PFC mode per interface. | ||
Syntax Description | force | Configures the PFC admin mode as on or auto with no confirmation needed if the port is admin enabled |
mode | The interface PFC mode. Possible values:
| |
Default | auto – PFC mode is established by traffic pool configuration (not a directly configurable mode) | |
Configuration Mode | config interface ethernet | |
History | 3.1.0000 | |
3.3.4500 | Added MPO configuration mode | |
3.6.6000 | Added “force” parameter | |
3.6.6102 | Added “mode” parameter | |
3.6.7100 | Updated “mode” parameter description | |
Example | switch (config interface ethernet 1/1) # dcb priority-flow-control mode on | |
Related Commands | show dcb priority-flow-control | |
Notes |
|
pfc-wd
pfc-wd Enables PFC watchdog on interface. | ||
Syntax Description | N/A | |
Default | Disabled | |
Configuration Mode | config interface ethernet | |
History | 3.6.6000 | |
Example | switch (config interface ethernet 1/1) # pfc-wd | |
Related Commands | show interface pfc-wd | |
Notes | When a user enables both "flowcontrol receive on" and "pfc-wd" on specific port, global-flow-control-watchdog |
show dcb priority-flow-control
show dcb priority-flow-control [interface <type> <inf>] [detail] Displays DCB priority flow control configuration and status. | ||
Syntax Description | type |
|
inf | The interface number | |
detail | Adds details information to the show output | |
Default | N/A | |
Configuration Mode | Any command mode | |
History | 3.1.0000 | |
Example | switch (config) # show dcb priority-flow-control PFC enabled Priority Enabled List : 0 Priority Disabled List : 1 2 3 4 5 6 7 TC Lossless --- ---------- 0 N 1 Y 2 Y 3 N Interface PFC admin PFC oper ------------ -------------- ------------- 1/1 On Enabled 1/2 Disabled Disabled 1/3 Disabled Disabled 1/4 Disabled Disabled ... | |
Related Commands | ||
Notes |
show dcb priority-flow-control interface mlag-port-channel
show dcb priority-flow-control interface mlag-port-channel <inf> [detail] Displays DCB priority flow control configuration and status for MPO interfaces. | ||
Syntax Description | inf | The interface number. |
detail | Adds details information to the show output. | |
Default | N/A | |
Configuration Mode | Any command mode | |
History | 3.1.0000 | |
3.6.6000 | Updated example | |
Example | switch (config) # show dcb priority-flow-control interface mlag-port-channel 1 detail | |
Related Commands | ||
Notes |
show interface pfc-wd
| show interface <type> <id> pfc-wd Displays PFC watchdog information. | |
Syntax Description | type | Interface type:
|
id | Interface ID | |
Default | N/A | |
Configuration Mode | Any command mode | |
History | 3.6.6000 | |
3.8.1300 | Updated example | |
Example | switch (config) # show interfaces ethernet 1/1 pfc-wd Interface ethernet 1/1: PFC-WD admin : enable PFC-WD mode : global / per-priority / n/a Traffic Class 0 state: OK Traffic Class 1 state: OK Traffic Class 2 state: OK Traffic Class 3 state: OK Traffic Class 4 state: OK Traffic Class 5 state: OK Traffic Class 6 state: OK Traffic Class 7 state: OK switch (config) # | |
Related Commands | pfc-wd | |
Notes | When PFC-watchdog mode is activated, display "per-priority" in "PFC-WD mode". While global flow control watchdog activated, display "global". Otherwise, display "n/a". |