This documentation is for the extended support release (ESR) version of Cumulus Linux. We will continue to keep this content up to date until 21 February, 2023, when ESR support ends. For more information about ESR, please read this knowledge base article.

If you are using the current version of Cumulus Linux, the content on this page may not be up to date. The current version of the documentation is available here. If you are redirected to the main page of the user guide, then this page may have been renamed; please search for it there.

VXLAN Tunnel DSCP Operations

Cumulus Linux 3.7.4 and later provides configuration options to control DSCP operations during VXLAN encapsulation and decapsulation, specifically for solutions that require end-to-end quality of service, such as RDMA over Converged Ethernet.

The configuration options propagate explicit congestion notification (ECN) between the underlay and overlay and are based on RFC 6040, which describes how to construct the IP header of an ECN field on both ingress to and egress from an IP-in-IP tunnel.

VXLAN Tunnel DSCP operations are supported on Mellanox Spectrum switches only.

Configure DSCP Operations

You can set the following DSCP operations by editing the /etc/cumulus/switchd.conf file.

vxlan.def_encap_dscp_actionSets the VXLAN outer DSCP action during encapsulation. You can specify one of the following options:
- copy (if the inner packet is IP)
- set (to a specific value)
- derive (from the switch priority).
The default setting is derive.
vxlan.def_encap_dscp_valueIf the vxlan.def_encap_dscp_action option is set, you must specify a value.
xlan.def_decap_dscp_actionSets the VXLAN decapsulation DSCP/COS action. You can specify one of the following options:
- copy (if the inner packet is IP)
- preserve (the inner DSCP is unchanged)
- derive (from the switch priority)

After you modify /etc/cumulus/switchd.conf file, you must restart switchd for the changes to take effect.

cumulus@switch:~$ sudo systemctl restart switchd.service

Restarting the switchd service causes all network ports to reset, interrupting network services, in addition to resetting the switch hardware configuration.

The following example shows that the VXLAN outer DSCP action during encapsulation is set with a value of 16.

cumulus@switch:~$ sudo nano /etc/cumulus/switchd.conf
# default vxlan outer dscp action during encap
# {copy | set | derive}
# copy: only if inner packet is IP
# set: to specific value
# derive: from switch priority
vxlan.def_encap_dscp_action = set

# default vxlan encap dscp value, only applicable if action is 'set'
vxlan.def_encap_dscp_value = 16

# default vxlan decap dscp/cos action
# {copy | preserve | derive}
# copy: only if inner packet is IP
# preserve: inner dscp unchanged
# derive: from switch priority
#vxlan.def_decap_dscp_action = derive

You can also set the DSCP operations from the command line. Use the echo command to change the settings in the /etc/cumulus/switchd.conf file. For example, to change the encapsulation action to copy:

cumulus@switch:~$ echo "copy" > /cumulus/switchd/config/vxlan/def_encap_dscp_action

To change the VXLAN outer DSCP action during encapsulation to set with a value of 32:

cumulus@switch:~$ echo "32" > /cumulus/switchd/config/vxlan/def_encap_dscp_value
cumulus@switch:~$ echo "set" > /cumulus/switchd/config/vxlan/def_encap_dscp_action

Caveats and Errata

Cumulus Linux supports only the default global settings. Per-VXLAN and per-tunnel granularity are not supported.