RSS Monitoring
The RSS monitoring feature allows users to track the packets received by the Mellanox adapters on their machine.
The feature is comprised of two parts:
Non-RSS Packet Capturing
A command-line interface for capturing received non-RSS packets.
Packet monitoring through the performance monitor
A counter in the performance monitor for tracking the received non-RSS packets
Both parts of the feature must be enabled (per adapter instance), using the registry key: "RssMonitoringEnabled". Restart the adapter for these changes to take effect.
Registry key location:
HKLM\SYSTEM\CurrentControlSet\Control\Class\{4d36e972-e325-11ce-bfc1-08002be10318}\<nn>\RssMonitoringEnabled
For instructions on how to identify the interface index of a network interface in the registry <nn>, please refer to Finding the Index Value of the Network Interface.
Controlling the RSS Monitoring Registry Key Feature
Key Name |
Key Type |
Values |
Description |
RssMonitoringEnabled |
REG_DWORD |
[0, 1] Default - Registry Key is not created (0) |
Disable and enable the RSS Monitoring Feature (including packet capturing and tracking through the performance monitor) Value 0 - Disable the feature Value 1 - Enable the feature |
This feature is added as a flag to "mlxtool.exe", which is installed as part of the WinOF package. The tool can be used to capture the Non-RSS packets received by the machine, and dump the captured traffic into a PCAP file.
Tool Default Path:
C:\Program Files\Mellanox\MLNX_VPI\Tools\mlxtool.exe
Using the Tool
Step 1. Enable the feature through the registry, as described in RSS Monitoring above.
Step 2. Obtain the name of the Mellanox adapter on your machine.
> mlxtool.exe show ports
Step 3. Query the status of the packet capturing. This can be used to tell if the tool is currently running:
> mlxtool.exe dbg nonrss-capture "Mellanox Adapter Name"
query
Step 4. Start the packet capturing:
When activating the feature, the user can specify two parameters:
Circular Buffer Size>: The number of packets to store in the capture buffer. This buffer is flushed when "collect" is called (see #5 for an description of collect). This is a circular buffer, so once it is full, it will override previous entries. If not specified, the default will be used. Default: 1024 packets.
<Number of Bytes to Capture per Packet>: The number of bytes to capture from each packet. Users can choose to only capture a "snip" of each packet. If not specified, the default will be used. Default: 128 bytes.
In case of VPort mode (when NicSwitch is enabled), these parameters are global for all VPorts. The feature is either enabled on all VPorts or disabled on all VPorts. The circular buffer size and number of bytes to capture per packet cannot be specified per VPort.
> mlxtool.exe dbg nonrss-capture "Mellanox Adapter Name"
start <Circular Buffer Size> <Number of Bytes to Capture per Packet>
Step 5. Collect the captured packets:
Packets are stored in a circular buffer. They are dumped to a PCAP file when "collect" is called. This must be called before calling "stop", since the buffer gets cleared once the tool is stopped. The status of the feature can be retrieved using the "query" command described in #3.
When capturing the dump, the user can specify two parameters:
<PCAP file name>: The name of the packet dump file. If not specified, the default will be used. Default: nonrss.pcap.
<VPort ID>: The Vport ID to collect for. This is only valid when the driver is operating in VPort mode (NicSwitch is enabled). If not specified, the default will be used. Default: 0.
> mlxtool.exe dbg nonrss-capture "Mellanox Adapter Name"
collect <PCAP File Name> <VPort ID>
Step 6. Stop the packet capture.
If "collect" is not called before the packet capture is stopped, the buffer that contains the captured packets will be cleared. Call "collect" before stopping the feature to dump the packets to a PCAP file.
> mlxtool.exe dbg nonrss-capture "Mellanox Adapter Name"
stop
Packet Monitoring through the Performance Monitor
This feature activates counters in the performance monitor which can be used to monitor the packets that are being received by the Mellanox adapter(s). The feature allows the user to monitor incoming traffic for each adapter. It is split up per VPort (if NicSwitch is enabled) and per CPU.
All available counters are listed in the following figure:
Using the Tool
Step 1. Enable the feature through the registry, as described in RSS Monitoring above. In addition to the "RssMonitoringEnabled" registry key, this feature has an additional registry key "RssCountersActivatedAtStartup". This counter will begin counting received packets as soon as the adapter is connected. If this is not set, the counter will begin counting received packets as soon as the counter is added to the performance monitor. Restart the adapter for these changes to take effect.
Registry key location:
HKLM\SYSTEM\CurrentControlSet\Control\Class\{4d36e972-e325-11ce-bfc1-08002be10318}\<nn>\RssCountersActivatedAtStartup
For instructions on how to identify the interface index of a network interface in the registry <nn>, please refer to Finding the Index Value of the Network Interface.
Disable and Enable RSS Monitoring On Adapter Startup with the RssCountersActivatedAtStartup Registry Key
Key Name |
Key Type |
Values |
Description |
RssCountersActivatedAtStartup |
REG_DWORD |
[0, 1] Default - Registry Key is not created (0) |
Choose when to begin counting received packets. Value 0 - Begin counting packets when the counter is added to the performance monitor Value 1 - Begin counting packets on adapter startup |
Step 2. Open up the Windows performance monitor and add a new counter. The counter will be called "Mellanox Adapter Rss Counters".
Step 3. Select and add the desired instances of the "Mellanox Adapter Rss Counters".
Note: There will be instances for each Mellanox Adapter that has the proper registry keys set (see introduction to Rss Monitoring for more information on registry keys).
Note: In VPort mode (when NicSwitch is enabled), there will be a counter available per VPort for each CPU. There is also a "Total" for each VPort, which will report the total packets received by that VPort.
In native mode, there will be a counter available per CPU. There will also be a "Total" for each adapter, which will report the total packets received by that adapter.
Step 4. Switch the graph type in the performance monitor to "Report".
Step 5. Once the counters are added, the counters will get incremented based on the traffic that is received.
The Proprietary Adapter Mellanox RSS counter set consists of software counters per vport (in case of native RSS it is the physical port) and per CPU. These counters collect information about the RSS and non-RSS traffic received on a specific vport (or physical port for native RSS) and on a specific CPU. The counter set also defines a "Total" instance to collect on all CPUs for each available vport.
Mellanox Adapter RSS Counters |
Description |
Encapsulated NonRss IPv4 Only |
Number of encapsulated Ipv4Only packets that have no RSS hash calculated by HW |
Encapsulated NonRss IPv4/Tcp |
Number of encapsulated Ipv4Tcp packets that have no RSS hash calculated by HW |
Encapsulated NonRss IPv4/Udp |
Number of encapsulated Ipv4Udp packets that have no RSS hash calculated by HW |
Encapsulated NonRss IPv6 Only |
Number of encapsulated Ipv6Only packets that have no RSS hash calculated by HW |
Encapsulated NonRss IPv6/Tcp |
Number of encapsulated Ipv6Tcp packets that have no RSS hash calculated by HW |
Encapsulated NonRss IPv6/Udp |
Number of encapsulated Ipv6Udp packets that have no RSS hash calculated by HW |
Encapsulated NonRss Misc |
Number of encapsulated packets that have no RSS hash calculated by HW without clear reason for that |
Encapsulated Rss IPv4 Only |
Number of received encapsulated packets that have RSS hash calculated on IPv4 header only |
Encapsulated Rss IPv4/Tcp |
Number of received encapsulated packets that have RSS hash calculated on IPv4 and Tcp headers |
Encapsulated Rss IPv4/Udp |
Number of received encapsulated packets that have RSS hash calculated on IPv4 and Udp headers |
Encapsulated Rss IPv6 Only |
Number of received encapsulated packets that have RSS hash calculated on IPv6 header only |
Encapsulated Rss IPv6/Tcp |
Number of received encapsulated packets that have RSS hash calculated on IPv6 and Tcp headers |
Encapsulated Rss IPv6/Udp |
Number of received encapsulated packets that have RSS hash calculated on IPv6 and Udp headers |
Encapsulated Rss Misc |
Number of received encapsulated packets that have RSS hash calculated with unknown RSS hash type |
NonRss IPv4 Only |
Number of Ipv4only packets that have no RSS hash calculated by HW |
NonRss IPv4/Tcp |
Number of Ipv4Tcp packets that have no RSS hash calculated by HW |
NonRss IPv4/Udp |
Number of Ipv4Udp packets that have no RSS hash calculated by HW |
NonRss IPv6 Only |
Number of Ipv6Only packets that have no RSS hash calculated by HW |
NonRss IPv6/Tcp |
Number of Ipv6Tcp packets that have no RSS hash calculated by HW |
NonRss IPv6/Udp |
Number of Ipv6Udp packets that have no RSS hash calculated by HW |
NonRss Misc |
Number of packets that have no RSS hash calculated by HW without clear reason for that |
Rss IPv4 Only |
Number of received packets that have RSS hash calculated on IPv4 header only |
Rss IPv4/Tcp |
Number of received packets that have RSS hash calculated on IPv4 and Tcp headers |
Rss IPv4/Udp |
Number of received packets that have RSS hash calculated on IPv4 and Udp headers |
Rss IPv6 Only |
Number of received packets that have RSS hash calculated on IPv6 header only |
Rss IPv6/Tcp |
Number of received packets that have RSS hash calculated on IPv6 and Tcp headers |
Rss IPv6/Udp |
Number of received packets that have RSS hash calculated on IPv6 and Udp headers |
Rss Misc |
Number of received packets that have RSS hash calculated with unknown RSS hash type |
Controlling the Counters
As of Version 5.30, the Adapter RSS Counter set is disabled by default. In order to enable it, the following registry value must be set to "1":
HKLM\SYSTEM\CurrentControlSet\Control\Class\{4d36e972-e325-11ce-bfc1-08002be10318}\<nn>\RssMonitoringEnabled
Since these are software counters, and may have impact on the performance, they are activated only when the user has explicitly added the desired counters instance in Perfmon (or any other PCW application).
If "always on" behavior is desired (start counting upon driver load), the following registry key must be set to "1":
HKLM\SYSTEM\CurrentControlSet\Control\Class\{4d36e972-e325-11ce-bfc1-08002be10318}\<nn>\RssCountersActivatedAtStartup
Since the Adapter RSS Counter set consists of software counters that are collected by the driver, unlike other adapter counters, they are not persistent and are cleared/reset upon driver restart.
Proprietary RDMA Activity
Proprietary RDMA Activity counter set consists of NDK and NDSPI performance counters. These performance counters allow you to track Network Direct Kernel (RDMA) activity, including traffic rates, errors, and control plane activity.
This counters set is relevant only for ETH ports.
RDMA Activity Counters |
Description |
RDMA Accepted Connectionsa |
The number of inbound RDMA connections established. |
RDMA Active Connectionsa |
The number of active RDMA connections. |
RDMA Completion Queue Errorsa |
This counter is not supported, and always is set to zero. |
RDMA Connection Errorsa |
The number of established connections with an error before a consumer disconnected the connection. |
RDMA Failed Connection Attemptsa |
The number of inbound and outbound RDMA connection attempts that failed. |
RDMA Inbound Bytes/sec |
The number of bytes for all incoming RDMA traffic. This includes additional layer two protocol overhead. |
RDMA Inbound Frames/sec |
The number, in frames, of layer two frames that carry incoming RDMA traffic. |
RDMA Initiated Connectionsa |
The number of outbound connections established. |
RDMA Outbound Bytes/sec |
The number of bytes for all outgoing RDMA traffic. This includes addi- tional layer two protocol overhead. |
RDMA Outbound Frames/sec |
The number, in frames, of layer two frames that carry outgoing RDMA traffic. |
Note a. These counters are only implemented in NDK and are not implemented in NDSPI.