Buffer Histograms Monitoring
This feature is only supported on EDR switches.
As it is becoming increasingly complex to manage networks, and network administrators need more tools to understand network behavior, it is necessary to provide basic information about network performance, identify network bottlenecks, and provide information for the purposes of network optimization and future planning.
Therefore, network administrators are required to constantly review network port behavior, record port buffer consumption, and identify shortage in buffer resources and record flows which lead to the excessive buffer consumption. MLNX-OS provides the following mechanisms to perform these tasks:
Sampling (histograms)—a network administrator can enable a sampling of the port buffer occupancy, record occupancy changes over time, and provide information for different levels of buffer occupancy, and amount of time the buffer has been occupied during the observation period.
Thresholds—thresholds may be enabled per port to record the network time when port buffer occupancy crosses the defined threshold and when buffer occupancy drops below it.
Flow recording—a record of the most active flows which cause an excessive usage of the port buffers may be kept. Once enabled, the system may identify flow patterns and present a user with a list of flows, based on which a network administrator can rearrange distribution of the data flows in the network and minimize data loss.
protocol telemetry
protocol telemetry no protocol telemetry Unhides telemetry config CLIs. The no form of the command hides telemetry config CLIs. | ||
Syntax Description | N/A | |
Default | Hidden | |
Configuration Mode | config | |
History | 3.6.3004 | |
Example | switch (config) # protocol telemetry | |
Related Commands | ||
Notes | ||
telemetry shutdown
telemetry shutdown no telemetry shutdown Disables the telemetry protocol, threshold detection, and histogram fetching for all sampling enabled interfaces without changing any internal configuration. The no form of the command enables telemetry protocol. | ||
Syntax Description | N/A | |
Default | Disabled | |
Configuration Mode | config | |
History | 3.6.3004 | |
Example | switch (config) # no telemetry shutdown | |
Related Commands | protocol telemetry | |
Notes | ||
telemetry sampling
interface ib <slot>/<port> telemetry sampling no interface ib <slot>/<port> telemetry sampling Enables sampling (histogram fetching) for a specific InfiniBand interface. The no form of the command disables sampling (histogram fetching). | ||
Syntax Description | N/A | |
Default | N/A | |
Configuration Mode | config interface ib | |
History | 3.6.3004 | |
Example | switch (config interface ib 1/1) # telemetry sampling | |
Related Commands | protocol telemetry | |
Notes | ||
telemetry sampling log
telemetry sampling log <time> no telemetry sampling log <time> Enables the log interval value (histogram fetching) from device. The no form of the command disables the log interval value. | ||
Syntax Description | time | Input range: 100-60000 (in msec) |
Default | 1000 millisecond | |
Configuration Mode | config | |
History | 3.6.3004 | |
Example | switch (config) # telemetry sampling log 1000 | |
Related Commands | protocol telemetry | |
Notes | ||
telemetry threshold
telemetry threshold no telemetry threshold Enables telemetry threshold on hardware. The no form of the command disables threshold. | ||
Syntax Description | N/A | |
Default | false | |
Configuration Mode | config interface ib | |
History | 3.6.4006 | |
Example | switch (config interface ib 1/1) # telemetry threshold | |
Related Commands | ||
Notes | ||
telemetry threshold level
telemetry threshold level <level> no telemetry threshold level Configures threshold level in hardware per port. The no form of the command resets threshold to default value. | ||
Syntax Description | level | Input range: 96-1000000 (in bytes and in increments of 96) |
Default | 69984 | |
Configuration Mode | config interface ib | |
History | 3.6.4006 | |
Example | switch (config interface ib 1/1) # telemetry threshold level 288 | |
Related Commands | ||
Notes | ||
telemetry threshold log
telemetry threshold log no telemetry threshold log Enables logging of threshold events in syslog. The no form of the command disables logging. | ||
Syntax Description | N/A | |
Default | Disabled | |
Configuration Mode | config | |
History | 3.6.4006 | |
Example | switch (config) # telemetry threshold log | |
Related Commands | ||
Notes | ||
telemetry threshold record
telemetry threshold record no telemetry threshold record Enables top talker configuration. The no form of the command disables top talker configuration. | ||
Syntax Description | N/A | |
Default | Disabled | |
Configuration Mode | config interface ib | |
History | 3.6.6105 | |
Example | switch (config interfaces ib 1/2) # telemetry threshold record | |
Related Commands | clear telemetry threshold record show telemetry threshold record | |
Notes |
| |
telemetry threshold syslog
telemetry threshold syslog <time> no telemetry threshold syslog <time> The command sets threshold events logging rate on per hour basis. The no form of the command sets the logging rate back to default. | ||
Syntax Description | time | Max rate per hour Range: 1-3600 |
Default | 100 | |
Configuration Mode | config | |
History | 3.6.4006 | |
Example | switch (config) # telemetry threshold syslog 400 | |
Related Commands | ||
Notes | ||
clear telemetry
clear telemetry {threshold | sampling} [interface ib <port-id>] Clears telemetry data. | ||
Syntax Description | port-id | InfiniBand interface ID |
Default | N/A | |
Configuration Mode | config interface ib | |
History | 3.6.5000 | |
Example | switch (config interface ib 1/12) # clear telemetry threshold level 288 | |
Related Commands | ||
Notes | ||
clear telemetry threshold
clear telemetry threshold [interface <type> <if>] Clears threshold and top talker data. | ||
Syntax Description | type | Available values:ib |
Default | N/A | |
Configuration Mode | config | |
History | 3.6.6105 | |
Example | switch (config) # clear telemetry threshold interface ib 1/34-1/36 | |
Related Commands | ||
Notes | ||
clear telemetry threshold record
clear telemetry threshold record [interface ib <if>] Clears top talker data. | ||
Syntax Description | if | InfiniBand interface ID |
Default | N/A | |
Configuration Mode | config | |
History | 3.6.6105 | |
Example | switch (config) # clear telemetry threshold record interface ib 1/34-1/36 | |
Related Commands | telemetry threshold record show telemetry threshold record | |
Notes | ||
stats export csv telemetry
stats export csv telemetry <slot>/<port>[/<subport>][filename <name>] [after * *] [before * *] Exports histograms collected by stats to a csv file. | ||
Syntax Description | slot/port | Port number |
subport | Subport number to be used if a port is split | |
Default | N/A | |
Configuration Mode | Any command mode | |
History | 3.6.3004 | |
3.9.0500 | Updated example | |
Example |
switch (config) # stats export csv telemetry 1/1/4-ucast after 2020/03/16 10:54:58 before 2020/03/16 11:16:24
Generated report file: telemetry-20200316-111704.csv
| |
Related Commands | ||
Notes | ||
file stats telemetry delete
file stats telemetry delete <filename> Deletes the given .csv file created by “stats export” command to user directory. | ||
Syntax Description | N/A | |
Default | N/A | |
Configuration Mode | config | |
History | 3.6.3004 | |
Example | switch (config) # file stats telemetry delete telemetry-20171006-102158.csv | |
Related Commands | ||
Notes | ||
file stats telemetry delete latest
file stats telemetry delete latest Delete the latest stats telemetry file. | ||
Syntax Description | N/A | |
Default | N/A | |
Configuration Mode | Configure terminal | |
History | 3.8.1000 | |
Example | (config) # file stats telemetry delete latest | |
Related Commands | file stats telemetry delete <file_name> file stats telemetry delete all | |
Notes | ||
file stats telemetry delete all
file stats telemetry delete all Deletes all stats telemetry files from machine. | ||
Syntax Description | N/A | |
Default | N/A | |
Configuration Mode | Configure terminal | |
History | 3.8.1000 | |
Example | (config) # file stats telemetry delete all | |
Related Commands | file stats telemetry delete <file_name> file stats telemetry delete latest | |
Notes | ||
file stats telemetry upload
file stats telemetry upload <filename> <upload-url> Uploads .csv file created by “stats export” command to user directory. | ||
Syntax Description | N/A | |
Default | N/A | |
Configuration Mode | config | |
History | 3.6.3004 | |
Example | switch (config) # file stats telemetry upload telemetry-20170119-102715.csv scp://username:password@server//directoryPassword (if required): ****** | |
Related Commands | ||
Notes | ||
file stats telemetry upload latest
file stats telemetry upload latest <upload-url> Upload the latest stats telemetry file to a remote host. | ||
Syntax Description | N/A | |
Default | N/A | |
Configuration Mode | Configure terminal | |
History | 3.8.1000 | |
Example | (config) # file stats telemetry upload latest scp://user:pass@10.135.155.8/tmp | |
Related Commands | file stats telemetry upload <file_name> file stats telemetry upload all | |
Notes | ||
file stats telemetry upload all
file stats telemetry upload all <upload_url> Upload all stats telemetry files to a remote host. | ||
Syntax Description | ||
Default | N/A | |
Configuration Mode | Configure terminal | |
History | 3.8.1000 | |
Example | (config) # file stats telemetry upload all scp://user:pass@10.135.155.8/tmp | |
Related Commands | file stats telemetry upload <file_name> file stats telemetry upload latest | |
Notes | ||
show telemetry
show telemetry Displays the global configuration of telemetry properties. | ||
Syntax Description | N/A | |
Default | N/A | |
Configuration Mode | config | |
History | 3.6.4000 | |
Example | ||
switch (config) # show telemetry Telemetry Status : EnabledH/W Sampling Interval(nsec) : 512S/W Sampling Interval(ms) : 1000Threshold Logging : DisabledThreshold Logging(rate per hour) : 100--------------------------------------------------------------------------------------------Interface Sampling Threshold Record Level (bytes)--------------------------------------------------------------------------------------------IB1/1 Disabled Enabled Enabled 100 (96)IB1/2 Disabled Enabled Enabled 100 (96)IB1/3 Disabled Disabled Disabled N/A IB1/4 Disabled Disabled Disabled N/A IB1/5 Disabled Disabled Disabled N/A IB1/6 Disabled Disabled Disabled N/A IB1/7 Disabled Disabled Disabled N/A ...IB1/36 Disabled Disabled Disabled N/A | ||
Related Commands | ||
Notes | ||
show telemetry threshold record
show telemetry threshold record [interface ib <interface-id> | <interface-id-range>] Displays top talker events for all configured ports. | ||
Syntax Description | N/A | |
Default | N/A | |
Configuration Mode | Any command mode | |
History | 3.6.4006 | |
3.6.6105 | Updated example | |
3.6.8100 | Updated example | |
Example | ||
switch (config) # show telemetry threshold record interface ib 1/11-1/12-----------------------------------------------------------------------------------------------------------------Event-id Date Time Port Level Duration(100 usec) Repeated DestQP DLID SLID Percent(%)-----------------------------------------------------------------------------------------------------------------1 07/10/18 14:00:31 IB 1/11 69984 48749.77 1 2741 29 32 62.302 07/10/18 14:01:47 IB 1/11 69984 63936.16 1 2745 29 32 54.55 | ||
Related Commands | clear telemetry threshold | |
Notes |
| |
show telemetry sampling interface ib
show telemetry sampling interface ib <slot>/<port> Displays telemetry histogram samples for a specific InfiniBand interface. | ||
Syntax Description | slot/port | InfiniBand interface number |
Default | N/A | |
Configuration Mode | Any command mode | |
History | 3.6.3004 | |
Example | ||
switch (config) # show telemetry sampling interface ib 1/32 -------------------------------------------------------------------------------------------------------------------------------------Telemetry histogram: IB1/32System-time Bin sizes (128 nsec tx buffer was occupied in bytes range)-------------------------------------------------------------------------------------------------------------------------------------02/09/17 <2976 35744 68512 101280 134048 166816 199584 232352 265120 265120<12:19:03.41948 1883 8538 7802080 0 0 0 0 0 0 012:19:04.42107 830 9001 7802670 0 0 0 0 0 0 012:19:05.42249 96 9705 7802700 0 0 0 0 0 0 012:19:06.42388 32 9035 7803434 0 0 0 0 0 0 012:19:07.42573 80 9461 7802960 0 0 0 0 0 0 012:19:08.42761 160 9302 7803040 0 0 0 0 0 0 012:19:09.42915 304 9369 7802829 0 0 0 0 0 0 012:19:10.43071 96 8906 7803500 0 0 0 0 0 0 012:19:11.43215 463 8907 7803132 0 0 0 0 0 0 012:19:12.43369 256 8571 7803675 0 0 0 0 0 0 0 | ||
Related Commands | ||
Notes | If the requested entries are more than what the DB contains, it prints the amount in the table. | |
show telemetry sampling interface ib last
show telemetry sampling interface ib <slot>/<port> last <num_of_entries> Displays fetched unicast histogram details for an InfiniBand interface. | ||
Syntax Description | slot/port | InfiniBand interface number |
num_of_entries | Range: 0-1000 | |
Default | N/A | |
Configuration Mode | Any command mode | |
History | 3.6.3004 | |
Example | ||
switch (config) # show telemetry sampling interface ib 1/36 last 20 Legend: 2976 bytes - between 0 - 2976 of tx bytes buffer consumed 35744 bytes - between 2977 - 35744 of tx bytes buffer consumed-------------------------------------------------------------------------------------------------------------------------------------Telemetry histogram: IB1/36System-time Bin sizes (128 nsec tx buffer was occupied in bytes range)-------------------------------------------------------------------------------------------------------------------------------------02/09/17 <2976 35744 68512 101280 134048 166816 199584 232352 265120 265120<12:19:03.41948 1883 8538 7802080 0 0 0 0 0 0 012:19:04.42107 830 9001 7802670 0 0 0 0 0 0 012:19:05.42249 96 9705 7802700 0 0 0 0 0 0 012:19:06.42388 32 9035 7803434 0 0 0 0 0 0 012:19:07.42573 80 9461 7802960 0 0 0 0 0 0 0 | ||
Related Commands | ||
Notes | If requested entries are more than what the DB contains, it prints the amount in the table. | |
show files stats telemetry
show files stats telemetry [filename] Displays all files created by the command “stats export csv telemetry”. | ||
Syntax Description | filename | Displays stats for the specified file |
Default | N/A | |
Configuration Mode | Any command mode | |
History | 3.6.3004 | |
3.6.8008 | Updated example | |
Example | switch (config) # show files stats telemetry telemetry-20180527-102715.csvHostname :test-switchReport :telemetry histogramTime lower bound(UTC) :2018/05/28 05:58:10Time upper bound(UTC) :2018/05/28 05:58:25Export time(UTC) :2018/05/28 06:00:06Time lower bound :2018/05/28 08:58:10 +0300Time upper bound :2018/05/28 08:58:25 +0300Export time :2018/05/28 09:00:06 +0300System version :X86_64 sys_test 2018-05-15 04:02:13 x86_64 | |
Related Commands | stats export csv telemetry | |
Notes | ||