Buffer Histograms Monitoring

NVIDIA Docs Hub NVIDIA Networking Networking Software Switch Software NVIDIA MLNX-OS User Manual v3.10.5000 Buffer Histograms Monitoring

As it is becoming increasingly complex to manage networks, and network administrators need more tools to understand network behavior, it is necessary to provide basic information about network performance, identify network bottlenecks, and provide information for the purposes of network optimization and future planning.

Therefore, network administrators are required to constantly review network port behavior, record port buffer consumption, and identify shortage in buffer resources and record flows which lead to the excessive buffer consumption. MLNX-OS provides the following mechanisms to perform these tasks:

Sampling (histograms)—a network administrator can enable a sampling of the port buffer occupancy, record occupancy changes over time, and provide information for different levels of buffer occupancy, and amount of time the buffer has been occupied during the observation period.
Thresholds—thresholds may be enabled per port to record the network time when port buffer occupancy crosses the defined threshold and when buffer occupancy drops below it.
Flow recording—a record of the most active flows which cause an excessive usage of the port buffers may be kept. Once enabled, the system may identify flow patterns and present a user with a list of flows, based on which a network administrator can rearrange distribution of the data flows in the network and minimize data loss.

Buffer Histograms and Thresholds Commands

protocol telemetry

	protocol telemetry no protocol telemetry Unhides telemetry config CLIs. The no form of the command hides telemetry config CLIs.
Syntax Description	N/A
Default	Hidden
Configuration Mode	config
History	3.6.3004
Example	switch (config) # protocol telemetry
Related Commands
Notes

telemetry shutdown

	telemetry shutdown no telemetry shutdown Disables the telemetry protocol, threshold detection, and histogram fetching for all sampling enabled interfaces without changing any internal configuration. The no form of the command enables telemetry protocol.
Syntax Description	N/A
Default	Disabled
Configuration Mode	config
History	3.6.3004
Example	switch (config) # no telemetry shutdown
Related Commands	protocol telemetry
Notes

telemetry sampling

	interface ib <slot>/<port> telemetry sampling no interface ib <slot>/<port> telemetry sampling Enables sampling (histogram fetching) for a specific InfiniBand interface. The no form of the command disables sampling (histogram fetching).
Syntax Description	N/A
Default	N/A
Configuration Mode	config interface ib
History	3.6.3004
Example	switch (config interface ib 1/1) # telemetry sampling
Related Commands	protocol telemetry
Notes

telemetry sampling log

	telemetry sampling log <time> no telemetry sampling log <time> Enables the log interval value (histogram fetching) from device. The no form of the command disables the log interval value.
Syntax Description	time	Input range: 100-60000 (in msec)
Default	1000 millisecond
Configuration Mode	config
History	3.6.3004
Example	switch (config) # telemetry sampling log 1000
Related Commands	protocol telemetry
Notes

telemetry threshold

	telemetry threshold no telemetry threshold Enables telemetry threshold on hardware. The no form of the command disables threshold.
Syntax Description	N/A
Default	false
Configuration Mode	config interface ib
History	3.6.4006
Example	switch (config interface ib 1/1) # telemetry threshold
Related Commands
Notes

telemetry threshold level

	telemetry threshold level <level> no telemetry threshold level Configures threshold level in hardware per port. The no form of the command resets threshold to default value.
Syntax Description	level	Input range: 96-1000000 (in bytes and in increments of 96)
Default	69984
Configuration Mode	config interface ib
History	3.6.4006
Example	switch (config interface ib 1/1) # telemetry threshold level 288
Related Commands
Notes

telemetry threshold log

	telemetry threshold log no telemetry threshold log Enables logging of threshold events in syslog. The no form of the command disables logging.
Syntax Description	N/A
Default	Disabled
Configuration Mode	config
History	3.6.4006
Example	switch (config) # telemetry threshold log
Related Commands
Notes

telemetry threshold record

	telemetry threshold record no telemetry threshold record Enables top talker configuration. The no form of the command disables top talker configuration.
Syntax Description	N/A
Default	Disabled
Configuration Mode	config interface ib
History	3.6.6105
Example	switch (config interfaces ib 1/2) # telemetry threshold record
Related Commands	clear telemetry threshold record show telemetry threshold record
Notes	When top talker is enabled, the minimal threshold window supported is 20 msecs Due to event timing issues, very short threshold events may not gather sufficient traffic samples to allow top-talker analysis. As a result, top-talkers may not be fully displayed in the relevant show command.

telemetry threshold syslog

	telemetry threshold syslog <time> no telemetry threshold syslog <time> The command sets threshold events logging rate on per hour basis. The no form of the command sets the logging rate back to default.
Syntax Description	time	Max rate per hour Range: 1-3600
Default	100
Configuration Mode	config
History	3.6.4006
Example	switch (config) # telemetry threshold syslog 400
Related Commands
Notes

clear telemetry

	clear telemetry {threshold \| sampling} [interface ib <port-id>] Clears telemetry data.
Syntax Description	port-id	InfiniBand interface ID
Default	N/A
Configuration Mode	config interface ib
History	3.6.5000
Example	switch (config interface ib 1/12) # clear telemetry threshold level 288
Related Commands
Notes

clear telemetry threshold

	clear telemetry threshold [interface <type> <if>] Clears threshold and top talker data.
Syntax Description	type	Available values:ib
Default	N/A
Configuration Mode	config
History	3.6.6105
Example	switch (config) # clear telemetry threshold interface ib 1/34-1/36
Related Commands
Notes

clear telemetry threshold record

	clear telemetry threshold record [interface ib <if>] Clears top talker data.
Syntax Description	if	InfiniBand interface ID
Default	N/A
Configuration Mode	config
History	3.6.6105
Example	switch (config) # clear telemetry threshold record interface ib 1/34-1/36
Related Commands	telemetry threshold record show telemetry threshold record
Notes

stats export csv telemetry

	stats export csv telemetry <slot>/<port>[/<subport>][filename <name>] [after * ] [before *] Exports histograms collected by stats to a csv file.
Syntax Description	slot/port	Port number
Syntax Description	subport	Subport number to be used if a port is split
Default	N/A
Configuration Mode	Any command mode
History	3.6.3004
History	3.9.0500	Updated example
Example	switch (config) # stats export csv telemetry 1/1/4-ucast after 2020/03/16 10:54:58 before 2020/03/16 11:16:24 Generated report file: telemetry-20200316-111704.csv
Related Commands
Notes

file stats telemetry delete

	file stats telemetry delete <filename> Deletes the given .csv file created by “stats export” command to user directory.
Syntax Description	N/A
Default	N/A
Configuration Mode	config
History	3.6.3004
Example	switch (config) # file stats telemetry delete telemetry-20171006-102158.csv
Related Commands
Notes

file stats telemetry delete latest

	file stats telemetry delete latest Delete the latest stats telemetry file.
Syntax Description	N/A
Default	N/A
Configuration Mode	Configure terminal
History	3.8.1000
Example	(config) # file stats telemetry delete latest
Related Commands	file stats telemetry delete <file_name> file stats telemetry delete all
Notes

file stats telemetry delete all

	file stats telemetry delete all Deletes all stats telemetry files from machine.
Syntax Description	N/A
Default	N/A
Configuration Mode	Configure terminal
History	3.8.1000
Example	(config) # file stats telemetry delete all
Related Commands	file stats telemetry delete <file_name> file stats telemetry delete latest
Notes

file stats telemetry upload

	file stats telemetry upload <filename> <upload-url> Uploads .csv file created by “stats export” command to user directory.
Syntax Description	N/A
Default	N/A
Configuration Mode	config
History	3.6.3004
History
Example	switch (config) # file stats telemetry upload telemetry-20170119-102715.csv scp://username:password@server//directory Password (if required): ******
Related Commands
Notes

file stats telemetry upload latest

	file stats telemetry upload latest <upload-url> Upload the latest stats telemetry file to a remote host.
Syntax Description	N/A
Default	N/A
Configuration Mode	Configure terminal
History	3.8.1000
History
Example	(config) # file stats telemetry upload latest scp://user:pass@10.135.155.8/tmp
Related Commands	file stats telemetry upload <file_name> file stats telemetry upload all
Notes

file stats telemetry upload all

	file stats telemetry upload all <upload_url> Upload all stats telemetry files to a remote host.
Syntax Description
Default	N/A
Configuration Mode	Configure terminal
History	3.8.1000
History
Example	(config) # file stats telemetry upload all scp://user:pass@10.135.155.8/tmp
Related Commands	file stats telemetry upload <file_name> file stats telemetry upload latest
Notes

show telemetry

	show telemetry Displays the global configuration of telemetry properties.
Syntax Description	N/A
Default	N/A
Configuration Mode	config
History	3.6.4000
Example
switch (config) # show telemetry Telemetry Status : Enabled H/W Sampling Interval(nsec) : 512 S/W Sampling Interval(ms) : 1000 Threshold Logging : Disabled Threshold Logging(rate per hour) : 100 -------------------------------------------------------------------------------------------- Interface Sampling Threshold Record Level (bytes) -------------------------------------------------------------------------------------------- IB1/1 Disabled Enabled Enabled 100 (96) IB1/2 Disabled Enabled Enabled 100 (96) IB1/3 Disabled Disabled Disabled N/A IB1/4 Disabled Disabled Disabled N/A IB1/5 Disabled Disabled Disabled N/A IB1/6 Disabled Disabled Disabled N/A IB1/7 Disabled Disabled Disabled N/A ... IB1/36 Disabled Disabled Disabled N/A
Related Commands
Notes

show telemetry threshold record

	show telemetry threshold record [interface ib <interface-id> \| <interface-id-range>] Displays top talker events for all configured ports.
Syntax Description	N/A
Default	N/A
Configuration Mode	Any command mode
History	3.6.4006
	3.6.6105	Updated example
	3.6.8100	Updated example

Example
switch (config) # show telemetry threshold record interface ib 1/11-1/12 ----------------------------------------------------------------------------------------------------------------- Event-id Date Time Port Level Duration(100 usec) Repeated DestQP DLID SLID Percent(%) ----------------------------------------------------------------------------------------------------------------- 1 07/10/18 14:00:31 IB 1/11 69984 48749.77 1 2741 29 32 62.30 2 07/10/18 14:01:47 IB 1/11 69984 63936.16 1 2745 29 32 54.55
Related Commands	clear telemetry threshold
Notes	The values displayed of the SLID, DLID, and QP fields are in decimal The command supports displaying up to 1000 threshold events. As a result, if more than 1000 thresholds configured in total, some interfaces may not be displayed. Therefore, to query thresholds for a specific interface, please use the command “show telemetry threshold interface ib <interface>”.

show telemetry sampling interface ib

	show telemetry sampling interface ib <slot>/<port> Displays telemetry histogram samples for a specific InfiniBand interface.
Syntax Description	slot/port	InfiniBand interface number
Default	N/A
Configuration Mode	Any command mode
History	3.6.3004
Example
switch (config) # show telemetry sampling interface ib 1/32 ------------------------------------------------------------------------------------------------------------------------------------- Telemetry histogram: IB1/32 System-time Bin sizes (128 nsec tx buffer was occupied in bytes range) ------------------------------------------------------------------------------------------------------------------------------------- 02/09/17 <2976 35744 68512 101280 134048 166816 199584 232352 265120 265120< 12:19:03.41948 1883 8538 7802080 0 0 0 0 0 0 0 12:19:04.42107 830 9001 7802670 0 0 0 0 0 0 0 12:19:05.42249 96 9705 7802700 0 0 0 0 0 0 0 12:19:06.42388 32 9035 7803434 0 0 0 0 0 0 0 12:19:07.42573 80 9461 7802960 0 0 0 0 0 0 0 12:19:08.42761 160 9302 7803040 0 0 0 0 0 0 0 12:19:09.42915 304 9369 7802829 0 0 0 0 0 0 0 12:19:10.43071 96 8906 7803500 0 0 0 0 0 0 0 12:19:11.43215 463 8907 7803132 0 0 0 0 0 0 0 12:19:12.43369 256 8571 7803675 0 0 0 0 0 0 0
Related Commands
Notes	If the requested entries are more than what the DB contains, it prints the amount in the table.

show telemetry sampling interface ib last

	show telemetry sampling interface ib <slot>/<port> last <num_of_entries> Displays fetched unicast histogram details for an InfiniBand interface.
Syntax Description	slot/port	InfiniBand interface number
Syntax Description	num_of_entries	Range: 0-1000
Default	N/A
Configuration Mode	Any command mode
History	3.6.3004
Example
switch (config) # show telemetry sampling interface ib 1/36 last 20 Legend: 2976 bytes - between 0 - 2976 of tx bytes buffer consumed 35744 bytes - between 2977 - 35744 of tx bytes buffer consumed ------------------------------------------------------------------------------------------------------------------------------------- Telemetry histogram: IB1/36 System-time Bin sizes (128 nsec tx buffer was occupied in bytes range) ------------------------------------------------------------------------------------------------------------------------------------- 02/09/17 <2976 35744 68512 101280 134048 166816 199584 232352 265120 265120< 12:19:03.41948 1883 8538 7802080 0 0 0 0 0 0 0 12:19:04.42107 830 9001 7802670 0 0 0 0 0 0 0 12:19:05.42249 96 9705 7802700 0 0 0 0 0 0 0 12:19:06.42388 32 9035 7803434 0 0 0 0 0 0 0 12:19:07.42573 80 9461 7802960 0 0 0 0 0 0 0
Related Commands
Notes	If requested entries are more than what the DB contains, it prints the amount in the table.

show files stats telemetry

	show files stats telemetry [filename] Displays all files created by the command “stats export csv telemetry”.
Syntax Description	filename	Displays stats for the specified file
Default	N/A
Configuration Mode	Any command mode
History	3.6.3004
History	3.6.8008	Updated example
Example	switch (config) # show files stats telemetry telemetry-20180527-102715.csv Hostname :test-switch Report :telemetry histogram Time lower bound(UTC) :2018/05/28 05:58:10 Time upper bound(UTC) :2018/05/28 05:58:25 Export time(UTC) :2018/05/28 06:00:06 Time lower bound :2018/05/28 08:58:10 +0300 Time upper bound :2018/05/28 08:58:25 +0300 Export time :2018/05/28 09:00:06 +0300 System version :X86_64 sys_test 2018-05-15 04:02:13 x86_64
Related Commands	stats export csv telemetry
Notes