Buffer Histograms Monitoring

NVIDIA MLNX-OS User Manual v3.11.2224 LTS

As it is becoming increasingly complex to manage networks, and network administrators need more tools to understand network behavior, it is necessary to provide basic information about network performance, identify network bottlenecks, and provide information for the purposes of network optimization and future planning.

Therefore, network administrators are required to constantly review network port behavior, record port buffer consumption, and identify shortage in buffer resources and record flows which lead to the excessive buffer consumption. MLNX-OS provides the following mechanisms to perform these tasks:

  • Sampling (histograms)—a network administrator can enable a sampling of the port buffer occupancy, record occupancy changes over time, and provide information for different levels of buffer occupancy, and amount of time the buffer has been occupied during the observation period.

  • Thresholds—thresholds may be enabled per port to record the network time when port buffer occupancy crosses the defined threshold and when buffer occupancy drops below it.

  • Flow recording—a record of the most active flows which cause an excessive usage of the port buffers may be kept. Once enabled, the system may identify flow patterns and present a user with a list of flows, based on which a network administrator can rearrange distribution of the data flows in the network and minimize data loss.

protocol telemetry

protocol telemetry no protocol telemetry

Unhides telemetry config CLIs. The no form of the command hides telemetry config CLIs.

Syntax Description

N/A

Default

Hidden

Configuration Mode

config

History

3.6.3004

Example

switch (config) # protocol telemetry

Related Commands

Notes


telemetry shutdown

telemetry shutdown

no telemetry shutdown

Disables the telemetry protocol, threshold detection, and histogram fetching for all sampling enabled interfaces without changing any internal configuration.

The no form of the command enables telemetry protocol.

Syntax Description

N/A

Default

Disabled

Configuration Mode

config

History

3.6.3004

Example

switch (config) # no telemetry shutdown

Related Commands

protocol telemetry

Notes

telemetry sampling

interface ib <slot>/<port> telemetry sampling

no interface ib <slot>/<port> telemetry sampling

Enables sampling (histogram fetching) for a specific InfiniBand interface.

The no form of the command disables sampling (histogram fetching).

Syntax Description

N/A

Default

N/A

Configuration Mode

config interface ib

History

3.6.3004

Example

switch (config interface ib 1/1) # telemetry sampling

Related Commands

protocol telemetry

Notes

telemetry sampling log

telemetry sampling log <time>

no telemetry sampling log <time>

Enables the log interval value (histogram fetching) from device.

The no form of the command disables the log interval value.

Syntax Description

time

Input range: 100-60000 (in msec)

Default

1000 millisecond

Configuration Mode

config

History

3.6.3004

Example

switch (config) # telemetry sampling log 1000

Related Commands

protocol telemetry

Notes

telemetry threshold

telemetry threshold

no telemetry threshold

Enables telemetry threshold on hardware.

The no form of the command disables threshold.

Syntax Description

N/A

Default

false

Configuration Mode

config interface ib

History

3.6.4006

Example

switch (config interface ib 1/1) # telemetry threshold

Related Commands

Notes


telemetry threshold level

telemetry threshold level <level>

no telemetry threshold level

Configures threshold level in hardware per port.

The no form of the command resets threshold to default value.

Syntax Description

level

Input range: 96-1000000 (in bytes and in increments of 96)

Default

69984

Configuration Mode

config interface ib

History

3.6.4006

Example

switch (config interface ib 1/1) # telemetry threshold level 288

Related Commands

Notes

telemetry threshold log

telemetry threshold log

no telemetry threshold log

Enables logging of threshold events in syslog.

The no form of the command disables logging.

Syntax Description

N/A

Default

Disabled

Configuration Mode

config

History

3.6.4006

Example

switch (config) # telemetry threshold log

Related Commands

Notes

telemetry threshold record

telemetry threshold record

no telemetry threshold record

Enables top talker configuration.

The no form of the command disables top talker configuration.

Syntax Description

N/A

Default

Disabled

Configuration Mode

config interface ib

History

3.6.6105

Example

switch (config interfaces ib 1/2) # telemetry threshold record

Related Commands

clear telemetry threshold record

show telemetry threshold record

Notes

  • When top talker is enabled, the minimal threshold window supported is 20 msecs

  • Due to event timing issues, very short threshold events may not gather sufficient traffic samples to allow top-talker analysis. As a result, top-talkers may not be fully displayed in the relevant show command.

telemetry threshold syslog

telemetry threshold syslog <time>

no telemetry threshold syslog <time>

The command sets threshold events logging rate on per hour basis.

The no form of the command sets the logging rate back to default.

Syntax Description

time

Max rate per hour

Range: 1-3600

Default

100

Configuration Mode

config

History

3.6.4006

Example

switch (config) # telemetry threshold syslog 400

Related Commands

Notes

clear telemetry

clear telemetry {threshold | sampling} [interface ib <port-id>]

Clears telemetry data.

Syntax Description

port-id

InfiniBand interface ID

Default

N/A

Configuration Mode

config interface ib

History

3.6.5000

Example

switch (config interface ib 1/12) # clear telemetry threshold level 288

Related Commands

Notes

clear telemetry threshold

clear telemetry threshold [interface <type> <if>]

Clears threshold and top talker data.

Syntax Description

type

Available values:ib

Default

N/A

Configuration Mode

config

History

3.6.6105

Example

switch (config) # clear telemetry threshold interface ib 1/34-1/36

Related Commands

Notes

clear telemetry threshold record

clear telemetry threshold record [interface ib <if>]

Clears top talker data.

Syntax Description

if

InfiniBand interface ID

Default

N/A

Configuration Mode

config

History

3.6.6105

Example

switch (config) # clear telemetry threshold record interface ib 1/34-1/36

Related Commands

telemetry threshold record

show telemetry threshold record

Notes

stats export csv telemetry

stats export csv telemetry <slot>/<port>[/<subport>][filename <name>] [after * *] [before * *]

Exports histograms collected by stats to a csv file.

Syntax Description

slot/port

Port number

subport

Subport number to be used if a port is split

Default

N/A

Configuration Mode

Any command mode

History

3.6.3004

3.9.0500

Updated example

Example

    
switch (config) # stats export csv telemetry 1/1/4-ucast after 2020/03/16 10:54:58 before 2020/03/16 11:16:24    

Generated report file: telemetry-20200316-111704.csv

Related Commands

Notes


file stats telemetry delete

file stats telemetry delete <filename>

Deletes the given .csv file created by “stats export” command to user directory.

Syntax Description

N/A

Default

N/A

Configuration Mode

config

History

3.6.3004

Example

switch (config) # file stats telemetry delete telemetry-20171006-102158.csv

Related Commands

Notes


file stats telemetry delete latest

file stats telemetry delete latest

Delete the latest stats telemetry file.

Syntax Description

N/A

Default

N/A

Configuration Mode

Configure terminal

History

3.8.1000

Example

(config) # file stats telemetry delete latest

Related Commands

file stats telemetry delete <file_name>

file stats telemetry delete all

Notes


file stats telemetry delete all

file stats telemetry delete all

Deletes all stats telemetry files from machine.

Syntax Description

N/A

Default

N/A

Configuration Mode

Configure terminal

History

3.8.1000

Example

(config) # file stats telemetry delete all

Related Commands

file stats telemetry delete <file_name>

file stats telemetry delete latest

Notes


file stats telemetry upload

file stats telemetry upload <filename> <upload-url>

Uploads .csv file created by “stats export” command to user directory.

Syntax Description

N/A

Default

N/A

Configuration Mode

config

History

3.6.3004

Example

switch (config) # file stats telemetry upload telemetry-20170119-102715.csv scp://username:password@server//directory

Password (if required): ******

Related Commands

Notes


file stats telemetry upload latest

file stats telemetry upload latest <upload-url>

Upload the latest stats telemetry file to a remote host.

Syntax Description

N/A

Default

N/A

Configuration Mode

Configure terminal

History

3.8.1000

Example

(config) # file stats telemetry upload latest scp://user:pass@10.135.155.8/tmp

Related Commands

file stats telemetry upload <file_name>

file stats telemetry upload all

Notes


file stats telemetry upload all

file stats telemetry upload all <upload_url>

Upload all stats telemetry files to a remote host.

Syntax Description

Default

N/A

Configuration Mode

Configure terminal

History

3.8.1000

Example

(config) # file stats telemetry upload all scp://user:pass@10.135.155.8/tmp

Related Commands

file stats telemetry upload <file_name>

file stats telemetry upload latest

Notes

show telemetry

show telemetry

Displays the global configuration of telemetry properties.

Syntax Description

N/A

Default

N/A

Configuration Mode

config

History

3.6.4000

Example

switch (config) # show telemetry 
Telemetry Status                   : Enabled

H/W Sampling Interval(nsec) : 512

S/W Sampling Interval(ms) : 1000

Threshold Logging : Disabled

Threshold Logging(rate per hour) : 100

--------------------------------------------------------------------------------------------

Interface Sampling Threshold Record Level (bytes)

--------------------------------------------------------------------------------------------

IB1/1 Disabled Enabled Enabled 100 (96)

IB1/2 Disabled Enabled Enabled 100 (96)

IB1/3 Disabled Disabled Disabled N/A

IB1/4 Disabled Disabled Disabled N/A

IB1/5 Disabled Disabled Disabled N/A

IB1/6 Disabled Disabled Disabled N/A

IB1/7 Disabled Disabled Disabled N/A

...

IB1/36 Disabled Disabled Disabled N/A

Related Commands

Notes


show telemetry threshold record

show telemetry threshold record [interface ib <interface-id> | <interface-id-range>]

Displays top talker events for all configured ports.

Syntax Description

N/A

Default

N/A

Configuration Mode

Any command mode

History

3.6.4006

3.6.6105

Updated example

3.6.8100

Updated example

Example

switch (config) # show telemetry threshold record interface ib 1/11-1/12

-----------------------------------------------------------------------------------------------------------------

Event-id Date Time Port Level Duration(100 usec) Repeated DestQP DLID SLID Percent(%)

-----------------------------------------------------------------------------------------------------------------

1 07/10/18 14:00:31 IB 1/11 69984 48749.77 1 2741 29 32 62.30

2 07/10/18 14:01:47 IB 1/11 69984 63936.16 1 2745 29 32 54.55

Related Commands

clear telemetry threshold

Notes

  • The values displayed of the SLID, DLID, and QP fields are in decimal

  • The command supports displaying up to 1000 threshold events. As a result, if more than 1000 thresholds configured in total, some interfaces may not be displayed. Therefore, to query thresholds for a specific interface, please use the command “show telemetry threshold interface ib <interface>”.


show telemetry sampling interface ib

show telemetry sampling interface ib <slot>/<port>

Displays telemetry histogram samples for a specific InfiniBand interface.

Syntax Description

slot/port

InfiniBand interface number

Default

N/A

Configuration Mode

Any command mode

History

3.6.3004

Example

switch (config) # show telemetry sampling interface ib 1/32
-------------------------------------------------------------------------------------------------------------------------------------

Telemetry histogram: IB1/32

System-time Bin sizes (128 nsec tx buffer was occupied in bytes range)

-------------------------------------------------------------------------------------------------------------------------------------

02/09/17 <2976 35744 68512 101280 134048 166816 199584 232352 265120 265120<

12:19:03.41948 1883 8538 7802080 0 0 0 0 0 0 0

12:19:04.42107 830 9001 7802670 0 0 0 0 0 0 0

12:19:05.42249 96 9705 7802700 0 0 0 0 0 0 0

12:19:06.42388 32 9035 7803434 0 0 0 0 0 0 0

12:19:07.42573 80 9461 7802960 0 0 0 0 0 0 0

12:19:08.42761 160 9302 7803040 0 0 0 0 0 0 0

12:19:09.42915 304 9369 7802829 0 0 0 0 0 0 0

12:19:10.43071 96 8906 7803500 0 0 0 0 0 0 0

12:19:11.43215 463 8907 7803132 0 0 0 0 0 0 0

12:19:12.43369 256 8571 7803675 0 0 0 0 0 0 0

Related Commands

Notes

If the requested entries are more than what the DB contains, it prints the amount in the table.


show telemetry sampling interface ib last

show telemetry sampling interface ib <slot>/<port> last <num_of_entries>

Displays fetched unicast histogram details for an InfiniBand interface.

Syntax Description

slot/port

InfiniBand interface number

num_of_entries

Range: 0-1000

Default

N/A

Configuration Mode

Any command mode

History

3.6.3004

Example

switch (config) # show telemetry sampling interface ib 1/36 last 20
Legend:

2976 bytes - between 0 - 2976 of tx bytes buffer consumed

35744 bytes - between 2977 - 35744 of tx bytes buffer consumed

-------------------------------------------------------------------------------------------------------------------------------------

Telemetry histogram: IB1/36

System-time Bin sizes (128 nsec tx buffer was occupied in bytes range)

-------------------------------------------------------------------------------------------------------------------------------------

02/09/17 <2976 35744 68512 101280 134048 166816 199584 232352 265120 265120<

12:19:03.41948 1883 8538 7802080 0 0 0 0 0 0 0

12:19:04.42107 830 9001 7802670 0 0 0 0 0 0 0

12:19:05.42249 96 9705 7802700 0 0 0 0 0 0 0

12:19:06.42388 32 9035 7803434 0 0 0 0 0 0 0

12:19:07.42573 80 9461 7802960 0 0 0 0 0 0 0

Related Commands

Notes

If requested entries are more than what the DB contains, it prints the amount in the table.

show files stats telemetry

show files stats telemetry [filename]

Displays all files created by the command “stats export csv telemetry”.

Syntax Description

filename

Displays stats for the specified file

Default

N/A

Configuration Mode

Any command mode

History

3.6.3004

3.6.8008

Updated example

Example

switch (config) # show files stats telemetry telemetry-20180527-102715.csv

Hostname :test-switch

Report :telemetry histogram

Time lower bound(UTC) :2018/05/28 05:58:10

Time upper bound(UTC) :2018/05/28 05:58:25

Export time(UTC) :2018/05/28 06:00:06

Time lower bound :2018/05/28 08:58:10 +0300

Time upper bound :2018/05/28 08:58:25 +0300

Export time :2018/05/28 09:00:06 +0300

System version :X86_64 sys_test 2018-05-15 04:02:13 x86_64

Related Commands

stats export csv telemetry

Notes


© Copyright 2024, NVIDIA. Last updated on May 30, 2024.