NVIDIA Docs Hub Homepage NVIDIA Networking Accelerator Software NVIDIA Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) Rev 3.3.0 Bug Fixes History

Bug Fixes History

The following table provides a list of bugs fixed in this SHARP version.

Internal Ref.	Issue
3236363	Description: Fixed the way physical link failures between switches are handled. In the event of a link failure, a SHARP job utilizing the link has to be stopped; however, this will bear no effect on the other present or future jobs.
	Keywords: Aggregation Manager; sharp_am; Link Failure
	Discovered in Release: 3.1.0
	Fixed in Release: 3.2.0
3230585	Description: Fixed the issue of when operating in Dynamic trees mode, ibdiagnet may have printed warning messages about the existence of multiple distinct trees with the same tree ID.
	Keywords: Dynamic tree; ibdiagnet
	Discovered in Version: 3.1.0
	Fixed in Release: 3.2.0
3226743	Description: Fixed the issue of when a management host was not connected to a leaf switch, sharp_am might have printed a number of warning messages about trees that could not reach all aggregation nodes. As of SHARP v3.2.0, the active management host is automatically identified and is not treated as a potential compute host. However, please note that this does not include standby management hosts for which a warning message would still appear. These management hosts can be mentioned in a list of GUIDs to ignore via the parameter ignore_host_guids_file.
	Keywords: Aggregation Manager; sharp_am; leaf; GUID
	Discovered in Release: 3.0.1
	Fixed in Release: 3.2.0
3274564	Description: Fixed an issue where sharp_benchmark bash script failed to operate on all bash versions.
	Keywords: sharp_benchmark
	Discovered in Release: 3.1.1
	Fixed in Release: 3.2.0
3262936	Description: Fixed the issue where a crash took place during sharp_am reboot while physical links were hanging between switches in the fabric.
	Keywords: sharp_am; physical links; crash
	Discovered in Release: 3.1.0
	Fixed in Release: 3.1.1 LTS
3192770	Description: Fixed the issue where SHARP jobs failed when using virtual interfaces configured with SR-IOV.
	Keywords: SR-IOV
	Discovered in Release: 3.0.0
	Fixed in Release: 3.1.0
3163697	Description: Fixed the issue of when the client application used more than 1024 file descriptors (range limit defined by FD_SETSIZE), libsharp was prevented from using any more file descriptors. Using poll() instead of select() enables using the full range of allowed file descriptors by Linux.
	Keywords: File descriptor; libsharp; HCOLL; HPC-X
	Discovered in Release: 3.0.0
	Fixed in Release: 3.1.0
3192770	Description: Fixed the issue where SHARP jobs failed when using virtual interfaces configured with SR-IOV.
	Keywords: SR-IOV
	Discovered in Release: 3.0.0
	Fixed in Release: 3.0.1
3163697	Description: Fixed the issue of when the client application used more than 1024 file descriptors (range limit defined by FD_SETSIZE), libsharp was prevented from using any more file descriptors. Using poll() instead of select() enables using the full range of allowed file descriptors by Linux.
	Keywords: File descriptor; libsharp; HCOLL
	Discovered in Release: 3.0.0
	Fixed in Release: 3.0.1
2995739	Description: Sharp_am daemon is no longer removed when performing rpm upgrade and is overridden instead.
	Keywords: Aggregation Manager; rpm
	Discovered in Release: 2.6.1
	Fixed in Release: 2.7.0
2972970	Description: Fixed the issue where completion of SHARP installation using sharp_daemons_setup.sh script depended on python availability.
	Keywords: Aggregation Manager
	Discovered in Release: 2.6.1
	Fixed in Release: 2.7.0
2749073	Description: SHARP AM reports the rediscovery of aggregation nodes on every topology change.
	Keywords: Aggregation Manager
	Workaround: N/A
	Discovered in Release: 2.5.0
2736102	Description: SHARP AM and SHARPD overrides backlog files after restart when log rotation is enabled.
	Keywords: Aggregation Manager, SHARPD, log file
	Workaround: N/A
	Discovered in Release: 2.5.0
2700530	Description: Terminating a job process during job initialization before sending a job request to Aggregation Manager, might result in job resource leakage in the SHARP Aggregation Manager.
	Workaround: N/A
	Keywords: SHARPD, Aggregation Manager
	Discovered in Release: 2.5.0
2726821	Description: Terminating SHARPD while the job process is still running will result in job resource leakage in SHARP Aggregation Manager.
	Workaround: Terminate SHARPD after terminating the job processes.
	Keywords: SHARPD, Aggregation Manager
2795902	Description: SHARPD might allocate handlers on GPU when running with UCX.
	Keywords: SHARPD, SMX, UCX
	Workaround: N/A
	Discovered in Release: 2.5.0
	Workaround: Disable UCX
2770210	Description: Syslog verbosity depends on log file verbosity.
	Keywords: SHARPD, Aggregation Manager
	Discovered in Release: 2.5.0
	Workaround: None
2825519	Description: Aggregation Manager continue to run after SM failover.
	Keywords: Aggregation Manager
	Discovered in Release: 2.5.0
	Workaround: Stop AM daemon manually
2754175	Description: SHARP Aggregation Manger might allocate bad links for jobs after receiving timeouts from Aggregation Nodes.
	Workaround: Restart corresponding switch or restart SHARP Aggregation Manager.
	Keywords: Aggregation Manager
	Discovered in Release: 2.5.0
2796317	Description: SHARP jobs may hang when running in reservations mode (i.e. SHARP allocation is enabled), and reservation is created with limited PKEY, and configuring reservation PKEY on tree is enabled.
	Workaround: The PKEY used for creating the reservation should be "full" (the most significant bit should be on e.g. 0x805c instead of 0x5a).
	Keywords: Aggregation Manager, Reservations, PKEY, UFM
	Discovered in Release: 2.5.0