Note

For training, evaluation, and inference, we expose two variables for each task: num_gpus and gpu_ids, which default to 1 and [0], respectively. If both are passed, but are inconsistent, for example num_gpus = 1, gpu_ids = [0, 1], then they are modified to follow the setting that implies more GPUs; in the same example num_gpus is modified from 1 to 2.

In some cases multi-GPU training may result in a segmentation fault. You can circumvent this by exporting OMP_NUM_THREADS=1 in the shell that launches your TAO agent. The TAO Execution SDK forwards the variable into the training container at dispatch time.

export OMP_NUM_THREADS=1