Launcher Multirun

There are scenarios where you might want to run the NeMo Launcher with multiple different configurations, such as conducting a performance test across various sets of hyperparameters.

The NeMo launcher integrates the multirun feature from Hydra. This feature allows you to sweep over a range of parameters or perform grid searches efficiently. To utilize this functionality, use the --multirun (or -m) option from the command line. Below is an example demonstrating how to execute a multirun with different configurations:

Copy
Copied!
            

python3 main.py -m \ stages=[training] \ training.trainer.num_nodes=6 \ training.run.name="5b_6nodes_tp_\${training.model.tensor_model_parallel_size}" \ training.model.tensor_model_parallel_size=1,2,4,8

Previous Logs and Results
Next Debugging
© Copyright 2023-2024, NVIDIA. Last updated on Jan 19, 2024.