Writeable File System Required for Index Creation#
BioNeMo is currently based on NeMo Toolkit v23.03. This version of NeMo requires index files to be created in the same directory as the data files at the start of training, which means that the location of the data must be writable. BCP Dataset file systems are read-only, thus we suggest copying the files to a Workspace before initializing training. This issue will be fixed in a future version of NeMo.
BioNeMo multi-node training is supported, but if more nodes are requested than available, the process will hang and appear to make no progress. Be sure that trainer.num_nodes is set to the number of available nodes.