NeMo Megatron and Megatron-LM share many underlying technology. You should be able to convert your GPT model checkpoints trained with Megatron-LM into NeMo Megatron. Example conversion script:
<NeMo_ROOT_FOLDER>/examples/nlp/language_modeling/megatron_lm_ckpt_to_nemo.py \
--checkpoint_folder <path_to_PTL_checkpoints_folder> \
--checkpoint_name megatron_gpt--val_loss=99.99-step={steps}-consumed_samples={consumed}.0 \
--nemo_file_path <path_to_output_nemo_file> \
--model_type <megatron model type> \
--tensor_model_parallel_size <tensor_model_parallel_size> \
--pipeline_model_parallel_size <pipeline_model_parallel_size> \
--gpus_per_node <gpus per node>
To resume the training from converted MegatronLM checkpoint, make sure to set the trainer.max_steps=round(lr-warmup-fraction * lr-decay-iters + lr-decay-iters) where lr-warmup-fraction and lr-decay-iters are arguments from MegatronLM training so the learning rate scheduler will follow the same curve.