Tutorials for NeMo Framework

Before You Start

Before starting the tutorials, ensure you have:

• NeMo Framework Container: Running the latest NeMo Framework container

• Model Checkpoint: Access to a NeMo 2.0 checkpoint (tutorials use Llama 3.2 1B Instruct)

• GPU Resources: CUDA-compatible GPU with sufficient memory

• Jupyter Environment: Ability to run Jupyter notebooks

---

Available Tutorials

Build your expertise with these progressive tutorials:

Orchestrating evaluations with NeMo Run

Launch deployment and evaluation jobs using NeMo Run.

Run Evaluations with NeMo Run

Basic evaluation with MMLU Evaluation

Deploy models and run evaluations with the MMLU benchmark for both completions and chat endpoints.

https://github.com/NVIDIA-NeMo/Eval/tree/main/tutorials/mmlu.ipynb

Enable additional evaluation harnesses

Discover how to extend evaluation capabilities by installing additional harnesses and running HumanEval coding assessments.

https://github.com/NVIDIA-NeMo/Eval/tree/main/tutorials/simple-evals.ipynb

Configure custom tasks

Master custom evaluation workflows by running WikiText benchmark with advanced configuration and log-probability analysis.

https://github.com/NVIDIA-NeMo/Eval/tree/main/tutorials/wikitext.ipynb

Run the Notebook Tutorials

1. Start NeMo Framework Container:

   docker run --rm -it -w /workdir -v $(pwd):/workdir \
     --entrypoint bash --gpus all \
     nvcr.io/nvidia/nemo:${TAG}
   

2. Launch Jupyter:

   jupyter lab --ip=0.0.0.0 --port=8888 --allow-root
   

3. Navigate to the tutorials/ directory and open the desired notebook