Launching Workloads with Dynamic GPU Memory Fractions

Browse to the provided NVIDIA Run:ai user interface and log in with your credentials.

1. Go to the Workload manager → Workloads

2. Click +NEW WORKLOAD and select Workspace

3. Select under which cluster to create the workload

4. Select the project in which your workspace will run

5. Select Start from scratch to launch a new workspace quickly

6. Enter a name for the workspace (if the name already exists in the project, you will be requested to submit a different name)

7. Click CONTINUE

   In the next step:

8. Under Environment, click the load icon. A side pane appears, displaying a list of available environments. To add a new environment:

   • Click the + icon to create a new environment

   • Enter quick-start as the name for the environment. The name must be unique.

   • Enter the Image URL - quay.io/jupyter/pytorch-notebook

   • Tools - Set the connection for your tool:

     • Click +TOOL
     • Select Jupyter tool from the list

   • Set the runtime settings for the environment. Click +COMMAND & ARGUMENTS and add the following:

     • Enter the command - start-notebook.sh
     • Enter the arguments - --NotebookApp.token=''

     Note: If path-based routing is enabled on the cluster, enter --NotebookApp.base_url=/${RUNAI_PROJECT}/${RUNAI_JOB_NAME} --NotebookApp.token=''.

   • Click CREATE ENVIRONMENT

   • Select the newly created environment from the side pane

9. Under Compute resources, click the load icon. A side pane appears, displaying a list of available compute resources. To add a new compute resource:

   • Click the + icon to create a new compute resource
   • Enter request-limit as the name for the compute resource. The name must be unique.
   • Set GPU devices per pod - 1
   • Enable GPU fractioning to set the GPU memory per device:
     • Select GB - Fraction of a GPU device’s memory
     • Set the memory Request - 4GB (the workload will allocate 4GB of the GPU memory)
     • Set the memory Limit - 12GB
   • Optional: set the CPU compute per pod - 0.1 cores (default)
   • Optional: set the CPU memory per pod - 100 MB (default)
   • Select More settings and toggle Increase shared memory size
   • Click CREATE COMPUTE RESOURCE
   • Select the newly created compute resource from the side pane

10. Click CREATE WORKSPACE

1. Go to the Workload manager → Workloads

2. Click +NEW WORKLOAD and select Workspace

3. Select the cluster where the previous workspace was created

4. Select the project where the previous workspace was created

5. Select Start from scratch to launch a new workspace quickly

6. Enter a name for the workspace (if the name already exists in the project, you will be requested to submit a different name)

7. Click CONTINUE

   In the next step:

8. Under Environment, click the load icon. A side pane appears, displaying a list of available environments. Select the environment created in Step 2.

9. Under Compute resources, click the load icon. A side pane appears, displaying a list of available compute resources. Select the compute resources created in Step 2.

10. Click CREATE WORKSPACE

1. Select the newly created workspace with the Jupyter application that you want to connect to
2. Click CONNECT
3. Select the Jupyter tool. The selected tool is opened in a new tab on your browser.
4. Open a terminal and use the watch nvidia-smi to get a constant reading of the memory consumed by the pod. The free memory shown always covers the rest of the 4 GB Request, and grows beyond that when the GPU has memory to spare, up to the 12 GB Limit.
5. Open the file Untitled.ipynb and move the frame so you can see both tabs
6. Execute both cells in Untitled.ipynb. This will consume about 3 GB of GPU memory and be well below the 4GB of the GPU Memory Request value.
7. In the second cell, edit the value after --image-size from 100 to 200 and run the cell. This will increase the GPU memory utilization to about 11.5 GB which is above the Request value.