Step #6: Workspace Use

Build an AI Center of Excellence with Base Command (Latest Version)

Now that we have created a workspace, we can use it across both the CLI and Web UI either in our jobs or, in the case of the CLI, even as an interactive mount point on our local system.

The command line interface provides the interesting capability to mount a workspace to your local machine. From your terminal, create a directory to mount your workspace to.

Copy
Copied!
            

$ mkdir test-ngc-mount

Use the ngc workspace mount command to mount the workspace created in the previous section to this new directory - mount as readable and writable via the --mode RW flag and argument to allow data to be copied to it.

Copy
Copied!
            

$ ngc workspace mount workspace-from-cli ./test-ngc-mount --mode RW Mounting complete.

A df -h command will show all active mounts on the system - look under the Mounted on column for the test-ngc-mount path.

Copy
Copied!
            

$ df -h Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/disk1s5s1 466Gi 14Gi 279Gi 5% 500637 2926965480 0% / devfs 227Ki 227Ki 0Bi 100% 784 0 100% /dev /dev/disk1s4 466Gi 20Ki 279Gi 1% 0 2926965480 0% /System/Volumes/VM /dev/disk1s2 466Gi 259Mi 279Gi 1% 841 2926965480 0% /System/Volumes/Preboot /dev/disk1s6 466Gi 296Ki 279Gi 1% 16 2926965480 0% /System/Volumes/Update /dev/disk1s1 466Gi 171Gi 279Gi 38% 1475182 2926965480 0% /System/Volumes/Data map auto_home 0Bi 0Bi 0Bi 100% 0 0 100% /System/Volumes/Data/home YjNOamIyMXlNR1p0YWpkeGJtSnVielZyTlcwM2FHUTNiVFE2TURReVlUZ3pZbUl0TnpjMVppMDBaVEUwTFdFMVkyS 1.0Ti 0Bi 1.0Ti 0% 0 1000000000 0% /Users/jhandzik/test-ngc-mount

The workspace can be used just like any other mount point on your system - let’s copy the test directory and test file from the previous dataset upload through the command line tool section.

Copy
Copied!
            

$ cp -r base-command-test test-ngc-mount/

Prove that the directory and underlying data was copied over with two basic ls commands.

Copy
Copied!
            

$ ls test-ngc-mount/ base-command-test

Copy
Copied!
            

$ ls test-ngc-mount/base-command-test/ base-command-test-data

Finally, unmount the directory by using the ngc workspace unmount command. The local directory will be the correct argument - not the workspace name as defined via the NGC CLI.

Copy
Copied!
            

$ ngc workspace unmount test-ngc-mount/ Successfully unmounted workspace.

Use df -h again to confirm the workspace mount is no longer active.

Copy
Copied!
            

$ df -h Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/disk1s5s1 466Gi 14Gi 279Gi 5% 500637 2927127200 0% / devfs 227Ki 227Ki 0Bi 100% 784 0 100% /dev /dev/disk1s4 466Gi 20Ki 279Gi 1% 0 2927127200 0% /System/Volumes/VM /dev/disk1s2 466Gi 259Mi 279Gi 1% 841 2927127200 0% /System/Volumes/Preboot /dev/disk1s6 466Gi 296Ki 279Gi 1% 16 2927127200 0% /System/Volumes/Update /dev/disk1s1 466Gi 171Gi 279Gi 38% 1475268 2927127200 0% /System/Volumes/Data map auto_home 0Bi 0Bi 0Bi 100% 0 0 100% /System/Volumes/Data/home

Let’s go ahead and launch a batch job that is similar to the job from the dataset creation section where data is downloaded from the internet - this time, we will mount the workspace to the job at /mount/workspace and download data to it, instead of to the /results directory (which still exists for this job - just not to hold user-created data).

Copy
Copied!
            

$ ngc batch run --name "load-linux-into-workspace" --preempt RUNONCE --min-timeslice 0s --total-runtime 0s --instance dgxa100.80g.1.norm --commandline "cd /mount/workspace; wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.17.3.tar.xz; tar -xvf linux-5.17.3.tar.xz; rm -rf linux-5.17.3.tar.xz" --result /results --image "nvidia/pytorch:22.04-py3" --workspace workspace-from-cli:/mount/workspace:RW -------------------------------------------------- Job Information Id: 2873341 Name: load-linux-into-workspace Number of Replicas: 1 Job Type: BATCH Submitted By: Joseph Handzik Job Container Information Docker Image URL: nvidia/pytorch:22.04-py3 Job Commands Command: cd /mount/workspace; wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.17.3.tar.xz; tar -xvf linux-5.17.3.tar.xz; rm -rf linux-5.17.3.tar.xz Dockerfile Image Entrypoint: False Datasets, Workspaces and Results Workspace ID: JyrRwRmXQCuptjLaCFumkQ Workspace Name: workspace-from-cli Workspace Mount Point: /mount/workspace Workspace Mount Mode: RW Result Mount Point: /results Job Resources Instance Type: dgxa100.80g.1.norm Instance Details: 1 GPU, 30 CPU, 244 GB System Memory ACE: nv-launchpad-bc-sjc-ace Team: nvbc-tme Job Status Created at: 2022-05-03 17:23:38 UTC Status: QUEUED Status Type: OK Preempt Class: RUNONCE --------------------------------------------------

Verify the job status with ngc batch list again - note that you may see multiple jobs listed, so be sure to check the job name to track the correct job.

Copy
Copied!
            

$ ngc batch list +---------+----------+---------------------------+----------+------------------+----------+----------------+ | Id | Replicas | Name | Team | Status | Duration | Status Details | +---------+----------+---------------------------+----------+------------------+----------+----------------+ | 2873341 | 1 | load-linux-into-workspace | nvbc-tme | FINISHED_SUCCESS | 0:02:01 | | | 2870152 | 1 | aws-dl-cli | nvbc-tme | FINISHED_SUCCESS | 0:00:21 | | +---------+----------+---------------------------+----------+------------------+----------+----------------+

To verify the workspace appears as expected, we can mount it locally again and do a light investigation of its contents. This time, we can mount using the mode argument RO, which stands for read only - doing so will ensure we do not accidentally modify the workspace.

Copy
Copied!
            

$ ngc workspace mount workspace-from-cli ./test-ngc-mount --mode RO Mounting complete.

Basic ls commands confirm that the additional directory is present and appears as expected.

Copy
Copied!
            

$ ls test-ngc-mount/ base-command-test linux-5.17.3

Copy
Copied!
            

$ ls test-ngc-mount/linux-5.17.3/ COPYING Kbuild MAINTAINERS arch crypto include kernel net security usr CREDITS Kconfig Makefile block drivers init lib samples sound virt Documentation LICENSES README certs fs ipc mm scripts tools

Remember to unmount the workspace before moving forward.

Copy
Copied!
            

$ ngc workspace unmount test-ngc-mount/ Successfully unmounted workspace.

Taking a look at the output from ngc workspace list again will confirm the augmented capacity usage as well.

Copy
Copied!
            

$ ngc workspace list --owned +------------------------+--------------------+-------------+----------------------+------------------+--------+---------------------+-------+---------+ | Id | Name | Description | ACE | Creator Username | Shared | Created Date | Owned | Size | +------------------------+--------------------+-------------+----------------------+------------------+--------+---------------------+-------+---------+ | JyrRwRmXQCuptjLaCFumkQ | workspace-from-cli | | nv-launchpad-bc-sjc- | Joseph Handzik | No | 2022-05-02 19:34:40 | Yes | 1.02 GB | | | | | ace | | | UTC | | | | a6CxaJmmQgK1OIcXR8-d5A | s3-demo-workspace | | nv-launchpad-bc-sjc- | Joseph Handzik | No | 2022-04-19 19:29:12 | Yes | 0 B | | | | | ace | | | UTC | | | +------------------------+--------------------+-------------+----------------------+------------------+--------+---------------------+-------+---------+

To leverage the workspace created through the Web UI, start again from the Base Command Dashboard and click the Create Job button.

step-06-image-001.png

Fill out the Create Job web form again - note that the main difference between previous jobs and this job will be the use of a workspace.

  1. Select your ACE from the list.

  2. Select dgxa100.80g.1.norm in the INSTANCE table.

  3. Select the WORKSPACES tab - click the box next to workspace-from-web-ui, and type /mount/workspace for its mount point, ensuring that the mount mode is RW.

  4. Enter /results for the Data Output field.

  5. Select the nvidia/pytorch container.

  6. Select the tag value of 22.04-py3.

  7. For the command, copy the following:

Copy
Copied!
            

curl 'https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip' -o 'awscliv2.zip'; unzip awscliv2.zip; ./aws/install -i /usr/local/aws-cli -b /usr/local/bin; aws s3 sync --no-sign-request --region=us-east-1 's3://safecast-opendata-public-us-east-1' '/mount/workspace'


  1. The remainder of the form can be left with the default values.

To verify that the correct options were selected, check the ngc batch run command on your screen with the values in the screenshot below (the --name argument will differ, and --ace and --org arguments may differ too). When ready, click the Launch Job button near the top right corner of the page.

step-06-image-002.png

The job shouldn’t take long to run - around 5 minutes assuming the ACE is relatively idle. You may need to refresh the page to verify that the job status is Finished Success.

Navigate to the Workspaces page instead of viewing the job - remember, we did not move any data into the /results directory, so it will not show anything in addition to a log file.

step-06-image-003.png

The workspace we used should, after a couple minutes, show a non-zero data size, as seen in the screenshot below. If you would like to interact with the directory structure, you can mount this workspace using the NGC CLI tool - just remember to unmount it when you are done!

step-06-image-004.png

© Copyright 2022-2023, NVIDIA. Last updated on Jan 10, 2023.