Step #7: Workspace Conversion to a Dataset
After iterating on a workspace over a period of time, part or all of the workspace may be ready to be turned into a dataset. Similar to most other dataset creation processes, we can use a job results directory to stage data movement between locations - in the following example, the workspace we’ve been working with so far will be the source for new datasets.
At a high level, the goal here is to combine a few different workflows that we have used separately so far:
Mount a workspace into a job
Copy data into a job result directory
Convert a job result directory into a dataset
We can either simply copy the data or move it completely - a good way to think about this is to copy if the data needs to be iterated on again into another distinct dataset, and move the data if it would never need to be interacted with outside of a read-only dataset context again. For this job example, the data will be moved to avoid unnecessary data duplication.
$ ngc batch run --name "move-data-out-of-workspace" --preempt RUNONCE --min-timeslice 0s --total-runtime 0s --instance dgxa100.80g.1.norm --commandline "mv /mount/workspace/* /results" --result /results --image "nvidia/pytorch:22.04-py3" --workspace workspace-from-cli:/mount/workspace:RW
--------------------------------------------------
Job Information
Id: 2873764
Name: move-data-out-of-workspace
Number of Replicas: 1
Job Type: BATCH
Submitted By: Joseph Handzik
Job Container Information
Docker Image URL: nvidia/pytorch:22.04-py3
Job Commands
Command: mv /mount/workspace/* /results
Dockerfile Image Entrypoint: False
Datasets, Workspaces and Results
Workspace ID: JyrRwRmXQCuptjLaCFumkQ
Workspace Name: workspace-from-cli
Workspace Mount Point: /mount/workspace
Workspace Mount Mode: RW
Result Mount Point: /results
Job Resources
Instance Type: dgxa100.80g.1.norm
Instance Details: 1 GPU, 30 CPU, 244 GB System Memory
ACE: nv-launchpad-bc-sjc-ace
Team: nvbc-tme
Job Status
Created at: 2022-05-03 20:33:05 UTC
Status: STARTING
Status Type: OK
Preempt Class: RUNONCE
--------------------------------------------------
As in other sections, checking the output of ngc batch list for this job’s status to become FINISHED_SUCCESS will indicate we can proceed.
$ ngc batch list
+---------+----------+----------------------------+----------+------------------+----------+----------------+
| Id | Replicas | Name | Team | Status | Duration | Status Details |
+---------+----------+----------------------------+----------+------------------+----------+----------------+
| 2873764 | 1 | move-data-out-of-workspace | nvbc-tme | FINISHED_SUCCESS | 0:03:05 | |
| 2873341 | 1 | load-linux-into-workspace | nvbc-tme | FINISHED_SUCCESS | 0:02:01 | |
| 2870152 | 1 | aws-dl-cli | nvbc-tme | FINISHED_SUCCESS | 0:00:21 | |
+---------+----------+----------------------------+----------+------------------+----------+----------------+
Check the workspace size again - it should be 0 bytes, because all data has been copied out of it.
$ ngc workspace list --owned
+------------------------+--------------------+-------------+----------------------+------------------+--------+---------------------+-------+------+
| Id | Name | Description | ACE | Creator Username | Shared | Created Date | Owned | Size |
+------------------------+--------------------+-------------+----------------------+------------------+--------+---------------------+-------+------+
| JyrRwRmXQCuptjLaCFumkQ | workspace-from-cli | | nv-launchpad-bc-sjc- | Joseph Handzik | No | 2022-05-02 19:34:40 | Yes | 0 B |
| | | | ace | | | UTC | | |
| a6CxaJmmQgK1OIcXR8-d5A | s3-demo-workspace | | nv-launchpad-bc-sjc- | Joseph Handzik | No | 2022-04-19 19:29:12 | Yes | 0 B |
| | | | ace | | | UTC | | |
+------------------------+--------------------+-------------+----------------------+------------------+--------+---------------------+-------+------+
We will convert the previous job’s results directory as we have before, naming it clearly to indicate how it has been created differently from other datasets.
$ ngc dataset convert --from-result 2873764 linux-kernel-from-workspace
Dataset with ID: '99967' created in ACE: 'nv-launchpad-bc-sjc-ace'.
A final ngc dataset list run will confirm the new dataset exists as expected.
$ ngc dataset list --owned
+--------------------+------------+--------------------+--------------------+------------------+--------+-----------+-----------+--------------+-------+---------+
| Id | Integer Id | Name | Description | ACE | Shared | Size | Status | Created Date | Owned | Pre-pop |
+--------------------+------------+--------------------+--------------------+------------------+--------+-----------+-----------+--------------+-------+---------+
| 8brma-E4RX6xVMmpP7 | 99967 | linux-kernel-from- | | nv-launchpad-bc- | No | 1.02 GB | COMPLETED | 2022-05-03 | Yes | No |
| OfpQ | | workspace | | sjc-ace | | | | 20:45:49 UTC | | |
| r3KPjT13TgCFw3a8Jv | 99885 | s3-dl-from-cli | | nv-launchpad-bc- | No | 260.82 MB | COMPLETED | 2022-05-02 | Yes | No |
| ljow | | | | sjc-ace | | | | 17:29:25 UTC | | |
| D9xMj6VNR- | 99209 | linux-kernel-from- | | nv-launchpad-bc- | No | 1.03 GB | COMPLETED | 2022-04-19 | Yes | No |
| OitmadRHwFig | | web-ui | | sjc-ace | | | | 17:27:47 UTC | | |
| cHCMiESgRVeduwq8vF | 99164 | linux-kernel-from- | | nv-launchpad-bc- | No | 1.03 GB | COMPLETED | 2022-04-18 | Yes | No |
| fIRQ | | cli | | sjc-ace | | | | 21:11:38 UTC | | |
| CzgFjq__SSqAb46hSj | 99150 | linux-nvbug-triage | | nv-launchpad-bc- | No | 1.03 GB | COMPLETED | 2022-04-18 | Yes | No |
| k5Nw | | | | sjc-ace | | | | 14:47:47 UTC | | |
| a5LuVPZPQhO7SUWMIz | 98828 | linux-kernel | My favorite kernel | nv-launchpad-bc- | No | 1.03 GB | COMPLETED | 2022-04-14 | Yes | No |
| cpZw | | | | sjc-ace | | | | 22:32:50 UTC | | |
+--------------------+------------+--------------------+--------------------+------------------+--------+-----------+-----------+--------------+-------+---------+
To accomplish the migration of data from a workspace to a dataset in the Web UI, start at the Base Command Dashboard again, and click the Create Job button.
Populate the Create Job as follows.
Select your ACE from the list.
Select dgxa100.80g.1.norm in the INSTANCE table.
Select the WORKSPACES tab - click the box next to
workspace-from-web-ui, and type/mount/workspacefor its mount point, ensuring that the mount mode isRW.Enter
/resultsfor the Data Output field.Select the
nvidia/pytorchcontainer.Select the tag value of
22.04-py3.For the command, copy the following:
mv /mount/workspace/* /results
The remainder of the form can be left with the default values.
To verify that the correct options were selected, check the ngc batch run command on your screen with the values in the screenshot below (the --name argument will differ, and --ace and --org arguments may differ too). When ready, click the Launch Job button near the top right corner of the page.
The job shouldn’t take long to run - around 5 minutes assuming the ACE is relatively idle. You may need to refresh the page to verify that the job status is Finished Success. Click on the newly-completed job.
Navigate to the Results tab to verify that the data has been moved out of the workspace.
To convert the results directory to a dataset, click on the vertical ellipsis near the top right corner of the page and click the Convert Results button.
A dialog box will appear - enter s3-dl-from-workspace in the Name field, and click the Convert button.
After a short period of time, the Results directory for the job used to create the dataset will become empty, which indicates the dataset conversion has been executed. Click the Datasets link in the navigation pane, just below the Jobs link that has been previously used.
The new dataset should be present in the table on the Datasets page on the top data row.
