Manage Files

NeMo Helix provides a file storage interface through the Files service. The Files service supports multiple storage backends and can be used to store datasets for training, evaluation results, model artifacts, and other files.

Concepts

• Fileset: A named container that holds files.

Filesets are uniquely identified by a name within a given workspace.

• Storage Backend: Each fileset is backed by a storage backend where the files are actually persisted. Supported backends include:
• local: Local filesystem storage (default, read/write)
• s3: Amazon S3 or S3-compatible storage such as MinIO (read/write)
• ngc: NVIDIA GPU Cloud storage (read-only)
• huggingface: HuggingFace Hub repositories (read-only)

Read-only backends allow you to create a fileset that acts as a handle to external resources. This provides a unified interface to access files from different sources using the same SDK methods, and allows other Helix services to reference external data through a fileset.

• Purpose: A fileset field that indicates the intended use. Each purpose enables specific metadata fields under the corresponding key. Select a tab below to see the available metadata fields for each purpose:

  generic

  dataset

  model

  Use purpose="generic" (default) for other files that don’t fit the dataset or model categories.

  Metadata fields: No purpose-specific metadata fields.

  These fields are merged into the model entity spec by the model-spec background task.

• Custom Fields: Arbitrary key-value data attached to a fileset via custom_fields for user-defined metadata.

Managing Filesets

Fileset management operations (create, retrieve, list, delete) are available through the CLI (nemo files filesets) or the SDK (client.files, for example create_fileset, list_filesets, get_fileset, delete_fileset).

nemo files filesets list --workspace my-workspace

All Python SDK examples that follow assume this client initialization:

from nemo_helix_plugin.client.client import NemoClient
client = NemoClient.from_config()

Creating Filesets

Creating a fileset involves specifying a name and workspace. You can optionally provide a description, purpose, and custom storage configuration.

nemo files filesets create my-files \
--description "Training data for model fine-tuning"

{
  "id": "fileset-TeufFfapeKBrMtpBb42zdv",
  "created_at": "2026-01-20T03:00:00",
  "custom_fields": {},
  "description": "Training data for model fine-tuning",
  "metadata": {
    "dataset": null
  },
  "name": "my-files",
  "project": "",
  "purpose": "generic",
  "storage": {
    "path": "/var/mnt/filesets/default/my-files",
    "read_chunk_size": 16777216,
    "type": "local",
    "write_buffer_size": 16777216
  },
  "updated_at": "2026-01-20T03:00:00",
  "workspace": "default"
}

Listing Filesets

List all filesets in a given workspace:

nemo files filesets list

┏━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name ┃ workspace ┃ created_at ┃
┡━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ my-files │ default │ 2026-01-20T03:00:00 │
└──────────┴───────────┴────────────────────────────┘

Filter filesets by purpose or storage type:

# List only dataset filesets
nemo files filesets list --filter.purpose dataset
# List filesets using local storage
nemo files filesets list --filter.storage-type local

Use pagination for large result sets:

# The "-" prefix sorts in descending order (newest first)
nemo files filesets list --page 1 --page-size 10 --sort "-created_at"

Inspect and Filter by Creator or Updater

Fileset responses include created_by (the principal ID that created the fileset) and updated_by (the principal ID recorded for its last update). These are server-managed attribution fields, alongside created_at and updated_at, not entries in metadata or custom_fields. They are not editable creator/owner settings. The values can be null when attribution is unavailable; do not assume a principal ID is an email address or display name.

First inspect a fileset you can read. The following uses my-files from Creating Filesets, in the workspace selected by your CLI context:

nemo files filesets get my-files --output-format json

Copy a non-null created_by or updated_by value from the response. To find filesets attributed to that principal, replace principal-id-from-response below with that exact value:

nemo files filesets list --filter '{"created_by": "principal-id-from-response"}' --output-format json
nemo files filesets list --filter '{"updated_by": "principal-id-from-response"}' --output-format json

Use the JSON --filter option: the 0.9 CLI does not have --filter.created-by or --filter.updated-by shortcuts for filesets. Filtering does not grant access to other users’ data; results remain limited by the workspace and your permissions. Add --workspace consistently for another workspace, and use --all-pages if you need every matching page rather than the first page. JSON output exposes attribution fields that the default list table does not display.

The Python SDK can reuse a returned identity without manually copying or JSON-escaping it:

import json
from nemo_helix_plugin.client.client import NemoClient
from nemo_helix_plugin.files.types import ListFilesetsQueryParams
client = NemoClient.from_config()
fileset = client.files.get_fileset(name="my-files").data()
print("Created by:", fileset.created_by, "Last updated by:", fileset.updated_by)
if fileset.created_by is not None:
    same_creator = client.files.list_filesets(
        query_params=ListFilesetsQueryParams(
            filter=json.dumps({"created_by": fileset.created_by})
        )
    )
    for match in same_creator.items():
        print(match.name, match.created_by, match.updated_by)
else:
    print("This fileset has no creator attribution to filter by.")

To filter by the last updater instead, use a non-null fileset.updated_by and the updated_by filter key. Supplying both keys matches filesets satisfying both conditions, not either condition. For more operators and filter composition, see Filtering.

These fields describe the fileset entity’s recorded attribution, not a per-file audit trail or proof of who authored every stored byte.

Deleting Filesets

Create and delete a disposable fileset, leaving my-files available for the file-management examples below:

nemo files filesets create fileset-to-delete
nemo files filesets delete fileset-to-delete

✓ Deleted successfully

Managing Files Within Filesets

File operations are available through the CLI (nemo files) or the SDK (client.files), which provide methods for uploading, downloading, and listing files.

For advanced use cases, an fsspec-compatible filesystem is available through the filesets package (FilesetFileSystem). Refer to the fsspec documentation for additional methods.

Uploading Files

Upload files to a fileset:

# Upload a single file
nemo files upload ./data.jsonl my-files --remote-path training/data.jsonl
# Upload an entire directory
nemo files upload ./training_data/ my-files --remote-path training/

Uploading ━━━━━━━━━━━━━━━━ 100% • 3/3 files
Completed upload to my-files#training/

# Auto-creates a new fileset with a generated name (fileset-<8 hex chars>)
nemo files upload ./data.jsonl

Uploading ━━━━━━━━━━━━━━━━ 100% • 1/1 files
Completed upload to fileset-a1b2c3d4

Listing Files

List all files in a fileset:

nemo files list my-files

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━┓
┃ PATH ┃ SIZE ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━┩
│ training/data.jsonl │ 1024 │
│ training/validation.jsonl │ 512 │
└────────────────────────────┴──────┘

List files under a specific directory:

nemo files list my-files --remote-path training/

Downloading Files

Download files to a local path:

# Download a single file
nemo files download my-files --remote-path training/data.jsonl -o ./data.jsonl
# Download an entire directory
nemo files download my-files --remote-path training/ -o ./training_data/

Downloading ━━━━━━━━━━━━━━━━ 100% • 2/2 files
Downloaded my-files#training/ to './training_data/'

Read file content into memory (SDK only):

content = client.files.download_file(
    name="my-files",
    path="training/data.jsonl",
).read()
print(content.decode("utf-8"))

Deleting Files

Delete files from a fileset:

nemo files delete my-files --remote-path training/data.jsonl

Deleted my-files#training/data.jsonl

Using Progress Callbacks

Track progress during large file transfers using the RichProgressCallback context manager:

from filesets import FilesetFileSystem, RichProgressCallback
from nemo_helix_plugin.files.client import FilesClient
files = client.files
assert isinstance(files, FilesClient)
fs = FilesetFileSystem(client=files)
# Create a small directory for the upload example.
Path("./large_dataset").mkdir(exist_ok=True)
Path("./large_dataset/sample.jsonl").write_text(
    '{"text": "Example training record"}\n', encoding="utf-8"
)
# Upload a directory with progress bar
with RichProgressCallback(description="Uploading dataset") as callback:
    fs.put(
        "./large_dataset/",
        "fileset://default/my-files",
        recursive=True,
        callback=callback,
    )
# Download all files from a fileset with progress bar
with RichProgressCallback(description="Downloading dataset") as callback:
    fs.get(
        "fileset://default/my-files",
        "./downloaded_data/",
        callback=callback,
    )

Use Cases

Using External Storage Backends

Connect to files stored in NVIDIA GPU Cloud (NGC):

# Create a secret to store your NGC API key
echo "$NGC_API_KEY" | nemo secrets create my-ngc-api-key --from-file -
# Create a fileset pointing to NGC storage
nemo files filesets create my-nemotron-personas-dataset-en_us \
--description "Nemotron Personas USA" \
--storage '{
"type": "ngc",
"org": "nvidia",
"team": "nemotron-personas",
"target": "nemotron-personas-dataset-en_us",
"version": "0.0.2",
"api_key_secret": "my-ngc-api-key"
}'

Connect to a HuggingFace repository:

# Create a secret to store your HuggingFace token (needed for gated and private repos)
echo "$HF_TOKEN" | nemo secrets create hf_token --from-file -
# Create a fileset pointing to a HuggingFace repo
nemo files filesets create hf-dataset \
--description "Dataset from HuggingFace" \
--storage '{
"type": "huggingface",
"repo_id": "nvidia/Nemotron-Personas-Japan",
"repo_type": "dataset",
"token_secret": "hf_token"
}'

Connect to an S3 bucket or S3-compatible storage (e.g., MinIO, Ceph):

These optional S3/MinIO examples are not exercised by default in E2E. Set NHX_RUN_S3_TUTORIAL=true to run them.

if [ "${NHX_RUN_S3_TUTORIAL:-false}" = "true" ]; then
# Create a fileset backed by S3 storage using SDK credential chain
# (uses AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY env vars, IRSA, instance profiles, etc.)
# The "prefix" field is optional - use it to scope the fileset to a folder within the bucket
nemo files filesets create s3-training-data \
--description "Training data stored in S3" \
--storage '{
"type": "s3",
"bucket": "my-ml-bucket",
"prefix": "datasets/training",
"region": "us-east-1",
"use_sdk_auth": true
}'
# Upload data to S3
nemo files upload ./training_data/ s3-training-data
# Download data from S3
nemo files download s3-training-data -o ./downloaded_data/
else
    echo "SKIPPED S3 tutorial: set NHX_RUN_S3_TUTORIAL=true to run."
fi

For S3-compatible storage like MinIO, use explicit credentials and a custom endpoint:

if [ "${NHX_RUN_S3_TUTORIAL:-false}" = "true" ]; then
# Create secrets to store your S3 credentials
echo "$S3_ACCESS_KEY" | nemo secrets create s3_access_key --from-file -
echo "$S3_SECRET_KEY" | nemo secrets create s3_secret_key --from-file -
nemo files filesets create minio-fileset \
--description "Data stored in MinIO" \
--storage '{
"type": "s3",
"bucket": "my-bucket",
"endpoint_url": "http://minio.example.com:9000",
"region": "us-east-1",
"use_sdk_auth": false,
"access_key_id_secret": "s3_access_key",
"secret_access_key_secret": "s3_secret_key"
}'
else
    echo "SKIPPED MinIO tutorial: set NHX_RUN_S3_TUTORIAL=true to run."
fi