Filesystem Spec Loader

DataLoader module is configured to use this loader function. It is responsible for loading data from external sources using the fsspec library, and returns the updated ControlMessage object with payload as MessageMeta, which contains dataframe (with filenames).

Copy
Copied!
            

{ "loaders": [{ "id": "fsspec" }] }

Note : Loaders can receive configuration from the load task via control message during runtime.

The parameters that can be configured for this specific loader at load task level:

Parameter

Type

Description

Example Value

Default Value

files array List of files to load [“/your/input/filepath”] []
loader_id string Unique identifier for the loader “file_to_df” [Required]

Below JSON configuration specifies how to pass additional configuration to the loader through a control message task at runtime.

Copy
Copied!
            

{ "type": "load", "properties": { "loader_id": "file_to_df", "files": ["/your/input/filepath"], } }

Previous File to DataFrame Loader
Next SQL Loader
© Copyright 2023, NVIDIA. Last updated on Feb 2, 2024.