Filesystem Spec Loader
DataLoader module is configured to use this loader function. It is responsible for loading data from external sources using the fsspec library, and returns the updated ControlMessage object with payload as MessageMeta, which contains dataframe (with filenames).
{
"loaders": [{
"id": "fsspec"
}]
}
Note : Loaders can receive configuration from the load
task via control message during runtime.
The parameters that can be configured for this specific loader at load task level:
Parameter |
Type |
Description |
Example Value |
Default Value |
---|---|---|---|---|
|
array |
List of files to load |
[“/your/input/filepath”] |
|
|
string |
Unique identifier for the loader |
“file_to_df” |
|
Below JSON configuration specifies how to pass additional configuration to the loader through a control message task at runtime.
{
"type": "load",
"properties": {
"loader_id": "file_to_df",
"files": ["/your/input/filepath"],
}
}