Filesystem Spec Loader#
DataLoader module is configured to use this loader function. It is responsible for loading data from external sources using the fsspec
library, and returns the updated ControlMessage object with payload as MessageMeta, which contains DataFrame (with filenames).
Example Loader Configuration#
{
"loaders": [{
"id": "fsspec"
}]
}
Note : Loaders can receive configuration from the load
task via control message during runtime.
Task Configurable Parameters#
The parameters that can be configured for this specific loader at load task level:
Parameter |
Type |
Description |
Example Value |
Default Value |
---|---|---|---|---|
|
array |
List of files to load |
|
|
|
string |
Unique identifier for the loader |
|
|
Example Load Task Configuration#
Below JSON configuration specifies how to pass additional configuration to the loader through a control message task at runtime.
{
"type": "load",
"properties": {
"loader_id": "file_to_df",
"files": ["/your/input/filepath"],
}
}