Functions
fsspec_loader(control_message, task) |
Loads data from external sources using the fsspec library, and returns an updated ControlMessage object with payload as MessageMeta, which contains a dataframe with file names and data. |
- fsspec_loader(control_message, task)[source]
Loads data from external sources using the fsspec library, and returns an updated ControlMessage object with payload as MessageMeta, which contains a dataframe with file names and data.
- Parameters
- control_message
- task
The ControlMessage object containing the pipeline control message.
A dictionary representing the current task in the pipeline control message.
- Returns
- control_message
An updated ControlMessage object with payload as a MessageMeta containing a dataframe with file names and data.
- Raises
- RuntimeError:
If no files matched the input strings specified in the task, or if there was an error loading the data.