Filesystem Spec Loader

DataLoader module is configured to use this loader function. It is responsible for loading data from external sources using the <a href="https://filesystem-spec.readthedocs.io/en/latest/">fsspec</a> library, and returns the updated ControlMessage object with payload as MessageMeta, which contains DataFrame (with filenames).

            

            
{
	"loaders": [{
		"id": "fsspec"
	}]
}
        

Note : Loaders can receive configuration from the load task via control message during runtime.

The parameters that can be configured for this specific loader at load task level:

Below JSON configuration specifies how to pass additional configuration to the loader through a control message task at runtime.

            

            
{
	"type": "load",
	"properties": {
		"loader_id": "file_to_df",
		"files": ["/your/input/filepath"],
	}
}