morpheus.io.data_storage.file_system.FileSystemStorage
- class FileSystemStorage(file_path, file_format)[source]
Bases:
morpheus.io.data_storage_interface.RecordStorageInterfaceA class to manage storage of data in various file formats.
- Attributes
backing_sourcenum_rowsowner
Get the backing source file path.
Get the number of rows in the data.
Get whether this instance is the owner of the backing source file.
Methods
delete()Delete the backing source file if this instance is the owner. load()Load data from the backing source file. store(data_source[, copy_from_source])Store data in the backing source file. - property backing_source: str
Get the backing source file path.
- Returns:
The path to the backing source file.
- delete()[source]
Delete the backing source file if this instance is the owner.
- load()[source]
Load data from the backing source file.
- Returns:
The loaded data as a cudf.DataFrame.
- property num_rows: int
Get the number of rows in the data.
- Returns:
The number of rows in the data.
- property owner: bool
Get whether this instance is the owner of the backing source file.
- Returns:
True if this instance is the owner of the backing source file, False otherwise.
- store(data_source, copy_from_source=False)[source]
Store data in the backing source file.
- Parameters
data_source (Union[pd.DataFrame, cudf.DataFrame, str]) – The data to store. Can be a pandas.DataFrame, cudf.DataFrame, or a file path.
copy_from_source (bool, optional) – If True, data_source is assumed to be a file path and the data will be copied from this file to the backing source file. If False, the backing source file will simply be updated to point to data_source (which is assumed to be a file path).