morpheus.io.data_storage.file_system.FileSystemStorage#

class FileSystemStorage(file_path, file_format)[source]#

Bases: RecordStorageInterface

A class to manage storage of data in various file formats.

Attributes:
backing_source

Get the backing source file path.

num_rows

Get the number of rows in the data.

owner

Get whether this instance is the owner of the backing source file.

Methods

delete()

Delete the backing source file if this instance is the owner.

load()

Load data from the backing source file.

store(data_source[, copy_from_source])

Store data in the backing source file.
property backing_source: str#

Get the backing source file path.

Returns:

The path to the backing source file.

delete()[source]#

Delete the backing source file if this instance is the owner.

load()[source]#

Load data from the backing source file.

Returns:

The loaded data as a cudf.DataFrame.

property num_rows: int#

Get the number of rows in the data.

Returns:

The number of rows in the data.

property owner: bool#

Get whether this instance is the owner of the backing source file.

Returns:

True if this instance is the owner of the backing source file, False otherwise.

store(data_source, copy_from_source=False)[source]#

Store data in the backing source file.

Parameters:

  • data_source (Union[pd.DataFrame, cudf.DataFrame, str]) – The data to store. Can be a pandas.DataFrame, cudf.DataFrame, or a file path.

  • copy_from_source (bool, optional) – If True, data_source is assumed to be a file path and the data will be copied from this file to the backing source file. If False, the backing source file will simply be updated to point to data_source (which is assumed to be a file path).