morpheus.io.data_storage.file_system.FileSystemStorage

(Latest Version)
class FileSystemStorage(file_path, file_format)[source]

Bases: morpheus.io.data_storage_interface.RecordStorageInterface

A class to manage storage of data in various file formats.

Attributes
backing_source

Get the backing source file path.

num_rows

Get the number of rows in the data.

owner

Get whether this instance is the owner of the backing source file.

Methods

delete() Delete the backing source file if this instance is the owner.
load() Load data from the backing source file.
store(data_source[, copy_from_source]) Store data in the backing source file.
property backing_source: str

Get the backing source file path.

Returns:

The path to the backing source file.

delete()[source]

Delete the backing source file if this instance is the owner.

load()[source]

Load data from the backing source file.

Returns:

The loaded data as a cudf.DataFrame.

property num_rows: int

Get the number of rows in the data.

Returns:

The number of rows in the data.

property owner: bool

Get whether this instance is the owner of the backing source file.

Returns:

True if this instance is the owner of the backing source file, False otherwise.

store(data_source, copy_from_source=False)[source]

Store data in the backing source file.

Parameters
  • data_source (Union[pd.DataFrame, cudf.DataFrame, str]) – The data to store. Can be a pandas.DataFrame, cudf.DataFrame, or a file path.

  • copy_from_source (bool, optional) – If True, data_source is assumed to be a file path and the data will be copied from this file to the backing source file. If False, the backing source file will simply be updated to point to data_source (which is assumed to be a file path).

Previous morpheus.io.data_storage.file_system
Next morpheus.io.data_storage.in_memory
© Copyright 2024, NVIDIA. Last updated on Apr 25, 2024.