morpheus.models.dfencoder.dataloader.FileSystemDataset

class FileSystemDataset(*args, **kwargs)[source]

Bases: torch.utils.data.Dataset

A dataset class that reads data in batches from a folder and applies preprocessing to each batch. * This class assumes that the data is saved in small csv files in one folder.

Attributes
batch_size

num_samples

Returns the number of samples in the dataset.

preprocess_fn

shuffle_batch_indices

shuffle_rows_in_batch

Methods

__call__(*args, **kwargs) Call self as a function.
get_preloaded_data() Loads all data from the files into memory and returns it as a pandas.DataFrame.
get_preloaded_data()[source]

Loads all data from the files into memory and returns it as a pandas.DataFrame.

property num_samples

Returns the number of samples in the dataset.

Previous morpheus.models.dfencoder.dataloader.DataframeDataset
Next morpheus.models.dfencoder.distributed_ae
© Copyright 2023, NVIDIA. Last updated on Feb 2, 2024.