- class FileSystemDataset(*args, **kwargs)[source]
Bases:
torch.utils.data.DatasetA dataset class that reads data in batches from a folder and applies preprocessing to each batch. * This class assumes that the data is saved in small csv files in one folder.
- Attributes
- batch_size
num_samples- preprocess_fn
- shuffle_batch_indices
- shuffle_rows_in_batch
Returns the number of samples in the dataset.
Methods
__call__(*args, **kwargs)Call self as a function. get_preloaded_data()Loads all data from the files into memory and returns it as a pandas.DataFrame. - get_preloaded_data()[source]
Loads all data from the files into memory and returns it as a pandas.DataFrame.
- property num_samples
Returns the number of samples in the dataset.
morpheus.models.dfencoder.dataloader.FileSystemDataset
(Latest Version)
Previous
morpheus.models.dfencoder.dataloader.DataframeDataset
Next
morpheus.models.dfencoder.distributed_ae