Is this page helpful?

pylibwholegraph.torch.create_wholememory_tensor_from_filelist#

pylibwholegraph.torch.create_wholememory_tensor_from_filelist(
comm: WholeMemoryCommunicator,
memory_type: str,
memory_location: str,
filelist: List[str] | str,
dtype: dtype,
last_dim_size: int | None = None,
last_dim_strides: int = -1,
tensor_entry_partition: List[int] | None = None,
file_format: str = 'binary',
expected_entry_count: int | None = None,
expected_shape: List[int] | tuple | None = None,
fail_on_dtype_mismatch: bool = False,
)#

Create WholeMemory Tensor from a list of files. :param comm: WholeMemoryCommunicator :param memory_type: WholeMemory type, should be continuous, chunked or distributed :param memory_location: WholeMemory location, should be cpu or cuda :param filelist: list of files :param dtype: data type of the tensor :param last_dim_size: 0 creates a 1-D array and a positive value creates a

matrix with that column count. Required for binary input and inferred from Parquet metadata when omitted.

Parameters:

  • last_dim_strides – stride of last_dim, -1 for same as size of last dim.

  • tensor_entry_partition – rank partition based on entry; tensor_entry_partition[i] determines the entry count of rank i and shoud be a positive integer; the sum of tensor_entry_partition should equal to total entry count; entries will be equally partitioned if None

  • file_format – file format, one of binary, parquet, or auto

  • expected_entry_count – optional expected number of rows. An error is raised before allocation when the files contain a different row count.

  • expected_shape – optional expected 1-D or 2-D shape. A one-column Parquet file defaults to 1-D unless (N, 1) is specified here.

  • fail_on_dtype_mismatch – raise an error instead of warning and converting when Parquet column dtypes differ from dtype.

Returns:

WholeMemoryTensor