Parquet IO Utils

IO utilities for Parquet.

pylibcudf.io.parquet_io_utils.fetch_byte_ranges_to_device(

SourceInfo source_info,

list byte_ranges,

stream=None,

DeviceMemoryResource mr=None,

) → list

Fetch byte ranges from a Parquet source into device memory.

Parameters:

source_infoSourceInfo

Source describing a single Parquet file.

byte_rangeslist[ByteRangeInfo]

Byte ranges to fetch, as returned by filter_column_chunks_byte_ranges(), payload_column_chunks_byte_ranges(), or all_column_chunks_byte_ranges().

streamStream, optional

CUDA stream.

mrDeviceMemoryResource, optional

Device memory resource.

Returns:

list[gpumemoryview]

One view per byte range. Each view holds a reference to the DeviceBuffer that owns its memory, keeping the allocation alive for as long as the view is referenced.

Raises:

ValueError

If source_info does not describe exactly one source.

pylibcudf.io.parquet_io_utils.fetch_page_index_to_host(

SourceInfo source_info,

ByteRangeInfo page_index_range,

) → bytes

Fetch parquet page index bytes to host memory.

Parameters:

source_infoSourceInfo

Source describing a single Parquet file.

page_index_rangeByteRangeInfo

Byte range of the page index, as returned by page_index_byte_range().

Returns:

bytes

Raw page index bytes copied to Python host memory.

Raises:

ValueError

If source_info does not describe exactly one source.