Experimental#

APIs in this namespace are experimental and may change without warning in the future.

class pylibcudf.io.experimental.FileMetaData#

Parquet file footer metadata.

For details, see cudf::io::parquet::FileMetaData

Attributes

`columnchunk_metadata`	FileMetaData.columnchunk_metadata: dict[str, list[int]]
`created_by`	FileMetaData.created_by: str
`num_rows`	FileMetaData.num_rows: int
`row_group_num_rows`	FileMetaData.row_group_num_rows: list[int]
`row_groups`	FileMetaData.row_groups: list[RowGroup]
`version`	FileMetaData.version: int

Methods

from_bytes(cls, const uint8_t[)

Build FileMetaData from parquet footer bytes.

See also

pylibcudf.io.parquet_metadata.read_parquet_footers: Read one FileMetaData per source directly from pylibcudf.io.types.SourceInfo.

columnchunk_metadata#

FileMetaData.columnchunk_metadata: dict[str, list[int]]

Get a map of dotted column paths to lists of total_uncompressed_size values from every column chunk in this file.

Returns:

dict[str, list[int]]: Map of dotted column paths (".".join(path_in_schema)) to lists of total_uncompressed_size metadata from all their column chunks.

Notes

Equivalent to, but faster than, walking each row group’s columns:

>>> result: dict[str, list[int]] = {}
>>> for rg in file_metadata.row_groups:
...     for col in rg.columns:
...         name = ".".join(col.meta_data.path_in_schema)
...         result.setdefault(name, []).append(
...             col.meta_data.total_uncompressed_size
...         )

created_by#

FileMetaData.created_by: str

Get the application that created the file.

classmethod from_bytes( cls, const uint8_t[::1] footer_bytes: Buffer, ) → FileMetaData#

Build FileMetaData from parquet footer bytes.

Parameters:

footer_bytesBuffer: A contiguous bytes-like object containing parquet footer bytes. The bytes are forwarded as-is to cudf::io::parquet::experimental::hybrid_scan_reader without Python-side preprocessing. This method does not strip the parquet footer suffix (4-byte footer length + PAR1 magic), so callers should generally pass only the footer region bytes.

Returns:

FileMetaData: Parsed parquet file footer metadata.

num_rows#

FileMetaData.num_rows: int

Get the total number of rows.

row_group_num_rows#

FileMetaData.row_group_num_rows: list[int]

Get row counts for each row group in this file.

Returns:

list: A list with the row count per row group in this file.

Notes

Equivalent to, but faster than, checking each row groups’ num_rows:

>>> [rg.num_rows for rg in file_metadata.row_groups]

row_groups#

FileMetaData.row_groups: list[RowGroup]

Get row group metadata in this file.

version#

FileMetaData.version: int

Get the file format version.

class pylibcudf.io.experimental.HybridScanMetadata#

Shareable, pre-parsed Parquet file metadata for the hybrid scan reader.

This class enables parsing the metadata of a Parquet file once, then constructing multiple HybridScanReader instances that share it (one per row-group range of the file) instead of each re-parsing and copying the metadata.

For details, see cudf::io::parquet::experimental::hybrid_scan_metadata

Methods

`from_footer_bytes`(const uint8_t[, ...)	Parse shareable metadata from Parquet footer bytes.
`from_parquet_metadata`(FileMetaData metadata, ...)	Build shareable metadata from a pre-populated `FileMetaData`.

Examples

>>> import pylibcudf as plc
>>> metadata = plc.io.experimental.HybridScanMetadata.from_parquet_metadata(
...     file_metadata, options)
>>> reader = plc.io.experimental.HybridScanReader.from_metadata(metadata)

static from_footer_bytes( const uint8_t[::1] footer_bytes, ParquetReaderOptions options, )#

Parse shareable metadata from Parquet footer bytes.

Parameters:

footer_bytesBuffer: Parquet file footer bytes
optionsParquetReaderOptions: Parquet reader options

Returns:

HybridScanMetadata

static from_parquet_metadata( FileMetaData metadata, ParquetReaderOptions options, )#

Build shareable metadata from a pre-populated FileMetaData.

Parameters:

metadataFileMetaData: Pre-populated Parquet file metadata
optionsParquetReaderOptions: Parquet reader options

Returns:

HybridScanMetadata

class pylibcudf.io.experimental.HybridScanReader( const uint8_t[::1] footer_bytes, ParquetReaderOptions options, )#

Experimental Parquet reader optimized for highly selective filters.

This class implements a hybrid scan operation for reading Parquet files with highly selective filters. It reads in two passes: first reading filter columns to build a row mask, then reading payload columns using that mask for optimization.

For details, see cudf::io::parquet::experimental::hybrid_scan_reader

Parameters:

footer_bytesBuffer: Parquet file footer bytes
optionsParquetReaderOptions: Parquet reader options

Methods

`all_column_chunks_byte_ranges`(self, ...)	Get byte ranges of column chunks of all columns.
`all_row_groups`(self, ...)	Get all available row groups from the parquet file.
`build_all_true_row_mask`(self, ...[, stream])	Build an all-true boolean survival column for the given row groups.
`build_row_mask_with_page_index_stats`(self, ...)	Build a boolean column indicating surviving rows from page stats.
`construct_row_group_passes`(self, ...)	Partition row groups into passes such that the GPU memory required to materialize a pass is bounded by the specified limit.
`filter_column_chunks_byte_ranges`(self, ...)	Get byte ranges of column chunks of filter columns.
`filter_row_groups_with_bloom_filters`(self, ...)	Filter row groups using column chunk bloom filters.
`filter_row_groups_with_dictionary_pages`(...)	Filter row groups using column chunk dictionary pages.
`filter_row_groups_with_stats`(self, ...)	Filter row groups using column chunk statistics.
`from_metadata`(HybridScanMetadata metadata)	Create a HybridScanReader that shares pre-parsed metadata.
`from_parquet_metadata`(FileMetaData metadata, ...)	Create a HybridScanReader from pre-populated metadata.
`has_next_table_chunk`(self)	Check if there is any parquet data left to read.
`materialize_all_columns`(self, ...)	Materialize all columns.
`materialize_filter_columns`(self, ...)	Materialize filter columns and update the row mask.
`materialize_filter_columns_chunk`(self, ...)	Materialize a chunk of filter columns.
`materialize_payload_columns`(self, ...)	Materialize payload columns and apply the row mask.
`materialize_payload_columns_chunk`(self, ...)	Materialize a chunk of payload columns.
`page_index_byte_range`(self)	Get the byte range of the page index.
`parquet_metadata`(self)	Get the Parquet file footer metadata.
`payload_column_chunks_byte_ranges`(self, ...)	Get byte ranges of column chunks of payload columns.
`reset_column_selection`(self)	Reset the column selection state.
`secondary_filters_byte_ranges`(self, ...)	Get byte ranges of bloom filters and dictionary pages.
`setup_chunking_for_filter_columns`(self, ...)	Setup chunking information for filter columns.
`setup_chunking_for_payload_columns`(self, ...)	Setup chunking information for payload columns.
`setup_page_index`(self, const uint8_t[)	Setup the page index within the Parquet file metadata.
`total_rows_in_row_groups`(self, ...)	Get the total number of top-level rows in the row groups.

Examples

>>> import pylibcudf as plc
>>> # Create reader from footer bytes
>>> reader = plc.io.hybrid_scan.HybridScanReader(footer_bytes, options)
>>> # Get metadata
>>> metadata = reader.parquet_metadata()
>>> # Get all row groups
>>> row_groups = reader.all_row_groups(options)

all_column_chunks_byte_ranges(self, list row_group_indices: list[int], ParquetReaderOptions options) → list[ByteRangeInfo]#

Get byte ranges of column chunks of all columns.

Parameters:

row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options

Returns:

list[ByteRangeInfo]: Byte ranges to column chunks of all columns

all_row_groups( self, ParquetReaderOptions options, ) → list[int]#

Get all available row groups from the parquet file.

Parameters:

optionsParquetReaderOptions: Parquet reader options

Returns:

list[int]: List of row group indices

build_all_true_row_mask( self, list row_group_indices, stream=None, DeviceMemoryResource mr=None, )#

Build an all-true boolean survival column for the given row groups.

Parameters:

row_group_indiceslist[int]: Input row group indices
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

Returns:

Column: All-true boolean column with one entry per row across all row groups

build_row_mask_with_page_index_stats(self, list row_group_indices: list[int], ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None) → Column#

Build a boolean column indicating surviving rows from page stats.

Parameters:

row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

Returns:

Column: Boolean column indicating surviving rows

construct_row_group_passes(self, list row_group_indices: list[int], size_t pass_read_limit) → list[list[int]]#

Partition row groups into passes such that the GPU memory required to materialize a pass is bounded by the specified limit.

Note that pass_read_limit is a hint, not an absolute limit. i.e. if a row group cannot fit within the limit, it will still constitute a valid pass.

Parameters:

row_group_indiceslist[int]: Input row group indices
pass_read_limitint: Limit on the amount of memory used for reading and decompressing data
or 0 if there is no limit.

Returns:

list[list[int]]: Lists of row group indices, one per pass.

Raises:

ValueError: If row_group_indices is empty.

filter_column_chunks_byte_ranges(self, list row_group_indices: list[int], ParquetReaderOptions options) → list[ByteRangeInfo]#

Get byte ranges of column chunks of filter columns.

Parameters:

row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options

Returns:

list[ByteRangeInfo]: Byte ranges to column chunks of filter columns

filter_row_groups_with_bloom_filters(self, list bloom_filter_data, list row_group_indices: list[int], ParquetReaderOptions options, stream: CudaStreamLike | None = None) → list[int]#

Filter row groups using column chunk bloom filters.

Parameters:

bloom_filter_dataSequence: Span-like objects containing bloom filter data
row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream

Returns:

list[int]: Filtered row group indices

filter_row_groups_with_dictionary_pages(self, list dictionary_page_data, list row_group_indices: list[int], ParquetReaderOptions options, stream: CudaStreamLike | None = None) → list[int]#

Filter row groups using column chunk dictionary pages.

Parameters:

dictionary_page_dataSequence: Span-like objects containing dictionary page data
row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream

Returns:

list[int]: Filtered row group indices

filter_row_groups_with_stats(self, list row_group_indices: list[int], ParquetReaderOptions options, stream: CudaStreamLike | None = None) → list[int]#

Filter row groups using column chunk statistics.

Parameters:

row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream

Returns:

list[int]: Filtered row group indices

static from_metadata(HybridScanMetadata metadata)#

Create a HybridScanReader that shares pre-parsed metadata.

Constructs a lightweight reader that borrows metadata instead of re-parsing and copying the file metadata. Use one shared HybridScanMetadata to read row-group ranges of a single file. Overlapping row-group ranges across readers produce duplicate rows.

Parameters:

metadataHybridScanMetadata: Shared, pre-parsed Parquet file metadata

Returns:

HybridScanReader

static from_parquet_metadata( FileMetaData metadata, ParquetReaderOptions options, ) → HybridScanReader#

Create a HybridScanReader from pre-populated metadata.

Parameters:

metadataFileMetaData: Pre-populated Parquet file metadata
optionsParquetReaderOptions: Parquet reader options

Returns:

HybridScanReader

has_next_table_chunk(self) → bool#

Check if there is any parquet data left to read.

Returns:

bool: True if there is data left to read

materialize_all_columns(self, list row_group_indices: list[int], list column_chunk_data, ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None) → TableWithMetadata#

Materialize all columns.

Parameters:

row_group_indiceslist[int]: Input row group indices
column_chunk_dataSequence: Span-like objects containing column chunk data of all columns
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

Returns:

TableWithMetadata: Table of materialized all columns and metadata

materialize_filter_columns(self, list row_group_indices: list[int], list column_chunk_data, Column row_mask, use_data_page_mask mask_data_pages, ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None) → TableWithMetadata#

Materialize filter columns and update the row mask.

Parameters:

row_group_indiceslist[int]: Input row group indices
column_chunk_dataSequence: Span-like objects containing column chunk data of filter columns
row_maskColumn: Mutable boolean column indicating surviving rows
mask_data_pagesUseDataPageMask: Whether to use a data page mask
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

Returns:

TableWithMetadata: Table of materialized filter columns and metadata

materialize_filter_columns_chunk( self, Column row_mask, ) → TableWithMetadata#

Materialize a chunk of filter columns.

Parameters:

row_maskColumn: Mutable boolean column indicating surviving rows
Returns
——-
TableWithMetadata: Table chunk of materialized filter columns and metadata

materialize_payload_columns(self, list row_group_indices: list[int], list column_chunk_data, Column row_mask, use_data_page_mask mask_data_pages, ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None) → TableWithMetadata#

Materialize payload columns and apply the row mask.

Parameters:

row_group_indiceslist[int]: Input row group indices
column_chunk_dataSequence: Span-like objects containing column chunk data of payload columns
row_maskColumn: Boolean column indicating surviving rows
mask_data_pagesUseDataPageMask: Whether to use a data page mask
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

Returns:

TableWithMetadata: Table of materialized payload columns and metadata

materialize_payload_columns_chunk( self, Column row_mask, ) → TableWithMetadata#

Materialize a chunk of payload columns.

Parameters:

row_maskColumn: Boolean column indicating surviving rows
Returns
——-
TableWithMetadata: Table chunk of materialized payload columns and metadata

page_index_byte_range(self) → ByteRangeInfo#

Get the byte range of the page index.

Returns:

ByteRangeInfo: Byte range of the page index

parquet_metadata(self) → FileMetaData#

Get the Parquet file footer metadata.

Returns:

FileMetaData: Parquet file footer metadata

payload_column_chunks_byte_ranges(self, list row_group_indices: list[int], ParquetReaderOptions options) → list[ByteRangeInfo]#

Get byte ranges of column chunks of payload columns.

Parameters:

row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options

Returns:

list[ByteRangeInfo]: Byte ranges to column chunks of payload columns

reset_column_selection(self) → None#

Reset the column selection state.

Resets the internal column selection state forcing re-selection of columns in subsequent filter and read operations

secondary_filters_byte_ranges(self, list row_group_indices: list[int], ParquetReaderOptions options) → tuple[list[ByteRangeInfo], list[ByteRangeInfo]]#

Get byte ranges of bloom filters and dictionary pages.

Parameters:

row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options

Returns:

tuple[list[ByteRangeInfo], list[ByteRangeInfo]]: Tuple of (bloom_filter_ranges, dictionary_page_ranges)

setup_chunking_for_filter_columns(self, size_t chunk_read_limit, size_t pass_read_limit, list row_group_indices: list[int], Column row_mask, use_data_page_mask mask_data_pages, column_chunk_data, ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None) → None#

Setup chunking information for filter columns.

Parameters:

chunk_read_limitint: Limit on bytes returned per chunk (0 for no limit)
pass_read_limitint: Limit on memory for reading/decompressing (0 for no limit)
row_group_indiceslist[int]: Input row group indices
row_maskColumn: Boolean column indicating surviving rows
mask_data_pagesUseDataPageMask: Whether to use a data page mask
column_chunk_dataSequence: Span-like objects containing column chunk data of filter columns
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

setup_chunking_for_payload_columns(self, size_t chunk_read_limit, size_t pass_read_limit, list row_group_indices: list[int], Column row_mask, use_data_page_mask mask_data_pages, column_chunk_data, ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None) → None#

Setup chunking information for payload columns.

Parameters:

chunk_read_limitint: Limit on bytes returned per chunk (0 for no limit)
pass_read_limitint: Limit on memory for reading/decompressing (0 for no limit)
row_group_indiceslist[int]: Input row group indices
row_maskColumn: Boolean column indicating surviving rows
mask_data_pagesUseDataPageMask: Whether to use a data page mask
column_chunk_dataSequence: Span-like objects containing column chunk data of payload columns
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

setup_page_index( self, const uint8_t[::1] page_index_bytes: Buffer, ) → None#

Setup the page index within the Parquet file metadata.

Parameters:

page_index_bytesBuffer: Parquet page index buffer bytes

total_rows_in_row_groups(self, list row_group_indices: list[int]) → int#

Get the total number of top-level rows in the row groups.

Parameters:

row_group_indiceslist[int]: Input row group indices

Returns:

int: Total number of top-level rows

pylibcudf.io.experimental.UseDataPageMask#