Experimental#

APIs in this namespace are experimental and may change without warning in the future.

class pylibcudf.io.experimental.FileMetaData#

Parquet file footer metadata.

For details, see cudf::io::parquet::FileMetaData

Attributes

`columnchunk_metadata`	FileMetaData.columnchunk_metadata: dict[str, list[int]]
`created_by`	FileMetaData.created_by: str
`num_rows`	FileMetaData.num_rows: int
`row_group_num_rows`	FileMetaData.row_group_num_rows: list[int]
`row_groups`	FileMetaData.row_groups: list[RowGroup]
`schema`	FileMetaData.schema: list[SchemaElement]
`version`	FileMetaData.version: int

Methods

from_bytes(cls, const uint8_t[)

Build FileMetaData from parquet footer bytes.

See also

pylibcudf.io.parquet_metadata.read_parquet_footers: Read one FileMetaData per source directly from pylibcudf.io.types.SourceInfo.

columnchunk_metadata#

FileMetaData.columnchunk_metadata: dict[str, list[int]]

Get a map of dotted column paths to lists of total_uncompressed_size values from every column chunk in this file.

Returns:

dict[str, list[int]]: Map of dotted column paths (".".join(path_in_schema)) to lists of total_uncompressed_size metadata from all their column chunks.

Notes

Equivalent to, but faster than, walking each row group’s columns:

>>> result: dict[str, list[int]] = {}
>>> for rg in file_metadata.row_groups:
...     for col in rg.columns:
...         name = ".".join(col.meta_data.path_in_schema)
...         result.setdefault(name, []).append(
...             col.meta_data.total_uncompressed_size
...         )

created_by#

FileMetaData.created_by: str

Get the application that created the file.

classmethod from_bytes( cls, const uint8_t[::1] footer_bytes: Buffer, ) → FileMetaData#

Build FileMetaData from parquet footer bytes.

Parameters:

footer_bytesBuffer: A contiguous bytes-like object containing parquet footer bytes. The bytes are forwarded as-is to cudf::io::parquet::experimental::hybrid_scan_reader without Python-side preprocessing. This method does not strip the parquet footer suffix (4-byte footer length + PAR1 magic), so callers should generally pass only the footer region bytes.

Returns:

FileMetaData: Parsed parquet file footer metadata.

num_rows#

FileMetaData.num_rows: int

Get the total number of rows.

row_group_num_rows#

FileMetaData.row_group_num_rows: list[int]

Get row counts for each row group in this file.

Returns:

list: A list with the row count per row group in this file.

Notes

Equivalent to, but faster than, checking each row groups’ num_rows:

>>> [rg.num_rows for rg in file_metadata.row_groups]

row_groups#

FileMetaData.row_groups: list[RowGroup]

Get row group metadata in this file.

schema#

FileMetaData.schema: list[SchemaElement]

Get the file’s schema tree, flattened by a depth-first traversal.

The first element is the root. Each element’s num_children gives the number of elements that follow it at the next level, which is what allows the tree to be reconstructed from the flat list.

Returns:

list[SchemaElement]: One entry per schema element, in depth-first order.

version#

FileMetaData.version: int

Get the file format version.

class pylibcudf.io.experimental.HybridScanMetadata#

Shareable, pre-parsed Parquet file metadata for the hybrid scan reader.

This class enables parsing the metadata of a Parquet file once, then constructing multiple HybridScanReader instances that share it (one per row-group range of the file) instead of each re-parsing and copying the metadata.

For details, see cudf::io::parquet::experimental::hybrid_scan_metadata

Methods

`from_footer_bytes`(const uint8_t[, ...)	Parse shareable metadata from Parquet footer bytes.
`from_parquet_metadata`(FileMetaData metadata, ...)	Build shareable metadata from a pre-populated `FileMetaData`.

Examples

>>> import pylibcudf as plc
>>> metadata = plc.io.experimental.HybridScanMetadata.from_parquet_metadata(
...     file_metadata, options)
>>> reader = plc.io.experimental.HybridScanReader.from_metadata(metadata)

static from_footer_bytes( const uint8_t[::1] footer_bytes, ParquetReaderOptions options, )#

Parse shareable metadata from Parquet footer bytes.

Parameters:

footer_bytesBuffer: Parquet file footer bytes
optionsParquetReaderOptions: Parquet reader options

Returns:

HybridScanMetadata

static from_parquet_metadata( FileMetaData metadata, ParquetReaderOptions options, )#

Build shareable metadata from a pre-populated FileMetaData.

Parameters:

metadataFileMetaData: Pre-populated Parquet file metadata
optionsParquetReaderOptions: Parquet reader options

Returns:

HybridScanMetadata

class pylibcudf.io.experimental.HybridScanMultiFile#

Experimental multi-source Parquet reader for highly selective filters.

Vectorizes the HybridScanReader APIs over multiple Parquet sources. Inputs and outputs are indexed by source order, except for the row mask which is a single boolean column spanning all rows of all sources concatenated in source order, then in row-group order within a source.

For details, see cudf::io::parquet::experimental::hybrid_scan_multifile

Methods

`all_column_chunks_byte_ranges`(self, ...)	Get byte ranges of column chunks of all columns.
`all_row_groups`(self, ...)	Get all available per-source row group indices.
`bloom_filters_byte_ranges`(self, ...)	Get byte ranges of bloom filters for row group pruning.
`build_all_true_row_mask`(self, ...)	Build an all-true boolean column spanning all selected rows.
`build_row_mask_with_page_index_stats`(self, ...)	Build a boolean column indicating surviving rows from page stats.
`construct_row_group_passes`(self, ...)	Partition row groups into passes such that the GPU memory required to materialize a pass for selected columns is bounded by the specified limit.
`dictionary_pages_byte_ranges`(self, ...)	Get byte ranges of column chunk dictionary pages for row group pruning.
`filter_row_groups_with_byte_range`(self, ...)	Filter row groups using the byte range from the options.
`filter_row_groups_with_stats`(self, ...)	Filter row groups using column chunk statistics.
`from_parquet_metadatas`(parquet_metadatas, ...)	Create a HybridScanMultiFile from pre-populated metadata.
`has_next_table_chunk`(self)	Check if there is any parquet data left to read.
`materialize_all_columns`(self, ...)	Materialize all columns.
`materialize_payload_columns_chunk`(self, ...)	Materialize a chunk of payload columns.
`page_index_byte_ranges`(self)	Get the byte range of the page index of all sources.
`parquet_metadatas`(self)	Get the Parquet file footer metadata of all sources.
`payload_pages_byte_ranges`(self, ...)	Get byte ranges of the pages of payload columns.
`reset_column_selection`(self)	Reset the column selection state.
`setup_chunking_for_payload_pages`(self, ...)	Setup chunking information for payload columns read page by page.
`setup_page_indexes`(self, page_index_bytes)	Setup the page index within the Parquet file metadata of all sources.
`total_rows_in_row_groups`(self, ...)	Get the total number of top-level rows in the row groups.

Examples

>>> import pylibcudf as plc
>>> reader = plc.io.experimental.HybridScanMultiFile.from_parquet_metadatas(
...     metadatas, options)
>>> row_groups = reader.all_row_groups(options)

all_column_chunks_byte_ranges( self, RowGroupIndices row_group_indices, ParquetReaderOptions options, ) → tuple[list[ByteRangeInfo], list[int]]#

Get byte ranges of column chunks of all columns.

Parameters:

row_group_indicesRowGroupIndices: Input row group indices, one list per source
optionsParquetReaderOptions: Parquet reader options

Returns:

tuple[list[ByteRangeInfo], list[int]]: Flattened byte ranges to column chunks of all columns and their corresponding source indices

all_row_groups( self, ParquetReaderOptions options, ) → RowGroupIndices#

Get all available per-source row group indices.

Parameters:

optionsParquetReaderOptions: Parquet reader options

Returns:

RowGroupIndices: Row group indices, one inner list per source

bloom_filters_byte_ranges( self, RowGroupIndices row_group_indices, ParquetReaderOptions options, ) → tuple[list[ByteRangeInfo], list[int]]#

Get byte ranges of bloom filters for row group pruning.

Parameters:

row_group_indicesRowGroupIndices: Input row group indices, one list per source
optionsParquetReaderOptions: Parquet reader options

Returns:

tuple[list[ByteRangeInfo], list[int]]: Flattened bloom filter byte ranges and their corresponding source indices

build_all_true_row_mask( self, RowGroupIndices row_group_indices, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None, ) → Column#

Build an all-true boolean column spanning all selected rows.

Parameters:

row_group_indicesRowGroupIndices: Input row group indices, one list per source
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

Returns:

Column: An all-true boolean column spanning all selected rows across all sources

build_row_mask_with_page_index_stats( self, RowGroupIndices row_group_indices, ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None, ) → Column#

Build a boolean column indicating surviving rows from page stats.

Parameters:

row_group_indicesRowGroupIndices: Input row group indices, one list per source
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

Returns:

Column: Boolean column indicating surviving rows across all sources

construct_row_group_passes( self, read_columns_mode columns_mode, RowGroupIndices row_group_indices, size_t pass_read_limit, ParquetReaderOptions options, ) → list[RowGroupIndices]#

Partition row groups into passes such that the GPU memory required to materialize a pass for selected columns is bounded by the specified limit.

Note that pass_read_limit is a hint, not an absolute limit. i.e. if a row group cannot fit within the limit, it will still constitute a valid pass.

Parameters:

columns_modeReadColumnsMode: Columns selection to use for pass memory estimation
row_group_indicesRowGroupIndices: Input row group indices, one list per source.
pass_read_limitint: Limit on the amount of memory used for reading and decompressing data, or 0 if there is no limit.
optionsParquetReaderOptions: Parquet reader options used to select columns.

Returns:

list[RowGroupIndices]: Per-source row group indices, one list per pass.

Raises:

ValueError: If row_group_indices is empty.

dictionary_pages_byte_ranges( self, RowGroupIndices row_group_indices, ParquetReaderOptions options, ) → tuple[list[ByteRangeInfo], list[int]]#

Get byte ranges of column chunk dictionary pages for row group pruning.

Parameters:

row_group_indicesRowGroupIndices: Input row group indices, one list per source
optionsParquetReaderOptions: Parquet reader options

Returns:

tuple[list[ByteRangeInfo], list[int]]: Flattened dictionary page byte ranges and their corresponding source indices

filter_row_groups_with_byte_range( self, RowGroupIndices row_group_indices, ParquetReaderOptions options, ) → RowGroupIndices#

Filter row groups using the byte range from the options.

Parameters:

row_group_indicesRowGroupIndices: Input row group indices, one list per source
optionsParquetReaderOptions: Parquet reader options

Returns:

RowGroupIndices: Filtered row group indices, one inner list per source

filter_row_groups_with_stats( self, RowGroupIndices row_group_indices, ParquetReaderOptions options, stream: CudaStreamLike | None = None, ) → RowGroupIndices#

Filter row groups using column chunk statistics.

Parameters:

row_group_indicesRowGroupIndices: Input row group indices, one list per source
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream

Returns:

RowGroupIndices: Filtered row group indices, one inner list per source

static from_parquet_metadatas(parquet_metadatas: Sequence[FileMetaData], ParquetReaderOptions options) → HybridScanMultiFile#

Create a HybridScanMultiFile from pre-populated metadata.

Parameters:

parquet_metadatasSequence[FileMetaData]: Pre-populated Parquet file metadata, one per source
optionsParquetReaderOptions: Parquet reader options

Returns:

HybridScanMultiFile

has_next_table_chunk(self) → bool#

Check if there is any parquet data left to read.

Returns:

bool: True if there is data left to read

materialize_all_columns(self, RowGroupIndices row_group_indices, list column_chunk_data: list[Span], ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None) → TableWithMetadata#

Materialize all columns.

Parameters:

row_group_indicesRowGroupIndices: Input row group indices, one list per source
column_chunk_datalist[Span]: Span-like objects containing flattened column chunk data of all columns, in the same order as all_column_chunks_byte_ranges
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

Returns:

TableWithMetadata: Table of materialized all columns and metadata

materialize_payload_columns_chunk( self, Column row_mask, ) → TableWithMetadata#

Materialize a chunk of payload columns.

Parameters:

row_maskColumn: Boolean column indicating which rows need to be read

Returns:

TableWithMetadata: Table chunk of materialized payload columns and metadata

page_index_byte_ranges( self, ) → list[ByteRangeInfo]#

Get the byte range of the page index of all sources.

Returns:

list[ByteRangeInfo]: Byte range of the page index, one per source

parquet_metadatas(self) → list[FileMetaData]#

Get the Parquet file footer metadata of all sources.

Returns:

list[FileMetaData]: Parquet file footer metadata, one per source

payload_pages_byte_ranges( self, RowGroupIndices row_group_indices, Column row_mask, ParquetReaderOptions options, stream: CudaStreamLike | None = None, ) → tuple[list[ByteRangeInfo], list[int]]#

Get byte ranges of the pages of payload columns.

Byte ranges are flattened in source, row group, column chunk, and page order. Dictionary pages precede data pages within a column chunk, and pruned pages are returned as empty byte ranges.

Parameters:

row_group_indicesRowGroupIndices: Input row group indices, one list per source
row_maskColumn: Boolean column indicating which rows need to be read
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream

Returns:

tuple[list[ByteRangeInfo], list[int]]: Flattened byte ranges to the pages of payload columns and the source index of each byte range

reset_column_selection(self) → None#

Reset the column selection state.

Resets the internal column selection state forcing re-selection of columns in subsequent filter and read operations

setup_chunking_for_payload_pages( self, size_t chunk_read_limit, size_t pass_read_limit, RowGroupIndices row_group_indices, Column row_mask, page_data: Sequence, ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None, ) → None#

Setup chunking information for payload columns read page by page.

The data page mask is inferred from page_data, which must have the same shape as the byte ranges returned by payload_pages_byte_ranges().

Parameters:

chunk_read_limitint: Limit on bytes returned per chunk (0 for no limit)
pass_read_limitint: Limit on memory for reading/decompressing (0 for no limit)
row_group_indicesRowGroupIndices: Input row group indices, one list per source
row_maskColumn: Boolean column indicating which rows need to be read
page_dataSequence: Span-like objects containing the page data of payload columns, in the same order as the byte ranges returned by payload_pages_byte_ranges(). None indicates a pruned page
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

setup_page_indexes( self, page_index_bytes: Sequence[Buffer], ) → None#

Setup the page index within the Parquet file metadata of all sources.

Parameters:

page_index_bytesSequence[Buffer]: Parquet page index buffer bytes, one per source

total_rows_in_row_groups( self, RowGroupIndices row_group_indices, ) → int#

Get the total number of top-level rows in the row groups.

Parameters:

row_group_indicesRowGroupIndices: Input row group indices, one list per source

Returns:

int: Total number of top-level rows across all sources

class pylibcudf.io.experimental.HybridScanReader( const uint8_t[::1] footer_bytes: Buffer, ParquetReaderOptions options, )#

Experimental Parquet reader optimized for highly selective filters.

This class implements a hybrid scan operation for reading Parquet files with highly selective filters. It reads in two passes: first reading filter columns to build a row mask, then reading payload columns using that mask for optimization.

For details, see cudf::io::parquet::experimental::hybrid_scan_reader

Parameters:

footer_bytesBuffer: Parquet file footer bytes
optionsParquetReaderOptions: Parquet reader options

Methods

`all_column_chunks_byte_ranges`(self, ...)	Get byte ranges of column chunks of all columns.
`all_row_groups`(self, ...)	Get all available row groups from the parquet file.
`bloom_filters_byte_ranges`(self, ...)	Get byte ranges of bloom filters for row group pruning.
`build_all_true_row_mask`(self, ...)	Build an all-true boolean survival column for the given row groups.
`build_row_mask_with_page_index_stats`(self, ...)	Build a boolean column indicating surviving rows from page stats.
`construct_row_group_passes`(self, ...)	Partition row groups into passes such that the GPU memory required to materialize a pass for selected columns is bounded by the specified limit.
`dictionary_pages_byte_ranges`(self, ...)	Get byte ranges of column chunk dictionary pages for row group pruning.
`filter_column_chunks_byte_ranges`(self, ...)	Get byte ranges of column chunks of filter columns.
`filter_row_groups_with_bloom_filters`(self, ...)	Filter row groups using column chunk bloom filters.
`filter_row_groups_with_dictionary_pages`(...)	Filter row groups using column chunk dictionary pages.
`filter_row_groups_with_stats`(self, ...)	Filter row groups using column chunk statistics.
`from_metadata`(HybridScanMetadata metadata)	Create a HybridScanReader that shares pre-parsed metadata.
`from_parquet_metadata`(FileMetaData metadata, ...)	Create a HybridScanReader from pre-populated metadata.
`has_next_table_chunk`(self)	Check if there is any parquet data left to read.
`materialize_all_columns`(self, ...)	Materialize all columns.
`materialize_filter_columns`(self, ...)	Materialize filter columns and update the row mask.
`materialize_filter_columns_chunk`(self, ...)	Materialize a chunk of filter columns.
`materialize_payload_columns`(self, ...)	Materialize payload columns and apply the row mask.
`materialize_payload_columns_chunk`(self, ...)	Materialize a chunk of payload columns.
`page_index_byte_range`(self)	Get the byte range of the page index.
`parquet_metadata`(self)	Get the Parquet file footer metadata.
`payload_column_chunks_byte_ranges`(self, ...)	Get byte ranges of column chunks of payload columns.
`payload_pages_byte_ranges`(self, ...)	Get byte ranges of pages of payload columns.
`reset_column_selection`(self)	Reset the column selection state.
`setup_chunking_for_filter_columns`(self, ...)	Setup chunking information for filter columns.
`setup_chunking_for_payload_columns`(self, ...)	Setup chunking information for payload columns.
`setup_chunking_for_payload_pages`(self, ...)	Setup chunking information for payload columns read page by page.
`setup_page_index`(self, const uint8_t[)	Setup the page index within the Parquet file metadata.
`total_rows_in_row_groups`(self, ...)	Get the total number of top-level rows in the row groups.

Examples

>>> import pylibcudf as plc
>>> # Create reader from footer bytes
>>> reader = plc.io.hybrid_scan.HybridScanReader(footer_bytes, options)
>>> # Get metadata
>>> metadata = reader.parquet_metadata()
>>> # Get all row groups
>>> row_groups = reader.all_row_groups(options)

all_column_chunks_byte_ranges(self, list row_group_indices: list[int], ParquetReaderOptions options) → list[ByteRangeInfo]#

Get byte ranges of column chunks of all columns.

Parameters:

row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options

Returns:

list[ByteRangeInfo]: Byte ranges to column chunks of all columns

all_row_groups( self, ParquetReaderOptions options, ) → list[int]#

Get all available row groups from the parquet file.

Parameters:

optionsParquetReaderOptions: Parquet reader options

Returns:

list[int]: List of row group indices

bloom_filters_byte_ranges(self, list row_group_indices: list[int], ParquetReaderOptions options) → list[ByteRangeInfo]#

Get byte ranges of bloom filters for row group pruning.

Parameters:

row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options

Returns:

list[ByteRangeInfo]: Byte ranges to column chunk bloom filters subject to the filter predicate

Notes

Device buffers for bloom filter byte ranges must be allocated using a 32 byte aligned memory resource.

build_all_true_row_mask( self, list row_group_indices, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None, ) → Column#

Build an all-true boolean survival column for the given row groups.

Parameters:

row_group_indiceslist[int]: Input row group indices
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

Returns:

Column: All-true boolean column with one entry per row across all row groups

build_row_mask_with_page_index_stats(self, list row_group_indices: list[int], ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None) → Column#

Build a boolean column indicating surviving rows from page stats.

Parameters:

row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

Returns:

Column: Boolean column indicating surviving rows

construct_row_group_passes(self, read_columns_mode columns_mode, list row_group_indices: list[int], size_t pass_read_limit, ParquetReaderOptions options) → list[list[int]]#

Partition row groups into passes such that the GPU memory required to materialize a pass for selected columns is bounded by the specified limit.

Note that pass_read_limit is a hint, not an absolute limit. i.e. if a row group cannot fit within the limit, it will still constitute a valid pass.

Parameters:

columns_modeReadColumnsMode: Columns selection to use for pass memory estimation
row_group_indiceslist[int]: Input row group indices
pass_read_limitint: Limit on the amount of memory used for reading and decompressing data, or 0 if there is no limit.
optionsParquetReaderOptions: Parquet reader options used to select columns.

Returns:

list[list[int]]: Lists of row group indices, one per pass.

Raises:

ValueError: If row_group_indices is empty.

dictionary_pages_byte_ranges(self, list row_group_indices: list[int], ParquetReaderOptions options) → list[ByteRangeInfo]#

Get byte ranges of column chunk dictionary pages for row group pruning.

Parameters:

row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options

Returns:

list[ByteRangeInfo]: Byte ranges to column chunk dictionary pages subject to the filter predicate

filter_column_chunks_byte_ranges(self, list row_group_indices: list[int], ParquetReaderOptions options) → list[ByteRangeInfo]#

Get byte ranges of column chunks of filter columns.

Parameters:

row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options

Returns:

list[ByteRangeInfo]: Byte ranges to column chunks of filter columns

filter_row_groups_with_bloom_filters(self, list bloom_filter_data, list row_group_indices: list[int], ParquetReaderOptions options, stream: CudaStreamLike | None = None) → list[int]#

Filter row groups using column chunk bloom filters.

Parameters:

bloom_filter_dataSequence: Span-like objects containing bloom filter data
row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream

Returns:

list[int]: Filtered row group indices

filter_row_groups_with_dictionary_pages(self, list dictionary_page_data, list row_group_indices: list[int], ParquetReaderOptions options, stream: CudaStreamLike | None = None) → list[int]#

Filter row groups using column chunk dictionary pages.

Parameters:

dictionary_page_dataSequence: Span-like objects containing dictionary page data
row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream

Returns:

list[int]: Filtered row group indices

filter_row_groups_with_stats(self, list row_group_indices: list[int], ParquetReaderOptions options, stream: CudaStreamLike | None = None) → list[int]#

Filter row groups using column chunk statistics.

Parameters:

row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream

Returns:

list[int]: Filtered row group indices

static from_metadata(HybridScanMetadata metadata)#

Create a HybridScanReader that shares pre-parsed metadata.

Constructs a lightweight reader that borrows metadata instead of re-parsing and copying the file metadata. Use one shared HybridScanMetadata to read row-group ranges of a single file. Overlapping row-group ranges across readers produce duplicate rows.

Parameters:

metadataHybridScanMetadata: Shared, pre-parsed Parquet file metadata

Returns:

HybridScanReader

static from_parquet_metadata( FileMetaData metadata, ParquetReaderOptions options, ) → HybridScanReader#

Create a HybridScanReader from pre-populated metadata.

Parameters:

metadataFileMetaData: Pre-populated Parquet file metadata
optionsParquetReaderOptions: Parquet reader options

Returns:

HybridScanReader

has_next_table_chunk(self) → bool#

Check if there is any parquet data left to read.

Returns:

bool: True if there is data left to read

materialize_all_columns(self, list row_group_indices: list[int], list column_chunk_data, ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None) → TableWithMetadata#

Materialize all columns.

Parameters:

row_group_indiceslist[int]: Input row group indices
column_chunk_dataSequence: Span-like objects containing column chunk data of all columns
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

Returns:

TableWithMetadata: Table of materialized all columns and metadata

materialize_filter_columns(self, list row_group_indices: list[int], list column_chunk_data, Column row_mask, use_data_page_mask mask_data_pages, ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None) → TableWithMetadata#

Materialize filter columns and update the row mask.

Parameters:

row_group_indiceslist[int]: Input row group indices
column_chunk_dataSequence: Span-like objects containing column chunk data of filter columns
row_maskColumn: Mutable boolean column indicating surviving rows
mask_data_pagesUseDataPageMask: Whether to use a data page mask
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

Returns:

TableWithMetadata: Table of materialized filter columns and metadata

materialize_filter_columns_chunk( self, Column row_mask, ) → TableWithMetadata#

Materialize a chunk of filter columns.

Parameters:

row_maskColumn: Mutable boolean column indicating surviving rows
Returns
——-
TableWithMetadata: Table chunk of materialized filter columns and metadata

materialize_payload_columns(self, list row_group_indices: list[int], list column_chunk_data, Column row_mask, use_data_page_mask mask_data_pages, ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None) → TableWithMetadata#

Materialize payload columns and apply the row mask.

Parameters:

row_group_indiceslist[int]: Input row group indices
column_chunk_dataSequence: Span-like objects containing column chunk data of payload columns
row_maskColumn: Boolean column indicating surviving rows
mask_data_pagesUseDataPageMask: Whether to use a data page mask
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

Returns:

TableWithMetadata: Table of materialized payload columns and metadata

materialize_payload_columns_chunk( self, Column row_mask, ) → TableWithMetadata#

Materialize a chunk of payload columns.

Parameters:

row_maskColumn: Boolean column indicating surviving rows
Returns
——-
TableWithMetadata: Table chunk of materialized payload columns and metadata

page_index_byte_range(self) → ByteRangeInfo#

Get the byte range of the page index.

Returns:

ByteRangeInfo: Byte range of the page index

parquet_metadata(self) → FileMetaData#

Get the Parquet file footer metadata.

Returns:

FileMetaData: Parquet file footer metadata

payload_column_chunks_byte_ranges(self, list row_group_indices: list[int], ParquetReaderOptions options) → list[ByteRangeInfo]#

Get byte ranges of column chunks of payload columns.

Parameters:

row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options

Returns:

list[ByteRangeInfo]: Byte ranges to column chunks of payload columns

payload_pages_byte_ranges(self, list row_group_indices: list[int], Column row_mask, ParquetReaderOptions options, stream: CudaStreamLike | None = None) → list[ByteRangeInfo]#

Get byte ranges of pages of payload columns.

Byte ranges are ordered by row group, column chunk, and page. Dictionary pages precede data pages within each column chunk, and pruned pages are returned as empty byte ranges.

Parameters:

row_group_indiceslist[int]: Input row group indices
row_maskColumn: Boolean column indicating which rows need to be read
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream

Returns:

list[ByteRangeInfo]: Byte ranges to pages of payload columns

reset_column_selection(self) → None#

Reset the column selection state.

Resets the internal column selection state forcing re-selection of columns in subsequent filter and read operations

setup_chunking_for_filter_columns(self, size_t chunk_read_limit, size_t pass_read_limit, list row_group_indices: list[int], Column row_mask, use_data_page_mask mask_data_pages, column_chunk_data: Sequence, ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None) → None#

Setup chunking information for filter columns.

Parameters:

chunk_read_limitint: Limit on bytes returned per chunk (0 for no limit)
pass_read_limitint: Limit on memory for reading/decompressing (0 for no limit)
row_group_indiceslist[int]: Input row group indices
row_maskColumn: Boolean column indicating surviving rows
mask_data_pagesUseDataPageMask: Whether to use a data page mask
column_chunk_dataSequence: Span-like objects containing column chunk data of filter columns
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

setup_chunking_for_payload_columns(self, size_t chunk_read_limit, size_t pass_read_limit, list row_group_indices: list[int], Column row_mask, use_data_page_mask mask_data_pages, column_chunk_data: Sequence, ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None) → None#

Setup chunking information for payload columns.

Parameters:

chunk_read_limitint: Limit on bytes returned per chunk (0 for no limit)
pass_read_limitint: Limit on memory for reading/decompressing (0 for no limit)
row_group_indiceslist[int]: Input row group indices
row_maskColumn: Boolean column indicating surviving rows
mask_data_pagesUseDataPageMask: Whether to use a data page mask
column_chunk_dataSequence: Span-like objects containing column chunk data of payload columns
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

setup_chunking_for_payload_pages(self, size_t chunk_read_limit, size_t pass_read_limit, list row_group_indices: list[int], Column row_mask, page_data: Sequence, ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None) → None#

Setup chunking information for payload columns read page by page.

The data page mask is inferred from page_data, which must be in the same order as the byte ranges returned by payload_pages_byte_ranges(). Table chunks are read with materialize_payload_columns_chunk().

Parameters:

chunk_read_limitint: Limit on bytes returned per chunk (0 for no limit)
pass_read_limitint: Limit on memory for reading/decompressing (0 for no limit)
row_group_indiceslist[int]: Input row group indices
row_maskColumn: Boolean column indicating which rows need to be read
page_dataSequence: Span-like objects containing the page data of payload columns, in the same order as the byte ranges returned by payload_pages_byte_ranges(). None indicates a pruned page
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

setup_page_index( self, const uint8_t[::1] page_index_bytes: Buffer, ) → None#

Setup the page index within the Parquet file metadata.

Parameters:

page_index_bytesBuffer: Parquet page index buffer bytes

total_rows_in_row_groups(self, list row_group_indices: list[int]) → int#

Get the total number of top-level rows in the row groups.

Parameters:

row_group_indiceslist[int]: Input row group indices

Returns:

int: Total number of top-level rows

pylibcudf.io.experimental.ReadColumnsMode#

`from_lists`(row_group_indices)	Create RowGroupIndices from per-source row group indices.
`tolist`(self)	Return the indices as a `list[list[int]]`.