Experimental#

APIs in this namespace are experimental and may change without warning in the future.

class pylibcudf.io.experimental.FileMetaData#

Parquet file footer metadata.

For details, see cudf::io::parquet::FileMetaData

Attributes

`columnchunk_metadata`	FileMetaData.columnchunk_metadata: dict[str, list[int]]
`created_by`	FileMetaData.created_by: str
`num_rows`	FileMetaData.num_rows: int
`row_group_num_rows`	FileMetaData.row_group_num_rows: list[int]
`row_groups`	FileMetaData.row_groups: list[RowGroup]
`schema`	FileMetaData.schema: list[SchemaElement]
`version`	FileMetaData.version: int

Methods

from_bytes(cls, const uint8_t[)

Build FileMetaData from parquet footer bytes.

See also

pylibcudf.io.parquet_metadata.read_parquet_footers: Read one FileMetaData per source directly from pylibcudf.io.types.SourceInfo.

columnchunk_metadata#

FileMetaData.columnchunk_metadata: dict[str, list[int]]

Get a map of dotted column paths to lists of total_uncompressed_size values from every column chunk in this file.

Returns:

dict[str, list[int]]: Map of dotted column paths (".".join(path_in_schema)) to lists of total_uncompressed_size metadata from all their column chunks.

Notes

Equivalent to, but faster than, walking each row group’s columns:

>>> result: dict[str, list[int]] = {}
>>> for rg in file_metadata.row_groups:
...     for col in rg.columns:
...         name = ".".join(col.meta_data.path_in_schema)
...         result.setdefault(name, []).append(
...             col.meta_data.total_uncompressed_size
...         )

created_by#

FileMetaData.created_by: str

Get the application that created the file.

classmethod from_bytes( cls, const uint8_t[::1] footer_bytes: Buffer, ) → FileMetaData#

Build FileMetaData from parquet footer bytes.

Parameters:

footer_bytesBuffer: A contiguous bytes-like object containing parquet footer bytes. The bytes are forwarded as-is to cudf::io::parquet::experimental::hybrid_scan_reader without Python-side preprocessing. This method does not strip the parquet footer suffix (4-byte footer length + PAR1 magic), so callers should generally pass only the footer region bytes.

Returns:

FileMetaData: Parsed parquet file footer metadata.

num_rows#

FileMetaData.num_rows: int

Get the total number of rows.

row_group_num_rows#

FileMetaData.row_group_num_rows: list[int]

Get row counts for each row group in this file.

Returns:

list: A list with the row count per row group in this file.

Notes

Equivalent to, but faster than, checking each row groups’ num_rows:

>>> [rg.num_rows for rg in file_metadata.row_groups]

row_groups#

FileMetaData.row_groups: list[RowGroup]

Get row group metadata in this file.

schema#

FileMetaData.schema: list[SchemaElement]

Get the file’s schema tree, flattened by a depth-first traversal.

The first element is the root. Each element’s num_children gives the number of elements that follow it at the next level, which is what allows the tree to be reconstructed from the flat list.

Returns:

list[SchemaElement]: One entry per schema element, in depth-first order.

version#

FileMetaData.version: int

Get the file format version.

class pylibcudf.io.experimental.HybridScanMetadata#

Shareable, pre-parsed Parquet file metadata for the hybrid scan reader.

This class enables parsing the metadata of a Parquet file once, then constructing multiple HybridScanReader instances that share it (one per row-group range of the file) instead of each re-parsing and copying the metadata.

For details, see cudf::io::parquet::experimental::hybrid_scan_metadata

Methods

`from_footer_bytes`(const uint8_t[, ...)	Parse shareable metadata from Parquet footer bytes.
`from_parquet_metadata`(FileMetaData metadata, ...)	Build shareable metadata from a pre-populated `FileMetaData`.

Examples

>>> import pylibcudf as plc
>>> metadata = plc.io.experimental.HybridScanMetadata.from_parquet_metadata(
...     file_metadata, options)
>>> reader = plc.io.experimental.HybridScanReader.from_metadata(metadata)

static from_footer_bytes( const uint8_t[::1] footer_bytes, ParquetReaderOptions options, )#

Parse shareable metadata from Parquet footer bytes.

Parameters:

footer_bytesBuffer: Parquet file footer bytes
optionsParquetReaderOptions: Parquet reader options

Returns:

HybridScanMetadata

static from_parquet_metadata( FileMetaData metadata, ParquetReaderOptions options, )#

Build shareable metadata from a pre-populated FileMetaData.

Parameters:

metadataFileMetaData: Pre-populated Parquet file metadata
optionsParquetReaderOptions: Parquet reader options

Returns:

HybridScanMetadata

class pylibcudf.io.experimental.HybridScanMultiFile#

Experimental multi-source Parquet reader for highly selective filters.

Vectorizes the HybridScanReader APIs over multiple Parquet sources. Inputs and outputs are indexed by source order, except for the row mask which is a single boolean column spanning all rows of all sources concatenated in source order, then in row-group order within a source.

For details, see cudf::io::parquet::experimental::hybrid_scan_multifile

Methods

`all_column_chunks_byte_ranges`(self, ...)	Get byte ranges of column chunks of all columns.
`all_row_groups`(self, ...)	Get all available per-source row group indices.
`bloom_filters_byte_ranges`(self, ...)	Get byte ranges of bloom filters for row group pruning.
`build_all_true_row_mask`(self, ...)	Build an all-true boolean column spanning all selected rows.
`build_row_mask_with_page_index_stats`(self, ...)	Build a boolean column indicating surviving rows from page stats.
`construct_row_group_passes`(self, ...)	Partition row groups into passes such that the GPU memory required to materialize a pass for selected columns is bounded by the specified limit.
`dictionary_pages_byte_ranges`(self, ...)	Get byte ranges of column chunk dictionary pages for row group pruning.
`filter_row_groups_with_byte_range`(self, ...)	Filter row groups using the byte range from the options.
`filter_row_groups_with_stats`(self, ...)	Filter row groups using column chunk statistics.
`from_parquet_metadatas`(parquet_metadatas, ...)	Create a HybridScanMultiFile from pre-populated metadata.
`has_next_table_chunk`(self)	Check if there is any parquet data left to read.
`materialize_all_columns`(self, ...)	Materialize all columns.
`materialize_payload_columns_chunk`(self, ...)	Materialize a chunk of payload columns.
`page_index_byte_ranges`(self)	Get the byte range of the page index of all sources.
`parquet_metadatas`(self)	Get the Parquet file footer metadata of all sources.
`payload_pages_byte_ranges`(self, ...)	Get byte ranges of the pages of payload columns.
`reset_column_selection`(self)	Reset the column selection state.
`setup_chunking_for_payload_columns`(self, ...)	Setup chunking information for payload columns read page by page.
`setup_page_indexes`(self, page_index_bytes)	Setup the page index within the Parquet file metadata of all sources.
`total_rows_in_row_groups`(self, ...)	Get the total number of top-level rows in the row groups.

Examples

>>> import pylibcudf as plc
>>> reader = plc.io.experimental.HybridScanMultiFile.from_parquet_metadatas(
...     metadatas, options)
>>> row_groups = reader.all_row_groups(options)

all_column_chunks_byte_ranges( self, RowGroupIndices row_group_indices, ParquetReaderOptions options, ) → tuple[list[ByteRangeInfo], list[int]]#

Get byte ranges of column chunks of all columns.

Parameters:

row_group_indicesRowGroupIndices: Input row group indices, one list per source
optionsParquetReaderOptions: Parquet reader options

Returns:

tuple[list[ByteRangeInfo], list[int]]: Flattened byte ranges to column chunks of all columns and their corresponding source indices

all_row_groups( self, ParquetReaderOptions options, ) → RowGroupIndices#

Get all available per-source row group indices.

Parameters:

optionsParquetReaderOptions: Parquet reader options

Returns:

RowGroupIndices: Row group indices, one inner list per source

bloom_filters_byte_ranges( self, RowGroupIndices row_group_indices, ParquetReaderOptions options, ) → tuple[list[ByteRangeInfo], list[int]]#

Get byte ranges of bloom filters for row group pruning.

Parameters:

row_group_indicesRowGroupIndices: Input row group indices, one list per source
optionsParquetReaderOptions: Parquet reader options

Returns:

tuple[list[ByteRangeInfo], list[int]]: Flattened bloom filter byte ranges and their corresponding source indices

build_all_true_row_mask( self, RowGroupIndices row_group_indices, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None, ) → Column#

Build an all-true boolean column spanning all selected rows.

Parameters:

row_group_indicesRowGroupIndices: Input row group indices, one list per source
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

Returns:

Column: An all-true boolean column spanning all selected rows across all sources

build_row_mask_with_page_index_stats( self, RowGroupIndices row_group_indices, ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None, ) → Column#

Build a boolean column indicating surviving rows from page stats.

Parameters:

row_group_indicesRowGroupIndices: Input row group indices, one list per source
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

Returns:

Column: Boolean column indicating surviving rows across all sources

construct_row_group_passes( self, read_columns_mode columns_mode, RowGroupIndices row_group_indices, size_t pass_read_limit, ParquetReaderOptions options, ) → list[RowGroupIndices]#

Partition row groups into passes such that the GPU memory required to materialize a pass for selected columns is bounded by the specified limit.

Note that pass_read_limit is a hint, not an absolute limit. i.e. if a row group cannot fit within the limit, it will still constitute a valid pass.

Parameters:

columns_modeReadColumnsMode: Columns selection to use for pass memory estimation
row_group_indicesRowGroupIndices: Input row group indices, one list per source.
pass_read_limitint: Limit on the amount of memory used for reading and decompressing data, or 0 if there is no limit.
optionsParquetReaderOptions: Parquet reader options used to select columns.

Returns:

list[RowGroupIndices]: Per-source row group indices, one list per pass.

Raises:

ValueError: If row_group_indices is empty.

dictionary_pages_byte_ranges( self, RowGroupIndices row_group_indices, ParquetReaderOptions options, ) → tuple[list[ByteRangeInfo], list[int]]#

Get byte ranges of column chunk dictionary pages for row group pruning.

Parameters:

row_group_indicesRowGroupIndices: Input row group indices, one list per source
optionsParquetReaderOptions: Parquet reader options

Returns:

tuple[list[ByteRangeInfo], list[int]]: Flattened dictionary page byte ranges and their corresponding source indices

filter_row_groups_with_byte_range( self, RowGroupIndices row_group_indices, ParquetReaderOptions options, ) → RowGroupIndices#

Filter row groups using the byte range from the options.

Parameters:

row_group_indicesRowGroupIndices: Input row group indices, one list per source
optionsParquetReaderOptions: Parquet reader options

Returns:

RowGroupIndices: Filtered row group indices, one inner list per source

filter_row_groups_with_stats( self, RowGroupIndices row_group_indices, ParquetReaderOptions options, stream: CudaStreamLike | None = None, ) → RowGroupIndices#

Filter row groups using column chunk statistics.

Parameters:

row_group_indicesRowGroupIndices: Input row group indices, one list per source
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream

Returns:

RowGroupIndices: Filtered row group indices, one inner list per source

static from_parquet_metadatas(parquet_metadatas: Sequence[FileMetaData], ParquetReaderOptions options) → HybridScanMultiFile#

Create a HybridScanMultiFile from pre-populated metadata.

Parameters:

parquet_metadatasSequence[FileMetaData]: Pre-populated Parquet file metadata, one per source
optionsParquetReaderOptions: Parquet reader options

Returns:

HybridScanMultiFile

has_next_table_chunk(self) → bool#

Check if there is any parquet data left to read.

Returns:

bool: True if there is data left to read

materialize_all_columns(self, RowGroupIndices row_group_indices, list column_chunk_data: list[Span], ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None) → TableWithMetadata#

Materialize all columns.

Parameters:

row_group_indicesRowGroupIndices: Input row group indices, one list per source
column_chunk_datalist[Span]: Span-like objects containing flattened column chunk data of all columns, in the same order as all_column_chunks_byte_ranges
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

Returns:

TableWithMetadata: Table of materialized all columns and metadata

materialize_payload_columns_chunk( self, Column row_mask, ) → TableWithMetadata#

Materialize a chunk of payload columns.

Parameters:

row_maskColumn: Boolean column indicating which rows need to be read

Returns:

TableWithMetadata: Table chunk of materialized payload columns and metadata

page_index_byte_ranges( self, ) → list[ByteRangeInfo]#

Get the byte range of the page index of all sources.

Returns:

list[ByteRangeInfo]: Byte range of the page index, one per source

parquet_metadatas(self) → list[FileMetaData]#

Get the Parquet file footer metadata of all sources.

Returns:

list[FileMetaData]: Parquet file footer metadata, one per source

payload_pages_byte_ranges( self, RowGroupIndices row_group_indices, Column row_mask, ParquetReaderOptions options, stream: CudaStreamLike | None = None, ) → tuple[list[ByteRangeInfo], list[int]]#

Get byte ranges of the pages of payload columns.

Byte ranges are flattened in source, row group, column chunk, and page order. Dictionary pages precede data pages within a column chunk, and pruned pages are returned as empty byte ranges.

Parameters:

row_group_indicesRowGroupIndices: Input row group indices, one list per source
row_maskColumn: Boolean column indicating which rows need to be read
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream

Returns:

tuple[list[ByteRangeInfo], list[int]]: Flattened byte ranges to the pages of payload columns and the source index of each byte range

reset_column_selection(self) → None#

Reset the column selection state.

Resets the internal column selection state forcing re-selection of columns in subsequent filter and read operations

setup_chunking_for_payload_columns( self, size_t chunk_read_limit, size_t pass_read_limit, RowGroupIndices row_group_indices, Column row_mask, page_data: Sequence, ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None, ) → None#

Setup chunking information for payload columns read page by page.

The data page mask is inferred from page_data, which must have the same shape as the byte ranges returned by payload_pages_byte_ranges().

Parameters:

chunk_read_limitint: Limit on bytes returned per chunk (0 for no limit)
pass_read_limitint: Limit on memory for reading/decompressing (0 for no limit)
row_group_indicesRowGroupIndices: Input row group indices, one list per source
row_maskColumn: Boolean column indicating which rows need to be read
page_dataSequence: Span-like objects containing the page data of payload columns, in the same order as the byte ranges returned by payload_pages_byte_ranges(). None indicates a pruned page
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

setup_page_indexes( self, page_index_bytes: Sequence[Buffer], ) → None#

Setup the page index within the Parquet file metadata of all sources.

Parameters:

page_index_bytesSequence[Buffer]: Parquet page index buffer bytes, one per source

total_rows_in_row_groups( self, RowGroupIndices row_group_indices, ) → int#

Get the total number of top-level rows in the row groups.

Parameters:

row_group_indicesRowGroupIndices: Input row group indices, one list per source

Returns:

int: Total number of top-level rows across all sources

class pylibcudf.io.experimental.HybridScanReader( const uint8_t[::1] footer_bytes: Buffer, ParquetReaderOptions options, )#

Experimental Parquet reader optimized for highly selective filters.

This class implements a hybrid scan operation for reading Parquet files with highly selective filters. It reads in two passes: first reading filter columns to build a row mask, then reading payload columns using that mask for optimization.

For details, see cudf::io::parquet::experimental::hybrid_scan_reader

Parameters:

footer_bytesBuffer: Parquet file footer bytes
optionsParquetReaderOptions: Parquet reader options

Methods

`all_column_chunks_byte_ranges`(self, ...)	Get byte ranges of column chunks of all columns.
`all_row_groups`(self, ...)	Get all available row groups from the parquet file.
`bloom_filters_byte_ranges`(self, ...)	Get byte ranges of bloom filters for row group pruning.
`build_all_true_row_mask`(self, ...)	Build an all-true boolean survival column for the given row groups.
`build_row_mask_with_page_index_stats`(self, ...)	Build a boolean column indicating surviving rows from page stats.
`construct_row_group_passes`(self, ...)	Partition row groups into passes such that the GPU memory required to materialize a pass for selected columns is bounded by the specified limit.
`dictionary_pages_byte_ranges`(self, ...)	Get byte ranges of column chunk dictionary pages for row group pruning.
`filter_column_chunks_byte_ranges`(self, ...)	Get byte ranges of column chunks of filter columns.
`filter_row_groups_with_bloom_filters`(self, ...)	Filter row groups using column chunk bloom filters.
`filter_row_groups_with_dictionary_pages`(...)	Filter row groups using column chunk dictionary pages.
`filter_row_groups_with_stats`(self, ...)	Filter row groups using column chunk statistics.
`from_metadata`(HybridScanMetadata metadata)	Create a HybridScanReader that shares pre-parsed metadata.
`from_parquet_metadata`(FileMetaData metadata, ...)	Create a HybridScanReader from pre-populated metadata.
`has_next_table_chunk`(self)	Check if there is any parquet data left to read.
`materialize_all_columns`(self, ...)	Materialize all columns.
`materialize_filter_columns`(self, ...)	Materialize filter columns and update the row mask.
`materialize_filter_columns_chunk`(self, ...)	Materialize a chunk of filter columns.
`materialize_payload_columns`(self, ...)	Materialize payload columns and apply the row mask.
`materialize_payload_columns_chunk`(self, ...)	Materialize a chunk of payload columns.
`page_index_byte_range`(self)	Get the byte range of the page index.
`parquet_metadata`(self)	Get the Parquet file footer metadata.
`payload_column_chunks_byte_ranges`(self, ...)	Get byte ranges of column chunks of payload columns.
`reset_column_selection`(self)	Reset the column selection state.
`setup_chunking_for_filter_columns`(self, ...)	Setup chunking information for filter columns.
`setup_chunking_for_payload_columns`(self, ...)	Setup chunking information for payload columns.
`setup_page_index`(self, const uint8_t[)	Setup the page index within the Parquet file metadata.
`total_rows_in_row_groups`(self, ...)	Get the total number of top-level rows in the row groups.

Examples

>>> import pylibcudf as plc
>>> # Create reader from footer bytes
>>> reader = plc.io.hybrid_scan.HybridScanReader(footer_bytes, options)
>>> # Get metadata
>>> metadata = reader.parquet_metadata()
>>> # Get all row groups
>>> row_groups = reader.all_row_groups(options)

all_column_chunks_byte_ranges(self, list row_group_indices: list[int], ParquetReaderOptions options) → list[ByteRangeInfo]#

Get byte ranges of column chunks of all columns.

Parameters:

row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options

Returns:

list[ByteRangeInfo]: Byte ranges to column chunks of all columns

all_row_groups( self, ParquetReaderOptions options, ) → list[int]#

Get all available row groups from the parquet file.

Parameters:

optionsParquetReaderOptions: Parquet reader options

Returns:

list[int]: List of row group indices

bloom_filters_byte_ranges(self, list row_group_indices: list[int], ParquetReaderOptions options) → list[ByteRangeInfo]#

Get byte ranges of bloom filters for row group pruning.

Parameters:

row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options

Returns:

list[ByteRangeInfo]: Byte ranges to column chunk bloom filters subject to the filter predicate

Notes

Device buffers for bloom filter byte ranges must be allocated using a 32 byte aligned memory resource.

build_all_true_row_mask( self, list row_group_indices, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None, ) → Column#

Build an all-true boolean survival column for the given row groups.

Parameters:

row_group_indiceslist[int]: Input row group indices
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

Returns:

Column: All-true boolean column with one entry per row across all row groups

build_row_mask_with_page_index_stats(self, list row_group_indices: list[int], ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None) → Column#

Build a boolean column indicating surviving rows from page stats.

Parameters:

row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

Returns:

Column: Boolean column indicating surviving rows

construct_row_group_passes(self, read_columns_mode columns_mode, list row_group_indices: list[int], size_t pass_read_limit, ParquetReaderOptions options) → list[list[int]]#

Partition row groups into passes such that the GPU memory required to materialize a pass for selected columns is bounded by the specified limit.

Note that pass_read_limit is a hint, not an absolute limit. i.e. if a row group cannot fit within the limit, it will still constitute a valid pass.

Parameters:

columns_modeReadColumnsMode: Columns selection to use for pass memory estimation
row_group_indiceslist[int]: Input row group indices
pass_read_limitint: Limit on the amount of memory used for reading and decompressing data, or 0 if there is no limit.
optionsParquetReaderOptions: Parquet reader options used to select columns.

Returns:

list[list[int]]: Lists of row group indices, one per pass.

Raises:

ValueError: If row_group_indices is empty.

dictionary_pages_byte_ranges(self, list row_group_indices: list[int], ParquetReaderOptions options) → list[ByteRangeInfo]#

Get byte ranges of column chunk dictionary pages for row group pruning.

Parameters:

row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options

Returns:

list[ByteRangeInfo]: Byte ranges to column chunk dictionary pages subject to the filter predicate

filter_column_chunks_byte_ranges(self, list row_group_indices: list[int], ParquetReaderOptions options) → list[ByteRangeInfo]#

Get byte ranges of column chunks of filter columns.

Parameters:

row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options

Returns:

list[ByteRangeInfo]: Byte ranges to column chunks of filter columns

filter_row_groups_with_bloom_filters(self, list bloom_filter_data, list row_group_indices: list[int], ParquetReaderOptions options, stream: CudaStreamLike | None = None) → list[int]#

Filter row groups using column chunk bloom filters.

Parameters:

bloom_filter_dataSequence: Span-like objects containing bloom filter data
row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream

Returns:

list[int]: Filtered row group indices

filter_row_groups_with_dictionary_pages(self, list dictionary_page_data, list row_group_indices: list[int], ParquetReaderOptions options, stream: CudaStreamLike | None = None) → list[int]#

Filter row groups using column chunk dictionary pages.

Parameters:

dictionary_page_dataSequence: Span-like objects containing dictionary page data
row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream

Returns:

list[int]: Filtered row group indices

filter_row_groups_with_stats(self, list row_group_indices: list[int], ParquetReaderOptions options, stream: CudaStreamLike | None = None) → list[int]#

Filter row groups using column chunk statistics.

Parameters:

row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream

Returns:

list[int]: Filtered row group indices

static from_metadata(HybridScanMetadata metadata)#

Create a HybridScanReader that shares pre-parsed metadata.

Constructs a lightweight reader that borrows metadata instead of re-parsing and copying the file metadata. Use one shared HybridScanMetadata to read row-group ranges of a single file. Overlapping row-group ranges across readers produce duplicate rows.

Parameters:

metadataHybridScanMetadata: Shared, pre-parsed Parquet file metadata

Returns:

HybridScanReader

static from_parquet_metadata( FileMetaData metadata, ParquetReaderOptions options, ) → HybridScanReader#

Create a HybridScanReader from pre-populated metadata.

Parameters:

metadataFileMetaData: Pre-populated Parquet file metadata
optionsParquetReaderOptions: Parquet reader options

Returns:

HybridScanReader

has_next_table_chunk(self) → bool#

Check if there is any parquet data left to read.

Returns:

bool: True if there is data left to read

materialize_all_columns(self, list row_group_indices: list[int], list column_chunk_data, ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None) → TableWithMetadata#

Materialize all columns.

Parameters:

row_group_indiceslist[int]: Input row group indices
column_chunk_dataSequence: Span-like objects containing column chunk data of all columns
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

Returns:

TableWithMetadata: Table of materialized all columns and metadata

materialize_filter_columns(self, list row_group_indices: list[int], list column_chunk_data, Column row_mask, use_data_page_mask mask_data_pages, ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None) → TableWithMetadata#

Materialize filter columns and update the row mask.

Parameters:

row_group_indiceslist[int]: Input row group indices
column_chunk_dataSequence: Span-like objects containing column chunk data of filter columns
row_maskColumn: Mutable boolean column indicating surviving rows
mask_data_pagesUseDataPageMask: Whether to use a data page mask
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

Returns:

TableWithMetadata: Table of materialized filter columns and metadata

materialize_filter_columns_chunk( self, Column row_mask, ) → TableWithMetadata#

Materialize a chunk of filter columns.

Parameters:

row_maskColumn: Mutable boolean column indicating surviving rows
Returns
——-
TableWithMetadata: Table chunk of materialized filter columns and metadata

materialize_payload_columns(self, list row_group_indices: list[int], list column_chunk_data, Column row_mask, use_data_page_mask mask_data_pages, ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None) → TableWithMetadata#

Materialize payload columns and apply the row mask.

Parameters:

row_group_indiceslist[int]: Input row group indices
column_chunk_dataSequence: Span-like objects containing column chunk data of payload columns
row_maskColumn: Boolean column indicating surviving rows
mask_data_pagesUseDataPageMask: Whether to use a data page mask
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

Returns:

TableWithMetadata: Table of materialized payload columns and metadata

materialize_payload_columns_chunk( self, Column row_mask, ) → TableWithMetadata#

Materialize a chunk of payload columns.

Parameters:

row_maskColumn: Boolean column indicating surviving rows
Returns
——-
TableWithMetadata: Table chunk of materialized payload columns and metadata

page_index_byte_range(self) → ByteRangeInfo#

Get the byte range of the page index.

Returns:

ByteRangeInfo: Byte range of the page index

parquet_metadata(self) → FileMetaData#

Get the Parquet file footer metadata.

Returns:

FileMetaData: Parquet file footer metadata

payload_column_chunks_byte_ranges(self, list row_group_indices: list[int], ParquetReaderOptions options) → list[ByteRangeInfo]#

Get byte ranges of column chunks of payload columns.

Parameters:

row_group_indiceslist[int]: Input row group indices
optionsParquetReaderOptions: Parquet reader options

Returns:

list[ByteRangeInfo]: Byte ranges to column chunks of payload columns

reset_column_selection(self) → None#

Reset the column selection state.

Resets the internal column selection state forcing re-selection of columns in subsequent filter and read operations

setup_chunking_for_filter_columns(self, size_t chunk_read_limit, size_t pass_read_limit, list row_group_indices: list[int], Column row_mask, use_data_page_mask mask_data_pages, column_chunk_data: Sequence, ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None) → None#

Setup chunking information for filter columns.

Parameters:

chunk_read_limitint: Limit on bytes returned per chunk (0 for no limit)
pass_read_limitint: Limit on memory for reading/decompressing (0 for no limit)
row_group_indiceslist[int]: Input row group indices
row_maskColumn: Boolean column indicating surviving rows
mask_data_pagesUseDataPageMask: Whether to use a data page mask
column_chunk_dataSequence: Span-like objects containing column chunk data of filter columns
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

setup_chunking_for_payload_columns(self, size_t chunk_read_limit, size_t pass_read_limit, list row_group_indices: list[int], Column row_mask, use_data_page_mask mask_data_pages, column_chunk_data: Sequence, ParquetReaderOptions options, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None) → None#

Setup chunking information for payload columns.

Parameters:

chunk_read_limitint: Limit on bytes returned per chunk (0 for no limit)
pass_read_limitint: Limit on memory for reading/decompressing (0 for no limit)
row_group_indiceslist[int]: Input row group indices
row_maskColumn: Boolean column indicating surviving rows
mask_data_pagesUseDataPageMask: Whether to use a data page mask
column_chunk_dataSequence: Span-like objects containing column chunk data of payload columns
optionsParquetReaderOptions: Parquet reader options
streamStream, optional: CUDA stream
mrDeviceMemoryResource, optional: Device memory resource

setup_page_index( self, const uint8_t[::1] page_index_bytes: Buffer, ) → None#

Setup the page index within the Parquet file metadata.

Parameters:

page_index_bytesBuffer: Parquet page index buffer bytes

total_rows_in_row_groups(self, list row_group_indices: list[int]) → int#

Get the total number of top-level rows in the row groups.

Parameters:

row_group_indiceslist[int]: Input row group indices

Returns:

int: Total number of top-level rows

pylibcudf.io.experimental.ReadColumnsMode#

`from_lists`(row_group_indices)	Create RowGroupIndices from per-source row group indices.
`tolist`(self)	Return the indices as a `list[list[int]]`.