HybridScanReader#

public class HybridScanReader implements AutoCloseable #

Experimental Parquet hybrid-scan reader.

This class is the Java binding for cudf::io::parquet::experimental::hybrid_scan_reader. It is designed for highly selective filter expressions over Parquet files and exposes the multi-step pipeline that the C++ reader uses internally:

Read the Parquet footer (and optional page index) to drive row-group / page-level pruning.
Filter the row groups using statistics, bloom filters, and dictionary pages.
Materialize the filter columns; the reader builds an initial row mask (all true or seeded from page index stats when page-level pruning is enabled) and then mutates it down to only the rows that survive the AST filter.
Materialize the payload columns using the surviving row mask.

A two-step convenience flow is provided for selective filters (materializeFilterColumns(int[], DeviceMemoryBuffer[], boolean) then materializePayloadColumns(int[], DeviceMemoryBuffer[], ColumnVector, boolean)), and a one-shot materializeAllColumns(int[], DeviceMemoryBuffer[]) is provided for small files or broad filters.

Chunked / streaming materialization is also supported via the setupChunkingFor* + materialize*Chunk family of methods, mirroring the C++ chunked reader pipeline.

The filter and payload materialization paths accept a boolean that toggles page-level pruning: skips decode of pages the filter (or row mask) proves empty. Enable when the workload prunes many pages. Pruning requirements for the two column materializations differ:

With page-level pruning enabled, filter columns require setupPageIndex(HostMemoryBuffer) to seed the row mask from page-index statistics.
Once a row mask exists, filter and payload columns use page row boundaries from the OffsetIndex when available, otherwise from decoded page headers.

The reader is created with no filter expression installed. Filter-related APIs behave as though nothing has been filtered out unless a filter is first supplied via setFilter(CompiledExpression).

The APIs in this file are experimental and subject to change.

Constructor Summary

HybridScanReader(HostMemoryBuffer footerBuffer, ParquetOptions opts)

Create a hybrid scan reader from the bytes of a Parquet file footer.

Method Summary

`allColumnChunksByteRanges(int[] rowGroupIndices)`
`allRowGroups()`
`bloomFiltersByteRanges(int[] rowGroupIndices)`	Get the byte ranges in the source file that hold the column-chunk bloom filter data needed for the next round of row-group pruning.
`close()`
`constructRowGroupPasses(ReadColumnsMode columnsMode, int[] rowGroupIndices, long passReadLimit)`	Partition the supplied row groups into passes whose estimated uncompressed size over the selected columns is bounded by the given limit.
`dictionaryPageLengths(HostMemoryBuffer[] pageData)`	The length of the dictionary page at the front of each buffer, its page header included, or 0 for a buffer that does not begin with a whole dictionary page.
`dictionaryPagesByteRanges(int[] rowGroupIndices)`	Get the byte ranges in the source file that hold the column-chunk dictionary page data used for row-group pruning of (in)equality predicates.
`dictionaryPagesByteRangesIncludeUnbounded(int[] rowGroupIndices)`	Like dictionaryPagesByteRanges(int[]) but also returns a range for a column chunk whose footer does not pin its dictionary page down exactly.
`filterColumnChunksByteRanges(int[] rowGroupIndices)`
`filterRowGroupsWithDictionaryPages(DeviceMemoryBuffer[] dictionaryPageData, int[] rowGroupIndices)`	Filter row groups using column-chunk dictionary pages loaded into device memory.
`filterRowGroupsWithStats(int[] rowGroupIndices)`	Filter row groups using column-chunk statistics from the file footer.
`hasNextTableChunk()`
`materializeAllColumns(int[] rowGroupIndices, DeviceMemoryBuffer[] columnChunkData)`	Materialize all (or selected) columns in a single pass.
`materializeAllColumnsChunk()`
`materializeFilterColumns(int[] rowGroupIndices, DeviceMemoryBuffer[] columnChunkData, boolean usePageLevelPruning)`	Build the initial row mask, materialize the filter columns, and evaluate the compiled filter expression against them.
`materializeFilterColumnsChunk()`
`materializePayloadColumns(int[] rowGroupIndices, DeviceMemoryBuffer[] columnChunkData, ColumnVector rowMask, boolean usePageLevelPruning)`	Materialize only the payload columns, applying the supplied row mask to the output.
`materializePayloadColumnsChunk(ColumnVector rowMask)`
`pageIndexByteRange()`
`payloadColumnChunksByteRanges(int[] rowGroupIndices)`
`setFilter(CompiledExpression filter)`	Install or replace the filter expression used by all subsequent filter-related APIs on this reader; pass null to clear.
`setupChunkingForAllColumns(long chunkReadLimit, long passReadLimit, int[] rowGroupIndices, DeviceMemoryBuffer[] columnChunkData)`	Set up chunking state for all-column materialization (single-pass mode).
`setupChunkingForFilterColumns(long chunkReadLimit, long passReadLimit, int[] rowGroupIndices, boolean usePageLevelPruning, DeviceMemoryBuffer[] columnChunkData)`	Build the initial row mask and set up chunking state for filter-column materialization.
`setupChunkingForPayloadColumns(long chunkReadLimit, long passReadLimit, int[] rowGroupIndices, ColumnVector rowMask, boolean usePageLevelPruning, DeviceMemoryBuffer[] columnChunkData)`	Set up chunking state for payload-column materialization.
`setupPageIndex(HostMemoryBuffer pageIndexBuffer)`	Materialize the ColumnIndex / OffsetIndex structs (collectively, the page index) from the supplied bytes.
`takeFilterRowMask()`	Transfer ownership of the row mask produced by the chunked filter-column materialization to the caller.
`totalRowsInRowGroups(int[] rowGroupIndices)`

Constructors

public HybridScanReader( HostMemoryBuffer footerBuffer, ParquetOptions opts )#

Create a hybrid scan reader from the bytes of a Parquet file footer.

The footerBuffer can be obtained by reading the last few bytes of a Parquet file. See the hybrid_scan_io example for a helper that does exactly that.

The reader is created with no filter expression installed. Filter-related APIs (e.g. filterRowGroupsWithStats(int[]), materializeFilterColumns(int[], DeviceMemoryBuffer[], boolean)) behave as though nothing has been filtered out unless a filter is first supplied via setFilter(CompiledExpression).

Parameters:

footerBuffer – host-resident footer bytes (must remain valid until this constructor returns; the JNI reads the bytes synchronously)
opts – Parquet reader options. ParquetOptions.DEFAULT by default.

Methods

public void setFilter(CompiledExpression filter)#

Install or replace the filter expression used by all subsequent filter-related APIs on this reader; pass null to clear.

Cheap to call, so callers can sweep the same file with several candidate predicates in a loop.

The caller retains ownership of any previously installed CompiledExpression and must close it themselves — this method replaces this reader’s strong reference only.

Parameters:: filter – the new filter expression, or null to clear

public ByteRange pageIndexByteRange()#

Returns:: the byte range of the page index in the Parquet file.

public void setupPageIndex(HostMemoryBuffer pageIndexBuffer)#

Materialize the ColumnIndex / OffsetIndex structs (collectively, the page index) from the supplied bytes.

Required before any filter column materialization with usePageLevelPruning == true.

Parameters:: pageIndexBuffer – host-resident page index bytes

public int[] allRowGroups()#

Returns:: all row group indices in the Parquet file.

public long totalRowsInRowGroups(int[] rowGroupIndices)#

Returns:: the total number of top-level rows in the supplied row groups.

public int[] filterRowGroupsWithStats(int[] rowGroupIndices)#: Filter row groups using column-chunk statistics from the file footer.

public ByteRange[] bloomFiltersByteRanges(int[] rowGroupIndices)#

Get the byte ranges in the source file that hold the column-chunk bloom filter data needed for the next round of row-group pruning.

The ordering follows the C++ reader’s ordering and is meaningful: the i-th entry corresponds to the i-th column chunk needing a bloom filter. The result may be empty.

Device buffers for these ranges must be allocated using a 32-byte aligned memory resource.

public ByteRange[] dictionaryPagesByteRanges(int[] rowGroupIndices)#

Get the byte ranges in the source file that hold the column-chunk dictionary page data used for row-group pruning of (in)equality predicates.

The ordering follows the C++ reader’s ordering and is meaningful: the i-th entry corresponds to the i-th column chunk needing a dictionary page. The result may be empty.

public DictionaryByteRange[] dictionaryPagesByteRangesIncludeUnbounded( int[] rowGroupIndices )#

Like dictionaryPagesByteRanges(int[]) but also returns a range for a column chunk whose footer does not pin its dictionary page down exactly.

Such a range only bounds a page that may not be there: some writers omit dictionary_page_offset even with a dictionary page present. A caller that opts into these ranges caps how much of each it reads with DictionaryByteRange.byteRangeToRead(long), cuts what it read down to the one page with dictionaryPageLengths(HostMemoryBuffer[]), and passes an empty buffer for a chunk whose page turns out not to be there.

The ordering follows the C++ reader’s ordering and is meaningful: the i-th entry corresponds to the i-th column chunk needing a dictionary page. The result may be empty.

public int[] filterRowGroupsWithDictionaryPages( DeviceMemoryBuffer[] dictionaryPageData, int[] rowGroupIndices )#: Filter row groups using column-chunk dictionary pages loaded into device memory.

public ByteRange[] filterColumnChunksByteRanges(int[] rowGroupIndices)#

Returns:: byte ranges for the column chunks of filter columns.

public ByteRange[] payloadColumnChunksByteRanges( int[] rowGroupIndices )#

If a filter is set, the columns referenced by the filter are always excluded, regardless of whether any filter column methods have been called on this reader. Without a filter, the ranges cover the full set of columns that would be read i.e. the columns projected via ParquetOptions, or all columns in the file when no projection was set.

Returns:: byte ranges for the column chunks of payload columns.

public ByteRange[] allColumnChunksByteRanges(int[] rowGroupIndices)#

Returns:: byte ranges for all column chunks (filter + payload) of the selected columns.

public FilterMaterializationResult materializeFilterColumns( int[] rowGroupIndices, DeviceMemoryBuffer[] columnChunkData, boolean usePageLevelPruning )#

Build the initial row mask, materialize the filter columns, and evaluate the compiled filter expression against them.

The row mask and the resulting filter table are returned together as a FilterMaterializationResult; close it via try-with-resources.

Set usePageLevelPruning = true to skip decompression and decode of pages the filter proves empty. Requires prior setupPageIndex(HostMemoryBuffer).

Cost: a per-page stats scan of filter columns and a carried row-mask column. Payoff: pruned pages are skipped entirely, typically the dominant read cost. Enable when a meaningful fraction of pages can be pruned; otherwise leave false.

Parameters:

rowGroupIndices – row groups to read
columnChunkData – device buffers holding the filter column chunks, in the order returned by filterColumnChunksByteRanges(int[])
usePageLevelPruning – seed the row mask from page-index stats and enable the data page mask; requires prior setupPageIndex(HostMemoryBuffer)

Returns:

combined filter table and mutated row mask; caller must close this result

public TableWithColumnNames materializePayloadColumns( int[] rowGroupIndices, DeviceMemoryBuffer[] columnChunkData, ColumnVector rowMask, boolean usePageLevelPruning )#

Materialize only the payload columns, applying the supplied row mask to the output.

Set usePageLevelPruning = true to skip decompression and decode of pages containing no rows surviving rowMask. Adds a small mask-build cost; wins when rowMask prunes a meaningful fraction of pages.

Parameters:

rowGroupIndices – row groups to read
columnChunkData – device buffers holding the payload column chunks, in the order returned by payloadColumnChunksByteRanges(int[])
rowMask – row mask (read-only)
usePageLevelPruning – enable the data page mask to skip decode of pages the row mask proves empty

Returns:

the materialized payload column table with its column names; caller must close

public Table materializeAllColumns( int[] rowGroupIndices, DeviceMemoryBuffer[] columnChunkData )#

Materialize all (or selected) columns in a single pass.

The filter expression is applied after reading, so this is most efficient for small files or when most rows survive the filter. For highly-selective filters, prefer the explicit two-step flow.

public void setupChunkingForFilterColumns( long chunkReadLimit, long passReadLimit, int[] rowGroupIndices, boolean usePageLevelPruning, DeviceMemoryBuffer[] columnChunkData )#

Build the initial row mask and set up chunking state for filter-column materialization.

The row mask is owned internally by the reader for the duration of the chunked filter pipeline; subsequent calls to materializeFilterColumnsChunk() mutate it in place. After all chunks are drained, call takeFilterRowMask() to transfer ownership of the row mask to the caller (typically to feed it into materializePayloadColumns(int[], DeviceMemoryBuffer[], ColumnVector, boolean)).

Set usePageLevelPruning = true to skip decompression and decode of pages the filter proves empty. Requires prior setupPageIndex(HostMemoryBuffer).

Calling this method again before takeFilterRowMask() discards the previous chunked-filter row mask.

Caller must keep columnChunkData open until takeFilterRowMask() (or a re-setup); the native reader holds references to it across chunk calls.

Parameters:

chunkReadLimit – soft limit (in bytes) on each output chunk returned by the matching materialize*Chunk call, or 0 for no limit
passReadLimit – soft limit (in bytes) on the working memory used to decompress and decode a single subpass (a page-level slice of the selected row groups), or 0 for no limit. Hybrid scan hardcodes the number of row-group passes to 1, so this parameter only bounds the subpass (decode) working set; it does not repartition row groups. To split row groups across multiple passes up front, call constructRowGroupPasses(ReadColumnsMode, int[], long) with ReadColumnsMode.FILTER_COLUMNS first and issue a separate chunked run per returned partition.
rowGroupIndices – row groups to read
usePageLevelPruning – seed the row mask from page-index stats and enable the data page mask; requires prior setupPageIndex(HostMemoryBuffer)
columnChunkData – device buffers holding the filter column chunks, in the order returned by filterColumnChunksByteRanges(int[])

public Table materializeFilterColumnsChunk()#

Returns:: the next filter-column chunk; throws if no chunk is available or if no chunked filter pipeline is active. The internal row mask owned by this reader is updated in place.

public ColumnVector takeFilterRowMask()#

Transfer ownership of the row mask produced by the chunked filter-column materialization to the caller.

Typically called after all materializeFilterColumnsChunk() calls complete and immediately before materializePayloadColumns(int[], DeviceMemoryBuffer[], ColumnVector, boolean).

After this call the reader has no chunked-filter row mask; subsequent materializeFilterColumnsChunk() calls will fail until setupChunkingForFilterColumns(long, long, int[], boolean, DeviceMemoryBuffer[]) is invoked again. If never called, the row mask is freed when the reader is closed.

Returns:: the owned row mask ColumnVector; caller must close it.

public void setupChunkingForPayloadColumns( long chunkReadLimit, long passReadLimit, int[] rowGroupIndices, ColumnVector rowMask, boolean usePageLevelPruning, DeviceMemoryBuffer[] columnChunkData )#

Set up chunking state for payload-column materialization.

Subsequent calls to materializePayloadColumnsChunk(ColumnVector) will yield successive chunks.

Caller must keep columnChunkData open until hasNextTableChunk() returns false; the native reader holds references to it across chunk calls.

Parameters:

chunkReadLimit – soft limit (in bytes) on each output chunk returned by the matching materialize*Chunk call, or 0 for no limit
passReadLimit – soft limit (in bytes) on the working memory used to decompress and decode a single subpass (a page-level slice of the selected row groups), or 0 for no limit. Hybrid scan hardcodes the number of row-group passes to 1, so this parameter only bounds the subpass (decode) working set; it does not repartition row groups. To split row groups across multiple passes up front, call constructRowGroupPasses(ReadColumnsMode, int[], long) with ReadColumnsMode.PAYLOAD_COLUMNS first and issue a separate chunked run per returned partition.
rowGroupIndices – row groups to read
rowMask – row mask (read-only)
usePageLevelPruning – enable the data page mask to skip decode of pages the row mask proves empty
columnChunkData – device buffers holding the payload column chunks, in the order returned by payloadColumnChunksByteRanges(int[])

public Table materializePayloadColumnsChunk(ColumnVector rowMask)#

Returns:: the next payload-column chunk; throws if no chunk is available.

public void setupChunkingForAllColumns( long chunkReadLimit, long passReadLimit, int[] rowGroupIndices, DeviceMemoryBuffer[] columnChunkData )#

Set up chunking state for all-column materialization (single-pass mode).

Subsequent calls to materializeAllColumnsChunk() will yield successive chunks.

Caller must keep columnChunkData open until hasNextTableChunk() returns false; the native reader holds references to it across chunk calls.

Parameters:

chunkReadLimit – soft limit (in bytes) on each output chunk returned by the matching materialize*Chunk call, or 0 for no limit
passReadLimit – soft limit (in bytes) on the working memory used to decompress and decode a single subpass (a page-level slice of the selected row groups), or 0 for no limit. Hybrid scan hardcodes the number of row-group passes to 1, so this parameter only bounds the subpass (decode) working set; it does not repartition row groups. To split row groups across multiple passes up front, call constructRowGroupPasses(ReadColumnsMode, int[], long) with ReadColumnsMode.ALL_COLUMNS first and issue a separate chunked run per returned partition.
rowGroupIndices – row groups to read
columnChunkData – device buffers holding all column chunks, in the order returned by allColumnChunksByteRanges(int[])

public Table materializeAllColumnsChunk()#

Returns:: the next all-columns chunk; throws if no chunk is available.

public boolean hasNextTableChunk()#

Returns:: true when a subsequent materialize*Chunk call would return data.

public int[][] constructRowGroupPasses( ReadColumnsMode columnsMode, int[] rowGroupIndices, long passReadLimit )#

Partition the supplied row groups into passes whose estimated uncompressed size over the selected columns is bounded by the given limit.

The limit is a hint, not a strict memory bound: a pass always contains whole row groups, so a single row group larger than the limit still constitutes its own pass. The returned array contains one inner array per pass.

Parameters:

columnsMode – columns selection to use for pass memory estimation
rowGroupIndices – row groups to partition
passReadLimit – limit on the memory used by a single pass, or 0 for no limit. Each returned pass can then be fed to a setupChunkingFor* call, which will further stream that pass in subpass-sized chunks bounded by its own passReadLimit argument.

Returns:

an array of arrays of row group indices, one per pass

public void close()#

Static Methods

public static long[] dictionaryPageLengths( HostMemoryBuffer[] pageData )#

The length of the dictionary page at the front of each buffer, its page header included, or 0 for a buffer that does not begin with a whole dictionary page.

What was read of a range that only bounds its dictionary page begins at that page and runs past it, so it has to be cut down to the page before filterRowGroupsWithDictionaryPages is given it. The page’s own header says how long the page is, and this reads that header off the front of each buffer. A 0 means the chunk cannot be pruned with what was read, either because a writer claimed dictionary encoding and wrote no dictionary page, or because the page is longer than what was read; such a chunk is passed on as an empty buffer.

The buffers are read on the host, and nothing here touches the GPU.

Parameters:: pageData – one buffer per dictionary page range, read from the start of the range
Returns:: the length of the page in each buffer, in the order the buffers were given

public enum ReadColumnsMode#

Column selection used when constructing row-group passes.

Enum Constant Summary

`ALL_COLUMNS`	All selected columns.
`FILTER_COLUMNS`	Only the filter columns.
`PAYLOAD_COLUMNS`	Only the payload columns.

Enum Constants

FILTER_COLUMNS#: Only the filter columns.

PAYLOAD_COLUMNS#: Only the payload columns.

ALL_COLUMNS#: All selected columns.

public static final class TableWithColumnNames implements AutoCloseable #

A materialized Table and the names of its columns, in the same order.

The table is owned by this object; close via try-with-resources to release it.

Method Summary

`close()`
`columnNames()`
`table()`

Methods

public Table table()#

Returns:: the materialized table.

public String[] columnNames()#

Returns:: the names of the columns of table(), in the same order.

public void close()#

public static final class FilterMaterializationResult implements AutoCloseable #

The result of a combined row-mask-build + filter-column-materialization call.

Both the filter TableWithColumnNames and the mutated row ColumnVector mask are owned by this object. Close via try-with-resources to release both.

Method Summary

`close()`
`rowMask()`
`tableWithColumnNames()`

Methods

public TableWithColumnNames tableWithColumnNames()#

Returns:: the materialized filter column table with its column names.

public ColumnVector rowMask()#

Returns:: the (mutated) row mask after the filter expression was applied.

public void close()#