DeletionVector#

public class DeletionVector#

Provides JNI wrappers for reading Parquet files with deletion vector support.

Deletion vectors are used in Delta Lake and other table formats to track deleted rows without physically rewriting data files. This class provides APIs to read Parquet files while applying deletion vectors using 64-bit roaring bitmap serialization format.

The APIs in this file are experimental and subject to change.

Method Summary

`computeNumDeletedRows(DeletionVectorInfo deletionVectorInfo, int maxChunkRows)`	Computes (on the GPU) the number of rows deleted by a serialized deletion vector.
`computeNumDeletedRows(DeletionVectorInfo[] deletionVectorInfos, int maxChunkRows)`	Computes (on the GPU) the total number of rows deleted by serialized deletion vectors.
`newParquetChunkedReader(long chunkSizeByteLimit, long passReadLimit, ParquetOptions opts, HostMemoryBuffer[] dataBuffers, DeletionVectorInfo[] deletionVectorInfos)`	Construct the reader instance from a read limit and data in host memory buffers.
`newParquetChunkedReader(long chunkSizeByteLimit, long passReadLimit, ParquetOptions opts, HostMemoryBuffer[] dataBuffers, int[][] rowGroups, DeletionVectorInfo[] deletionVectorInfos)`	Construct the reader instance from a read limit and data in host memory buffers.
`newParquetChunkedReader(long chunkSizeByteLimit, long passReadLimit, ParquetOptions opts, String[] inputFilePaths, int[][] rowGroups, DeletionVectorInfo[] deletionVectorInfos)`	Construct the reader instance from a read limit and data in host memory buffers.
`readParquet(ParquetOptions opts, HostMemoryBuffer[] dataBuffers, DeletionVectorInfo[] deletionVectorInfos)`	Reads a Parquet file with deletion vector support.
`readParquet(ParquetOptions opts, HostMemoryBuffer[] dataBuffers, int[][] rowGroups, DeletionVectorInfo[] deletionVectorInfos)`	Reads a Parquet file with deletion vector support.
`readParquet(ParquetOptions opts, String[] inputFilePaths, int[][] rowGroups, DeletionVectorInfo[] deletionVectorInfos)`	Reads a Parquet file with deletion vector support.

Static Methods

public static long computeNumDeletedRows( DeletionVectorInfo deletionVectorInfo, int maxChunkRows )#

Computes (on the GPU) the number of rows deleted by a serialized deletion vector.

Parameters:

deletionVectorInfo – deletion vector and row-group metadata
maxChunkRows – maximum number of row indexes to process at once

Throws:

NullPointerException – if deletionVectorInfo is null
IllegalArgumentException – if row-group metadata is missing, empty, or contains a negative value, or if maxChunkRows is not positive

Returns:

number of deleted rows in the specified row groups

public static long computeNumDeletedRows( DeletionVectorInfo[] deletionVectorInfos, int maxChunkRows )#

Computes (on the GPU) the total number of rows deleted by serialized deletion vectors.

Parameters:

deletionVectorInfos – deletion vectors and row-group metadata
maxChunkRows – maximum number of row indexes to process at once

Throws:

NullPointerException – if deletionVectorInfos or one of its elements is null
IllegalArgumentException – if no deletion vectors are supplied, row-group metadata is missing, empty, or contains a negative value, deletion and retention vectors are mixed, or maxChunkRows is not positive

Returns:

total number of deleted rows in the specified row groups across all deletion vectors.

public static Table readParquet( ParquetOptions opts, HostMemoryBuffer[] dataBuffers, DeletionVectorInfo[] deletionVectorInfos )#

Reads a Parquet file with deletion vector support.

Reads a Parquet file, prepends an index column to the table, and applies the deletion vector filter. If row group metadata is not provided, the index column will be a simple sequence from 0 to the number of rows. If the deletion vector is null or empty, the table with the prepended index column is returned as-is without filtering.

Parameters:

opts – ParquetOptions
dataBuffers – Array of HostMemoryBuffers containing the Parquet file data.
deletionVectorInfos – Array of DeletionVectorInfo objects representing deletion vectors for each Parquet file to read.

Returns:

A Table containing the filtered data with a prepended UINT64 index column.

public static Table readParquet( ParquetOptions opts, HostMemoryBuffer[] dataBuffers, int[][] rowGroups, DeletionVectorInfo[] deletionVectorInfos )#

Reads a Parquet file with deletion vector support.

Parameters:

opts – ParquetOptions
dataBuffers – Array of HostMemoryBuffers containing the Parquet file data.
rowGroups – Row group indices to read
deletionVectorInfos – Array of DeletionVectorInfo objects representing deletion vectors for each Parquet file to read.

Returns:

A Table containing the filtered data with a prepended UINT64 index column.

public static Table readParquet( ParquetOptions opts, String[] inputFilePaths, int[][] rowGroups, DeletionVectorInfo[] deletionVectorInfos )#

Reads a Parquet file with deletion vector support.

Parameters:

opts – ParquetOptions
inputFilePaths – Array of input Parquet file paths.
rowGroups – Row group indices to read
deletionVectorInfos – Array of DeletionVectorInfo objects representing deletion vectors for each Parquet file to read.

Returns:

A Table containing the filtered data with a prepended UINT64 index column.

public static ParquetChunkedReader newParquetChunkedReader( long chunkSizeByteLimit, long passReadLimit, ParquetOptions opts, HostMemoryBuffer[] dataBuffers, DeletionVectorInfo[] deletionVectorInfos )#

Construct the reader instance from a read limit and data in host memory buffers.

Parameters:

chunkSizeByteLimit – Limit on total number of bytes to be returned per read, or 0 if there is no limit.
passReadLimit – Limit on the amount of memory used for reading and decompressing data or 0 if there is no limit
opts – The options for Parquet reading.
dataBuffers – Array of HostMemoryBuffers containing the Parquet file data.
rowGroups – Row group indices to read
deletionVectorInfos – Array of DeletionVectorInfo objects representing deletion vectors for each Parquet file to read.

Construct the reader instance from a read limit and data in host memory buffers.

Parameters:

chunkSizeByteLimit – Limit on total number of bytes to be returned per read, or 0 if there is no limit.
passReadLimit – Limit on the amount of memory used for reading and decompressing data or 0 if there is no limit
opts – The options for Parquet reading.
dataBuffers – Array of HostMemoryBuffers containing the Parquet file data.
rowGroups – Row group indices to read
deletionVectorInfos – Array of DeletionVectorInfo objects representing deletion vectors for each Parquet file to read.

public static ParquetChunkedReader newParquetChunkedReader( long chunkSizeByteLimit, long passReadLimit, ParquetOptions opts, String[] inputFilePaths, int[][] rowGroups, DeletionVectorInfo[] deletionVectorInfos )#

Construct the reader instance from a read limit and data in host memory buffers.

Parameters:

chunkSizeByteLimit – Limit on total number of bytes to be returned per read, or 0 if there is no limit.
passReadLimit – Limit on the amount of memory used for reading and decompressing data or 0 if there is no limit
opts – The options for Parquet reading.
inputFilePaths – Array of input file paths containing the Parquet file data.
rowGroups – Row group indices to read
deletionVectorInfos – Array of DeletionVectorInfo objects representing deletion vectors for each Parquet file to read.

public static class DeletionVectorInfo#

Information holder for a deletion vector associated with a Parquet file to read.

Field Summary

`isRetention`
`rowGroupNumRows`	Number of rows in each row group to read.
`rowGroupOffsets`	Row index offsets for each row group to read.
`serializedBitmap`	Serialized 64-bit roaring bitmap in portable format representing the deletion vector.
`totalNumRows`	Total number of rows to read from the Parquet file associated with this deletion vector.

Constructor Summary

DeletionVectorInfo(HostMemoryBuffer serializedBitmap, boolean isRetention, long[] rowGroupOffsets, int[] rowGroupNumRows)

Fields

public final HostMemoryBuffer serializedBitmap#: Serialized 64-bit roaring bitmap in portable format representing the deletion vector.

public final boolean isRetention#

public final long[] rowGroupOffsets#

Row index offsets for each row group to read.

Can be null to read all row groups.

public final int[] rowGroupNumRows#

Number of rows in each row group to read.

Can be null to read all row groups.

public final int totalNumRows#: Total number of rows to read from the Parquet file associated with this deletion vector.

Constructors

public DeletionVectorInfo( HostMemoryBuffer serializedBitmap, boolean isRetention, long[] rowGroupOffsets, int[] rowGroupNumRows )#

public static class ParquetChunkedReader implements AutoCloseable #

Method Summary

`close()`
`hasNext()`	Check if the given file has anything left to read.
`readChunk()`	Read a chunk of rows in the given Parquet file such that the returning data has total size does not exceed the given read limit.

Methods

public boolean hasNext()#

Check if the given file has anything left to read.

Returns:: A boolean value indicating if there is more data to read from file.

public Table readChunk()#

Read a chunk of rows in the given Parquet file such that the returning data has total size does not exceed the given read limit.

If the given file has no data, or all data has been read before by previous calls to this function, a null Table will be returned.

Returns:: A table of new rows reading from the given file.

public void close() throws Exception #