Parquet Metadata#

class pylibcudf.io.parquet_metadata.ColumnChunk#

Metadata for a row group’s column chunk.

Attributes

`column_index_length`	ColumnChunk.column_index_length: int
`column_index_offset`	ColumnChunk.column_index_offset: int
`file_offset`	ColumnChunk.file_offset: int
`file_path`	ColumnChunk.file_path: str
`meta_data`	ColumnChunk.meta_data: ColumnChunkMetaData
`offset_index_length`	ColumnChunk.offset_index_length: int
`offset_index_offset`	ColumnChunk.offset_index_offset: int
`schema_idx`	ColumnChunk.schema_idx: int

column_index_length#

ColumnChunk.column_index_length: int

Size of the chunk’s ColumnIndex, in bytes.

column_index_offset#

ColumnChunk.column_index_offset: int

File offset of the chunk’s ColumnIndex.

file_offset#

ColumnChunk.file_offset: int

Deprecated byte offset to column metadata.

file_path#

ColumnChunk.file_path: str

Relative file path for this column chunk.

meta_data#

ColumnChunk.meta_data: ColumnChunkMetaData

Column metadata for this chunk.

offset_index_length#

ColumnChunk.offset_index_length: int

Size of the chunk’s OffsetIndex, in bytes.

offset_index_offset#

ColumnChunk.offset_index_offset: int

File offset of the chunk’s OffsetIndex.

schema_idx#

ColumnChunk.schema_idx: int

Derived index in the flattened schema.

class pylibcudf.io.parquet_metadata.ColumnChunkMetaData#

Metadata payload for a column chunk.

Attributes

`num_values`	ColumnChunkMetaData.num_values: int
`path_in_schema`	ColumnChunkMetaData.path_in_schema: list[str]
`statistics`	ColumnChunkMetaData.statistics: ColumnChunkStatistics
`total_compressed_size`	ColumnChunkMetaData.total_compressed_size: int
`total_uncompressed_size`	ColumnChunkMetaData.total_uncompressed_size: int

num_values#

ColumnChunkMetaData.num_values: int

Number of values in this chunk.

path_in_schema#

ColumnChunkMetaData.path_in_schema: list[str]

Column path components in the flattened schema.

statistics#

ColumnChunkMetaData.statistics: ColumnChunkStatistics

Column chunk statistics.

total_compressed_size#

ColumnChunkMetaData.total_compressed_size: int

Total compressed page bytes for this chunk.

total_uncompressed_size#

ColumnChunkMetaData.total_uncompressed_size: int

Total uncompressed page bytes for this chunk.

class pylibcudf.io.parquet_metadata.ColumnChunkStatistics#

Column chunk statistics.

Attributes

`distinct_count`	ColumnChunkStatistics.distinct_count: int \| None
`has_min_max`	ColumnChunkStatistics.has_min_max: bool
`is_max_value_exact`	ColumnChunkStatistics.is_max_value_exact: bool \| None
`is_min_value_exact`	ColumnChunkStatistics.is_min_value_exact: bool \| None
`max_encoded`	ColumnChunkStatistics.max_encoded: bytes \| None
`min_encoded`	ColumnChunkStatistics.min_encoded: bytes \| None
`null_count`	ColumnChunkStatistics.null_count: int \| None

distinct_count#

ColumnChunkStatistics.distinct_count: int | None

Number of distinct values in the column chunk.

has_min_max#

ColumnChunkStatistics.has_min_max: bool

Whether this column chunk has encoded minimum and maximum values.

is_max_value_exact#

ColumnChunkStatistics.is_max_value_exact: bool | None

Whether max_value is the exact column-chunk maximum.

is_min_value_exact#

ColumnChunkStatistics.is_min_value_exact: bool | None

Whether min_value is the exact column-chunk minimum.

max_encoded#

ColumnChunkStatistics.max_encoded: bytes | None

Encoded maximum value, preferring max_value over deprecated max.

The bytes are the raw Parquet statistics payload and must be interpreted using the column’s Parquet physical and logical type metadata.

min_encoded#

ColumnChunkStatistics.min_encoded: bytes | None

Encoded minimum value, preferring min_value over deprecated min.

The bytes are the raw Parquet statistics payload and must be interpreted using the column’s Parquet physical and logical type metadata.

null_count#

ColumnChunkStatistics.null_count: int | None

Number of null values in the column chunk.

class pylibcudf.io.parquet_metadata.DecimalType#

Parameters of a DECIMAL logical type.

Attributes

`precision`	DecimalType.precision: int
`scale`	DecimalType.scale: int

precision#

DecimalType.precision: int

Total number of digits.

scale#

DecimalType.scale: int

Number of digits after the decimal point.

pylibcudf.io.parquet_metadata.FieldRepetitionType(*values)#

See also

pylibcudf.io.parquet_metadata.read_parquet_footers: Read one FileMetaData per source directly from pylibcudf.io.types.SourceInfo.

columnchunk_metadata#

FileMetaData.columnchunk_metadata: dict[str, list[int]]

Get a map of dotted column paths to lists of total_uncompressed_size values from every column chunk in this file.

Returns:

dict[str, list[int]]: Map of dotted column paths (".".join(path_in_schema)) to lists of total_uncompressed_size metadata from all their column chunks.

Notes

Equivalent to, but faster than, walking each row group’s columns:

>>> result: dict[str, list[int]] = {}
>>> for rg in file_metadata.row_groups:
...     for col in rg.columns:
...         name = ".".join(col.meta_data.path_in_schema)
...         result.setdefault(name, []).append(
...             col.meta_data.total_uncompressed_size
...         )

created_by#

FileMetaData.created_by: str

Get the application that created the file.

classmethod from_bytes( cls, const uint8_t[::1] footer_bytes: Buffer, ) → FileMetaData#

Build FileMetaData from parquet footer bytes.

Parameters:

footer_bytesBuffer: A contiguous bytes-like object containing parquet footer bytes. The bytes are forwarded as-is to cudf::io::parquet::experimental::hybrid_scan_reader without Python-side preprocessing. This method does not strip the parquet footer suffix (4-byte footer length + PAR1 magic), so callers should generally pass only the footer region bytes.

Returns:

FileMetaData: Parsed parquet file footer metadata.

num_rows#

FileMetaData.num_rows: int

Get the total number of rows.

row_group_num_rows#

FileMetaData.row_group_num_rows: list[int]

Get row counts for each row group in this file.

Returns:

list: A list with the row count per row group in this file.

Notes

Equivalent to, but faster than, checking each row groups’ num_rows:

>>> [rg.num_rows for rg in file_metadata.row_groups]

row_groups#

FileMetaData.row_groups: list[RowGroup]

Get row group metadata in this file.

schema#

FileMetaData.schema: list[SchemaElement]

Get the file’s schema tree, flattened by a depth-first traversal.

The first element is the root. Each element’s num_children gives the number of elements that follow it at the next level, which is what allows the tree to be reconstructed from the flat list.

Returns:

list[SchemaElement]: One entry per schema element, in depth-first order.

version#

FileMetaData.version: int

Get the file format version.

class pylibcudf.io.parquet_metadata.IntType#

Parameters of an INTEGER logical type.

Attributes

`bit_width`	IntType.bit_width: int
`is_signed`	IntType.is_signed: bool

bit_width#

IntType.bit_width: int

Number of bits: 8, 16, 32, or 64.

is_signed#

IntType.is_signed: bool

Whether the integer is signed.

class pylibcudf.io.parquet_metadata.LogicalType#

Logical type annotation of a Parquet schema element.

Attributes

`decimal_type`	LogicalType.decimal_type: DecimalType \| None
`int_type`	LogicalType.int_type: IntType \| None
`time_type`	LogicalType.time_type: TimeType \| None
`timestamp_type`	LogicalType.timestamp_type: TimestampType \| None
`type`	LogicalType.type: LogicalTypeId

decimal_type#

LogicalType.decimal_type: DecimalType | None

Parameters of a DECIMAL type, otherwise None.

int_type#

LogicalType.int_type: IntType | None

Parameters of an INTEGER type, otherwise None.

time_type#

LogicalType.time_type: TimeType | None

Parameters of a TIME type, otherwise None.

timestamp_type#

LogicalType.timestamp_type: TimestampType | None

Parameters of a TIMESTAMP type, otherwise None.

type#

LogicalType.type: LogicalTypeId

Kind of logical type.

pylibcudf.io.parquet_metadata.LogicalTypeId(*values)#

Enum members

UNDEFINED
STRING
MAP
LIST
ENUM
DECIMAL
DATE
TIME
TIMESTAMP
INTEGER
UNKNOWN
JSON
BSON
VARIANT

class pylibcudf.io.parquet_metadata.ParquetColumnSchema#

Schema of a parquet column, including the nested columns.

Parameters:

parquet_column_schema

Methods

`child`(self, int idx)	Returns schema of the child with the given index.
`children`(self)	Returns schemas of all child columns.
`cudf_type`(self)	Returns the cudf data type for this column.
`name`(self)	Returns parquet column name; can be empty.
`num_children`(self)	Returns the number of child columns.

child(self, int idx) → ParquetColumnSchema#

Returns schema of the child with the given index.

Parameters:

idxint: Child Index

Returns:

ParquetColumnSchema: Child schema

children(self) → list#

Returns schemas of all child columns.

Returns:

list[ParquetColumnSchema]: Child schemas.

cudf_type(self) → DataType#

Returns the cudf data type for this column.

This is the resolved cudf data type mapped from the parquet physical/logical types.

Returns:

DataType: cudf data type

name(self) → str#

Returns parquet column name; can be empty.

Returns:

str: Column name

num_children(self) → int#

Returns the number of child columns.

Returns:

int: Children count

class pylibcudf.io.parquet_metadata.ParquetMetadata#

Information about content of a parquet file.

Parameters:

parquet_metadata

Methods

`columnchunk_metadata`(self)	Returns a map of leaf column names to lists of total_uncompressed_size metadata from all column chunks in the file footer.
`metadata`(self)	Returns the key-value metadata in the file footer.
`num_rowgroups`(self)	Returns the total number of rowgroups in the file.
`num_rowgroups_per_file`(self)	Returns the number of rowgroups in each file.
`num_rows`(self)	Returns the number of rows of the root column.
`rowgroup_metadata`(self)	Returns the row group metadata in the file footer.
`schema`(self)	Returns the parquet schema.

columnchunk_metadata(self) → dict#

Returns a map of leaf column names to lists of total_uncompressed_size metadata from all column chunks in the file footer.

Returns:

dict[str, list[int]]: Map of leaf column names to lists of total_uncompressed_size metadata from all their column chunks.

metadata(self) → dict#

Returns the key-value metadata in the file footer.

Returns:

dict[str, str]: Key value metadata as a map.

num_rowgroups(self) → int#

Returns the total number of rowgroups in the file.

Returns:

int: Number of row groups.

num_rowgroups_per_file(self) → list#: Returns the number of rowgroups in each file.

num_rows(self) → int#

Returns the number of rows of the root column.

Returns:

int: Number of rows

rowgroup_metadata(self) → list#

Returns the row group metadata in the file footer.

Returns:

list[dict[str, int]]: Vector of row group metadata as maps.

schema(self) → ParquetSchema#

Returns the parquet schema.

Returns:

ParquetSchema: Parquet schema

class pylibcudf.io.parquet_metadata.ParquetSchema#

Schema of a parquet file.

Parameters:

parquet_schema

Methods

`column_types`(self)	Returns a dictionary mapping column names to their cudf data types.
`root`(self)	Returns the schema of the struct column that contains all columns as fields.

column_types(self) → dict#

Returns a dictionary mapping column names to their cudf data types.

Returns:

dict[str, DataType]: Dictionary mapping column names to DataType objects

root(self) → ParquetColumnSchema#

Returns the schema of the struct column that contains all columns as fields.

Returns:

ParquetColumnSchema: Root column schema

pylibcudf.io.parquet_metadata.PhysicalType#

See also Type.

Enum members

UNDEFINED
BOOLEAN
INT32
INT64
INT96
FLOAT
DOUBLE
BYTE_ARRAY
FIXED_LEN_BYTE_ARRAY

class pylibcudf.io.parquet_metadata.RowGroup#

Parquet row group metadata.

Attributes

`columns`	RowGroup.columns: list[ColumnChunk]
`file_offset`	RowGroup.file_offset: int \| None
`num_rows`	RowGroup.num_rows: int
`ordinal`	RowGroup.ordinal: int \| None
`sorting_columns`	RowGroup.sorting_columns: list[SortingColumn] \| None
`total_byte_size`	RowGroup.total_byte_size: int
`total_compressed_size`	RowGroup.total_compressed_size: int \| None

columns#

RowGroup.columns: list[ColumnChunk]

Column chunk metadata for each column in this row group.

file_offset#

RowGroup.file_offset: int | None

Optional byte offset to first page in this row group.

num_rows#

RowGroup.num_rows: int

Number of rows in this row group.

ordinal#

RowGroup.ordinal: int | None

Optional row group ordinal within the file.

sorting_columns#

RowGroup.sorting_columns: list[SortingColumn] | None

Optional row sort order metadata.

total_byte_size#

RowGroup.total_byte_size: int

Total uncompressed byte size in this row group.

total_compressed_size#

RowGroup.total_compressed_size: int | None

Optional total compressed bytes for this row group.

class pylibcudf.io.parquet_metadata.SchemaElement#

An element of a Parquet file’s schema tree.

Attributes

`arrow_type`	SchemaElement.arrow_type: TypeId \| None
`children_idx`	SchemaElement.children_idx: list[int]
`field_id`	SchemaElement.field_id: int \| None
`logical_type`	SchemaElement.logical_type: LogicalType \| None
`max_definition_level`	SchemaElement.max_definition_level: int
`max_repetition_level`	SchemaElement.max_repetition_level: int
`name`	SchemaElement.name: str
`num_children`	SchemaElement.num_children: int
`output_as_byte_array`	SchemaElement.output_as_byte_array: bool
`parent_idx`	SchemaElement.parent_idx: int
`repetition_type`	SchemaElement.repetition_type: FieldRepetitionType
`type`	SchemaElement.type: PhysicalType
`type_length`	SchemaElement.type_length: int

Methods

`is_one_level_list`(self, SchemaElement parent)	Whether the element is a list in the one-level list encoding.
`is_struct`(self)	Whether the element is a struct.
`is_stub`(self)	Whether the element is a repeated group that is not a nesting level.

arrow_type#

SchemaElement.arrow_type: TypeId | None

cudf type recorded in the file’s arrow schema, if any was applied.

children_idx#

SchemaElement.children_idx: list[int]

Indices of the child elements in FileMetaData.schema.

field_id#

SchemaElement.field_id: int | None

Field ID from the original schema, if the writer recorded one.

is_one_level_list(self, SchemaElement parent) → bool#

Whether the element is a list in the one-level list encoding.

Parameters:

parentSchemaElement: Parent of this element.

Returns:

bool: True for a repeated leaf whose parent is not a list.

is_struct(self) → bool#

Whether the element is a struct.

Returns:

bool: True for a group that is a level of struct nesting.

is_stub(self) → bool#

Whether the element is a repeated group that is not a nesting level.

Returns:

bool: True for the repeated wrapper of a three-level list encoding.

logical_type#

SchemaElement.logical_type: LogicalType | None

Logical type annotation, if the writer recorded one.

max_definition_level#

SchemaElement.max_definition_level: int

Maximum definition level of the field.

max_repetition_level#

SchemaElement.max_repetition_level: int

Maximum repetition level of the field.

name#

SchemaElement.name: str

Name of the field, as recorded by the writer.

num_children#

SchemaElement.num_children: int

Number of child elements; zero for leaf columns.

output_as_byte_array#

SchemaElement.output_as_byte_array: bool

Whether the writer stores the column as a byte array.

parent_idx#

SchemaElement.parent_idx: int

Index of the parent element in FileMetaData.schema.

repetition_type#

SchemaElement.repetition_type: FieldRepetitionType

Whether the field is required, optional, or repeated.

type#

SchemaElement.type: PhysicalType

Physical type of the field; UNDEFINED for non-leaf elements.

type_length#

SchemaElement.type_length: int

Byte length of FIXED_LEN_BYTE_ARRAY values, or the bit length.

class pylibcudf.io.parquet_metadata.SortingColumn#

Sort metadata for a row group column.

Attributes

`column_idx`	SortingColumn.column_idx: int
`descending`	SortingColumn.descending: bool
`nulls_first`	SortingColumn.nulls_first: bool

column_idx#

SortingColumn.column_idx: int

Column index (within the row group).

descending#

SortingColumn.descending: bool

Whether this column is sorted in descending order.

nulls_first#

SortingColumn.nulls_first: bool

Whether null values are ordered before non-null values.

class pylibcudf.io.parquet_metadata.TimeType#

Parameters of a TIME logical type.

Attributes

`is_adjusted_to_utc`	TimeType.is_adjusted_to_utc: bool
`unit`	TimeType.unit: TimeUnit

is_adjusted_to_utc#

TimeType.is_adjusted_to_utc: bool

Whether the values are adjusted to UTC.

unit#

TimeType.unit: TimeUnit

Unit of the values.

pylibcudf.io.parquet_metadata.TimeUnit(*values)#

Enum members

UNDEFINED
MILLIS
MICROS
NANOS

class pylibcudf.io.parquet_metadata.TimestampType#

Parameters of a TIMESTAMP logical type.

Attributes

`is_adjusted_to_utc`	TimestampType.is_adjusted_to_utc: bool
`unit`	TimestampType.unit: TimeUnit

is_adjusted_to_utc#

TimestampType.is_adjusted_to_utc: bool

Whether the values are adjusted to UTC.

unit#

TimestampType.unit: TimeUnit

Unit of the values.

pylibcudf.io.parquet_metadata.read_parquet_column_chunk_bounds( file_metadatas, columns, stream: CudaStreamLike | None = None, DeviceMemoryResource mr=None, ) → Table#

Decode parquet column-chunk min/max statistics for selected columns.

Missing min/max statistics are returned as nulls. Parquet min/max exactness flags are not interpreted by this function.

Parameters:

file_metadatasSequence[FileMetaData]: Parquet footer metadata objects, one per source.
columnsSequence[str]: Dotted leaf-column paths to decode statistics for.
streamCudaStreamLike, optional: CUDA stream used for device memory operations.
mrDeviceMemoryResource, optional: Device memory resource used for device memory allocation.

Returns:

Table: Table containing file indices in column 0, file-local row-group indices in column 1, and one (min, max) column pair per requested column after that. For columns[i], the minimum column is at 2 + 2 * i and the maximum column is at 3 + 2 * i.

pylibcudf.io.parquet_metadata.read_parquet_footers(SourceInfo src_info) → list#

Read parquet file footers as FileMetaData objects.

Parameters:

src_infoSourceInfo: Dataset source.

Returns:

list[FileMetaData]: One footer metadata object per input source.

pylibcudf.io.parquet_metadata.read_parquet_metadata(SourceInfo src_info) → ParquetMetadata#

Reads metadata of parquet dataset.

Parameters:

src_infoSourceInfo: Dataset source.

Returns:

ParquetMetadata: Parquet_metadata with parquet schema, number of rows, number of row groups and key-value metadata.

`columnchunk_metadata`	FileMetaData.columnchunk_metadata: dict[str, list[int]]
`created_by`	FileMetaData.created_by: str
`num_rows`	FileMetaData.num_rows: int
`row_group_num_rows`	FileMetaData.row_group_num_rows: list[int]
`row_groups`	FileMetaData.row_groups: list[RowGroup]
`schema`	FileMetaData.schema: list[SchemaElement]
`version`	FileMetaData.version: int