ParquetWriterOptions#

public final class ParquetWriterOptions extends CompressionMetadataWriterOptions#

This class represents settings for writing Parquet files.

It includes meta data information that will be used by the Parquet writer to write the file

Method Summary

`builder()`
`getDictionaryPolicy()`
`getMaxDictionarySize()`
`getRowGroupSizeBytes()`
`getRowGroupSizeRows()`
`getStatisticsFrequency()`

Fields inherited from class ai.rapids.cudf.ColumnWriterOptions: childColumnOptions, UNKNOWN_PRECISION

Methods inherited from class ai.rapids.cudf.ColumnWriterOptions: getChildColumnOptions, getColumnName, getFlatBooleans, getFlatColumnNames, getFlatInts, getParquetFieldId, getPrecision, hasParquetFieldId, isBinary, isNullable, isTimestampTypeInt96, listBuilder, mapColumn, structBuilder

Methods inherited from class ai.rapids.cudf.CompressionMetadataWriterOptions: getCompressionType, getMetadata, getTopLevelChildren

Methods

public int getRowGroupSizeRows()#

public long getRowGroupSizeBytes()#

public long getMaxDictionarySize()#

Returns:: the maximum per-column-chunk dictionary size in bytes. Only used when the dictionary policy is DictionaryPolicy.ADAPTIVE.

public DictionaryPolicy getDictionaryPolicy()#

Returns:: the dictionary policy in effect for the Parquet writer.

public StatisticsFrequency getStatisticsFrequency()#

Static Methods

public static Builder builder()#

public enum StatisticsFrequency#

Enum Constant Summary

`COLUMN`	Generate full column and offset indices (page index).
`NONE`	Do not generate statistics.
`PAGE`	Generate column statistics for each page.
`ROWGROUP`	Generate column statistics for each rowgroup.

Enum Constants

NONE#: Do not generate statistics.

ROWGROUP#: Generate column statistics for each rowgroup.

PAGE#: Generate column statistics for each page.

COLUMN#

Generate full column and offset indices (page index).

Implies ROWGROUP statistics. Required for page-level pruning in HybridScanReader.

public enum DictionaryPolicy#

Mirror of cudf::io::dictionary_policy.

Controls when the Parquet writer is allowed to use dictionary encoding.

Enum Constant Summary

`ADAPTIVE`	Use dictionary encoding when it does not impact compression.
`ALWAYS`	Use dictionary encoding even if it disables compression for affected columns.
`NEVER`	Never use dictionary encoding.

Enum Constants

NEVER#: Never use dictionary encoding.

ADAPTIVE#

Use dictionary encoding when it does not impact compression.

The writer disables dictionary encoding for any column chunk whose dictionary would exceed Builder.withMaxDictionarySize(long).

ALWAYS#: Use dictionary encoding even if it disables compression for affected columns.

public static class Builder extends CompressionMetadataWriterOptions.Builder <Builder, ParquetWriterOptions>#

Constructor Summary

Builder()

Method Summary

`build()`
`withDictionaryPolicy(DictionaryPolicy dictionaryPolicy)`	Sets the dictionary policy.
`withMaxDictionarySize(long maxDictionarySize)`	Sets the maximum dictionary size, in bytes.
`withRowGroupSizeBytes(long rowGroupSizeBytes)`
`withRowGroupSizeRows(int rowGroupSizeRows)`
`withStatisticsFrequency(StatisticsFrequency statsGranularity)`

Methods inherited from class ai.rapids.cudf.CompressionMetadataWriterOptions.Builder: withCompressionType, withMetadata

Constructors

public Builder()#

Methods

public Builder withRowGroupSizeRows(int rowGroupSizeRows)#

public Builder withRowGroupSizeBytes(long rowGroupSizeBytes)#

public Builder withMaxDictionarySize(long maxDictionarySize)#

Sets the maximum dictionary size, in bytes.

Only used when the dictionary policy is DictionaryPolicy.ADAPTIVE. Must be in [0, Integer.MAX_VALUE].

public Builder withDictionaryPolicy(DictionaryPolicy dictionaryPolicy)#

Sets the dictionary policy.

Must not be null.

public Builder withStatisticsFrequency( StatisticsFrequency statsGranularity )#

public ParquetWriterOptions build()#