ColumnWriterOptions#

public class ColumnWriterOptions#

Per column settings for writing Parquet/ORC files.

The native also uses the same “column_in_metadata” for both Parquet and ORC.

Subclassed by ai.rapids.cudf.ColumnWriterOptions.ListColumnWriterOptions, ai.rapids.cudf.ColumnWriterOptions.StructColumnWriterOptions

Field Summary

`childColumnOptions`
`UNKNOWN_PRECISION`

Constructor Summary

`ColumnWriterOptions(String columnName)`
`ColumnWriterOptions(String columnName, boolean isNullable)`
`ColumnWriterOptions(String columnName, boolean isNullable, int parquetFieldId)`
`ColumnWriterOptions(String columnName, boolean isTimestampTypeInt96, int precision, boolean isNullable)`
`ColumnWriterOptions(String columnName, boolean isTimestampTypeInt96, int precision, boolean isNullable, int parquetFieldId)`

Method Summary

`getChildColumnOptions()`	Return the child columnOptions for this column.
`getColumnName()`	Return if the column can have null values.
`getFlatBooleans(boolean[] ret, ByteArrayProducer producer)`
`getFlatColumnNames(String[] ret)`
`getFlatInts(int[] ret, IntArrayProducer producer)`
`getParquetFieldId()`	Return the Parquet field ID for this column.
`getPrecision()`	Return the precision for this column.
`hasParquetFieldId()`	Returns true if a Parquet field ID was set on this column.
`isBinary()`	Returns true if this column is a binary (byte array) column in Parquet.
`isNullable()`	Return if the column can have null values.
`isTimestampTypeInt96()`	Returns true if the writer is expected to write timestamps in INT96.
`listBuilder(String name)`	Creates a ListBuilder for column called ‘name’.
`listBuilder(String name, boolean isNullable)`	Creates a ListBuilder for column called ‘name’.
`listBuilder(String name, boolean isNullable, int parquetFieldId)`	Creates a ListBuilder for column called ‘name’ with a Parquet field ID set on the list itself.
`mapColumn(String name, ColumnWriterOptions key, ColumnWriterOptions value)`	Deprecated. Add a Map Column to the schema.
`mapColumn(String name, ColumnWriterOptions key, ColumnWriterOptions value, Boolean isNullable)`	Add a Map Column to the schema.
`mapColumn(String name, ColumnWriterOptions key, ColumnWriterOptions value, boolean isNullable, int parquetFieldId)`	Add a Map Column to the schema with a Parquet field ID set on the outer list.
`structBuilder(String name)`	Creates a StructBuilder for column called ‘name’.
`structBuilder(String name, boolean isNullable)`	Creates a StructBuilder for column called ‘name’.
`structBuilder(String name, boolean isNullable, int parquetFieldId)`	Creates a StructBuilder for column called ‘name’.

Static Fields

public static int UNKNOWN_PRECISION#

Protected Fields

protected ColumnWriterOptions[] childColumnOptions#

Constructors

public ColumnWriterOptions( String columnName, boolean isTimestampTypeInt96, int precision, boolean isNullable )#

public ColumnWriterOptions( String columnName, boolean isTimestampTypeInt96, int precision, boolean isNullable, int parquetFieldId )#

public ColumnWriterOptions(String columnName, boolean isNullable)#

public ColumnWriterOptions( String columnName, boolean isNullable, int parquetFieldId )#

public ColumnWriterOptions(String columnName)#

Methods

public String getColumnName()#: Return if the column can have null values.

public boolean isNullable()#: Return if the column can have null values.

public int getPrecision()#: Return the precision for this column.

public boolean isTimestampTypeInt96()#: Returns true if the writer is expected to write timestamps in INT96.

public ColumnWriterOptions[] getChildColumnOptions()#: Return the child columnOptions for this column.

public boolean isBinary()#: Returns true if this column is a binary (byte array) column in Parquet.

public boolean hasParquetFieldId()#: Returns true if a Parquet field ID was set on this column.

public int getParquetFieldId()#

Return the Parquet field ID for this column.

The value is only meaningful when hasParquetFieldId() returns true.

Static Methods

public static ColumnWriterOptions mapColumn( String name, ColumnWriterOptions key, ColumnWriterOptions value ) Deprecated#

Add a Map Column to the schema.

Maps are List columns with a Struct named ‘key_value’ with a child named ‘key’ and a child named ‘value’. The caller of this method doesn’t need to worry about this as this method will take care of this without the knowledge of the caller.

Note: This method always returns a nullabe column, cannot return non-nullable column. Do not use this, use the next function with the parameter isNullable.

public static ColumnWriterOptions mapColumn( String name, ColumnWriterOptions key, ColumnWriterOptions value, Boolean isNullable )#

Add a Map Column to the schema.

Note: If this map column is a key of another map, should pass isNullable = false. e.g.: map1(map2(int, int), int) the map2 should be non-nullable.

Parameters:: isNullable – is the returned map nullable.

public static ColumnWriterOptions mapColumn( String name, ColumnWriterOptions key, ColumnWriterOptions value, boolean isNullable, int parquetFieldId )#

Add a Map Column to the schema with a Parquet field ID set on the outer list.

Parameters:

isNullable – is the returned map nullable. Primitive boolean (not Boolean) so a null can’t slip through and trigger an unexpected NullPointerException when forwarded to listBuilder(String, boolean, int).
parquetFieldId – the Parquet field ID to attach to the outer (list) column.

public static ListBuilder listBuilder(String name)#: Creates a ListBuilder for column called ‘name’.

public static ListBuilder listBuilder(String name, boolean isNullable)#: Creates a ListBuilder for column called ‘name’.

public static ListBuilder listBuilder( String name, boolean isNullable, int parquetFieldId )#: Creates a ListBuilder for column called ‘name’ with a Parquet field ID set on the list itself.

public static StructBuilder structBuilder( String name, boolean isNullable )#: Creates a StructBuilder for column called ‘name’.

public static StructBuilder structBuilder( String name, boolean isNullable, int parquetFieldId )#: Creates a StructBuilder for column called ‘name’.

public static StructBuilder structBuilder(String name)#: Creates a StructBuilder for column called ‘name’.

Protected Methods

protected boolean[] getFlatBooleans( boolean[] ret, ByteArrayProducer producer )#

protected int[] getFlatInts(int[] ret, IntArrayProducer producer)#

protected String[] getFlatColumnNames(String[] ret)#

protected abstract static class AbstractStructBuilder<T extends AbstractStructBuilder, V extends ColumnWriterOptions> extends NestedBuilder<T, V>#

Subclassed by ai.rapids.cudf.CompressionMetadataWriterOptions.Builder< T extends Builder, V extends CompressionMetadataWriterOptions >

Constructor Summary

`AbstractStructBuilder()`
`AbstractStructBuilder(String name, boolean isNullable)`	Builder specific to build a Struct meta.
`AbstractStructBuilder(String name, boolean isNullable, int parquetFieldId)`

Fields inherited from class ai.rapids.cudf.ColumnWriterOptions.NestedBuilder: children, hasParquetFieldId, isNullable, name, parquetFieldId

Methods inherited from class ai.rapids.cudf.ColumnWriterOptions.NestedBuilder: build, withBinary, withBinaryColumn, withColumn, withColumns, withDecimal, withDecimalColumn, withListColumn, withMapColumn, withNonNullableColumns, withNullableColumns, withNullableDecimalColumn, withNullableTimestampColumn, withStructColumn, withTimestamp, withTimestampColumn

Constructors

public AbstractStructBuilder(String name, boolean isNullable)#: Builder specific to build a Struct meta.

public AbstractStructBuilder( String name, boolean isNullable, int parquetFieldId )#

Protected Constructors

protected AbstractStructBuilder()#

public static abstract class NestedBuilder<T extends NestedBuilder, V extends ColumnWriterOptions>#

Field Summary

`children`
`hasParquetFieldId`
`isNullable`
`name`
`parquetFieldId`

Constructor Summary

`NestedBuilder()`
`NestedBuilder(String name, boolean isNullable)`	Builder specific to build a Struct meta.
`NestedBuilder(String name, boolean isNullable, int parquetFieldId)`

Method Summary

`build()`
`withBinary(String name, boolean isNullable)`
`withBinary(String name, boolean isNullable, int parquetFieldId)`
`withBinaryColumn(String name, boolean nullable)`	Set a binary child meta data.
`withBinaryColumn(String name, boolean nullable, int parquetFieldId)`	Set a binary child meta data.
`withColumn(String name, boolean isNullable)`
`withColumn(String name, boolean isNullable, int parquetFieldId)`
`withColumn(boolean nullable, String name, int parquetFieldId)`	Set a simple child meta data.
`withColumns(boolean nullable, String... names)`	Set a simple child meta data.
`withDecimal(String name, int precision, boolean isNullable)`
`withDecimal(String name, int precision, boolean isNullable, int parquetFieldId)`
`withDecimalColumn(String name, int precision)`	Set a Decimal child meta data.
`withDecimalColumn(String name, int precision, boolean nullable)`	Set a Decimal child meta data.
`withDecimalColumn(String name, int precision, boolean nullable, int parquetFieldId)`	Set a Decimal child meta data.
`withListColumn(ListColumnWriterOptions child)`	Set the list column meta.
`withMapColumn(ColumnWriterOptions child)`	Set the map column meta.
`withNonNullableColumns(String... names)`	Set column name.
`withNullableColumns(String... names)`	Set nullable column meta data.
`withNullableDecimalColumn(String name, int precision)`	Set a Decimal child meta data.
`withNullableTimestampColumn(String name, boolean isInt96)`	Set a timestamp child meta data.
`withStructColumn(StructColumnWriterOptions child)`	Set a child struct meta data.
`withTimestamp(String name, boolean isInt96, boolean isNullable)`
`withTimestamp(String name, boolean isInt96, boolean isNullable, int parquetFieldId)`
`withTimestampColumn(String name, boolean isInt96)`	Set a timestamp child meta data.
`withTimestampColumn(String name, boolean isInt96, boolean nullable)`	Set a timestamp child meta data.
`withTimestampColumn(String name, boolean isInt96, boolean nullable, int parquetFieldId)`	Set a timestamp child meta data.

Protected Fields

protected List<ColumnWriterOptions> children#

protected boolean isNullable#

protected String name#

protected boolean hasParquetFieldId#

protected int parquetFieldId#

Methods

public T withListColumn(ListColumnWriterOptions child)#

Set the list column meta.

Lists should have only one child in ColumnVector, but the metadata expects a LIST column to have two children and the first child to be the ColumnWriterOptions.DUMMY_CHILD. This is the current behavior in cudf and will change in future

Returns:: this for chaining.

public T withMapColumn(ColumnWriterOptions child)#

Set the map column meta.

Returns:: this for chaining.

public T withStructColumn(StructColumnWriterOptions child)#

Set a child struct meta data.

Returns:: this for chaining.

public T withNonNullableColumns(String... names)#: Set column name.

public T withNullableColumns(String... names)#: Set nullable column meta data.

public T withColumns(boolean nullable, String... names)#

Set a simple child meta data.

Returns:: this for chaining.

public T withColumn(boolean nullable, String name, int parquetFieldId)#

Set a simple child meta data.

Returns:: this for chaining.

public T withDecimalColumn( String name, int precision, boolean nullable )#

Set a Decimal child meta data.

Returns:: this for chaining.

public T withDecimalColumn( String name, int precision, boolean nullable, int parquetFieldId )#

Set a Decimal child meta data.

Returns:: this for chaining.

public T withNullableDecimalColumn(String name, int precision)#

Set a Decimal child meta data.

Returns:: this for chaining.

public T withDecimalColumn(String name, int precision)#

Set a Decimal child meta data.

Returns:: this for chaining.

public T withBinaryColumn( String name, boolean nullable, int parquetFieldId )#

Set a binary child meta data.

Returns:: this for chaining.

public T withBinaryColumn(String name, boolean nullable)#

Set a binary child meta data.

Returns:: this for chaining.

public T withTimestampColumn( String name, boolean isInt96, boolean nullable, int parquetFieldId )#

Set a timestamp child meta data.

Returns:: this for chaining.

public T withTimestampColumn( String name, boolean isInt96, boolean nullable )#

Set a timestamp child meta data.

Returns:: this for chaining.

public T withTimestampColumn(String name, boolean isInt96)#

Set a timestamp child meta data.

Returns:: this for chaining.

public T withNullableTimestampColumn(String name, boolean isInt96)#

Set a timestamp child meta data.

Returns:: this for chaining.

public abstract V build()#

Protected Constructors

protected NestedBuilder(String name, boolean isNullable)#: Builder specific to build a Struct meta.

protected NestedBuilder( String name, boolean isNullable, int parquetFieldId )#

protected NestedBuilder()#

Protected Methods

protected ColumnWriterOptions withColumn( String name, boolean isNullable )#

protected ColumnWriterOptions withColumn( String name, boolean isNullable, int parquetFieldId )#

protected ColumnWriterOptions withDecimal( String name, int precision, boolean isNullable )#

protected ColumnWriterOptions withDecimal( String name, int precision, boolean isNullable, int parquetFieldId )#

protected ColumnWriterOptions withTimestamp( String name, boolean isInt96, boolean isNullable )#

protected ColumnWriterOptions withTimestamp( String name, boolean isInt96, boolean isNullable, int parquetFieldId )#

protected ColumnWriterOptions withBinary( String name, boolean isNullable )#

protected ColumnWriterOptions withBinary( String name, boolean isNullable, int parquetFieldId )#

protected interface ByteArrayProducer#

Method Summary

apply(ColumnWriterOptions opt)

Methods

public boolean[] apply(ColumnWriterOptions opt)#

protected interface IntArrayProducer#

Method Summary

apply(ColumnWriterOptions opt)

Methods

public int[] apply(ColumnWriterOptions opt)#

public static class StructColumnWriterOptions extends ColumnWriterOptions #

Subclassed by ai.rapids.cudf.CompressionMetadataWriterOptions

Constructor Summary

StructColumnWriterOptions(AbstractStructBuilder builder)

Fields inherited from class ai.rapids.cudf.ColumnWriterOptions: childColumnOptions, UNKNOWN_PRECISION

Methods inherited from class ai.rapids.cudf.ColumnWriterOptions: getChildColumnOptions, getColumnName, getFlatBooleans, getFlatColumnNames, getFlatInts, getParquetFieldId, getPrecision, hasParquetFieldId, isBinary, isNullable, isTimestampTypeInt96, listBuilder, mapColumn, structBuilder

Protected Constructors

protected StructColumnWriterOptions(AbstractStructBuilder builder)#

public static class ListColumnWriterOptions extends ColumnWriterOptions #

Constructor Summary

ListColumnWriterOptions(ListBuilder builder)

Fields inherited from class ai.rapids.cudf.ColumnWriterOptions: childColumnOptions, UNKNOWN_PRECISION

Protected Constructors

protected ListColumnWriterOptions(ListBuilder builder)#

public static class StructBuilder extends AbstractStructBuilder<StructBuilder, StructColumnWriterOptions>#

Constructor Summary

`StructBuilder(String name, boolean isNullable)`
`StructBuilder(String name, boolean isNullable, int parquetFieldId)`

Method Summary

build()

Constructors

public StructBuilder(String name, boolean isNullable)#

public StructBuilder( String name, boolean isNullable, int parquetFieldId )#

Methods

public StructColumnWriterOptions build()#

public static class ListBuilder extends NestedBuilder<ListBuilder, ListColumnWriterOptions>#

Constructor Summary

`ListBuilder(String name, boolean isNullable)`
`ListBuilder(String name, boolean isNullable, int parquetFieldId)`

Method Summary

build()

Fields inherited from class ai.rapids.cudf.ColumnWriterOptions.NestedBuilder: children, hasParquetFieldId, isNullable, name, parquetFieldId

Methods inherited from class ai.rapids.cudf.ColumnWriterOptions.NestedBuilder: withBinary, withBinaryColumn, withColumn, withColumns, withDecimal, withDecimalColumn, withListColumn, withMapColumn, withNonNullableColumns, withNullableColumns, withNullableDecimalColumn, withNullableTimestampColumn, withStructColumn, withTimestamp, withTimestampColumn

Constructors

public ListBuilder(String name, boolean isNullable)#

public ListBuilder( String name, boolean isNullable, int parquetFieldId )#

Methods

public ListColumnWriterOptions build()#