JCudfSerialization#

public class JCudfSerialization#

Serialize and deserialize CUDF tables and columns using a custom format.

The goal of this is to provide a way to efficiently serialize and deserialize cudf data for distributed processing within a single application. Typically after a partition like operation has happened. It is not intended for inter-application communication or for long term storage of data, there are much better standards based formats for all of that.

The goal is to transfer data from a local GPU to a remote GPU as quickly and efficiently as possible using build in java communication channels. There is no guarantee of compatibility between different releases of CUDF. This is to allow us to adapt if internal memory layouts and formats change.

This version optimizes for reduced memory transfers, and as such will try to do the fewest number of transfers possible when putting the data back onto the GPU. This means that it will slice a single large memory buffer into smaller buffers used by the resulting ColumnVectors. The downside of this is that generally none of the memory can be released until all of the ColumnVectors are closed. It is assumed that this will not be a problem because for processing efficiency after the data is transferred it will likely be combined with other similar batches from other processes into a single larger buffer.

Method Summary

`concatToContiguousTable(SerializedTableHeader[] headers, HostMemoryBuffer[] dataBuffers)`	Concatenate multiple tables in host memory into a contiguous table in device memory.
`concatToHostBuffer(SerializedTableHeader[] headers, HostMemoryBuffer[] dataBuffers)`
`concatToHostBuffer(SerializedTableHeader[] headers, HostMemoryBuffer[] dataBuffers, HostMemoryAllocator hostMemoryAllocator)`	Concatenate multiple tables in host memory into a single host table buffer.
`getSerializedSizeInBytes(HostColumnVector[] columns, long rowOffset, long numRows)`	Get the size in bytes needed to serialize the given data.
`readAndConcat(SerializedTableHeader[] headers, HostMemoryBuffer[] dataBuffers)`
`readTableFrom(InputStream in)`
`readTableFrom(InputStream in, HostMemoryAllocator hostMemoryAllocator)`	Read a serialize table from the given InputStream.
`readTableFrom(SerializedTableHeader header, HostMemoryBuffer hostBuffer)`
`readTableIntoBuffer(InputStream in, SerializedTableHeader header, HostMemoryBuffer buffer)`	After reading a header for a table read the data portion into a host side buffer.
`unpackHostColumnVectors(SerializedTableHeader header, HostMemoryBuffer hostBuffer)`	Deserialize a serialized contiguous table into an array of host columns.
`writeConcatedStream(SerializedTableHeader[] headers, HostMemoryBuffer[] dataBuffers, OutputStream out)`	Take the data from multiple batches stored in the parsed headers and the dataBuffer and write it out to out as if it were a single buffer.
`writeRowsToStream(OutputStream out, long numRows)`	Write a rowcount only header to the output stream in a case where a columnar batch with no columns but a non zero row count is received.
`writeToStream(ColumnVector[] columns, OutputStream out, long rowOffset, long numRows)`	Write all or part of a set of columns out in an internal format.
`writeToStream(HostColumnVector[] columns, OutputStream out, long rowOffset, long numRows)`	Write all or part of a set of columns out in an internal format.
`writeToStream(Table t, OutputStream out, long rowOffset, long numRows)`	Write all or part of a table out in an internal format.

Static Methods

public static long getSerializedSizeInBytes( HostColumnVector[] columns, long rowOffset, long numRows )#

Get the size in bytes needed to serialize the given data.

The columns should be in host memory before calling this.

Parameters:

columns – columns to be serialized.
rowOffset – the first row to serialize.
numRows – the number of rows to serialize.

Returns:

the size in bytes needed to serialize the data including the header.

public static void writeToStream( Table t, OutputStream out, long rowOffset, long numRows ) throws IOException #

Write all or part of a table out in an internal format.

Parameters:

t – the table to be written.
out – the stream to write the serialized table out to.
rowOffset – the first row to write out.
numRows – the number of rows to write out.

public static void writeToStream( ColumnVector[] columns, OutputStream out, long rowOffset, long numRows ) throws IOException #

Write all or part of a set of columns out in an internal format.

Parameters:

columns – the columns to be written.
out – the stream to write the serialized table out to.
rowOffset – the first row to write out.
numRows – the number of rows to write out.

public static void writeToStream( HostColumnVector[] columns, OutputStream out, long rowOffset, long numRows ) throws IOException #

Write all or part of a set of columns out in an internal format.

Parameters:

columns – the columns to be written.
out – the stream to write the serialized table out to.
rowOffset – the first row to write out.
numRows – the number of rows to write out.

public static void writeRowsToStream( OutputStream out, long numRows ) throws IOException #

Write a rowcount only header to the output stream in a case where a columnar batch with no columns but a non zero row count is received.

Parameters:

out – the stream to write the serialized table out to.
numRows – the number of rows to write out.

public static void writeConcatedStream( SerializedTableHeader[] headers, HostMemoryBuffer[] dataBuffers, OutputStream out ) throws IOException #

Take the data from multiple batches stored in the parsed headers and the dataBuffer and write it out to out as if it were a single buffer.

Parameters:

headers – the headers parsed from multiple streams.
dataBuffers – an array of buffers that hold the data, one per header.
out – what to write the data out to.

Throws:

IOException – on any error.

public static Table readAndConcat( SerializedTableHeader[] headers, HostMemoryBuffer[] dataBuffers ) throws IOException #

public static ContiguousTable concatToContiguousTable( SerializedTableHeader[] headers, HostMemoryBuffer[] dataBuffers ) throws IOException #

Concatenate multiple tables in host memory into a contiguous table in device memory.

Parameters:

headers – table headers corresponding to the host table buffers
dataBuffers – host table buffer for each input table to be concatenated

Returns:

contiguous table in device memory

public static HostConcatResult concatToHostBuffer( SerializedTableHeader[] headers, HostMemoryBuffer[] dataBuffers, HostMemoryAllocator hostMemoryAllocator ) throws IOException #

Concatenate multiple tables in host memory into a single host table buffer.

Parameters:

headers – table headers corresponding to the host table buffers
dataBuffers – host table buffer for each input table to be concatenated
hostMemoryAllocator – allocator for host memory buffers

Returns:

host table header and buffer

public static HostConcatResult concatToHostBuffer( SerializedTableHeader[] headers, HostMemoryBuffer[] dataBuffers ) throws IOException #

public static HostColumnVector[] unpackHostColumnVectors( SerializedTableHeader header, HostMemoryBuffer hostBuffer )#

Deserialize a serialized contiguous table into an array of host columns.

Parameters:

header – serialized table header
hostBuffer – buffer containing the data for all columns in the serialized table

Returns:

array of host columns representing the data from the serialized table

public static void readTableIntoBuffer( InputStream in, SerializedTableHeader header, HostMemoryBuffer buffer ) throws IOException #

After reading a header for a table read the data portion into a host side buffer.

Parameters:

in – the stream to read the data from.
header – the header that finished just moments ago.
buffer – the buffer to write the data into. If there is not enough room to store the data in buffer it will not be read and header will still have dataRead set to false.

Throws:

IOException –

public static TableAndRowCountPair readTableFrom( SerializedTableHeader header, HostMemoryBuffer hostBuffer )#

public static TableAndRowCountPair readTableFrom( InputStream in, HostMemoryAllocator hostMemoryAllocator ) throws IOException #

Read a serialize table from the given InputStream.

Parameters:

in – the stream to read the table data from.
hostMemoryAllocator – a host memory allocator for an intermediate host memory buffer

Throws:

IOException – on any error.
EOFException – if the data stream ended unexpectedly in the middle of processing.

Returns:

the deserialized table in device memory, or null if the stream has no table to read from, an end of the stream at the very beginning.

public static TableAndRowCountPair readTableFrom( InputStream in ) throws IOException #

public static final class SerializedTableHeader#

Holds the metadata about a serialized table.

If this is being read from a stream isInitialized will return true if the metadata was read correctly from the stream. It will return false if an EOF was encountered at the beginning indicating that there was no data to be read.

Constructor Summary

`SerializedTableHeader(DataInputStream din)`
`SerializedTableHeader(int numRows)`	Constructor for a row-count only table (no columns).

Method Summary

`getColumnHeader(int columnIndex)`	Get the column header for the corresponding column index.
`getDataLen()`	Returns the size of a buffer needed to read data into the stream.
`getNumColumns()`	Returns the number of columns stored in this table.
`getNumRows()`	Returns the number of rows stored in this table.
`getSerializedHeaderSizeInBytes()`	Returns the number of bytes needed to serialize this table header.
`getTotalSerializedSizeInBytes()`	Returns the number of bytes needed to serialize this table header and the table data.
`wasDataRead()`	Set to true once data is successfully read from a stream by readTableIntoBuffer.
`wasInitialized()`	Returns true if the metadata for this table was read, else false indicating an EOF was encountered.
`writeTo(DataWriter dout)`

Constructors

public SerializedTableHeader(DataInputStream din) throws IOException #

public SerializedTableHeader(int numRows)#: Constructor for a row-count only table (no columns).

Methods

public SerializedColumnHeader getColumnHeader(int columnIndex)#: Get the column header for the corresponding column index.

public boolean wasDataRead()#

Set to true once data is successfully read from a stream by readTableIntoBuffer.

Returns:: true if data was read, else false.

public long getDataLen()#: Returns the size of a buffer needed to read data into the stream.

public int getNumRows()#: Returns the number of rows stored in this table.

public int getNumColumns()#: Returns the number of columns stored in this table.

public boolean wasInitialized()#: Returns true if the metadata for this table was read, else false indicating an EOF was encountered.

public long getSerializedHeaderSizeInBytes()#

Returns the number of bytes needed to serialize this table header.

Note that this is only the metadata for the table (i.e.: column types, row counts, etc.) and does not include the bytes needed to serialize the table data.

public long getTotalSerializedSizeInBytes()#: Returns the number of bytes needed to serialize this table header and the table data.

public void writeTo(DataWriter dout) throws IOException #

public static final class SerializedColumnHeader#

Holds the metadata about a serialized column.

Field Summary

`children`
`dtype`
`nullCount`
`rowCount`

Method Summary

`getChildren()`	Get the metadata for any child columns or null if there are no children.
`getNullCount()`	Get the null count of the column.
`getNumChildren()`	Get the number of child columns.
`getRowCount()`	Get the row count of the column.
`getSerializedHeaderSizeInBytes()`	Return the number of bytes needed to store this column header in serialized form.
`getType()`	Get the data type of the column.
`writeTo(DataWriter dout)`	Write this column header to the specified writer.

Fields

public final DType dtype#

public final long nullCount#

public final long rowCount#

public final SerializedColumnHeader[] children#

Methods

public DType getType()#: Get the data type of the column.

public long getRowCount()#: Get the row count of the column.

public long getNullCount()#: Get the null count of the column.

public SerializedColumnHeader[] getChildren()#: Get the metadata for any child columns or null if there are no children.

public int getNumChildren()#: Get the number of child columns.

public long getSerializedHeaderSizeInBytes()#: Return the number of bytes needed to store this column header in serialized form.

public void writeTo(DataWriter dout) throws IOException #: Write this column header to the specified writer.

public static final class HostConcatResult implements AutoCloseable #

Class to hold the header and buffer pair result from host-side concatenation.

Constructor Summary

HostConcatResult(SerializedTableHeader tableHeader, HostMemoryBuffer tableBuffer)

Method Summary

`close()`
`getHostBuffer()`
`getTableHeader()`
`toContiguousTable()`	Build a contiguous table in device memory from this host-concatenated result.

Constructors

public HostConcatResult( SerializedTableHeader tableHeader, HostMemoryBuffer tableBuffer )#

Methods

public SerializedTableHeader getTableHeader()#

public HostMemoryBuffer getHostBuffer()#

public ContiguousTable toContiguousTable()#: Build a contiguous table in device memory from this host-concatenated result.

public void close()#

public static final class TableAndRowCountPair implements Closeable #

Holds the result of deserializing a table.

Constructor Summary

TableAndRowCountPair(int numRows, ContiguousTable table)

Method Summary

`close()`
`getContiguousTable()`	Get the ContiguousTable that was deserialized or null if there was no data (e.g.: rows without columns).
`getNumRows()`	Get the number of rows that were deserialized.
`getTable()`	Get the Table that was deserialized or null if there was no data (e.g.: rows without columns).

Constructors

public TableAndRowCountPair(int numRows, ContiguousTable table)#

Methods

public void close()#

public int getNumRows()#: Get the number of rows that were deserialized.

public Table getTable()#

Get the Table that was deserialized or null if there was no data (e.g.: rows without columns).

NOTE: Ownership of the table is not transferred by this method. The table is still owned by this instance and will be closed when this instance is closed.

public ContiguousTable getContiguousTable()#

Get the ContiguousTable that was deserialized or null if there was no data (e.g.: rows without columns).

NOTE: Ownership of the contiguous table is not transferred by this method. The contiguous table is still owned by this instance and will be closed when this instance is closed.