BatchedCompressor#
- public abstract class BatchedCompressor#
Multi-buffer compressor.
Subclassed by ai.rapids.cudf.nvcomp.BatchedLZ4Compressor, ai.rapids.cudf.nvcomp.BatchedZstdCompressor
Constructor Summary
BatchedCompressor(long chunkSize, long maxOutputChunkSize, long maxIntermediateBufferSize)Construct a batched compressor instance.
Method Summary
Asynchronously compress a batch of buffers.
batchedCompressGetTempSize(long batchSize, long maxChunkSize, long totalSize, long stream)Get the temporary workspace size required to perform compression of an entire batch.
compress(BaseDeviceMemoryBuffer[] origInputs, Cuda.Stream stream)Compress a batch of buffers.
Constructors
- public BatchedCompressor(
- long chunkSize,
- long maxOutputChunkSize,
- long maxIntermediateBufferSize
Construct a batched compressor instance.
- Parameters:
chunkSize – maximum amount of uncompressed data to compress as a single chunk. Inputs larger than this will be compressed in multiple chunks.
maxIntermediateBufferSize – desired maximum size of intermediate device buffers used during compression.
Methods
- public DeviceMemoryBuffer[] compress(
- BaseDeviceMemoryBuffer[] origInputs,
- Cuda.Stream stream
Compress a batch of buffers.
The input buffers will be closed.
- Parameters:
origInputs – buffers to compress
stream – CUDA stream to use
- Returns:
compressed buffers corresponding to the input buffers
Protected Methods
- protected abstract long batchedCompressGetTempSize(
- long batchSize,
- long maxChunkSize,
- long totalSize,
- long stream
Get the temporary workspace size required to perform compression of an entire batch.
- Parameters:
batchSize – number of chunks in the batch
maxChunkSize – maximum size of an uncompressed chunk in bytes
totalSize – Upper bound on the total uncompressed size of all chunks
stream – CUDA stream to use
- Returns:
The size of required temporary workspace in bytes to compress the batch.
- protected abstract void batchedCompressAsync(
- long devInPtrs,
- long devInSizes,
- long chunkSize,
- long batchSize,
- long tempPtr,
- long tempSize,
- long devOutPtrs,
- long compressedSizesOutPtr,
- long stream
Asynchronously compress a batch of buffers.
Note that compressedSizesOutPtr must point to pinned memory for this operation to be asynchronous.
- Parameters:
devInPtrs – device address of uncompressed buffer addresses vector
devInSizes – device address of uncompressed buffer sizes vector
chunkSize – maximum size of an uncompressed chunk in bytes
batchSize – number of chunks in the batch
tempPtr – device address of the temporary workspace buffer
tempSize – size of the temporary workspace buffer in bytes
devOutPtrs – device address of output buffer addresses vector
compressedSizesOutPtr – device address where to write the sizes of the compressed data written to the corresponding output buffers. Must point to a buffer with at least 8 bytes of memory per output buffer in the batch.
stream – CUDA stream to use