C++ API

This is the C++ API reference for the NVIDIA® nvCOMP library.

Generic

namespace nvcomp

Enums

enum class nvcompFormatType_t : uint8_t

nvCOMP supported compression formats.

Values:

enumerator LZ4

enumerator Snappy

enumerator ANS

enumerator GDeflate

enumerator Cascaded

enumerator Bitcomp

enumerator Zstd

enumerator Deflate

enumerator Gzip

enumerator NotSupportedError

class NVCompException : public runtime_error

#include <nvcomp.hpp>

The top-level exception thrown by nvcomp C++ methods.

Public Functions

inline NVCompException(nvcompStatus_t err, const std::string &msg)

Create a new NVCompException.

Parameters:

• err – [in] The error associated with the exception.

• msg – [in] The error message.

inline nvcompStatus_t get_error() const noexcept

Private Members

nvcompStatus_t m_err

namespace nvcomp

Typedefs

using AllocFn_t = std::function<void*(size_t)>

Custom allocator function type that receives a size in bytes to allocate in device-accessible memory.

using DeAllocFn_t = std::function<void(void*, size_t)>

Custom deallocation function type that receives a memory pointer and size in bytes to deallocate.

Enums

enum class BitstreamKind

Enumeration that defines how a buffer gets compressed by an nvCOMP manager.

Values:

enumerator NVCOMP_NATIVE

Each input buffer is chunked according to manager setting and compressed in parallel. Allows computation of checksums. Adds custom header with nvCOMP metadata at the beginning of the compressed data.

enumerator RAW

Compresses input data as is, just using underlying compression algorithm. Does not add header with nvCOMP metadata.

enumerator WITH_UNCOMPRESSED_SIZE

Similar to RAW, but adds custom header with just uncompressed size at the beginning of the compressed data.

enum ChecksumPolicy

Enumeration that defines the checksum policy used by an nvCOMP manager.

Values:

enumerator NoComputeNoVerify

During compression, do not compute checksums. During decompression, do not verify checksums.

enumerator ComputeAndNoVerify

During compression, compute checksums. During decompression, do not attempt to verify checksums.

enumerator NoComputeAndVerifyIfPresent

During compression, do not compute checksums. During decompression, verify checksums if they were included.

enumerator ComputeAndVerifyIfPresent

During compression, compute checksums. During decompression, verify checksums if they were included.

enumerator ComputeAndVerify

During compression, compute checksums. During decompression, verify checksums. A runtime error will be thrown upon configure_decompression if checksums were not included in the compressed buffer.

struct CompressionConfig

#include <nvcompManager.hpp>

Configuration used to aggregate information about the compression of a particular buffer.

Public Functions

CompressionConfig()

Default constructor that initializes members with default values.

Note

CompressionConfig should be retrieved via the configure_compression member function, and shouldn’t be constructed directly.

CompressionConfig(size_t uncompressed_buffer_size)

Construct the CompressionConfig using the uncompressed buffer size.

Note

CompressionConfig should be retrieved via the configure_compression member function, and shouldn’t be constructed directly.

nvcompStatus_t *get_status() const

Get the status of the overall compression.

Note

The data resides in host memory.

CompressionConfig(CompressionConfig &&other)

CompressionConfig(const CompressionConfig &other)

CompressionConfig &operator=(CompressionConfig &&other)

CompressionConfig &operator=(const CompressionConfig &other)

~CompressionConfig() noexcept

Public Members

std::shared_ptr<CompressionConfigImpl> impl

size_t uncompressed_buffer_size

size_t max_compressed_buffer_size

size_t num_chunks

bool compute_checksums

struct DecompressionConfig

#include <nvcompManager.hpp>

Configuration used to aggregate information about a particular decompression.

Public Functions

DecompressionConfig()

Default constructor that initializes members with default values.

Note

DecompressionConfig should be retrieved via the configure_decompression member function, and shouldn’t be constructed directly.

nvcompStatus_t *get_status() const

Get the status of the overall decompression.

Note

The data resides in host memory.

DecompressionConfig(DecompressionConfig &&other)

DecompressionConfig(const DecompressionConfig &other)

DecompressionConfig &operator=(DecompressionConfig &&other)

DecompressionConfig &operator=(const DecompressionConfig &other)

~DecompressionConfig() noexcept

Public Members

std::shared_ptr<DecompressionConfigImpl> impl

size_t decomp_data_size

size_t num_chunks

bool checksums_present

struct nvcompManagerBase

#include <nvcompManager.hpp>

Abstract base class that defines the nvCOMP high-level interface.

Note

This base class should not be directly constructed. One should rely on the explicit manager constructors (e.g., ANSManager), or the create_manager factory function.

Public Functions

virtual CompressionConfig configure_compression(

const size_t uncomp_buffer_size,

) = 0

Configure the compression of a single buffer.

This routine computes the size of the required result buffer. The result config also contains the nvcompStatus_t* that allows error checking.

Parameters:

uncomp_buffer_size – [in] The uncompressed input data size (in bytes).

Returns:

CompressionConfig for the size provided.

virtual std::vector<CompressionConfig> configure_compression(

const std::vector<size_t> &uncomp_buffer_sizes,

) = 0

Configure the compression of a batch of buffers.

This routine computes the size of the required result buffer for each element of the batch. The result config also contains the nvcompStatus_t* that allows error checking.

Parameters:

uncomp_buffer_sizes – [in] The vector of uncompressed input data sizes (in bytes) for each element of the batch.

Returns:

A vector with CompressionConfig for each of the size provided.

virtual void compress(

const uint8_t *uncomp_buffer,

uint8_t *comp_buffer,

const CompressionConfig &comp_config,

size_t *comp_size = nullptr,

) = 0

Perform compression asynchronously for a single buffer.

Parameters:

• uncomp_buffer – [in] The uncompressed input data. (a pointer to device continuous memory for GPU-based compression and host-accessible memory for CPU-based compression)

• comp_buffer – [out] The location to output the compressed data to. (a pointer to device continuous memory for GPU-based compression and host-accessible memory for CPU-based compression) Size requirement is provided in CompressionConfig.

• comp_config – [in] Generated for the current uncomp_buffer with configure_compression.

• comp_size – [out] The location to output size in bytes after compression. (a pointer to a single size_t variable on device for GPU-based compression and host variable for CPU-based (de)compression) Optional when bitstream kind is NVCOMP_NATIVE.

virtual void compress(

const uint8_t *const *uncomp_buffers,

uint8_t *const *comp_buffers,

const std::vector<CompressionConfig> &comp_configs,

size_t *comp_sizes = nullptr,

) = 0

Perform compression asynchronously for a batch of buffers. Batch size is inferred from comp_configs size.

Parameters:

• uncomp_buffers – [in] The uncompressed input data. (a pointer to a host array of pointers to device continuous memory for GPU-based compression and host array of pointers to host-accessible memory for CPU-based compression)

• comp_buffers – [out] The location to output the compressed data to. (a pointer to a host array of pointers to device continuous memory for GPU-based compression and host array of pointers to host-accessible memory for CPU-based compression) Size requirement is provided in CompressionConfig.

• comp_configs – [in] Generated for the current uncomp_buffers with configure_compression.

• comp_sizes – [out] The location to output size in bytes after compression. (a pointer to a device array (or host array for CPU-based compression), with size enough to contain batch_size elements of type size_t) Optional when bitstream kind is NVCOMP_NATIVE.

virtual DecompressionConfig configure_decompression(

const uint8_t *comp_buffer,

const size_t *comp_size = nullptr,

) = 0

Configure the decompression for a single buffer using a compressed buffer.

Synchronizes the user stream.

• If bitstream kind is NVCOMP_NATIVE, it will parse the header in comp_buffer.

• If bitstream kind is RAW, it may be required (e.g for LZ4) to parse the whole comp_buffer, which could be significantly slower that other options.

• If bitstream kind is WITH_UNCOMPRESSED_SIZE, it will read the size from the beginning of the comp_buffer.

Parameters:

• comp_buffer – [in] The compressed input data. (a pointer to device continuous memory for GPU-based (de)compression and host-accessible memory for CPU-based (de)compression)

• comp_size – [in] Size of the compressed input data. This is required only for RAW format. (a pointer to device variable with compressed size for GPU-based decompression and host variable for CPU-based decompression)

Returns:

DecompressionConfig for the comp_buffer provided.

virtual std::vector<DecompressionConfig> configure_decompression(

const uint8_t *const *comp_buffers,

size_t batch_size,

const size_t *comp_sizes = nullptr,

) = 0

Configure the decompression for a batch of buffers using a compressed buffer.

Synchronizes the user stream.

• If bitstream kind is NVCOMP_NATIVE, it will parse the header in comp_buffers.

• If bitstream kind is RAW, it may be required (e.g for LZ4) to parse the whole comp_buffers, which could be significantly slower that other options.

• If bitstream kind is WITH_UNCOMPRESSED_SIZE, it will read the size from the beginning of the comp_buffers.

Parameters:

• comp_buffers – [in] The compressed input data. (a pointer to host array of pointers to device continuous memory for GPU-based decompression and host array of pointers to host-accessible memory for CPU-based decompression)

• batch_size – [in] The size of the batch.

• comp_sizes – [in] Size of the compressed input data. (a pointer to device array (or host array for CPU-based (de)compression)) This is required only for RAW format.

Returns:

A vector of DecompressionConfig for each of the comp_buffer provided.

virtual DecompressionConfig configure_decompression(

const CompressionConfig &comp_config,

) = 0

Configure the decompression for a single buffer using a CompressionConfig object.

Does not synchronize the user stream.

Parameters:

comp_config – [in] The config used to compress a buffer.

Returns:

DecompressionConfig based on compression config provided.

virtual std::vector<DecompressionConfig> configure_decompression(

const std::vector<CompressionConfig> &comp_configs,

) = 0

Configure the decompression for a batch of buffers using a CompressionConfig objects.

Does not synchronize the user stream.

Parameters:

comp_configs – [in] A vector of configs used to compress a batch of buffers.

Returns:

A vector of DecompressionConfig based on compression configs provided.

virtual void decompress(

uint8_t *decomp_buffer,

const uint8_t *comp_buffer,

const DecompressionConfig &decomp_config,

size_t *comp_size = nullptr,

) = 0

Perform decompression asynchronously of a single buffer.

Parameters:

• decomp_buffer – [out] The location to output the decompressed data to. (a pointer to device continuous memory for GPU-based decompression and host-accessible memory for CPU-based decompression) Size requirement is provided in DecompressionConfig.

• comp_buffer – [in] The compressed input data. (a pointer to device continuous memory for GPU-based decompression and host-accessible memory for CPU-based decompression)

• decomp_config – [in] Resulted from configure_decompression given this comp_buffer. Contains nvcompStatus_t* in host/device-accessible memory to allow error checking in a synchronous manner. Must not be freed by the user until the decompression is complete.

• comp_size – [in] The size of compressed input data passed. (a pointer to a single size_t variable on device (or on host for CPU-based decompression)) Optional when bitstream kind is NVCOMP_NATIVE.

virtual void decompress(

uint8_t *const *decomp_buffers,

const uint8_t *const *comp_buffers,

const std::vector<DecompressionConfig> &decomp_configs,

const size_t *comp_sizes = nullptr,

) = 0

Perform decompression asynchronously of a batch of buffers.

Parameters:

• decomp_buffers – [out] The location to output the decompressed data to. (a pointer to a host array of pointers to device continuous memory for GPU-based decompression and host array of pointers to host-accessible memory for CPU-based decompression) Size requirement is provided in DecompressionConfig.

• comp_buffers – [in] The compressed input data. (a pointer to a host array of pointers to device continuous memory with compressed data for GPU-based decompression or host array of pointers to host-accessible memory with compressed data for CPU-based decompression)

• decomp_configs – [in] Resulted from configure_decompression given this comp_buffers. Contains nvcompStatus_t* in host/device-accessible memory to allow error checking in a synchronous manner. Must not be freed by the user until the decompression is complete.

• comp_sizes – [in] The size of compressed input data passed. (a pointer to a device array (or host array for CPU-based decompression), with size enough to contain batch_size elements of type size_t) Optional when bitstream kind is NVCOMP_NATIVE.

virtual void set_scratch_allocators(

const AllocFn_t &alloc_fn,

const DeAllocFn_t &dealloc_fn,

) = 0

Allows the user to provide a function for allocating / deallocating memory.

The manager requires scratch memory to perform its operations. By default, it will use internal allocators which make use of cudaMallocAsync / cudaFreeAsync The user can override the allocation functions with this API. The required signatures are void* alloc_fn(size_t alloc_size) and void dealloc_fn(void* buffer, size_t alloc_size)

This API copies the allocation functions. The copied functions must be valid until either 1) deallocate_gpu_mem() is called or 2) the nvcompManager instance is destroyed

If a scratch buffer was previously allocated, it is first deallocated using the prior dealloc_fn (or cudaFreeAsync if one wasn’t previously provided)

Note

This API is not supported for CPU (de)compression and will throw an exception if called.

Parameters:

• alloc_fn – [in] The host function to use to alloc a new scratch result buffer.

• dealloc_fn – [in] The host function to use to dealloc a scratch result buffer.

virtual size_t get_compressed_output_size(

const uint8_t *comp_buffer,

) = 0

Computes the compressed output size (in bytes) of a given buffer that was compressed using nvcomp. The buffer can be in device or host memory.

Synchronously copies the size of the compressed buffer to a host variable for return.

Can only be used with NVCOMP_NATIVE bitstream kind.

To obtain compressed sizes one can also cudaMemcpy sizes from comp_sizes passed to compress function.

Parameters:

comp_buffer – [in] A pointer to the compressed input data. (a pointer to device continuous memory for GPU and host-accessible memory for CPU)

Returns:

Size of the compressed buffer.

virtual std::vector<size_t> get_compressed_output_size(

const uint8_t *const *comp_buffers,

size_t batch_size,

) = 0

Computes the compressed output size (in bytes) of a given batch of buffers that were compressed using nvcomp. The buffers can be in device or host memory.

Synchronously copies the sizes of the compressed buffers to host variables for return.

Can only be used with NVCOMP_NATIVE bitstream kind.

To obtain compressed sizes one can also cudaMemcpy sizes from comp_sizes passed to compress function.

Parameters:

comp_buffers – [in] An array of pointers to the input compressed buffers. (a pointer to a host array of pointers to device continuous memory for GPU compression and host array of pointers to host-accessible memory for CPU compression)

Returns:

A vector with sizes of each compressed buffer in the batch.

virtual void deallocate_gpu_mem() = 0

Frees any internal GPU memory used by the nvCOMP HLIF.

Returns this memory back through the deallocator if one was specified through set_scratch_allocators()

virtual ~nvcompManagerBase() noexcept = default

Destructor of the base class.

virtual void decompress(

uint8_t *const *decomp_buffers,

const uint8_t *const *comp_buffers,

const std::vector<DecompressionConfig> &decomp_configs,

const size_t *comp_sizes,

const size_t batch_count,

const uint8_t *const *host_comp_buffers,

) = 0

Perform decompression synchronously of a batch of buffers with host setup. For asynchronous decompression, use the API that does not take host_comp_buffers.

Parameters:

• decomp_buffers – [out] The location to output the decompressed data to. (a pointer to a host array of pointers to device continuous memory) Size requirement is provided in DecompressionConfig.

• comp_buffers – [in] The compressed input data. (a pointer to a host array of pointers to device continuous memory with uncompressed data)

• decomp_configs – [in] Resulted from configure_decompression given this comp_buffers. (Optional) Contains nvcompStatus_t* in host/device-accessible memory to allow error checking in a synchronous manner. When using this API with host_comp_buffers, and host setup, decomp_configs is optional and can be an empty vector. It cannot be nullptr. If provided, decomp_configs must not be freed until the decompression is complete.

• batch_count – [in] The number of buffers in the batch

• comp_sizes – [in] The size of compressed input data passed. (a pointer to a device array, with size enough to contain batch_size elements of type size_t) Optional when bitstream kind is NVCOMP_NATIVE.

• host_comp_buffers – [in] Compressed buffers residing in host accessible memory. A pointer to a host array of pointers to device continuous memory with compressed data. The compressed buffers must reside in host-accessible memory. If not, it would lead to undefined behavior.

virtual size_t get_decompressed_output_size(

const uint8_t *comp_buffer,

) = 0

Computes the decompressed output size (in bytes) of a given buffer that was compressed using nvcomp. The buffer can be in device or host memory.

Synchronously copies the sizes of the decompressed buffers to host variables for return.

Can only be used with NVCOMP_NATIVE bitstream kind.

Parameters:

comp_buffer – [in] The compressed input data (a pointer to the compressed buffer).

Returns:

Size of the decompressed buffer.

virtual std::vector<size_t> get_decompressed_output_size(

const uint8_t *const *comp_buffers,

size_t batch_size,

) = 0

Computes the decompressed output size (in bytes) of a given batch of buffers that were compressed using nvcomp. The buffers can be in device or host memory.

Synchronously copies the sizes of the decompressed buffers to host variables for return.

Can only be used with NVCOMP_NATIVE bitstream kind.

Parameters:

comp_buffers – [in] The compressed input data (an array of pointers to the compressed buffers).

Returns:

A vector with the decompressed sizes of each buffer in the batch.

namespace nvcomp

Functions

std::shared_ptr<nvcompManagerBase> create_manager(

const uint8_t *comp_buffer,

cudaStream_t stream = 0,

ChecksumPolicy checksum_policy = NoComputeNoVerify,

nvcompDecompressBackend_t backend = NVCOMP_DECOMPRESS_BACKEND_DEFAULT,

bool use_de_sort = false,

)

Construct a ManagerBase from a given compressed buffer.

Note

This operation synchronizes the host with the stream.

Parameters:

• comp_buffer – [in] The HLIF compressed buffer from which we intend to create the manager. The buffer can be either in host or device memory.

• stream – [in] The CUDA stream to perform the operation on.

• checksum_policy – [in] The checksum policy to use.

• backend – [in] The backend (CUDA / hardware decompress engine) to use.

• use_de_sort – [in] Whether to sort before hardware decompression for load balancing (for LZ4, Snappy, Deflate, and Gzip).

Returns:

The constructed manager instance.

nvcompFormatType_t get_compression_format(

const uint8_t *comp_buffer,

cudaStream_t stream = 0,

)

Returns the compression format for a given HLIF compressed buffer.

Note

This operation synchronizes the host with the stream.

Parameters:

• comp_buffer – [in] The HLIF compressed buffer from which we intend to create the manager. The buffer can be either in host or device memory.

• stream – [in] The CUDA stream to perform the operation on.

Returns:

Compression format.

namespace nvcomp

Functions

std::shared_ptr<nvcompManagerBase> create_cpu_manager(

const uint8_t *comp_buffer,

unsigned int num_threads = 0,

)

Construct an nvcompManagerBase instance from a given compressed buffer.

Parameters:

• comp_buffer – [in] The HLIF compressed buffer from which we intend to create the manager. The buffer needs to be host accessible.

• num_threads – [in] The number of threads to use.

Returns:

The constructed manager instance.

nvcompFormatType_t get_cpu_compression_format(

const uint8_t *comp_buffer,

)

Returns the compression format for a given HLIF compressed buffer.

Parameters:

comp_buffer – [in] The HLIF compressed buffer from which we intend to create the manager. The buffer needs to be host accessible.

Returns:

Compression format.

ANS

namespace nvcomp

struct ANSManager : public PimplManager

#include <ans.hpp>

High-level interface class for the ANS compressor.

Note

If user_stream is specified, the lifetime of the ANSManager instance must not extend beyond that of the user_stream.

Public Functions

ANSManager(

size_t uncomp_chunk_size,

const nvcompBatchedANSCompressOpts_t &compress_opts = nvcompBatchedANSCompressDefaultOpts,

const nvcompBatchedANSDecompressOpts_t &decompress_opts = nvcompBatchedANSDecompressDefaultOpts,

cudaStream_t user_stream = 0,

ChecksumPolicy checksum_policy = NoComputeNoVerify,

BitstreamKind bitstream_kind = BitstreamKind::NVCOMP_NATIVE,

)

Constructor of ANSManager.

Parameters:

• uncomp_chunk_size – [in] Internal chunk size used to partition the input data.

• compress_opts – [in] Compression options to use.

• decompress_opts – [in] Decompression options to use.

• user_stream – [in] The CUDA stream to operate on.

• checksum_policy – [in] The checksum policy to use during compression and decompression.

• bitstream_kind – [in] Setting to configure how the manager compresses the input.

~ANSManager() noexcept

Destructor of ANSManager.

Bitcomp

namespace nvcomp

struct BitcompManager : public PimplManager

#include <bitcomp.hpp>

High-level interface class for the Bitcomp compressor.

Note

If user_stream is specified, the lifetime of the BitcompManager instance must not extend beyond that of the user_stream.

Warning

Any uncompressed data buffer to be compressed MUST be a size that is a multiple of the data type size, else compression may crash or result in invalid output.

Public Functions

BitcompManager(

size_t uncomp_chunk_size,

const nvcompBatchedBitcompCompressOpts_t &compress_opts = nvcompBatchedBitcompCompressDefaultOpts,

const nvcompBatchedBitcompDecompressOpts_t &decompress_opts = nvcompBatchedBitcompDecompressDefaultOpts,

cudaStream_t user_stream = 0,

ChecksumPolicy checksum_policy = NoComputeNoVerify,

BitstreamKind bitstream_kind = BitstreamKind::NVCOMP_NATIVE,

)

Constructor of BitcompManager.

Parameters:

• uncomp_chunk_size – [in] Internal chunk size used to partition the input data.

• compress_opts – [in] Compression options to use.

• decompress_opts – [in] Decompression options to use.

• user_stream – [in] The CUDA stream to operate on.

• checksum_policy – [in] The checksum policy to use during compression and decompression.

• bitstream_kind – [in] Setting to configure how the manager compresses the input.

~BitcompManager() noexcept

Destructor of BitcompManager.

Cascaded

namespace nvcomp

struct CascadedManager : public PimplManager

#include <cascaded.hpp>

High-level interface class for the Cascaded compressor.

Note

If user_stream is specified, the lifetime of the CascadedManager instance must not extend beyond that of the user_stream.

Warning

Any uncompressed data buffer to be compressed MUST be a size that is a multiple of the data type size, else compression may crash or result in invalid output.

Public Functions

CascadedManager(

size_t uncomp_chunk_size,

const nvcompBatchedCascadedCompressOpts_t &compress_opts = nvcompBatchedCascadedCompressDefaultOpts,

const nvcompBatchedCascadedDecompressOpts_t &decompress_opts = nvcompBatchedCascadedDecompressDefaultOpts,

cudaStream_t user_stream = 0,

ChecksumPolicy checksum_policy = NoComputeNoVerify,

BitstreamKind bitstream_kind = BitstreamKind::NVCOMP_NATIVE,

)

Constructor of CascadedManager.

Parameters:

• uncomp_chunk_size – [in] Internal chunk size used to partition the input data.

• compress_opts – [in] Compression options to use.

• decompress_opts – [in] Decompression options to use.

• user_stream – [in] The CUDA stream to operate on.

• checksum_policy – [in] The checksum policy to use during compression and decompression.

• bitstream_kind – [in] Setting to configure how the manager compresses the input.

~CascadedManager() noexcept

Destructor of CascadedManager.

Deflate

namespace nvcomp

struct DeflateManager : public PimplManager

#include <deflate.hpp>

High-level interface class for the Deflate compressor.

Note

If user_stream is specified, the lifetime of the DeflateManager instance must not extend beyond that of the user_stream.

Public Functions

DeflateManager(

size_t uncomp_chunk_size,

const nvcompBatchedDeflateCompressOpts_t &compress_opts = nvcompBatchedDeflateCompressDefaultOpts,

const nvcompBatchedDeflateDecompressOpts_t &decompress_opts = nvcompBatchedDeflateDecompressDefaultOpts,

cudaStream_t user_stream = 0,

ChecksumPolicy checksum_policy = NoComputeNoVerify,

BitstreamKind bitstream_kind = BitstreamKind::NVCOMP_NATIVE,

)

Constructor of DeflateManager.

Parameters:

• uncomp_chunk_size – [in] Internal chunk size used to partition the input data.

• compress_opts – [in] Compression options to use.

• decompress_opts – [in] Decompression options to use.

• user_stream – [in] The CUDA stream to operate on.

• checksum_policy – [in] The checksum policy to use during compression and decompression.

• bitstream_kind – [in] Setting to configure how the manager compresses the input.

~DeflateManager() noexcept

Destructor of DeflateManager.

GDeflate

namespace nvcomp

struct GdeflateManager : public PimplManager

#include <gdeflate.hpp>

High-level interface class for the GDeflate compressor.

Note

If user_stream is specified, the lifetime of the GdeflateManager instance must not extend beyond that of the user_stream.

Public Functions

GdeflateManager(

size_t uncomp_chunk_size,

const nvcompBatchedGdeflateCompressOpts_t &compress_opts = nvcompBatchedGdeflateCompressDefaultOpts,

const nvcompBatchedGdeflateDecompressOpts_t &decompress_opts = nvcompBatchedGdeflateDecompressDefaultOpts,

cudaStream_t user_stream = 0,

ChecksumPolicy checksum_policy = NoComputeNoVerify,

BitstreamKind bitstream_kind = BitstreamKind::NVCOMP_NATIVE,

)

Constructor of GdeflateManager.

Parameters:

• uncomp_chunk_size – [in] Internal chunk size used to partition the input data.

• compress_opts – [in] Compression options to use.

• decompress_opts – [in] Decompression options to use.

• user_stream – [in] The CUDA stream to operate on.

• checksum_policy – [in] The checksum policy to use during compression and decompression.

• bitstream_kind – [in] Setting to configure how the manager compresses the input.

~GdeflateManager() noexcept

Destructor of GdeflateManager.

LZ4

This section documents the GPU LZ4 manager API (lz4.hpp) and the host-side LZ4CPUManager (lz4_cpu.hpp). For general background on the LZ4 format, see LZ4 Compression.

LZ4Manager (GPU-based (de)compressor)

namespace nvcomp

struct LZ4Manager : public PimplManager

#include <lz4.hpp>

High-level interface class for the LZ4 compressor.

Note

If user_stream is specified, the lifetime of the LZ4Manager must not extend beyond that of the user_stream.

Warning

Any uncompressed data buffer to be compressed MUST be a size that is a multiple of the data type size, else compression may crash or result in invalid output.

Public Functions

LZ4Manager(

size_t uncomp_chunk_size,

const nvcompBatchedLZ4CompressOpts_t &compress_opts = nvcompBatchedLZ4CompressDefaultOpts,

const nvcompBatchedLZ4DecompressOpts_t &decompress_opts = nvcompBatchedLZ4DecompressDefaultOpts,

cudaStream_t user_stream = 0,

ChecksumPolicy checksum_policy = NoComputeNoVerify,

BitstreamKind bitstream_kind = BitstreamKind::NVCOMP_NATIVE,

)

Constructor of LZ4Manager.

Parameters:

• uncomp_chunk_size – [in] Internal chunk size used to partition the input data.

• compress_opts – [in] Compression options to use.

• decompress_opts – [in] Decompression options to use.

• user_stream – [in] The CUDA stream to operate on.

• checksum_policy – [in] The checksum policy to use during compression and decompression.

• bitstream_kind – [in] Setting to configure how the manager compresses the input.

~LZ4Manager() noexcept

Destructor of LZ4Manager.

LZ4CPUManager (CPU-based (de)compressor)

The C++ type nvcomp::LZ4CPUManager (API reference below in lz4_cpu.hpp) provides a host-side high-level (HLIF) manager for LZ4 compression and decompression. For both directions, this manager uses the open-source LZ4 C library as its implementation backend. That is the same family of APIs as the upstream LZ4 project (for example the lz4.h / lz4hc.h interfaces), not the GPU LZ4 codec in lz4.hpp above.

When using the CPU manager factory (create_cpu_manager in nvcomp/nvcompCPUManagerFactory.hpp), selecting the LZ4 format yields an LZ4CPUManager built on that LZ4 library stack.

namespace nvcomp

struct LZ4CPUManager : public PimplManager

#include <lz4_cpu.hpp>

CPU-side high-level interface for LZ4 compression/decompression (HLIF format).

Public Functions

LZ4CPUManager(

size_t uncomp_chunk_size = 65536,

int compression_level = 9,

unsigned int num_threads = 0,

)

Constructor for LZ4CPUManager.

Note

CPU compression does not support calculating or verifying checksums and will skip the checksum data during decompression if it is provided in the HLIF compressed buffer. In addition, only NVCOMP_NATIVE bitstream kind is supported for CPU compression and decompression.

Parameters:

• uncomp_chunk_size – The uncompressed chunk size. Default is 65536.

• compression_level – The compression level. Default is 9.

• num_threads – The number of threads to use. Default is 0 (use maximum available hardware concurrency threads).

~LZ4CPUManager() noexcept

Destructor for LZ4CPUManager.

Snappy

namespace nvcomp

struct SnappyManager : public PimplManager

#include <snappy.hpp>

High-level interface class for the Snappy compressor.

Note

If user_stream is specified, the lifetime of the SnappyManager instance must not extend beyond that of the user_stream.

Public Functions

SnappyManager(

size_t uncomp_chunk_size,

const nvcompBatchedSnappyCompressOpts_t &compress_opts = nvcompBatchedSnappyCompressDefaultOpts,

const nvcompBatchedSnappyDecompressOpts_t &decompress_opts = nvcompBatchedSnappyDecompressDefaultOpts,

cudaStream_t user_stream = 0,

ChecksumPolicy checksum_policy = NoComputeNoVerify,

BitstreamKind bitstream_kind = BitstreamKind::NVCOMP_NATIVE,

)

Constructor of SnappyManager.

Parameters:

• uncomp_chunk_size – [in] Internal chunk size used to partition the input data.

• compress_opts – [in] Compression options to use.

• decompress_opts – [in] Decompression options to use.

• user_stream – [in] The CUDA stream to operate on.

• checksum_policy – [in] The checksum policy to use during compression and decompression.

• bitstream_kind – [in] Setting to configure how the manager compresses the input.

~SnappyManager() noexcept

Destructor of SnappyManager.

ZSTD

namespace nvcomp

struct ZstdManager : public PimplManager

#include <zstd.hpp>

High-level interface class for the Zstd compressor.

Note

uncomp_chunk_size must be <= nvcompZstdCompressionMaxAllowedChunkSize. Use 64-128 KB for best performance.

Note

If user_stream is specified, the lifetime of the ZstdManager instance must not extend beyond that of the user_stream.

Public Functions

ZstdManager(

size_t uncomp_chunk_size,

const nvcompBatchedZstdCompressOpts_t &compress_opts = nvcompBatchedZstdCompressDefaultOpts,

const nvcompBatchedZstdDecompressOpts_t &decompress_opts = nvcompBatchedZstdDecompressDefaultOpts,

cudaStream_t user_stream = 0,

ChecksumPolicy checksum_policy = NoComputeNoVerify,

BitstreamKind bitstream_kind = BitstreamKind::NVCOMP_NATIVE,

)

Constructor of ZstdManager.

Parameters:

• uncomp_chunk_size – [in] Internal chunk size used to partition the input data.

• compress_opts – [in] Compression options to use.

• decompress_opts – [in] Decompression options to use.

• user_stream – [in] The CUDA stream to operate on.

• checksum_policy – [in] The checksum policy to use during compression and decompression.

• bitstream_kind – [in] Setting to configure how the manager compresses the input.

~ZstdManager() noexcept

Destructor of ZstdManager.