byte_pair_encode#
- class pylibcudf.nvtext.byte_pair_encode.BPEMergePairs#
The table of merge pairs for the BPE encoder.
For details, see
nvtext::load_merge_pairs().- Parameters:
- merge_pairsColumn
Strings column of merge pairs. Each row contains the two halves of a pair separated by a single space. Pairs are ranked by their row position; earlier rows have higher priority. A duplicate pair keeps its earliest (highest priority) rank.
- streamStream | None
CUDA stream on which to perform the operation.
- mrDeviceMemoryResource | None
Device memory resource used to allocate the merge pairs table.
- pylibcudf.nvtext.byte_pair_encode.byte_pair_encoding(
- Column input,
- BPEMergePairs merge_pairs,
- Scalar separator=None,
- stream: CudaStreamLike | None = None,
- DeviceMemoryResource mr=None,
Byte pair encode the input strings.
For details, see
nvtext::byte_pair_encoding().- Parameters:
- inputColumn
Strings to encode.
- merge_pairsBPEMergePairs
The ranked merge pairs used to encode the strings.
- separatorScalar | None
Single-byte string inserted between the encoded tokens. Default is a space.
- streamStream | None
CUDA stream on which to perform the operation.
- mrDeviceMemoryResource | None
Device memory resource used to allocate the returned column’s device memory.
- Returns:
- Column
An encoded column of strings.