Nvtext Tokenize#

group Tokenizing

Functions

std::unique_ptr<bpe_merge_pairs> load_merge_pairs( cudf::strings_column_view const &merge_pairs, cuda::stream_ref stream = cudf::get_default_stream(), cudf::memory_resources mr = cudf::get_current_device_resource_ref() )#

Create a nvtext::bpe_merge_pairs from a strings column.

The input column should contain a pair of strings per row separated by a single space. An incorrect format will result in undefined behavior.

Example:

merge_pairs = ["e n", "i t", "i s", "e s", "en t", "c e", "es t", "en ce", "t est", "s ent"]
mps = load_merge_pairs(merge_pairs)
// the mps object can be passed to the byte_pair_encoding API

The pairs are expected to be ordered by their rank relative to each other. A pair in an earlier row has priority over any pairs in later rows. If a pair appears more than once, only its first (lowest ranked) row is used.

Throws:

cudf::logic_error – if merge_pairs is empty or contains nulls

Parameters:

merge_pairs – Column containing the merge pairs
stream – CUDA stream used for device memory operations and kernel launches
mr – Memory resources used for temporary allocations and the returned object’s device memory

Returns:

A nvtext::bpe_merge_pairs object

std::unique_ptr<cudf::column> byte_pair_encoding( cudf::strings_column_view const &input, bpe_merge_pairs const &merges_pairs, cudf::string_scalar const &separator = cudf::string_scalar(" "), cuda::stream_ref stream = cudf::get_default_stream(), cudf::memory_resources mr = cudf::get_current_device_resource_ref() )#

Byte pair encode the input strings.

Each string starts as a sequence of individual UTF-8 characters (tokens). The adjacent pair of tokens with the lowest rank in the merge_pairs table is merged (all non-overlapping occurrences, left to right) and this repeats until no adjacent pair appears in the table. The separator is then inserted between the remaining tokens to build the output string.

Characters such as spaces that do not appear in the table remain individual tokens and so are also surrounded by separators. Tables such as GPT-2’s expect the input to be byte-level mapped (e.g. space as Ġ) before encoding.

merge_pairs = ["e n", "i t", "i s", "e s", "en t", "c e", "es t", "en ce",
               "t h", "h i", "th is", "t est", "s i", "s ent"]
mps = load_merge_pairs(merge_pairs)
input = ["test sentence", "thisis test"]
result = byte_pair_encoding(input, mps)
result is now ["test   sent ence", "this is   test"]

Null rows result in null output rows.

Throws:

cudf::logic_error – if separator is invalid or is not a single byte

Parameters:

input – Strings to encode.
merges_pairs – Created by a call to nvtext::load_merge_pairs.
separator – String used to build the output after encoding. Default is a space.
stream – CUDA stream used for device memory operations and kernel launches
mr – Memory resources used for temporary allocations and the returned column

Returns:

An encoded column of strings.

std::unique_ptr<cudf::column> tokenize( cudf::strings_column_view const &input, cudf::string_scalar const &delimiter = cudf::string_scalar{""}, cuda::stream_ref stream = cudf::get_default_stream(), cudf::memory_resources mr = cudf::get_current_device_resource_ref() )#

Returns a single column of strings by tokenizing the input strings column using the provided characters as delimiters.

The delimiter may be zero or more characters. If the delimiter is empty, whitespace (character code-point <= ‘ ‘) is used for identifying tokens. Also, any consecutive delimiters found in a string are ignored. This means only non-empty tokens are returned.

Tokens are found by locating delimiter(s) starting at the beginning of each string. As each string is tokenized, the tokens are appended using input column row order to build the output column. That is, tokens found in input row[i] will be placed in the output column directly before tokens found in input row[i+1].

Example:
s = ["a", "b c", "d  e f "]
t = tokenize(s)
t is now ["a", "b", "c", "d", "e", "f"]

All null row entries are ignored and the output contains all valid rows.

Parameters:

input – Strings column to tokenize
delimiter – UTF-8 characters used to separate each string into tokens. The default of empty string will separate tokens using whitespace.
stream – CUDA stream used for device memory operations and kernel launches
mr – Memory resources used for temporary allocations and the returned column

Returns:

New strings columns of tokens

std::unique_ptr<cudf::column> tokenize( cudf::strings_column_view const &input, cudf::strings_column_view const &delimiters, cuda::stream_ref stream = cudf::get_default_stream(), cudf::memory_resources mr = cudf::get_current_device_resource_ref() )#

Returns a single column of strings by tokenizing the input strings column using multiple strings as delimiters.

Tokens are found by locating delimiter(s) starting at the beginning of each string. Any consecutive delimiters found in a string are ignored. This means only non-empty tokens are returned.

As each string is tokenized, the tokens are appended using input column row order to build the output column. That is, tokens found in input row[i] will be placed in the output column directly before tokens found in input row[i+1].

Example:
s = ["a", "b c", "d.e:f;"]
d = [".", ":", ";"]
t = tokenize(s,d)
t is now ["a", "b c", "d", "e", "f"]

All null row entries are ignored and the output contains all valid rows.

Throws:

cudf::logic_error – if the delimiters column is empty or contains nulls.

Parameters:

input – Strings column to tokenize
delimiters – Strings used to separate individual strings into tokens
stream – CUDA stream used for device memory operations and kernel launches
mr – Memory resources used for temporary allocations and the returned column

Returns:

New strings columns of tokens

std::unique_ptr<cudf::column> count_tokens( cudf::strings_column_view const &input, cudf::string_scalar const &delimiter = cudf::string_scalar{""}, cuda::stream_ref stream = cudf::get_default_stream(), cudf::memory_resources mr = cudf::get_current_device_resource_ref() )#

Returns the number of tokens in each string of a strings column.

Example:
s = ["a", "b c", " ", "d e f"]
t = count_tokens(s)
t is now [1, 2, 0, 3]

All null row entries are ignored and the output contains all valid rows. The number of tokens for a null element is set to 0 in the output column.

Parameters:

input – Strings column to count tokens
delimiter – Strings used to separate each string into tokens. The default of empty string will separate tokens using whitespace.
stream – CUDA stream used for device memory operations and kernel launches
mr – Memory resources used for temporary allocations and the returned column

Returns:

New column of token counts

std::unique_ptr<cudf::column> count_tokens( cudf::strings_column_view const &input, cudf::strings_column_view const &delimiters, cuda::stream_ref stream = cudf::get_default_stream(), cudf::memory_resources mr = cudf::get_current_device_resource_ref() )#

Returns the number of tokens in each string of a strings column by using multiple strings delimiters to identify tokens in each string.

Also, any consecutive delimiters found in a string are ignored. This means that only empty strings or null rows will result in a token count of 0.

Example:
s = ["a", "b c", "d.e:f;"]
d = [".", ":", ";"]
t = count_tokens(s,d)
t is now [1, 1, 3]

All null row entries are ignored and the output contains all valid rows. The number of tokens for a null element is set to 0 in the output column.

Throws:

cudf::logic_error – if the delimiters column is empty or contains nulls

Parameters:

input – Strings column to count tokens
delimiters – Strings used to separate each string into tokens
stream – CUDA stream used for device memory operations and kernel launches
mr – Memory resources used for temporary allocations and the returned column

Returns:

New column of token counts

std::unique_ptr<cudf::column> character_tokenize( cudf::strings_column_view const &input, cuda::stream_ref stream = cudf::get_default_stream(), cudf::memory_resources mr = cudf::get_current_device_resource_ref() )#

Returns a single column of strings by converting each character to a string.

Each string is converted to multiple strings – one for each character. Note that a character maybe more than one byte.

Example:
s = ["hello world", null, "goodbye"]
t = character_tokenize(s)
t is now ["h","e","l","l","o"," ","w","o","r","l","d","g","o","o","d","b","y","e"]

Throws:

std::invalid_argument – if input contains nulls
std::overflow_error – if the output would produce more than max size_type rows

Parameters:

input – Strings column to tokenize
stream – CUDA stream used for device memory operations and kernel launches
mr – Memory resources used for temporary allocations and the returned column

Returns:

New strings columns of tokens

std::unique_ptr<cudf::column> detokenize( cudf::strings_column_view const &input, cudf::column_view const &row_indices, cudf::string_scalar const &separator = cudf::string_scalar(" "), cuda::stream_ref stream = cudf::get_default_stream(), cudf::memory_resources mr = cudf::get_current_device_resource_ref() )#

Creates a strings column from a strings column of tokens and an associated column of row ids.

Multiple tokens from the input column may be combined into a single row (string) in the output column. The tokens are concatenated along with the separator string in the order in which they appear in the row_indices column.

Example:
s = ["hello", "world", "one", "two", "three"]
r = [0, 0, 1, 1, 1]
s1 = detokenize(s,r)
s1 is now ["hello world", "one two three"]
r = [0, 2, 1, 1, 0]
s2 = detokenize(s,r)
s2 is now ["hello three", "one two", "world"]

All null row entries are ignored and the output contains all valid rows. The values in row_indices are expected to have positive, sequential values without any missing row indices otherwise the output is undefined.

Throws:

cudf::logic_error – is separator is invalid
cudf::logic_error – if row_indices.size() != strings.size()
cudf::logic_error – if row_indices contains nulls

Parameters:

input – Strings column to detokenize
row_indices – The relative output row index assigned for each token in the input column
separator – String to append after concatenating each token to the proper output row
stream – CUDA stream used for device memory operations and kernel launches
mr – Memory resources used for temporary allocations and the returned column

Returns:

New strings columns of tokens

std::unique_ptr<tokenize_vocabulary> load_vocabulary( cudf::strings_column_view const &input, cuda::stream_ref stream = cudf::get_default_stream(), cudf::memory_resources mr = cudf::get_current_device_resource_ref() )#

Create a tokenize_vocabulary object from a strings column.

Token ids are the row indices within the vocabulary column. Each vocabulary entry is expected to be unique otherwise the behavior is undefined.

Throws:

cudf::logic_error – if vocabulary contains nulls or is empty

Parameters:

input – Strings for the vocabulary
stream – CUDA stream used for device memory operations and kernel launches
mr – Memory resources used for temporary allocations and the returned object’s device memory

Returns:

Object to be used with nvtext::tokenize_with_vocabulary

std::unique_ptr<cudf::column> tokenize_with_vocabulary( cudf::strings_column_view const &input, tokenize_vocabulary const &vocabulary, cudf::string_scalar const &delimiter, cudf::size_type default_id = -1, cuda::stream_ref stream = cudf::get_default_stream(), cudf::memory_resources mr = cudf::get_current_device_resource_ref() )#

Returns the token ids for the input string by looking up each delimited token in the given vocabulary.

Example:
s = ["hello world", "hello there", "there there world", "watch out world"]
v = load_vocabulary(["hello", "there", "world"])
r = tokenize_with_vocabulary(s,v)
r is now [[0,2], [0,1], [1,1,2], [-1,-1,2]]

Any null row entry results in a corresponding null entry in the output

Throws:

cudf::logic_error – if delimiter is invalid

Parameters:

input – Strings column to tokenize
vocabulary – Used to lookup tokens within input
delimiter – Used to identify tokens within input
default_id – The token id to be used for tokens not found in the vocabulary; Default is -1
stream – CUDA stream used for device memory operations and kernel launches
mr – Memory resources used for temporary allocations and the returned column

Returns:

Lists column of token ids

std::unique_ptr<wordpiece_vocabulary> load_wordpiece_vocabulary( cudf::strings_column_view const &input, cuda::stream_ref stream = cudf::get_default_stream(), cudf::memory_resources mr = cudf::get_current_device_resource_ref() )#

Create a tokenize_vocabulary object from a strings column.

Token ids are the row indices within the vocabulary column. Each vocabulary entry is expected to be unique otherwise the behavior is undefined.

Throws:

std::invalid_argument – if vocabulary contains nulls or is empty

Parameters:

input – Strings for the vocabulary
stream – CUDA stream used for device memory operations and kernel launches
mr – Memory resources used for temporary allocations and the returned object’s device memory

Returns:

Object to be used with nvtext::wordpiece_tokenize

std::unique_ptr<cudf::column> wordpiece_tokenize( cudf::strings_column_view const &input, wordpiece_vocabulary const &vocabulary, cudf::size_type max_words_per_row = 0, cuda::stream_ref stream = cudf::get_default_stream(), cudf::memory_resources mr = cudf::get_current_device_resource_ref() )#

Returns the token ids for the input string a wordpiece tokenizer algorithm with the given vocabulary.

Example:

vocabulary = ["[UNK]", "a", "have", "I", "new", "GP", "##U", "!"]
v = load_wordpiece_vocabulary(vocabulary)
input = ["I have a new GPU now !"]
t = wordpiece_tokenize(i,v)
t is now [[3, 2, 1, 4, 5, 6, 0, 7]]

The max_words_per_row also optionally limits the output by only processing a maximum number of words per row. Here a word is defined as consecutive sequence of characters delimited by space character(s).

Example:

vocabulary = ["[UNK]", "a", "have", "I", "new", "GP", "##U", "!"]
v = load_wordpiece_vocabulary(vocabulary)
input = ["I have  a new GPU now !"]
t4 = wordpiece_tokenize(i,v,4)
t4 is now [[3, 2, 1, 4]]
t5 = wordpiece_tokenize(i,v,5)
t5 is now [[3, 2, 1, 4, 5, 6]]

Any null row entry results in a corresponding null entry in the output.

Throws:

std::invalid_argument – If max_words_per_row is less than 0.

Parameters:

input – Strings column to tokenize
vocabulary – Used to lookup tokens within input
max_words_per_row – Maximum number of words to tokenize for each row. Default 0 tokenizes all words.
stream – CUDA stream used for device memory operations and kernel launches
mr – Memory resources used for temporary allocations and the returned column

Returns:

Lists column of token ids

struct bpe_merge_pairs#

#include <nvtext/byte_pair_encoding.hpp>

The table of merge pairs for the BPE encoder.

To create an instance, call nvtext::load_merge_pairs

Public Functions

bpe_merge_pairs( std::unique_ptr<cudf::column> &&input, cuda::stream_ref stream = cudf::get_default_stream(), cudf::memory_resources mr = cudf::get_current_device_resource_ref() )#

Construct a new bpe merge pairs object.

Each merge pair is stored as two consecutive rows of input: the left half at row 2i and the right half at row 2i+1 where i is the pair’s rank. A duplicate pair keeps its lowest rank.

Parameters:

input – Strings column of merge pair halves
stream – CUDA stream used for device memory operations and kernel launches.
mr – Memory resources used for temporary allocations and the returned object’s device memory

bpe_merge_pairs( cudf::strings_column_view const &input, cuda::stream_ref stream = cudf::get_default_stream(), cudf::memory_resources mr = cudf::get_current_device_resource_ref() )#

Construct a new bpe merge pairs object.

See nvtext::load_merge_pairs for the format of input.

Parameters:

input – Strings column with one merge pair per row
stream – CUDA stream used for device memory operations and kernel launches.
mr – Memory resources used for temporary allocations and the returned object’s device memory

struct tokenize_vocabulary#

#include <nvtext/tokenize.hpp>

Vocabulary object to be used with nvtext::tokenize_with_vocabulary.

Use nvtext::load_vocabulary to create this object.

Public Functions

tokenize_vocabulary( cudf::strings_column_view const &input, cuda::stream_ref stream = cudf::get_default_stream(), cudf::memory_resources mr = cudf::get_current_device_resource_ref() )#

Vocabulary object constructor.

Token ids are the row indices within the vocabulary column. Each vocabulary entry is expected to be unique otherwise the behavior is undefined.

Throws:

cudf::logic_error – if vocabulary contains nulls or is empty

Parameters:

input – Strings for the vocabulary
stream – CUDA stream used for device memory operations and kernel launches
mr – Memory resources used for temporary allocations and the returned object’s device memory

struct wordpiece_vocabulary#

#include <nvtext/wordpiece_tokenize.hpp>

Vocabulary object to be used with nvtext::wordpiece_tokenizer.

Use nvtext::load_wordpiece_vocabulary to create this object.

Public Functions

wordpiece_vocabulary( cudf::strings_column_view const &input, cuda::stream_ref stream = cudf::get_default_stream(), cudf::memory_resources mr = cudf::get_current_device_resource_ref() )#

Vocabulary object constructor.

Token ids are the row indices within the vocabulary column. Each vocabulary entry is expected to be unique otherwise the behavior is undefined.

Throws:

std::invalid_argument – if vocabulary contains nulls or is empty

Parameters:

input – Strings for the vocabulary
stream – CUDA stream used for device memory operations and kernel launches
mr – Memory resources used for temporary allocations and the returned object’s device memory