Template Class PreprocessNLPStage
Defined in File preprocess_nlp.hpp
Base Type
public mrc::pymrc::PythonNode< std::shared_ptr< InputT >, std::shared_ptr< OutputT > >
-
template<typename InputT, typename OutputT>
class PreprocessNLPStage : public mrc::pymrc::PythonNode<std::shared_ptr<InputT>, std::shared_ptr<OutputT>> NLP input data for inference.
Public Types
Public Functions
-
PreprocessNLPStage(std::string vocab_hash_file, uint32_t sequence_length, bool truncation, bool do_lower_case, bool add_special_token, int stride = -1, std::string column = "data")
Construct a new Preprocess NLP Stage object.
- Parameters
vocab_hash_file – : Path to hash file containing vocabulary of words with token-ids. This can be created from the raw vocabulary using the
cudf.utils.hash_vocab_utils.hash_vocab
function.sequence_length – : Sequence Length to use (We add to special tokens for NER classification job).
truncation – : If set to true, strings will be truncated and padded to max_length. Each input string will result in exactly one output sequence. If set to false, there may be multiple output sequences when the max_length is smaller than generated tokens.
do_lower_case – : If set to true, original text will be lowercased before encoding.
add_special_token – : Whether or not to encode the sequences with the special tokens of the BERT classification model.
stride – : If
truncation
== False and the tokenized string is larger than max_length, the sequences containing the overflowing token-ids can contain duplicated token-ids from the main sequence. If max_length is equal to stride there are no duplicated-id tokens. If stride is 80% of max_length, 20% of the first sequence will be repeated on the second sequence and so on until the entire sentence is encoded.column – : Name of the string column to operate on, defaults to “data”.
- source_type_t on_data(sink_type_t x)
Called every time a message is passed to this stage
-
PreprocessNLPStage(std::string vocab_hash_file, uint32_t sequence_length, bool truncation, bool do_lower_case, bool add_special_token, int stride = -1, std::string column = "data")