Struct PreprocessNLPStageInterfaceProxy

struct PreprocessNLPStageInterfaceProxy

Interface proxy, used to insulate python bindings.

Public Static Functions

static std::shared_ptr<mrc::segment::Object<PreprocessNLPStage>> init(mrc::segment::Builder &builder, const std::string &name, std::string vocab_hash_file, uint32_t sequence_length, bool truncation, bool do_lower_case, bool add_special_token, int stride = -1, std::string column = "data")

Create and initialize a ProcessNLPStage, and return the result.

  • builder – : Pipeline context object reference

  • name – : Name of a stage reference

  • vocab_hash_file – : Path to hash file containing vocabulary of words with token-ids. This can be created from the raw vocabulary using the cudf.utils.hash_vocab_utils.hash_vocab function.

  • sequence_length – : Sequence Length to use (We add to special tokens for NER classification job).

  • truncation – : If set to true, strings will be truncated and padded to max_length. Each input string will result in exactly one output sequence. If set to false, there may be multiple output sequences when the max_length is smaller than generated tokens.

  • do_lower_case – : If set to true, original text will be lowercased before encoding.

  • add_special_token – : Whether or not to encode the sequences with the special tokens of the BERT classification model.

  • stride – : If truncation == False and the tokenized string is larger than max_length, the sequences containing the overflowing token-ids can contain duplicated token-ids from the main sequence. If max_length is equal to stride there are no duplicated-id tokens. If stride is 80% of max_length, 20% of the first sequence will be repeated on the second sequence and so on until the entire sentence is encoded.

  • column – : Name of the string column to operate on, defaults to “data”.



© Copyright 2023, NVIDIA. Last updated on Apr 11, 2023.