Joint Intent and Slot Classification

Joint Intent and Slot classification is a method for classifying an Intent and detecting all relevant Slots (Entities) for the Intent in a query. For example, in the query “What is the weather in Santa Clara tomorrow morning?”, we would like to classify the query as a weather Intent, detect Santa Clara as a location Slot, and tomorrow morning as a date_time Slot. Intent and Slot names are usually task-specific and defined as labels in the training data. This is a fundamental step that is executed in any task-driven conversational assistant.

Our BERT-based model implementation allows you to train and detect both of these tasks together.

TAO Toolkit provides a sample notebook to outline the end-to-end workflow on how to train an Intent and Slot classification model using TAO Toolkit and deploy it in Riva format on NGC resources.

Before proceeding, let’s download sample spec files that we would need for the rest of the subtasks.

Copy
Copied!
            

tao intent_slot_classification download_specs -r /results/intent_slot_classification/default_specs/ \ -o /specs/nlp/intent_slot_classification

When training the model, the dataset should be first converted to the required data format, which requires these files:

  • dict.intents.csv - A list of all Intent names in the data. Only one line per intent name is allowed.

  • dict.slots.csv - A list of all Slot names in the data. Only one line per slot name is allowed.

  • train.tsv/test.tsv - A list of original queries, one per line, with the intent number separated by a tab (e.g., “what alarms do i have set right now <TAB> 0”). Intent numbers are set according to the intent line in the intent dictionary file (dict.intents.csv), starting from 0. The first line in these files should contain the header line “sentence <tab> label”.

  • train_slot.tvs/test_slot.tsv - A list that contains one line per a query, when each word from the original text queried, is replaced by a token number from the slots dictionary file (dict.slots.csv), counted starting from 0. All the words that do not contain a relevant slot are replaced by an ‘out-of-scope’ token number, which is also a part of the slot dictionary file, usually as the last entry there. Example of a line from these files: “54 0 0 54 54 12 12” (the numbers are separated by a space). These files do not contain any header line.

To convert to the format of the model data, use the dataset_convert utility, which implements conversion for the Assistant dataset, which you can download here. Or you can write your own converter for the format that you are using for data annotation.

For a dataset that follows your own annotation format, we recommend using one text file for all samples of the same intent, with the name of the file as the name of the intent. Use one line per query, with brackets to define slot names. This is very similar to the assistant format, and you can adapt this converter utility or your own format with small changes:

Copy
Copied!
            

did i set an alarm to [alarm_type : wake up] in the [timeofday : morning]

To run the dataset_converter, use this command:

Copy
Copied!
            

!tao intent_slot_classification dataset_convert \ source_data_dir=`source_data_dir` \ target_data_dir=`target_data_dir`

  • source_data_dir: The directory location of the your dataset

  • target_data_dir: The directory location where the converted dataset should be saved

After conversion, the target_data_dir should contain the following files:

Copy
Copied!
            

. |--target_data_dir |-- dict.intents.csv |-- dict.slots.csv |-- train.tsv |-- train_slots.tsv |-- test.tsv |-- test_slots.tsv

The following is an example of the config spec for training train.yaml file. You can change any of these parameters and pass them to the train command.

Copy
Copied!
            

# Name of the file where trained model will be saved. save_to: trained-model.tlt data_dir: ??? model: class_balancing: null # choose from [null, weighted_loss]. weighted_loss enables the weighted class balancing of the loss, may be used for handling unbalanced classes intent_loss_weight: 0.6 # relation of intent to slot loss in total loss (between 0 to 1) pad_label: -1 # value to pad the inputs ignore_extra_tokens: false ignore_start_end: true # do not use first and last token for slot training tokenizer: tokenizer_name: ${model.language_model.pretrained_model_name} # or sentencepiece vocab_file: null # path to vocab file tokenizer_model: null # only used if tokenizer is sentencepiece special_tokens: null language_model: max_seq_length: 50 pretrained_model_name: bert-base-uncased lm_checkpoint: null config_file: null # json file, precedence over config config: null head: num_output_layers: 2 fc_dropout: 0.1 training_ds: prefix: train batch_size: 32 num_workers: 2 validation_ds: prefix: dev batch_size: 32 num_workers: 2 optim: name: adam lr: 2e-5 # optimizer arguments betas: [0.9, 0.999] weight_decay: 0.01 # scheduler setup sched: name: WarmupAnnealing # Scheduler params warmup_steps: null warmup_ratio: 0.1 last_epoch: -1 # pytorch lightning args monitor: val_loss reduce_on_plateau: false

Parameter Data Type Default Description
save_to string trained-model.tlt The filename of the trained model
data_dir string The path of the data converted to the specified format
model.class_balancing string null Choose from [null, weighted_loss]. The weighted_loss enables weighted class balancing of the loss
model.intent_loss_weight float 0.6 The relation of intent-to-slot loss in the total loss
model.pad_label integer -1 A value to pad the inputs
model.ignore_extra_tokens boolean false A flag that specifies whether to ignore extra tokens
model.ignore_start_end boolean true A flag that specifies whether to not use the first and last token for slot training
model.tokenizer.tokenizer_name string Will be filled automatically based on model.language_model.pretrained_model_name The tokenizer name
model.tokenizer.vocab_file string null The path to the tokenizer vocabulary
model.tokenizer.tokenizer_model string null The path to tokenizer model (only for the sentencepiece tokenizer)
model.tokenizer.special_tokens string null Special tokens for the tokenizer (if they exist)
model.language_model.max_seq_length integer 50 The maximum length of the input queries (in tokens)

model.language_model.pretrained_model_name

string

bert-base-uncased

The pre-trained language model name (choose from bert-base-cased, bert-base-uncased,
megatron_bert_345m_cased, megatron_bert_345m_uncased, distilbert-base-uncased
biomegatron-bert-345m-uncased)

model.language_model.lm_checkpoint string null The path to the pre-trained language-model checkpoint
model.language_model.config_file string null The path to the pre-trained language-model config file
model.language_model.config dictionary null The config for the pre-trained language model
model.head.num_output_layers integer 2 The number of fully connected layers of the Classifier on top of the BERT model
model.head.fc_dropout float 0.1 The dropout ratio of the fully connected layers
training_ds.prefix string train A prefix for the training file names
training_ds.num_workers integer 2 The number of worker threads for training
training_ds.batch_size integer 32 The training data batch size
validation_ds.prefix string dev A prefix for the validation file names
validation_ds.num_workers integer 2 The number of worker threads for validation
validation_ds.batch_size integer 32 The validation data batch size
optim.name string adam An optimizer to use for training
optim.lr float 2e-5 The learning rate to use for training
optim.weight_decay float 0.01 The weight decay to use for training
optim.sched.name string WarmupAnnealing The warmup schedule
optim.sched.warmup_ratio float 0.1 The warmup ratio

The following is an example of the command for training the model:

Copy
Copied!
            

!tao intent_slot_classification train -e /specs/nlp/intent_slot_classification/train.yaml \ data_dir=PATH_TO_DATA \ trainer.max_epochs=50 \ -g 1 \ -k $KEY

Required Arguments for Training

  • -e: The experiment-specification file to set up training

  • data_dir: The dataset directory

  • -k: The encryption key

Optional Arguments

  • trainer.max_epochs: The number of training epochs

  • -g: The number of GPUs to use for training

Note

You can use other arguments to override fields in the specification file. To do so, use the name of the config parameter with a desired value and pass it as a parameter in the script call (e.g., model.class_balancing=weighted_loss).


Training Procedure

At the start of evaluation, TAO Toolkit will print out a log of the experiment specification, a summary of the training dataset, and the model architecture.

As the model starts training, you should see a progress bar per epoch. During training, after each epoch TAO Toolkit will display accuracy metrics on the validation dataset for every Intent and Slot separately, as well as the total accuracy. You can expect these numbers to grow up to 50-100 epochs, depending on the size of the trained data.

At the end of training, TAO Toolkit will save the best checkpoint on the validation dataset at the path specified by the experiment spec file before finishing.

Copy
Copied!
            

GPU available: True, used: True TPU available: None, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2] [NeMo W 2021-01-28 14:52:19 exp_manager:299] There was no checkpoint folder at checkpoint_dir :results/checkpoints. Training from scratch. [NeMo I 2021-01-28 14:52:19 exp_manager:186] Experiments will be logged at results ... label precision recall f1 support weather.weather (label_id: 0) 0.00 0.00 0.00 128 weather.temperature (label_id: 1) 0.00 0.00 0.00 0 weather.temperature_yes_no (label_id: 2) 0.00 0.00 0.00 0 weather.rainfall (label_id: 3) 0.00 0.00 0.00 0 weather.rainfall_yes_no (label_id: 4) 0.00 0.00 0.00 0 weather.snow (label_id: 5) 0.00 0.00 0.00 0 weather.snow_yes_no (label_id: 6) 0.00 0.00 0.00 0 weather.humidity (label_id: 7) 0.00 0.00 0.00 0 weather.humidity_yes_no (label_id: 8) 0.00 0.00 0.00 0 weather.windspeed (label_id: 9) 0.00 0.00 0.00 0 weather.sunny (label_id: 10) 0.00 0.00 0.00 0 weather.cloudy (label_id: 11) 0.00 0.00 0.00 0 weather.alert (label_id: 12) 0.00 0.00 0.00 0 context.weather (label_id: 13) 0.00 0.00 0.00 0 context.continue (label_id: 14) 0.00 0.00 0.00 0 context.navigation (label_id: 15) 0.00 0.00 0.00 0 context.rating (label_id: 16) 0.00 0.00 0.00 0 context.distance (label_id: 17) 0.00 0.00 0.00 0 ------------------- micro avg 0.00 0.00 0.00 128 macro avg 0.00 0.00 0.00 128 weighted avg 0.00 0.00 0.00 128


The fine-tuning process will continue training using a previously trained model; however, fine-tuning is not used much in this task since training is fast and the trained model is specific to a dataset.

The following is an example spec for fine-tuning of the model:

Copy
Copied!
            

# Name of the .tlt file where trained model will be saved or restored. save_to: finetuned-model.tlt restore_from: trained-model.tlt # Directory containing both fine-tuning and validation data. data_dir: ??? # Training data set configurations finetuning_ds: prefix: train batch_size: 32 shuffle: true num_samples: -1 num_workers: 2 # Validation data set configurations validation_ds: prefix: dev batch_size: 32 shuffle: false num_samples: -1 num_workers: 2 optim: name: adam lr: 2e-5 weight_decay: 0.01

Parameter Data Type Default Description
restore_from string trained-model.tlt The path to the pre-trained model
save_to string finetuned-model.tlt The path to saved, trained model
data_dir string The path to the data that is converted to the specified format
trainer.max_epochs integer 5 The maximum number of epochs to train the model
finetuning_ds.prefix string train A prefix for the training filenames
finetuning_ds.num_workers integer 2 The number of worker threads for training
finetuning_ds.batch_size integer 32 The training data batch size
finetuning_ds.shuffle bool True A flag specifying whether to shuffle the training data
finetuning_ds.num_samples integer -1 The number of samples to use from the training dataset (use -1 to specify all samples)
validation_ds.prefix string dev A prefix for the validation filenames
validation_ds.num_workers integer 2 The number of worker threads for validation
validation_ds.batch_size integer 32 The validation data batch size
validation_ds.shuffle bool False A flag specifying whether to shuffle the validation data
validation_ds.num_samples integer -1 The number of samples to use from the validation dataset (use -1 to specify all samples)
optim.name string adam The optimizer to use for training
optim.lr float 1e-5 The learning rate to use for training
optim.weight_decay float 0.01 The weight decay to use for training

Use the following command to fine-tune the model:

Copy
Copied!
            

!tao intent_slot_classification finetune \ -e /specs/nlp/intent_slot_classification/finetune.yaml \ -g 1 \ data_dir=PATH_TO_DATA \ trainer.max_epochs=5 \ -k $KEY

Required Arguments for Fine-tuning

  • -e: The experiment specification file to set up fine-tuning

  • data_dir: The dataset directory

  • -k: The encryption key

Optional Arguments

  • trainer.max_epochs: The number of training epochs

  • -g: The number of GPUs to use in evaluation in a multi-GPU scenario (default: 1)

Note

You can use other arguments to override fields in the specification file. To do so, use the name of the config parameter with a desired value and pass it as a parameter in the script call (e.g., model.class_balancing=weighted_loss).


Fine-tuning Procedure

Fine-tuning the procedure and logs will look similar to the procedure described in the Model Training section, with the addition of the model that

is initially loaded from a previously trained checkpoint.

The following is an example spec to evaluate the pre-trained model:

Copy
Copied!
            

restore_from: trained-model.tlt data_dir: ??? test_ds: prefix: test num_workers: 2 batch_size: 32 shuffle: false num_samples: -1

Parameter Data Type Default Description
restore_from string trained-model.tlt The path to the pre-trained model
data_dir string The path to the data converted to the specified format
test_ds.prefix string test A prefix for the training file names
test_ds.num_workers integer 2 The number of worker threads for training
test_ds.batch_size integer 32 The training data batch size
test_ds.shuffle bool False A flag specifying whether to shuffle the training data
test_ds.num_samples integer -1 The number of samples to use from the training dataset (use -1 to specify all samples)

Use the following command to evaluate the model:

Copy
Copied!
            

!tao intent_slot_classification evaluate \ -e /specs/nlp/intent_slot_classification/evaluate.yaml \ data_dir=PATH_TO_DATA

Required Arguments for Evaluation

  • -e: The experiment specification file to set up evaluation

  • data_dir: Path to the pre-processed data to run evaluation on

Evaluation Procedure

After the previously trained model is initialized, it will run evaluation against the provided test set. You should see metrics for each Intent and Slot in the dataset and accumulative metrics, as shown below:

Copy
Copied!
            

... social_post (label_id: 55) 100.00 36.84 53.85 19 social_query (label_id: 56) 57.69 83.33 68.18 18 takeaway_order (label_id: 57) 72.00 94.74 81.82 19 takeaway_query (label_id: 58) 87.50 73.68 80.00 19 transport_query (label_id: 59) 82.35 73.68 77.78 19 transport_taxi (label_id: 60) 100.00 100.00 100.00 18 transport_ticket (label_id: 61) 79.17 100.00 88.37 19 transport_traffic (label_id: 62) 100.00 89.47 94.44 19 weather_query (label_id: 63) 92.31 63.16 75.00 19 ------------------- micro avg 66.54 66.54 66.54 1076 macro avg 58.77 60.11 56.48 1076 weighted avg 64.62 66.54 62.32 1076


To run inference on the model, specify the list of examples in the spec:

Copy
Copied!
            

input_batch: - 'set alarm for seven thirty am' - 'lower volume by fifty percent' - 'what is my schedule for tomorrow'

In the output, for each input query, the recognized intent and a list of slots for each word will be printed, as shown in the following example:

Copy
Copied!
            

Query : set alarm for seven thirty am Predicted Intent: alarm_set Predicted Slots: O O O time time time

Use the following command to run inference:

Copy
Copied!
            

!tao intent_slot_classification infer \ -e /specs/nlp/intent_slot_classification/infer.yaml \ -m trained-model.tlt \

Required Arguments for Inference

  • -e: The experiment specification file to set up inference. This requires the input_batch with a list of examples to run inference on.

  • -m: The path to the pre-trained model checkpoint from which to infer. The file should have a .tlt extension.

Inference Procedure

After the trained model is loaded, it will run on the provided set of query examples and display the classified Intents and Slots.

Copy
Copied!
            

... [NeMo I 2021-01-29 09:52:21 infer:76] The prediction results of some sample queries with the trained model: [NeMo I 2021-01-29 09:52:21 infer:78] Query : set alarm for seven thirty am [NeMo I 2021-01-29 09:52:21 infer:79] Predicted Intent: alarm_set [NeMo I 2021-01-29 09:52:21 infer:80] Predicted Slots: O O O time time time [NeMo I 2021-01-29 09:52:21 infer:78] Query : lower volume by fifty percent [NeMo I 2021-01-29 09:52:21 infer:79] Predicted Intent: audio_volume_up [NeMo I 2021-01-29 09:52:21 infer:80] Predicted Slots: O O O O O [NeMo I 2021-01-29 09:52:21 infer:78] Query : what is my schedule for tomorrow [NeMo I 2021-01-29 09:52:21 infer:79] Predicted Intent: calendar_query [NeMo I 2021-01-29 09:52:21 infer:80] Predicted Slots: O O O O O date


The following is an example of the spec file for model export:

Copy
Copied!
            

# Name of the .tlt EFF archive to be loaded/model to be exported. restore_from: trained-model.tlt # Set export format: ONNX | RIVA export_format: RIVA # Output EFF archive containing RIVA. export_to: exported-model.riva

Parameter Data Type Default Description
restore_from string trained-model.tlt The path to the pre-trained model
export_format string RIVA The export format (either “ONNX” or “RIVA”)
export_to string exported-model.riva The path to the exported model

To export a pre-trained model, run the following:

Copy
Copied!
            

### For export to Riva !tao intent_slot_classification export \ -e /specs/nlp/intent_slot_classification/export.yaml \ -m finetuned-model.tlt \ -k $KEY

Required Arguments for Export

  • -e: The experiment specification file to set up inference. This requires the input_batch with the list of examples to run inference on.

  • -m: The path to the pre-trained model checkpoint from which to infer. The file should have a .tlt extension.

  • -k: The encryption key

You can use the Riva framework for the deployment of the trained model in the runtime. For more details, refer to the Riva documentation

© Copyright 2023, NVIDIA.. Last updated on Sep 5, 2023.