Joint Intent and Slot Classification¶
Joint Intent and Slot classification is a method for classifying an Intent and detecting all relevant Slots (Entities) for the Intent in a query. For example, in the query “What is the weather in Santa Clara tomorrow morning?”, we would like to classify the query as a weather Intent, detect Santa Clara as a location Slot, and tomorrow morning as a date_time Slot. Intent and Slot names are usually task-specific and defined as labels in the training data. This is a fundamental step that is executed in any task-driven conversational assistant.
Our BERT-based model implementation allows you to train and detect both of these tasks together.
TLT provides a sample notebook to outline the end-to-end workflow on how to train an Intent and Slot classification model using TLT and deploy it in Jarvis format on NGC resources.
Downloading Sample Spec files¶
Before proceeding, let’s download sample spec files that we would need for the rest of the subtasks.
tlt intent_slot_classification download_specs -r /results/intent_slot_classification/default_specs/ \
-o /specs/nlp/intent_slot_classification
Data Format¶
When training the model, the dataset should be first converted to the required data format, which requires these files:
dict.intents.csv- A list of all Intent names in the data. Only one line per intent name is allowed.dict.slots.csv- A list of all Slot names in the data. Only one line per slot name is allowed.train.tsv/test.tsv- A list of original queries, one per line, with the intent number separated by a tab (e.g., “what alarms do i have set right now <TAB> 0”). Intent numbers are set according to the intent line in the intent dictionary file (dict.intents.csv), starting from 0. The first line in these files should contain the header line “sentence <tab> label”.train_slot.tvs/test_slot.tsv- A list that contains one line per a query, when each word from the original text queried, is replaced by a token number from the slots dictionary file (dict.slots.csv), counted starting from 0. All the words that do not contain a relevant slot are replaced by an ‘out-of-scope’ token number, which is also a part of the slot dictionary file, usually as the last entry there. Example of a line from these files: “54 0 0 54 54 12 12” (the numbers are separated by a space). These files do not contain any header line.
Dataset Conversion¶
To convert to the format of the model data, use the dataset_convert utility, which implements conversion for the Assistant dataset, which you can download here. Or you can write your own converter for the format that you are using for data annotation.
For a dataset that follows your own annotation format, we recommend using one text file for all samples of the same intent, with the name of the file as the name of the intent. Use one line per query, with brackets to define slot names. This is very similar to the assistant format, and you can adapt this converter utility or your own format with small changes:
did i set an alarm to [alarm_type : wake up] in the [timeofday : morning]
To run the dataset_converter, use this command:
!tlt intent_slot_classification dataset_convert \
source_data_dir=`source_data_dir` \
target_data_dir=`target_data_dir`
source_data_dir: The directory location of the your datasettarget_data_dir: The directory location where the converted dataset should be saved
After conversion, the target_data_dir should contain the following files:
.
|--target_data_dir
|-- dict.intents.csv
|-- dict.slots.csv
|-- train.tsv
|-- train_slots.tsv
|-- test.tsv
|-- test_slots.tsv
Model Training¶
The following is an example of the config spec for training train.yaml file. You can change
any of these parameters and pass them to the train command.
# Name of the file where trained model will be saved.
save_to: trained-model.tlt
data_dir: ???
model:
class_balancing: null # choose from [null, weighted_loss]. weighted_loss enables the weighted class balancing of the loss, may be used for handling unbalanced classes
intent_loss_weight: 0.6 # relation of intent to slot loss in total loss (between 0 to 1)
pad_label: -1 # value to pad the inputs
ignore_extra_tokens: false
ignore_start_end: true # do not use first and last token for slot training
tokenizer:
tokenizer_name: ${model.language_model.pretrained_model_name} # or sentencepiece
vocab_file: null # path to vocab file
tokenizer_model: null # only used if tokenizer is sentencepiece
special_tokens: null
language_model:
max_seq_length: 50
pretrained_model_name: bert-base-uncased
lm_checkpoint: null
config_file: null # json file, precedence over config
config: null
head:
num_output_layers: 2
fc_dropout: 0.1
training_ds:
prefix: train
batch_size: 32
num_workers: 2
validation_ds:
prefix: dev
batch_size: 32
num_workers: 2
optim:
name: adam
lr: 2e-5
# optimizer arguments
betas: [0.9, 0.999]
weight_decay: 0.01
# scheduler setup
sched:
name: WarmupAnnealing
# Scheduler params
warmup_steps: null
warmup_ratio: 0.1
last_epoch: -1
# pytorch lightning args
monitor: val_loss
reduce_on_plateau: false
Parameter |
Data Type |
Default |
Description |
save_to |
string |
trained-model.tlt |
The filename of the trained model |
data_dir |
string |
– |
The path of the data converted to the specified format |
model.class_balancing |
string |
null |
Choose from [null, weighted_loss]. The weighted_loss enables weighted class balancing of the loss |
model.intent_loss_weight |
float |
0.6 |
The relation of intent-to-slot loss in the total loss |
model.pad_label |
integer |
-1 |
A value to pad the inputs |
model.ignore_extra_tokens |
boolean |
false |
A flag that specifies whether to ignore extra tokens |
model.ignore_start_end |
boolean |
true |
A flag that specifies whether to not use the first and last token for slot training |
model.tokenizer.tokenizer_name |
string |
Will be filled automatically based on model.language_model.pretrained_model_name |
The tokenizer name |
model.tokenizer.vocab_file |
string |
null |
The path to the tokenizer vocabulary |
model.tokenizer.tokenizer_model |
string |
null |
The path to tokenizer model (only for the sentencepiece tokenizer) |
model.tokenizer.special_tokens |
string |
null |
Special tokens for the tokenizer (if they exist) |
model.language_model.max_seq_length |
integer |
50 |
The maximum length of the input queries (in tokens) |
model.language_model.pretrained_model_name |
string |
bert-base-uncased |
The pre-trained language model name (choose from bert-base-cased, bert-base-uncased, megatron-bert-345m-cased and megatron-bert-345m-uncased) |
model.language_model.lm_checkpoint |
string |
null |
The path to the pre-trained language-model checkpoint |
model.language_model.config_file |
string |
null |
The path to the pre-trained language-model config file |
model.language_model.config |
dictionary |
null |
The config for the pre-trained language model |
model.head.num_output_layers |
integer |
2 |
The number of fully connected layers of the Classifier on top of the BERT model |
model.head.fc_dropout |
float |
0.1 |
The dropout ratio of the fully connected layers |
training_ds.prefix |
string |
train |
A prefix for the training file names |
training_ds.num_workers |
integer |
2 |
The number of worker threads for training |
training_ds.batch_size |
integer |
32 |
The training data batch size |
validation_ds.prefix |
string |
dev |
A prefix for the validation file names |
validation_ds.num_workers |
integer |
2 |
The number of worker threads for validation |
validation_ds.batch_size |
integer |
32 |
The validation data batch size |
optim.name |
string |
adam |
An optimizer to use for training |
optim.lr |
float |
2e-5 |
The learning rate to use for training |
optim.weight_decay |
float |
0.01 |
The weight decay to use for training |
optim.sched.name |
string |
WarmupAnnealing |
The warmup schedule |
optim.sched.warmup_ratio |
float |
0.1 |
The warmup ratio |
The following is an example of the command for training the model:
!tlt intent_slot_classification train -e /specs/nlp/intent_slot_classification/train.yaml \
data_dir=PATH_TO_DATA \
trainer.max_epochs=50 \
-g 1 \
-k $KEY
Required Arguments for Training¶
-e: The experiment-specification file to set up trainingdata_dir: The dataset directory-k: The encryption key
Optional Arguments¶
trainer.max_epochs: The number of training epochs-g: The number of GPUs to use for training
Note
You can use other arguments to override fields in the specification file.
To do so, use the name of the config parameter with a desired value and pass it as a parameter in
the script call (e.g., model.class_balancing=weighted_loss).
Training Procedure¶
At the start of evaluation, TLT will print out a log of the experiment specification, a summary of the training dataset, and the model architecture.
As the model starts training, you should see a progress bar per epoch. During training, after each epoch TLT will display accuracy metrics on the validation dataset for every Intent and Slot separately, as well as the total accuracy. You can expect these numbers to grow up to 50-100 epochs, depending on the size of the trained data.
At the end of training, TLT will save the best checkpoint on the validation dataset at the path specified by the experiment spec file before finishing.
GPU available: True, used: True
TPU available: None, using: 0 TPU cores
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2]
[NeMo W 2021-01-28 14:52:19 exp_manager:299] There was no checkpoint folder at checkpoint_dir :results/checkpoints. Training from scratch.
[NeMo I 2021-01-28 14:52:19 exp_manager:186] Experiments will be logged at results
...
label precision recall f1 support
weather.weather (label_id: 0) 0.00 0.00 0.00 128
weather.temprature (label_id: 1) 0.00 0.00 0.00 0
weather.temperature_yes_no (label_id: 2) 0.00 0.00 0.00 0
weather.rainfall (label_id: 3) 0.00 0.00 0.00 0
weather.rainfall_yes_no (label_id: 4) 0.00 0.00 0.00 0
weather.snow (label_id: 5) 0.00 0.00 0.00 0
weather.snow_yes_no (label_id: 6) 0.00 0.00 0.00 0
weather.humidity (label_id: 7) 0.00 0.00 0.00 0
weather.humidity_yes_no (label_id: 8) 0.00 0.00 0.00 0
weather.windspeed (label_id: 9) 0.00 0.00 0.00 0
weather.sunny (label_id: 10) 0.00 0.00 0.00 0
weather.cloudy (label_id: 11) 0.00 0.00 0.00 0
weather.alert (label_id: 12) 0.00 0.00 0.00 0
context.weather (label_id: 13) 0.00 0.00 0.00 0
context.continue (label_id: 14) 0.00 0.00 0.00 0
context.navigation (label_id: 15) 0.00 0.00 0.00 0
context.rating (label_id: 16) 0.00 0.00 0.00 0
context.distance (label_id: 17) 0.00 0.00 0.00 0
-------------------
micro avg 0.00 0.00 0.00 128
macro avg 0.00 0.00 0.00 128
weighted avg 0.00 0.00 0.00 128
Model Fine-tuning¶
The fine-tuning process will continue training using a previously trained model; however, fine-tuning is not used much in this task since training is fast and the trained model is specific to a dataset.
The following is an example spec for fine-tuning of the model:
# Name of the .tlt file where trained model will be saved or restored.
save_to: finetuned-model.tlt
restore_from: trained-model.tlt
# Directory containing both fine-tuning and validation data.
data_dir: ???
# Training data set configurations
finetuning_ds:
prefix: train
batch_size: 32
shuffle: true
num_samples: -1
num_workers: 2
# Validation data set configurations
validation_ds:
prefix: dev
batch_size: 32
shuffle: false
num_samples: -1
num_workers: 2
optim:
name: adam
lr: 2e-5
weight_decay: 0.01
Parameter |
Data Type |
Default |
Description |
restore_from |
string |
trained-model.tlt |
The path to the pre-trained model |
save_to |
string |
finetuned-model.tlt |
The path to saved, trained model |
data_dir |
string |
– |
The path to the data that is converted to the specified format |
trainer.max_epochs |
integer |
5 |
The maximum number of epochs to train the model |
finetuning_ds.prefix |
string |
train |
A prefix for the training filenames |
finetuning_ds.num_workers |
integer |
2 |
The number of worker threads for training |
finetuning_ds.batch_size |
integer |
32 |
The training data batch size |
finetuning_ds.shuffle |
bool |
True |
A flag specifying whether to shuffle the training data |
finetuning_ds.num_samples |
integer |
-1 |
The number of samples to use from the training dataset (use -1 to specify all samples) |
validation_ds.prefix |
string |
dev |
A prefix for the validation filenames |
validation_ds.num_workers |
integer |
2 |
The number of worker threads for validation |
validation_ds.batch_size |
integer |
32 |
The validation data batch size |
validation_ds.shuffle |
bool |
False |
A flag specifying whether to shuffle the validation data |
validation_ds.num_samples |
integer |
-1 |
The number of samples to use from the validation dataset (use -1 to specify all samples) |
optim.name |
string |
adam |
The optimizer to use for training |
optim.lr |
float |
1e-5 |
The learning rate to use for training |
optim.weight_decay |
float |
0.01 |
The weight decay to use for training |
Use the following command to fine-tune the model:
!tlt intent_slot_classification finetune \
-e /specs/nlp/intent_slot_classification/finetune.yaml \
-g 1 \
data_dir=PATH_TO_DATA \
trainer.max_epochs=5 \
-k $KEY
Required Arguments for Fine-tuning¶
-e: The experiment specification file to set up fine-tuningdata_dir: The dataset directory-k: The encryption key
Optional Arguments¶
trainer.max_epochs: The number of training epochs-g: The number of GPUs to use in evaluation in a multi-GPU scenario (default: 1)
Note
You can use other arguments to override fields in the specification file.
To do so, use the name of the config parameter with a desired value and pass it as a parameter in
the script call (e.g., model.class_balancing=weighted_loss).
Fine-tuning Procedure¶
Fine-tuning the procedure and logs will look similar to the procedure described in the Model Training section, with the addition of the model that
is initially loaded from a previously trained checkpoint.
Model Evaluation¶
The following is an example spec to evaluate the pre-trained model:
restore_from: trained-model.tlt
data_dir: ???
test_ds:
prefix: test
num_workers: 2
batch_size: 32
shuffle: false
num_samples: -1
Parameter |
Data Type |
Default |
Description |
restore_from |
string |
trained-model.tlt |
The path to the pre-trained model |
data_dir |
string |
– |
The path to the data converted to the specified format |
test_ds.prefix |
string |
test |
A prefix for the training file names |
test_ds.num_workers |
integer |
2 |
The number of worker threads for training |
test_ds.batch_size |
integer |
32 |
The training data batch size |
test_ds.shuffle |
bool |
False |
A flag specifying whether to shuffle the training data |
test_ds.num_samples |
integer |
-1 |
The number of samples to use from the training dataset (use -1 to specify all samples) |
Use the following command to evaluate the model:
!tlt intent_slot_classification evaluate \
-e /specs/nlp/intent_slot_classification/evaluate.yaml \
data_dir=PATH_TO_DATA
Required Arguments for Evaluation¶
-e: The experiment specification file to set up evaluationdata_dir: Path to the pre-processed data to run evaluation on
Evaluation Procedure¶
After the previously trained model is initialized, it will run evaluation against the provided test set. You should see metrics for each Intent and Slot in the dataset and accumulative metrics, as shown below:
...
social_post (label_id: 55) 100.00 36.84 53.85 19
social_query (label_id: 56) 57.69 83.33 68.18 18
takeaway_order (label_id: 57) 72.00 94.74 81.82 19
takeaway_query (label_id: 58) 87.50 73.68 80.00 19
transport_query (label_id: 59) 82.35 73.68 77.78 19
transport_taxi (label_id: 60) 100.00 100.00 100.00 18
transport_ticket (label_id: 61) 79.17 100.00 88.37 19
transport_traffic (label_id: 62) 100.00 89.47 94.44 19
weather_query (label_id: 63) 92.31 63.16 75.00 19
-------------------
micro avg 66.54 66.54 66.54 1076
macro avg 58.77 60.11 56.48 1076
weighted avg 64.62 66.54 62.32 1076
Model Inference¶
To run inference on the model, specify the list of examples in the spec:
input_batch:
- 'set alarm for seven thirty am'
- 'lower volume by fifty percent'
- 'what is my schedule for tomorrow'
In the output, for each input query, the recognized intent and a list of slots for each word will be printed, as shown in the following example:
Query : set alarm for seven thirty am
Predicted Intent: alarm_set
Predicted Slots: O O O time time time
Use the following command to run inference:
!tlt intent_slot_classification infer \
-e /specs/nlp/intent_slot_classification/infer.yaml \
-m trained-model.tlt \
Required Arguments for Inference¶
-e: The experiment specification file to set up inference. This requires theinput_batchwith a list of examples to run inference on.-m: The path to the pre-trained model checkpoint from which to infer. The file should have a.tltextension.
Inference Procedure¶
After the trained model is loaded, it will run on the provided set of query examples and display the classified Intents and Slots.
...
[NeMo I 2021-01-29 09:52:21 infer:76] The prediction results of some sample queries with the trained model:
[NeMo I 2021-01-29 09:52:21 infer:78] Query : set alarm for seven thirty am
[NeMo I 2021-01-29 09:52:21 infer:79] Predicted Intent: alarm_set
[NeMo I 2021-01-29 09:52:21 infer:80] Predicted Slots: O O O time time time
[NeMo I 2021-01-29 09:52:21 infer:78] Query : lower volume by fifty percent
[NeMo I 2021-01-29 09:52:21 infer:79] Predicted Intent: audio_volume_up
[NeMo I 2021-01-29 09:52:21 infer:80] Predicted Slots: O O O O O
[NeMo I 2021-01-29 09:52:21 infer:78] Query : what is my schedule for tomorrow
[NeMo I 2021-01-29 09:52:21 infer:79] Predicted Intent: calendar_query
[NeMo I 2021-01-29 09:52:21 infer:80] Predicted Slots: O O O O O date
Model Export¶
The following is an example of the spec file for model export:
# Name of the .tlt EFF archive to be loaded/model to be exported.
restore_from: trained-model.tlt
# Set export format: ONNX | JARVIS
export_format: JARVIS
# Output EFF archive containing JARVIS.
export_to: exported-model.ejrvs
Parameter |
Data Type |
Default |
Description |
restore_from |
string |
trained-model.tlt |
The path to the pre-trained model |
export_format |
string |
JARVIS |
The export format (either “ONNX” or “JARVIS”) |
export_to |
string |
exported-model.ejrvs |
The path to the exported model |
To export a pre-trained model, run the following:
### For export to Jarvis
!tlt intent_slot_classification export \
-e /specs/nlp/intent_slot_classification/export.yaml \
-m finetuned-model.tlt \
-k $KEY
Required Arguments for Export¶
-e: The experiment specification file to set up inference. This requires theinput_batchwith the list of examples to run inference on.-m: The path to the pre-trained model checkpoint from which to infer. The file should have a.tltextension.-k: The encryption key
Model Deployment¶
You can use the Jarvis framework for the deployment of the trained model in the runtime. For more details, refer to the Jarvis documentation