Joint Intent and Slot Classification

Joint Intent and Slot classification is a method for classifying an Intent and detecting all relevant Slots (Entities) for the Intent in a query. For example, in the query “What is the weather in Santa Clara tomorrow morning?”, we would like to classify the query as a weather Intent, detect Santa Clara as a location Slot, and tomorrow morning as a date_time Slot. Intent and Slot names are usually task-specific and defined as labels in the training data. This is a fundamental step that is executed in any task-driven conversational assistant.

Our BERT-based model implementation allows you to train and detect both of these tasks together.

TAO Toolkit provides a sample notebook to outline the end-to-end workflow on how to train an Intent and Slot classification model using TAO Toolkit and deploy it in Riva format on NGC resources.

Downloading Sample Spec files

Before proceeding, let’s download sample spec files that we would need for the rest of the subtasks.

tao intent_slot_classification download_specs -r /results/intent_slot_classification/default_specs/ \
                                              -o /specs/nlp/intent_slot_classification

Data Format

When training the model, the dataset should be first converted to the required data format, which requires these files:

  • dict.intents.csv - A list of all Intent names in the data. Only one line per intent name is allowed.

  • dict.slots.csv - A list of all Slot names in the data. Only one line per slot name is allowed.

  • train.tsv/test.tsv - A list of original queries, one per line, with the intent number separated by a tab (e.g., “what alarms do i have set right now <TAB> 0”). Intent numbers are set according to the intent line in the intent dictionary file (dict.intents.csv), starting from 0. The first line in these files should contain the header line “sentence <tab> label”.

  • train_slot.tvs/test_slot.tsv - A list that contains one line per a query, when each word from the original text queried, is replaced by a token number from the slots dictionary file (dict.slots.csv), counted starting from 0. All the words that do not contain a relevant slot are replaced by an ‘out-of-scope’ token number, which is also a part of the slot dictionary file, usually as the last entry there. Example of a line from these files: “54 0 0 54 54 12 12” (the numbers are separated by a space). These files do not contain any header line.

Dataset Conversion

To convert to the format of the model data, use the dataset_convert utility, which implements conversion for the Assistant dataset, which you can download here. Or you can write your own converter for the format that you are using for data annotation.

For a dataset that follows your own annotation format, we recommend using one text file for all samples of the same intent, with the name of the file as the name of the intent. Use one line per query, with brackets to define slot names. This is very similar to the assistant format, and you can adapt this converter utility or your own format with small changes:

did i set an alarm to [alarm_type : wake up] in the [timeofday : morning]

To run the dataset_converter, use this command:

!tao intent_slot_classification dataset_convert \
   source_data_dir=`source_data_dir` \
   target_data_dir=`target_data_dir`
  • source_data_dir: The directory location of the your dataset

  • target_data_dir: The directory location where the converted dataset should be saved

After conversion, the target_data_dir should contain the following files:

.
|--target_data_dir
  |-- dict.intents.csv
  |-- dict.slots.csv
  |-- train.tsv
  |-- train_slots.tsv
  |-- test.tsv
  |-- test_slots.tsv

Model Training

The following is an example of the config spec for training train.yaml file. You can change any of these parameters and pass them to the train command.

# Name of the file where trained model will be saved.
save_to: trained-model.tlt

data_dir: ???

model:
  class_balancing: null # choose from [null, weighted_loss]. weighted_loss enables the weighted class balancing of the loss, may be used for handling unbalanced classes
  intent_loss_weight: 0.6 # relation of intent to slot loss in total loss (between 0 to 1)
  pad_label: -1 # value to pad the inputs
  ignore_extra_tokens: false
  ignore_start_end: true # do not use first and last token for slot training

  tokenizer:
      tokenizer_name: ${model.language_model.pretrained_model_name} # or sentencepiece
      vocab_file: null # path to vocab file
      tokenizer_model: null # only used if tokenizer is sentencepiece
      special_tokens: null

  language_model:
    max_seq_length: 50
    pretrained_model_name: bert-base-uncased
    lm_checkpoint: null
    config_file: null # json file, precedence over config
    config: null

  head:
    num_output_layers: 2
    fc_dropout: 0.1

training_ds:
  prefix: train
  batch_size: 32
  num_workers: 2

validation_ds:
  prefix: dev
  batch_size: 32
  num_workers: 2

optim:
  name: adam
  lr: 2e-5
  # optimizer arguments
  betas: [0.9, 0.999]
  weight_decay: 0.01

  # scheduler setup
  sched:
    name: WarmupAnnealing
    # Scheduler params
    warmup_steps: null
    warmup_ratio: 0.1
    last_epoch: -1
    # pytorch lightning args
    monitor: val_loss
    reduce_on_plateau: false

Parameter

Data Type

Default

Description

save_to

string

trained-model.tlt

The filename of the trained model

data_dir

string

The path of the data converted to the specified format

model.class_balancing

string

null

Choose from [null, weighted_loss]. The weighted_loss enables weighted class balancing of the loss

model.intent_loss_weight

float

0.6

The relation of intent-to-slot loss in the total loss

model.pad_label

integer

-1

A value to pad the inputs

model.ignore_extra_tokens

boolean

false

A flag that specifies whether to ignore extra tokens

model.ignore_start_end

boolean

true

A flag that specifies whether to not use the first and last token for slot training

model.tokenizer.tokenizer_name

string

Will be filled automatically based on model.language_model.pretrained_model_name

The tokenizer name

model.tokenizer.vocab_file

string

null

The path to the tokenizer vocabulary

model.tokenizer.tokenizer_model

string

null

The path to tokenizer model (only for the sentencepiece tokenizer)

model.tokenizer.special_tokens

string

null

Special tokens for the tokenizer (if they exist)

model.language_model.max_seq_length

integer

50

The maximum length of the input queries (in tokens)

model.language_model.pretrained_model_name


string


bert-base-uncased


The pre-trained language model name (choose from bert-base-cased, bert-base-uncased,
megatron-bert-345m-cased, megatron-bert-345m-uncased, distilbert-base-uncased
biomegatron-bert-345m-uncased)

model.language_model.lm_checkpoint

string

null

The path to the pre-trained language-model checkpoint

model.language_model.config_file

string

null

The path to the pre-trained language-model config file

model.language_model.config

dictionary

null

The config for the pre-trained language model

model.head.num_output_layers

integer

2

The number of fully connected layers of the Classifier on top of the BERT model

model.head.fc_dropout

float

0.1

The dropout ratio of the fully connected layers

training_ds.prefix

string

train

A prefix for the training file names

training_ds.num_workers

integer

2

The number of worker threads for training

training_ds.batch_size

integer

32

The training data batch size

validation_ds.prefix

string

dev

A prefix for the validation file names

validation_ds.num_workers

integer

2

The number of worker threads for validation

validation_ds.batch_size

integer

32

The validation data batch size

optim.name

string

adam

An optimizer to use for training

optim.lr

float

2e-5

The learning rate to use for training

optim.weight_decay

float

0.01

The weight decay to use for training

optim.sched.name

string

WarmupAnnealing

The warmup schedule

optim.sched.warmup_ratio

float

0.1

The warmup ratio

The following is an example of the command for training the model:

!tao intent_slot_classification train -e /specs/nlp/intent_slot_classification/train.yaml \
                                data_dir=PATH_TO_DATA \
                                trainer.max_epochs=50 \
                                -g 1  \
                                -k $KEY

Required Arguments for Training

  • -e: The experiment-specification file to set up training

  • data_dir: The dataset directory

  • -k: The encryption key

Optional Arguments

  • trainer.max_epochs: The number of training epochs

  • -g: The number of GPUs to use for training

Note

You can use other arguments to override fields in the specification file. To do so, use the name of the config parameter with a desired value and pass it as a parameter in the script call (e.g., model.class_balancing=weighted_loss).

Training Procedure

At the start of evaluation, TAO Toolkit will print out a log of the experiment specification, a summary of the training dataset, and the model architecture.

As the model starts training, you should see a progress bar per epoch. During training, after each epoch TAO Toolkit will display accuracy metrics on the validation dataset for every Intent and Slot separately, as well as the total accuracy. You can expect these numbers to grow up to 50-100 epochs, depending on the size of the trained data.

At the end of training, TAO Toolkit will save the best checkpoint on the validation dataset at the path specified by the experiment spec file before finishing.

GPU available: True, used: True
TPU available: None, using: 0 TPU cores
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2]
[NeMo W 2021-01-28 14:52:19 exp_manager:299] There was no checkpoint folder at checkpoint_dir :results/checkpoints. Training from scratch.
[NeMo I 2021-01-28 14:52:19 exp_manager:186] Experiments will be logged at results
...
  label                                                precision    recall       f1           support
  weather.weather (label_id: 0)                            0.00       0.00       0.00        128
  weather.temprature (label_id: 1)                         0.00       0.00       0.00          0
  weather.temperature_yes_no (label_id: 2)                 0.00       0.00       0.00          0
  weather.rainfall (label_id: 3)                           0.00       0.00       0.00          0
  weather.rainfall_yes_no (label_id: 4)                    0.00       0.00       0.00          0
  weather.snow (label_id: 5)                               0.00       0.00       0.00          0
  weather.snow_yes_no (label_id: 6)                        0.00       0.00       0.00          0
  weather.humidity (label_id: 7)                           0.00       0.00       0.00          0
  weather.humidity_yes_no (label_id: 8)                    0.00       0.00       0.00          0
  weather.windspeed (label_id: 9)                          0.00       0.00       0.00          0
  weather.sunny (label_id: 10)                             0.00       0.00       0.00          0
  weather.cloudy (label_id: 11)                            0.00       0.00       0.00          0
  weather.alert (label_id: 12)                             0.00       0.00       0.00          0
  context.weather (label_id: 13)                           0.00       0.00       0.00          0
  context.continue (label_id: 14)                          0.00       0.00       0.00          0
  context.navigation (label_id: 15)                        0.00       0.00       0.00          0
  context.rating (label_id: 16)                            0.00       0.00       0.00          0
  context.distance (label_id: 17)                          0.00       0.00       0.00          0
  -------------------
  micro avg                                                0.00       0.00       0.00        128
  macro avg                                                0.00       0.00       0.00        128
  weighted avg                                             0.00       0.00       0.00        128

Model Fine-tuning

The fine-tuning process will continue training using a previously trained model; however, fine-tuning is not used much in this task since training is fast and the trained model is specific to a dataset.

The following is an example spec for fine-tuning of the model:

# Name of the .tlt file where trained model will be saved or restored.
save_to: finetuned-model.tlt
restore_from: trained-model.tlt

# Directory containing both fine-tuning and validation data.
data_dir: ???

# Training data set configurations
finetuning_ds:
  prefix: train
  batch_size: 32
  shuffle: true
  num_samples: -1
  num_workers: 2

# Validation data set configurations
validation_ds:
  prefix: dev
  batch_size: 32
  shuffle: false
  num_samples: -1
  num_workers: 2

optim:
  name: adam
  lr: 2e-5
  weight_decay: 0.01

Parameter

Data Type

Default

Description

restore_from

string

trained-model.tlt

The path to the pre-trained model

save_to

string

finetuned-model.tlt

The path to saved, trained model

data_dir

string

The path to the data that is converted to the specified format

trainer.max_epochs

integer

5

The maximum number of epochs to train the model

finetuning_ds.prefix

string

train

A prefix for the training filenames

finetuning_ds.num_workers

integer

2

The number of worker threads for training

finetuning_ds.batch_size

integer

32

The training data batch size

finetuning_ds.shuffle

bool

True

A flag specifying whether to shuffle the training data

finetuning_ds.num_samples

integer

-1

The number of samples to use from the training dataset (use -1 to specify all samples)

validation_ds.prefix

string

dev

A prefix for the validation filenames

validation_ds.num_workers

integer

2

The number of worker threads for validation

validation_ds.batch_size

integer

32

The validation data batch size

validation_ds.shuffle

bool

False

A flag specifying whether to shuffle the validation data

validation_ds.num_samples

integer

-1

The number of samples to use from the validation dataset (use -1 to specify all samples)

optim.name

string

adam

The optimizer to use for training

optim.lr

float

1e-5

The learning rate to use for training

optim.weight_decay

float

0.01

The weight decay to use for training

Use the following command to fine-tune the model:

!tao intent_slot_classification finetune \
                                -e /specs/nlp/intent_slot_classification/finetune.yaml \
                                -g 1 \
                                data_dir=PATH_TO_DATA \
                                trainer.max_epochs=5 \
                                -k $KEY

Required Arguments for Fine-tuning

  • -e: The experiment specification file to set up fine-tuning

  • data_dir: The dataset directory

  • -k: The encryption key

Optional Arguments

  • trainer.max_epochs: The number of training epochs

  • -g: The number of GPUs to use in evaluation in a multi-GPU scenario (default: 1)

Note

You can use other arguments to override fields in the specification file. To do so, use the name of the config parameter with a desired value and pass it as a parameter in the script call (e.g., model.class_balancing=weighted_loss).

Fine-tuning Procedure

Fine-tuning the procedure and logs will look similar to the procedure described in the Model Training section, with the addition of the model that

is initially loaded from a previously trained checkpoint.

Model Evaluation

The following is an example spec to evaluate the pre-trained model:

restore_from: trained-model.tlt

data_dir: ???

test_ds:
  prefix: test
  num_workers: 2
  batch_size: 32
  shuffle: false
  num_samples: -1

Parameter

Data Type

Default

Description

restore_from

string

trained-model.tlt

The path to the pre-trained model

data_dir

string

The path to the data converted to the specified format

test_ds.prefix

string

test

A prefix for the training file names

test_ds.num_workers

integer

2

The number of worker threads for training

test_ds.batch_size

integer

32

The training data batch size

test_ds.shuffle

bool

False

A flag specifying whether to shuffle the training data

test_ds.num_samples

integer

-1

The number of samples to use from the training dataset (use -1 to specify all samples)

Use the following command to evaluate the model:

!tao intent_slot_classification evaluate \
                                -e /specs/nlp/intent_slot_classification/evaluate.yaml \
                                data_dir=PATH_TO_DATA

Required Arguments for Evaluation

  • -e: The experiment specification file to set up evaluation

  • data_dir: Path to the pre-processed data to run evaluation on

Evaluation Procedure

After the previously trained model is initialized, it will run evaluation against the provided test set. You should see metrics for each Intent and Slot in the dataset and accumulative metrics, as shown below:

...
social_post (label_id: 55)                             100.00      36.84      53.85         19
social_query (label_id: 56)                             57.69      83.33      68.18         18
takeaway_order (label_id: 57)                           72.00      94.74      81.82         19
takeaway_query (label_id: 58)                           87.50      73.68      80.00         19
transport_query (label_id: 59)                          82.35      73.68      77.78         19
transport_taxi (label_id: 60)                          100.00     100.00     100.00         18
transport_ticket (label_id: 61)                         79.17     100.00      88.37         19
transport_traffic (label_id: 62)                       100.00      89.47      94.44         19
weather_query (label_id: 63)                            92.31      63.16      75.00         19
-------------------
micro avg                                               66.54      66.54      66.54       1076
macro avg                                               58.77      60.11      56.48       1076
weighted avg                                            64.62      66.54      62.32       1076

Model Inference

To run inference on the model, specify the list of examples in the spec:

input_batch:
    - 'set alarm for seven thirty am'
    - 'lower volume by fifty percent'
    - 'what is my schedule for tomorrow'

In the output, for each input query, the recognized intent and a list of slots for each word will be printed, as shown in the following example:

Query : set alarm for seven thirty am
Predicted Intent: alarm_set
Predicted Slots: O O O time time time

Use the following command to run inference:

!tao intent_slot_classification infer \
                                -e /specs/nlp/intent_slot_classification/infer.yaml \
                                -m trained-model.tlt \

Required Arguments for Inference

  • -e: The experiment specification file to set up inference. This requires the input_batch with a list of examples to run inference on.

  • -m: The path to the pre-trained model checkpoint from which to infer. The file should have a .tlt extension.

Inference Procedure

After the trained model is loaded, it will run on the provided set of query examples and display the classified Intents and Slots.

...
[NeMo I 2021-01-29 09:52:21 infer:76] The prediction results of some sample queries with the trained model:
[NeMo I 2021-01-29 09:52:21 infer:78] Query : set alarm for seven thirty am
[NeMo I 2021-01-29 09:52:21 infer:79] Predicted Intent: alarm_set
[NeMo I 2021-01-29 09:52:21 infer:80] Predicted Slots: O O O time time time
[NeMo I 2021-01-29 09:52:21 infer:78] Query : lower volume by fifty percent
[NeMo I 2021-01-29 09:52:21 infer:79] Predicted Intent: audio_volume_up
[NeMo I 2021-01-29 09:52:21 infer:80] Predicted Slots: O O O O O
[NeMo I 2021-01-29 09:52:21 infer:78] Query : what is my schedule for tomorrow
[NeMo I 2021-01-29 09:52:21 infer:79] Predicted Intent: calendar_query
[NeMo I 2021-01-29 09:52:21 infer:80] Predicted Slots: O O O O O date

Model Export

The following is an example of the spec file for model export:

# Name of the .tlt EFF archive to be loaded/model to be exported.
restore_from: trained-model.tlt

# Set export format: ONNX | RIVA
export_format: RIVA

# Output EFF archive containing RIVA.
export_to: exported-model.riva

Parameter

Data Type

Default

Description

restore_from

string

trained-model.tlt

The path to the pre-trained model

export_format

string

RIVA

The export format (either “ONNX” or “RIVA”)

export_to

string

exported-model.riva

The path to the exported model

To export a pre-trained model, run the following:

 ### For export to Riva
!tao intent_slot_classification export \
                                -e /specs/nlp/intent_slot_classification/export.yaml \
                                -m finetuned-model.tlt \
                                -k $KEY

Required Arguments for Export

  • -e: The experiment specification file to set up inference. This requires the input_batch with the list of examples to run inference on.

  • -m: The path to the pre-trained model checkpoint from which to infer. The file should have a .tlt extension.

  • -k: The encryption key

Model Deployment

You can use the Riva framework for the deployment of the trained model in the runtime. For more details, refer to the Riva documentation