NVIDIA TAO Toolkit v30.2108

Joint Intent and Slot Classification

Joint Intent and Slot classification is a method for classifying an Intent and detecting all relevant Slots (Entities) for the Intent in a query. For example, in the query “What is the weather in Santa Clara tomorrow morning?”, we would like to classify the query as a weather Intent, detect Santa Clara as a location Slot, and tomorrow morning as a date_time Slot. Intent and Slot names are usually task-specific and defined as labels in the training data. This is a fundamental step that is executed in any task-driven conversational assistant.

Our BERT-based model implementation allows you to train and detect both of these tasks together.

TAO Toolkit provides a sample notebook to outline the end-to-end workflow on how to train an Intent and Slot classification model using TAO Toolkit and deploy it in Riva format on NGC resources.

Before proceeding, let’s download sample spec files that we would need for the rest of the subtasks.

Copy
Copied!
            

tao intent_slot_classification download_specs -r /results/intent_slot_classification/default_specs/ \ -o /specs/nlp/intent_slot_classification

When training the model, the dataset should be first converted to the required data format, which requires these files:

  • dict.intents.csv - A list of all Intent names in the data. Only one line per intent name is allowed.

  • dict.slots.csv - A list of all Slot names in the data. Only one line per slot name is allowed.

  • train.tsv/test.tsv - A list of original queries, one per line, with the intent number separated by a tab (e.g., “what alarms do i have set right now <TAB> 0”). Intent numbers are set according to the intent line in the intent dictionary file (dict.intents.csv), starting from 0. The first line in these files should contain the header line “sentence <tab> label”.

  • train_slot.tvs/test_slot.tsv - A list that contains one line per a query, when each word from the original text queried, is replaced by a token number from the slots dictionary file (dict.slots.csv), counted starting from 0. All the words that do not contain a relevant slot are replaced by an ‘out-of-scope’ token number, which is also a part of the slot dictionary file, usually as the last entry there. Example of a line from these files: “54 0 0 54 54 12 12” (the numbers are separated by a space). These files do not contain any header line.

To convert to the format of the model data, use the dataset_convert utility, which implements conversion for the Assistant dataset, which you can download here. Or you can write your own converter for the format that you are using for data annotation.

For a dataset that follows your own annotation format, we recommend using one text file for all samples of the same intent, with the name of the file as the name of the intent. Use one line per query, with brackets to define slot names. This is very similar to the assistant format, and you can adapt this converter utility or your own format with small changes:

Copy
Copied!
            

did i set an alarm to [alarm_type : wake up] in the [timeofday : morning]

To run the dataset_converter, use this command:

Copy
Copied!
            

!tao intent_slot_classification dataset_convert \ source_data_dir=`source_data_dir` \ target_data_dir=`target_data_dir`

  • source_data_dir: The directory location of the your dataset

  • target_data_dir: The directory location where the converted dataset should be saved

After conversion, the target_data_dir should contain the following files:

Copy
Copied!
            

. |--target_data_dir |-- dict.intents.csv |-- dict.slots.csv |-- train.tsv |-- train_slots.tsv |-- test.tsv |-- test_slots.tsv

The following is an example of the config spec for training train.yaml file. You can change any of these parameters and pass them to the train command.

Copy
Copied!
            

# Name of the file where trained model will be saved. save_to: trained-model.tlt data_dir: ??? model: class_balancing: null # choose from [null, weighted_loss]. weighted_loss enables the weighted class balancing of the loss, may be used for handling unbalanced classes intent_loss_weight: 0.6 # relation of intent to slot loss in total loss (between 0 to 1) pad_label: -1 # value to pad the inputs ignore_extra_tokens: false ignore_start_end: true # do not use first and last token for slot training tokenizer: tokenizer_name: ${model.language_model.pretrained_model_name} # or sentencepiece vocab_file: null # path to vocab file tokenizer_model: null # only used if tokenizer is sentencepiece special_tokens: null language_model: max_seq_length: 50 pretrained_model_name: bert-base-uncased lm_checkpoint: null config_file: null # json file, precedence over config config: null head: num_output_layers: 2 fc_dropout: 0.1 training_ds: prefix: train batch_size: 32 num_workers: 2 validation_ds: prefix: dev batch_size: 32 num_workers: 2 optim: name: adam lr: 2e-5 # optimizer arguments betas: [0.9, 0.999] weight_decay: 0.01 # scheduler setup sched: name: WarmupAnnealing # Scheduler params warmup_steps: null warmup_ratio: 0.1 last_epoch: -1 # pytorch lightning args monitor: val_loss reduce_on_plateau: false

Parameter

Data Type

Default

Description

save_to

string

trained-model.tlt

The filename of the trained model

data_dir

string

The path of the data converted to the specified format

model.class_balancing

string

null

Choose from [null, weighted_loss]. The weighted_loss enables weighted class balancing of the loss

model.intent_loss_weight

float

0.6

The relation of intent-to-slot loss in the total loss

model.pad_label

integer

-1

A value to pad the inputs

model.ignore_extra_tokens

boolean

false

A flag that specifies whether to ignore extra tokens

model.ignore_start_end

boolean

true

A flag that specifies whether to not use the first and last token for slot training

model.tokenizer.tokenizer_name

string

Will be filled automatically based on model.language_model.pretrained_model_name

The tokenizer name

model.tokenizer.vocab_file

string

null

The path to the tokenizer vocabulary

model.tokenizer.tokenizer_model

string

null

The path to tokenizer model (only for the sentencepiece tokenizer)

model.tokenizer.special_tokens

string

null

Special tokens for the tokenizer (if they exist)

model.language_model.max_seq_length

integer

50

The maximum length of the input queries (in tokens)

model.language_model.pretrained_model_name

string

bert-base-uncased

The pre-trained language model name (choose from bert-base-cased, bert-base-uncased, megatron-bert-345m-cased and megatron-bert-345m-uncased)

model.language_model.lm_checkpoint

string

null

The path to the pre-trained language-model checkpoint

model.language_model.config_file

string

null

The path to the pre-trained language-model config file

model.language_model.config

dictionary

null

The config for the pre-trained language model

model.head.num_output_layers

integer

2

The number of fully connected layers of the Classifier on top of the BERT model

model.head.fc_dropout

float

0.1

The dropout ratio of the fully connected layers

training_ds.prefix

string

train

A prefix for the training file names

training_ds.num_workers

integer

2

The number of worker threads for training

training_ds.batch_size

integer

32

The training data batch size

validation_ds.prefix

string

dev

A prefix for the validation file names

validation_ds.num_workers

integer

2

The number of worker threads for validation

validation_ds.batch_size

integer

32

The validation data batch size

optim.name

string

adam

An optimizer to use for training

optim.lr

float

2e-5

The learning rate to use for training

optim.weight_decay

float

0.01

The weight decay to use for training

optim.sched.name

string

WarmupAnnealing

The warmup schedule

optim.sched.warmup_ratio

float

0.1

The warmup ratio

The following is an example of the command for training the model:

Copy
Copied!
            

!tao intent_slot_classification train -e /specs/nlp/intent_slot_classification/train.yaml \ data_dir=PATH_TO_DATA \ trainer.max_epochs=50 \ -g 1 \ -k $KEY

Required Arguments for Training

  • -e: The experiment-specification file to set up training

  • data_dir: The dataset directory

  • -k: The encryption key

Optional Arguments

  • trainer.max_epochs: The number of training epochs

  • -g: The number of GPUs to use for training

Note

You can use other arguments to override fields in the specification file. To do so, use the name of the config parameter with a desired value and pass it as a parameter in the script call (e.g., model.class_balancing=weighted_loss).


Training Procedure

At the start of evaluation, TAO Toolkit will print out a log of the experiment specification, a summary of the training dataset, and the model architecture.

As the model starts training, you should see a progress bar per epoch. During training, after each epoch TAO Toolkit will display accuracy metrics on the validation dataset for every Intent and Slot separately, as well as the total accuracy. You can expect these numbers to grow up to 50-100 epochs, depending on the size of the trained data.

At the end of training, TAO Toolkit will save the best checkpoint on the validation dataset at the path specified by the experiment spec file before finishing.

Copy
Copied!
            

GPU available: True, used: True TPU available: None, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2] [NeMo W 2021-01-28 14:52:19 exp_manager:299] There was no checkpoint folder at checkpoint_dir :results/checkpoints. Training from scratch. [NeMo I 2021-01-28 14:52:19 exp_manager:186] Experiments will be logged at results ... label precision recall f1 support weather.weather (label_id: 0) 0.00 0.00 0.00 128 weather.temprature (label_id: 1) 0.00 0.00 0.00 0 weather.temperature_yes_no (label_id: 2) 0.00 0.00 0.00 0 weather.rainfall (label_id: 3) 0.00 0.00 0.00 0 weather.rainfall_yes_no (label_id: 4) 0.00 0.00 0.00 0 weather.snow (label_id: 5) 0.00 0.00 0.00 0 weather.snow_yes_no (label_id: 6) 0.00 0.00 0.00 0 weather.humidity (label_id: 7) 0.00 0.00 0.00 0 weather.humidity_yes_no (label_id: 8) 0.00 0.00 0.00 0 weather.windspeed (label_id: 9) 0.00 0.00 0.00 0 weather.sunny (label_id: 10) 0.00 0.00 0.00 0 weather.cloudy (label_id: 11) 0.00 0.00 0.00 0 weather.alert (label_id: 12) 0.00 0.00 0.00 0 context.weather (label_id: 13) 0.00 0.00 0.00 0 context.continue (label_id: 14) 0.00 0.00 0.00 0 context.navigation (label_id: 15) 0.00 0.00 0.00 0 context.rating (label_id: 16) 0.00 0.00 0.00 0 context.distance (label_id: 17) 0.00 0.00 0.00 0 ------------------- micro avg 0.00 0.00 0.00 128 macro avg 0.00 0.00 0.00 128 weighted avg 0.00 0.00 0.00 128


The fine-tuning process will continue training using a previously trained model; however, fine-tuning is not used much in this task since training is fast and the trained model is specific to a dataset.

The following is an example spec for fine-tuning of the model:

Copy
Copied!
            

# Name of the .tlt file where trained model will be saved or restored. save_to: finetuned-model.tlt restore_from: trained-model.tlt # Directory containing both fine-tuning and validation data. data_dir: ??? # Training data set configurations finetuning_ds: prefix: train batch_size: 32 shuffle: true num_samples: -1 num_workers: 2 # Validation data set configurations validation_ds: prefix: dev batch_size: 32 shuffle: false num_samples: -1 num_workers: 2 optim: name: adam lr: 2e-5 weight_decay: 0.01

Parameter

Data Type

Default

Description

restore_from

string

trained-model.tlt

The path to the pre-trained model

save_to

string

finetuned-model.tlt

The path to saved, trained model

data_dir

string

The path to the data that is converted to the specified format

trainer.max_epochs

integer

5

The maximum number of epochs to train the model

finetuning_ds.prefix

string

train

A prefix for the training filenames

finetuning_ds.num_workers

integer

2

The number of worker threads for training

finetuning_ds.batch_size

integer

32

The training data batch size

finetuning_ds.shuffle

bool

True

A flag specifying whether to shuffle the training data

finetuning_ds.num_samples

integer

-1

The number of samples to use from the training dataset (use -1 to specify all samples)

validation_ds.prefix

string

dev

A prefix for the validation filenames

validation_ds.num_workers

integer

2

The number of worker threads for validation

validation_ds.batch_size

integer

32

The validation data batch size

validation_ds.shuffle

bool

False

A flag specifying whether to shuffle the validation data

validation_ds.num_samples

integer

-1

The number of samples to use from the validation dataset (use -1 to specify all samples)

optim.name

string

adam

The optimizer to use for training

optim.lr

float

1e-5

The learning rate to use for training

optim.weight_decay

float

0.01

The weight decay to use for training

Use the following command to fine-tune the model:

Copy
Copied!
            

!tao intent_slot_classification finetune \ -e /specs/nlp/intent_slot_classification/finetune.yaml \ -g 1 \ data_dir=PATH_TO_DATA \ trainer.max_epochs=5 \ -k $KEY

Required Arguments for Fine-tuning

  • -e: The experiment specification file to set up fine-tuning

  • data_dir: The dataset directory

  • -k: The encryption key

Optional Arguments

  • trainer.max_epochs: The number of training epochs

  • -g: The number of GPUs to use in evaluation in a multi-GPU scenario (default: 1)

Note

You can use other arguments to override fields in the specification file. To do so, use the name of the config parameter with a desired value and pass it as a parameter in the script call (e.g., model.class_balancing=weighted_loss).


Fine-tuning Procedure

Fine-tuning the procedure and logs will look similar to the procedure described in the Model Training section, with the addition of the model that

is initially loaded from a previously trained checkpoint.

The following is an example spec to evaluate the pre-trained model:

Copy
Copied!
            

restore_from: trained-model.tlt data_dir: ??? test_ds: prefix: test num_workers: 2 batch_size: 32 shuffle: false num_samples: -1

Parameter

Data Type

Default

Description

restore_from

string

trained-model.tlt

The path to the pre-trained model

data_dir

string

The path to the data converted to the specified format

test_ds.prefix

string

test

A prefix for the training file names

test_ds.num_workers

integer

2

The number of worker threads for training

test_ds.batch_size

integer

32

The training data batch size

test_ds.shuffle

bool

False

A flag specifying whether to shuffle the training data

test_ds.num_samples

integer

-1

The number of samples to use from the training dataset (use -1 to specify all samples)

Use the following command to evaluate the model:

Copy
Copied!
            

!tao intent_slot_classification evaluate \ -e /specs/nlp/intent_slot_classification/evaluate.yaml \ data_dir=PATH_TO_DATA

Required Arguments for Evaluation

  • -e: The experiment specification file to set up evaluation

  • data_dir: Path to the pre-processed data to run evaluation on

Evaluation Procedure

After the previously trained model is initialized, it will run evaluation against the provided test set. You should see metrics for each Intent and Slot in the dataset and accumulative metrics, as shown below:

Copy
Copied!
            

... social_post (label_id: 55) 100.00 36.84 53.85 19 social_query (label_id: 56) 57.69 83.33 68.18 18 takeaway_order (label_id: 57) 72.00 94.74 81.82 19 takeaway_query (label_id: 58) 87.50 73.68 80.00 19 transport_query (label_id: 59) 82.35 73.68 77.78 19 transport_taxi (label_id: 60) 100.00 100.00 100.00 18 transport_ticket (label_id: 61) 79.17 100.00 88.37 19 transport_traffic (label_id: 62) 100.00 89.47 94.44 19 weather_query (label_id: 63) 92.31 63.16 75.00 19 ------------------- micro avg 66.54 66.54 66.54 1076 macro avg 58.77 60.11 56.48 1076 weighted avg 64.62 66.54 62.32 1076


To run inference on the model, specify the list of examples in the spec:

Copy
Copied!
            

input_batch: - 'set alarm for seven thirty am' - 'lower volume by fifty percent' - 'what is my schedule for tomorrow'

In the output, for each input query, the recognized intent and a list of slots for each word will be printed, as shown in the following example:

Copy
Copied!
            

Query : set alarm for seven thirty am Predicted Intent: alarm_set Predicted Slots: O O O time time time

Use the following command to run inference:

Copy
Copied!
            

!tao intent_slot_classification infer \ -e /specs/nlp/intent_slot_classification/infer.yaml \ -m trained-model.tlt \

Required Arguments for Inference

  • -e: The experiment specification file to set up inference. This requires the input_batch with a list of examples to run inference on.

  • -m: The path to the pre-trained model checkpoint from which to infer. The file should have a .tlt extension.

Inference Procedure

After the trained model is loaded, it will run on the provided set of query examples and display the classified Intents and Slots.

Copy
Copied!
            

... [NeMo I 2021-01-29 09:52:21 infer:76] The prediction results of some sample queries with the trained model: [NeMo I 2021-01-29 09:52:21 infer:78] Query : set alarm for seven thirty am [NeMo I 2021-01-29 09:52:21 infer:79] Predicted Intent: alarm_set [NeMo I 2021-01-29 09:52:21 infer:80] Predicted Slots: O O O time time time [NeMo I 2021-01-29 09:52:21 infer:78] Query : lower volume by fifty percent [NeMo I 2021-01-29 09:52:21 infer:79] Predicted Intent: audio_volume_up [NeMo I 2021-01-29 09:52:21 infer:80] Predicted Slots: O O O O O [NeMo I 2021-01-29 09:52:21 infer:78] Query : what is my schedule for tomorrow [NeMo I 2021-01-29 09:52:21 infer:79] Predicted Intent: calendar_query [NeMo I 2021-01-29 09:52:21 infer:80] Predicted Slots: O O O O O date


The following is an example of the spec file for model export:

Copy
Copied!
            

# Name of the .tlt EFF archive to be loaded/model to be exported. restore_from: trained-model.tlt # Set export format: ONNX | RIVA export_format: RIVA # Output EFF archive containing RIVA. export_to: exported-model.riva

Parameter

Data Type

Default

Description

restore_from

string

trained-model.tlt

The path to the pre-trained model

export_format

string

RIVA

The export format (either “ONNX” or “RIVA”)

export_to

string

exported-model.riva

The path to the exported model

To export a pre-trained model, run the following:

Copy
Copied!
            

### For export to Riva !tao intent_slot_classification export \ -e /specs/nlp/intent_slot_classification/export.yaml \ -m finetuned-model.tlt \ -k $KEY

Required Arguments for Export

  • -e: The experiment specification file to set up inference. This requires the input_batch with the list of examples to run inference on.

  • -m: The path to the pre-trained model checkpoint from which to infer. The file should have a .tlt extension.

  • -k: The encryption key

You can use the Riva framework for the deployment of the trained model in the runtime. For more details, refer to the Riva documentation

© Copyright 2020, NVIDIA. Last updated on Aug 24, 2021.