Training Models

The ACE Agent Quick Start comes with a model helper script to easily train NVIDIA Riva Joint Intent & Slot Classification, Text Classification, and Named Entity Recognition NLP models with custom domain specific datasets, evaluate the models, and deploy them. Under the hood, ACE Agent uses the NVIDIA TAO Toolkit for training models and uses the Riva Skills for model deployment.

Currently, Joint Intent & Slot model training supports both yaml-based training datasets and NeMo Joint Intent & Slot datasets. For Text Classifiers and NER training, only NeMo format datasets are allowed. Run the following command to get detailed argument and documentation for the supported tasks in model utils from the ACE Agent Quick Start resource directory.

source deploy/docker/docker_init.sh

docker compose -f deploy/docker/docker-compose.yml run model-utils -h

The ACE Agent provides a sample pipeline as part of the Quick Start scripts in the samples/training directory. The sample pipeline executes the following tasks:

Exports multiple domain datasets in single NEMO dataset format.
Trains the model.
Evaluates the trained model.
Inferences the trained model on sample queries.
Exports the trained TLT model to .riva and .rmir formats.
[Optionally] Uploads the RMIR model to NGC.

The following procedure trains a Joint Intent & Slot recognizer for the food ordering or quick service restaurant domain. The following trained model can be used with the sample Food Ordering bot.

Prepare the training data.

To add an intent, we need to add queries which are examples of what you may ask the bot. You should provide multiple training examples per intent, so that your bot can understand a variety of expressions.

Create a file at nlu_data/qsr_assistant.yaml and add the queries related to add_item, query_item, contextual_query, yes_intent, no_intent, and checkout intents. Annotate the entities for all the intents. In the example queries, add entity values with square brackets and add an entity name enclosed in curly braces next to it without any space. You can add more variations to the list below.

version: "1.0"
domain: qsr
nlu:
- intent: add_item
queries: |
    - Add [french fries]{food_name}
    - Can I get a [cheeseburger]{food_name}
    - Please add [avocado]{food_name} to my cart
    - Add it please
    - Add [two]{food_quantity} [small]{food_size} ones please
    - Add a [medium]{food_size} [double protein burger]{food_name} and [two]{food_quantity} [sandwiches]{food_name}
    - Can I get a [extra large]{food_size} [cola]{food_name}
    - Please add [avocado]{food_name} to my cart
    - Add it please
    - Add [two]{food_quantity} [small]{food_size} ones please
- intent: query_item
queries: |
    - How much calories [french fries]{food_name} has
    - What all is there in [cheeseburger]{food_name}
    - How much does a [regular]{food_size} [pepsi]{food_name} cost
    - What is the [market salad]{food_name} made of
    - what ingredients are there in [cobb salad]{food_name}
    - what sizes is the [chicken sandwich]{food_name} available in
    - What toppings are there with the [pizza]{food_name}
    - what are [chicken nuggets]{food_name}
    - What healthy options do you have in menu
    - What sides are there in menu
    - show me your popular options
    - tell me the best selling items in menu
    - show me your drinks menu
    - what desserts do you have
- intent: contextual_query
queries: |
    - [french fries]{food_name}
    - [cheeseburger]{food_name} please
    - [two]{food_quantity} [small]{food_size} [pepsi]{food_name}
    - what about [sandwiches]{food_name}
    - what about [lemon tea]{food_name}
    - what about [french toast]{food_name}
    - umm [diet cola]{food_name}
    - [lemonade]{food_name}
    - [medium]{food_size} [burgers]{food_name}
    - [nuggets]{food_name} in [small]{food_size} size
- intent: checkout
queries: |
    - That's all
    - I would like to checkout now
    - complete the order
    - checkout please
    - nothing else for today
    - nothing else please
    - All done for today
    - I am all set
    - that completes my order
    - let's checkout now
- intent: yes_intent
queries: |
    - yes
    - Yup
    - yes confirmed
    - yes please
    - please yes
    - yup yup
    - yes yes
    - sure yes
    - yes let's go
    - sure
- intent: no_intent
queries: |
    - no
    - nope
    - no no
    - no please
    - no wait
    - nope nope
    - please no

Train the Joint Intent & Slot model.
After your training data is ready, the next step is to trigger the domain model training using the ACE Agent.

ACE Agent provides a sample pipeline at samples/training/pipeline_intent_slot.yaml as part of the Quick Start scripts for training the Joint Intent & Slot Classification model, exporting it to the required format as well as uploading the model to NGC. We need to update all fields marked as {UPDATE} and optionally update other arguments based on training requirements.

Make the following changes to the samples/training/pipeline_intent_slot.yaml file in order to trigger training for the qsr domain.
1. Replace {UPDATE} with the following parameters:
  model_name: qsr dataset_name: qsr_intent_slot_dataset dataset_paths: [ "nlu_data/qsr_assistant.yaml"]
2. Update the path where the model needs to be stored under global:
  global: result_path: "./results/models"
3. Update the training hyperparameters.
  - _name: task_train_intent_slot_1 epochs: 25 batch_size: 1
4. [Optional] Comment out the task_upload block since we do not want to upload the model to NGC.
  If model needs to be stored in NGC at the end of the training step, provide the proper NGC path as shown below:
  
  - _name: task_upload_model_7 ngc_path: ngc_org/[ngc_team/]model_name:version
5. Trigger the intent_slot model training using the Quick Start scripts after making the above changes to samples/training/pipeline_intent_slot.yaml. We used a small batch size and epochs due to the small dataset size.
  source deploy/docker/docker_init.sh docker compose -f deploy/docker/docker-compose.yml run model-utils pipeline --config_yaml samples/training/pipeline_intent_slot.yaml
6. After the end of the training, confirm that the Joint Intent & Slot model file in RMIR format has been generated in the result directory.
  find results/models/ -name *.rmir Output: results/models/qsr/1/model/intent_slot_qsr.rmir

Deploy the Joint Intent & Slot model.

You can deploy the Joint Intent & Slot model with the ACE Agent NLP server by creating a model_config.yaml file.
model_servers:
- name: riva
    nlp_models:
    - ./results/models/qsr/1/model/intent_slot_qsr.rmir
    url: localhost:8001
    riva_url: localhost:50051

Deploy the models.

export BOT_PATH="."
source deploy/docker/docker_init.sh

docker compose -f deploy/docker/docker-compose.yml up model-utils

Note

The accuracy of the model may be low since we used a small dataset for demonstration purposes. If using a larger dataset, ensure you are familiar with these best practices:

Minimum 20 utterances should be added for every intent and entity.
Avoid using tools or dataset formats that auto generate training data to produce a large number of examples quickly. It can cause overfitting of the model.
Each intent should only include unique utterances and should not overlap with other intents.
Always include an out-of-scope domain or intent for avoiding incorrect bot responses for unsupported queries.
Add utterances with misspellings and shorthands for matching user conversation language.

Alternatively, you can follow NeMo tutorials to train the model, use the instructions from Riva Custom Models documentation to create an RMIR format model file and use the same in NLP server models deployment.