AlignScore Integration

NeMo Guardrails provides out-of-the-box support for the AlignScore metric (Zha et al.), which uses a RoBERTa-based model for scoring factual consistency in model responses with respect to the knowledge base.

In our testing, we observed an average latency of ~220ms on hosting AlignScore as an HTTP service, and ~45ms on direct inference with the model loaded in-memory. This makes it much faster than the self-check method. However, this method requires an on-prem deployment of the publicly available AlignScore model. See Deploy AlignScore for deployment options.

Usage

To use the AlignScore-based fact-checking, you have to set the following configuration options in your config.yml:

1
rails:
2
  config:
3
    fact_checking:
4
      parameters:
5
        # Point to a running instance of the AlignScore server
6
        endpoint: "http://localhost:5000/alignscore_large"
7
8
  output:
9
    flows:
10
      - alignscore check facts

The Colang flow for AlignScore-based fact-checking rail is the same as that for the self-check fact-checking rail. To trigger the fact-checking rail, you have to set the $check_facts context variable to True before a bot message that requires fact-checking, e.g.:

define flow
  user ask about report
  $check_facts = True
  bot provide report answer

Deploy AlignScore

To deploy an AlignScore server from source, follow these steps:

1. Install the alignscore package from the GitHub repository:

   $
   git clone https://github.com/yuh-zha/AlignScore.git
   $
   cd AlignScore
   $
   pip install .

2. Install PyTorch version 2.0.1.

   $
   pip install torch==2.0.1

3. Download the spaCy en_core_web_sm model:

   $
   python -m spacy download en_core_web_sm

4. Download one or both of the AlignScore checkpoints:

   $
   curl -OL https://huggingface.co/yzha/AlignScore/resolve/main/AlignScore-base.ckpt
   $
   curl -OL https://huggingface.co/yzha/AlignScore/resolve/main/AlignScore-large.ckpt

5. Set the ALIGN_SCORE_PATH environment variable to point to the path where the checkpoints were downloaded.

6. Set the ALIGN_SCORE_DEVICE environment variable to "cpu" to run the AlignScore model on CPU, or to the corresponding GPU device, such as "cuda:0".

   $
   export ALIGN_SCORE_PATH=<path/to/folder_containing_ckpt>
   $
   export ALIGN_SCORE_DEVICE="cuda:0"

7. Start the AlignScore server.

   $
   python -m nemoguardrails.library.factchecking.align_score.server --port 5000 --models=base

By default, the AlignScore server listens on port 5000. You can change the port using the --port option. By default, the AlignScore server loads only the base model. You can load only the large model using --models=large or both models using --models=base --models=large.