GPT#

Added in version 2.3.

GPT Local Execution#

GPT for local execution is using the Nemotron4-Mini-Instruct model in instruct mode. The required template to create a correct prompt is specified below. It is currently not possible to change this model for a custom model nor use a cloud hosted model provider.

For more details, see the full model card: Nemotron-Mini-4B-Instruct

GPT Local Execution System Requirements#

Windows:

64-bit Windows 10 May 2020 Update or later
NVIDIA Ampere-class GPU or better with at least 8 GiB VRAM
- NVIDIA Ampere: NVIDIA GeForce RTX 30xx, NVIDIA RTX Axxxx, NVIDIA Ax GPU accelerator
- GPT feature itself needs 4 GiB VRAM
NVIDIA display driver 551.78 or later

Additional recommendations to get the best performance:

Windows 10 or 11
- If using Windows 10, Display Hardware-accelerated GPU Scheduling (HWS) must be enabled via Settings > System > Display > Graphics > Change default graphics settings. See https://devblogs.microsoft.com/directx/hardware-accelerated-gpu-scheduling
NVIDIA Ada-class GPU or better with at least 12 GiB VRAM
- NVIDIA Ada: NVIDIA GeForce RTX 40xx, NVIDIA RTX Ada Generation, NVIDIA L4x GPU accelerator
NVIDIA display driver 555.85 or later

Editor Setup Example#

The Send text to GPT node is asynchronous and will return OnResponse after inference is completed.

Editor - Send Text to GPT node

Instruct Mode#

In this mode we provide GPT with the full combination of system, user,and assistant input slots with instructions to produced an adequate response.

There are three different roles that are supported by Llama text models:

system: Explains the context in which the model interacts. It can include rules, guidelines, or other information to help the models response.
user: Represents input from the user interacting with the model. It can be comments, questions, or commands to the model.
assistant: Represents the generated responses from the model. The last tag is left empty to represent the upcoming reponse from the model.
tool: This role means “tool”. This is used represent messages with the output of a tool call when sent back.

Prompt Template#

The prompt template that was used to train the model must also be used for interacting with the model.

In addition to the different roles described above there are markers written within angled brackets, for example <extra_id_0>. These are special tokens to help the language model understand the boundary of the instruction, the user input, and the model output.

Here are a few examples of prompt templates. Actual examples of prompts can be seen below.

Example 1

--- Prompt START ---
<extra_id_0>System
{system prompt}

<extra_id_1>User
{prompt}
<extra_id_1>Assistant

--- Prompt END ---

Example 2

--- Prompt START ---
<extra_id_0>System
{system prompt}

<tool> ... </tool>
<context> ... </context>

<extra_id_1>User
{prompt}
<extra_id_1>Assistant
<toolcall> ... </toolcall>
<extra_id_1>Tool
{tool response}
<extra_id_1>Assistant

--- Prompt END ---

Example 3

--- Prompt START ---
<extra_id_0>System
{system prompt}

<extra_id_1>User
<tool> ... </tool>
<context> ... </context>
{prompt}
<extra_id_1>Assistant
<toolcall> ... </toolcall>
<extra_id_1>Tool
{tool response}
<extra_id_1>Assistant

--- Prompt END ---

Example 4

--- Prompt START ---
<extra_id_0>System
{system prompt}

<extra_id_1>User
{prompt}
<extra_id_1>Assistant
{reponse}
<extra_id_1>User
{prompt}
<extra_id_1>Assistant
{reponse}
<extra_id_1>User
{prompt}
<extra_id_1>Assistant

--- Prompt END ---

Make sure to include the last line break after <extra_id_1>Assistant.
One or more <tool> ... </tool> and <context> ... </context> blocks can be supplied either in the System prompt or in User turns.
To keep (or reset) a conversation, previous conversation history can be added or omitted in the prompt template as shown in example 4.

Examples#

Basic Questioning#

<extra_id_0>System
Answer all questions like a pirate.

<extra_id_1>User
Hi there! How are you doing?
<extra_id_1>Assistant

Tool and Function Calling#

Handling multiple tools in a system prompt:

<extra_id_0>System
<tool> { "name": "get_weather_forecast", "description": "Get the weather forecast for the next 7 days", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. New York, NY" } }, "required": [ "location" ] } } </tool>
<tool> { "name": "get_current_weather", "description": "Get the current weather", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA" } }, "required": [ "location" ] } } </tool>

<extra_id_1>User
What is the weather like in New York right now?
<extra_id_1>Assistant
<toolcall> {"name": "get_current_weather", "arguments": {"location": "New York"}} </toolcall>
<extra_id_1>Tool
{"temperature": "15°C", "condition": "Cloudy"}
<extra_id_1>Assistant
The current weather in New York is 15°C with some clouds.

Role-Play#

<extra_id_0>System
You are Brandon, a sports reporter with a polite demeanor and excited about the upcoming fight. You need to get information from Slim about how he feels about his upcoming fight.
Slim is playing the role of User. Slim is a boxer that speaks with an African American dialect.  He has just gotten finished training and is a bit winded.
The location is a boxing gym that is hot due to all the people working out. Brandon is interviewing Slim about his upcoming fight with Big Jim. Brandon wants to know how Slim is feeling, physically, if he's ready for the fight. Slim says he's ready and predicts he'll knockout Big Jim in the first part of the fight, perhaps the fourth round.

<extra_id_1>User
Whoa, didn't see you there. Who are you?
<extra_id_1>Assistant
I'm Brandon, a sports reporter. I'm here to interview you about your upcoming fight with Big Jim.
<extra_id_1>User
Just finished an intense training session. I'm feeling hungry now.
<extra_id_1>Assistant
That's great to hear. How do you feel physically? Are you ready for the fight?
<extra_id_1>User
I have been hitting the gym a lot. Twice everyday. The last time I did that, I was in Boston. Have you ever been to Boston?
<extra_id_1>Assistant
No, I haven't. But I've heard it's a great city. So, how do you feel about the fight?

Result in Application#

Using Simple Questioning above #

Editor - Simple Questioning example — Simple Questioning prompt (from ACE Plugin Sample Application)#