GPT#

Added in version 2.3.

GPT Local Execution#

GPT for local execution is using the Nemotron4-Mini-Instruct model in instruct mode. The required template to create a correct prompt is specified below.

For more details, see the full model card: Nemotron-Mini-4B-Instruct

GPT Local Execution System Requirements#

Windows:

  • 64-bit Windows 10 or later

  • NVIDIA RTX-class GPU (GeForce, Titan or Quadro)

  • NVIDIA display driver 551.78 or later

Linux:

  • 64-bit Linux OS with libstdc++ version 6.0.30 or later

    • Ubuntu 22.04 or later

    • Fedora 36 or later

    • Debian 12.0 or later

  • NVIDIA RTX-class GPU (GeForce, Titan or Quadro)

  • NVIDIA display driver 531.14 or later

Additional recommendations to get the best performance:

Instruct Mode#

In this mode we provice GPT with the full combination of system, user,and assistant input slots with instruction(s) to produced an adequate response.

There are 3 different roles that are supported by Llama text models:

  • system: Explains the context in which the model interacts. It can include rules, guidelines, or other information to help the models response.

  • user: Represents input from the user interacting with the model. It can be comments, questions, or commands to the model.

  • assistant: Represents the generated responses from the model. The last tag is left empty to represent the upcoming reponse from the model.

  • tool: This role means “tool”. This is used represent messages with the output of a tool call when sent back.

Prompt Template#

The prompt template that was used to train the model must also be used then interacting with the model.

In addition to the different roles described above there markers written within angled brancket, for example <extra_id_0>. These are special tokens to help the language model understand the boundary of the instruction, the user input, and the model output.

Here are a few examples of prompt templates. Actual examples of prompts can be seen below.

  • Example 1

--- Prompt START ---
<extra_id_0>System
{system prompt}

<extra_id_1>User
{prompt}
<extra_id_1>Assistant

--- Prompt END ---

  • Example 2

--- Prompt START ---
<extra_id_0>System
{system prompt}

<tool> ... </tool>
<context> ... </context>

<extra_id_1>User
{prompt}
<extra_id_1>Assistant
<toolcall> ... </toolcall>
<extra_id_1>Tool
{tool response}
<extra_id_1>Assistant

--- Prompt END ---

  • Example 3

--- Prompt START ---
<extra_id_0>System
{system prompt}

<extra_id_1>User
<tool> ... </tool>
<context> ... </context>
{prompt}
<extra_id_1>Assistant
<toolcall> ... </toolcall>
<extra_id_1>Tool
{tool response}
<extra_id_1>Assistant

--- Prompt END ---

  • Example 4

--- Prompt START ---
<extra_id_0>System
{system prompt}

<extra_id_1>User
{prompt}
<extra_id_1>Assistant
{reponse}
<extra_id_1>User
{prompt}
<extra_id_1>Assistant
{reponse}
<extra_id_1>User
{prompt}
<extra_id_1>Assistant

--- Prompt END ---

  • Make sure to include the last line break after <extra_id_1>Assistant.

  • One or more <tool> ... </tool> and <context> ... </context> blocks can be supplied either in the System prompt or in User turns.

  • To keep (or reset) a conversation, previous conversation history can be added or omitted in the prompt template as shown in example 4.

Examples#

Basic Questioning#

<extra_id_0>System
Answer all questions like a pirate.

<extra_id_1>User
Hi there! How are you doing?
<extra_id_1>Assistant

Tool and Function Calling#

Handling multiple tools in a system prompt:

<extra_id_0>System
<tool> { "name": "get_weather_forecast", "description": "Get the weather forecast for the next 7 days", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. New York, NY" } }, "required": [ "location" ] } } </tool>
<tool> { "name": "get_current_weather", "description": "Get the current weather", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA" } }, "required": [ "location" ] } } </tool>

<extra_id_1>User
What is the weather like in New York right now?
<extra_id_1>Assistant
<toolcall> {"name": "get_current_weather", "arguments": {"location": "New York"}} </toolcall>
<extra_id_1>Tool
{"temperature": "15°C", "condition": "Cloudy"}
<extra_id_1>Assistant
The current weather in New York is 15°C with some clouds.

Role-Play#

<extra_id_0>System
You are Brandon, a sports reporter with a polite demeanor and excited about the upcoming fight. You need to get information from Slim about how he feels about his upcoming fight.
Slim is playing the role of User. Slim is a boxer that speaks with an African American dialect.  He has just gotten finished training and is a bit winded.
The location is a boxing gym that is hot due to all the people working out. Brandon is interviewing Slim about his upcoming fight with Big Jim. Brandon wants to know how Slim is feeling, physically, if he's ready for the fight. Slim says he's ready and predicts he'll knockout Big Jim in the first part of the fight, perhaps the fourth round.

<extra_id_1>User
Whoa, didn't see you there. Who are you?
<extra_id_1>Assistant
I'm Brandon, a sports reporter. I'm here to interview you about your upcoming fight with Big Jim.
<extra_id_1>User
Just finished an intense training session. I'm feeling hungry now.
<extra_id_1>Assistant
That's great to hear. How do you feel physically? Are you ready for the fight?
<extra_id_1>User
I have been hitting the gym a lot. Twice everyday. The last time I did that, I was in Boston. Have you ever been to Boston?
<extra_id_1>Assistant
No, I haven't. But I've heard it's a great city. So, how do you feel about the fight?

Editor Setup Example#

The Send text to GPT node is asyncronous and will return OnReponse after inference is completed.

Editor - Send Text to GPT node

Result in Application#

Using Simple Questioning above#

Editor - Textbox example

Using Role-Play Example above#

Editor - Role-Play example
On this page