Function Calling
You can connect NIM to external tools and services using function calling (also known as tool calling). By providing a list of available functions, NIM can choose to output function arguments for the relevant function(s) which you can execute to augment the prompt with relevant external information.
Function calling is controlled using the tool_choice, tools, and parallel_tool_calls parameters. Only the following models support function calling, and only a subset of those models support parallel tool calling.
Llama-3.1-8b-Instruct
Llama-3.1-70b-Instruct
To use function calling, modify the tool_choice, tools, and parallel_tool_calls parameters.
Parameter |
Description |
|---|---|
tool_choice |
Specifies how the model should choose tools. Has four options: "none", "auto", "required", or named tool choice. Requires that tools is also set. |
tools |
The list of tool objects that define the functions the model can call. Requires that tool_choice is also set. |
parallel_tool_calls |
Boolean value (True or False) specifying whether to make tool calls in parallel. Default is False. Requires that the model supports it. |
tool_choice options
"none": Disables the use of tools."auto": Enables the model to decide whether to use tools and which ones to use."required": Forces the model to use a tool, but the model chooses which one.Named tool choice: Forces the model to use a specific tool. It must be in the following format:
{ "type": "function", "function": { "name": "name of the tool goes here" } }
Note: tool_choice can only be set when tools is also set, and vice versa. These parameters work together to define and control the use of tools in the model’s responses.
For further information on these parameters and their usage, see the OpenAI API documentation.
The following example uses NIM to create an agent that can connect to an external weather API and respond to weather queries.
client = OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="not-used")
# Make a list of available functions
available_tools = [
{
"type": "function",
"function": {
"name": "get_current_weather",
"description": "Get the current weather",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA"
},
"format": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "The temperature unit to use. Infer this from the user's location."
}
},
"required": ["location", "format"]
}
}
}
]
messages = [
{"role": "user", "content": "Is it hot in Pittsburgh, PA right now?"}
]
chat_response = client.chat.completions.create(
model="mistralai/Mistral-7B-Instruct-v0.3",
messages=messages,
tools=available_tools,
tool_choice="auto",
stream=False
)
assistant_message = chat_response.choices[0].message
messages.append(assistant_message)
print(assistant_message)
# Prints
# ChatCompletionMessage(content=None, role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='chatcmpl-tool-73995047e99c49f5a2f3d1349edc3561', function=Function(arguments='{"location": "Pittsburgh, PA", "format": "fahrenheit"}', name='get_current_weather'), type='function')])
# The model has responded with a ChatCompletionMessageToolCall. This is the point where we call the external function `get_current_weather`
# with the arguments `{"location": "Pittsburgh, PA", "format": "fahrenheit"}`
# Let's assume the external function returns 88 F as the current weather in Pittsburgh, PA
# We'll specify the result and use the returned tool call ID as follows
tool_call_result = 88
tool_call_id = assistant_message.tool_calls[0].id
tool_function_name = assistant_message.tool_calls[0].function.name
messages.append({"role": "tool", "content": str(tool_call_result), "tool_call_id": tool_call_id, "name": tool_function_name})
chat_response = client.chat.completions.create(
model="mistralai/Mistral-7B-Instruct-v0.3",
messages=messages,
tools=available_tools,
tool_choice="auto",
stream=False
)
# Check final response
assistant_message = chat_response.choices[0].message
print(assistant_message)
# Prints
# ChatCompletionMessage(content=' It is 88 degrees Fahrenheit in Pittsburgh, PA right now.', role='assistant', function_call=None, tool_calls=None)