Is this page helpful?

Tool Calling and MCP Integration#

NIM LLM supports OpenAI-compatible tool calling through vLLM’s tool calling engine. Tool calling lets models invoke external functions by returning structured tool calls instead of text responses, enabling integration with Model Context Protocol (MCP) servers and any client library that uses the OpenAI tools format.

Enable Tool Calling#

Tool calling requires two vLLM engine arguments: --enable-auto-tool-choice and a --tool-call-parser that matches your model. Pass these as CLI arguments to nim-serve:

docker run --gpus all \
  -e NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct \
  -p 8000:8000 \
  ${NIM_LLM_MODEL_FREE_IMAGE}:2.0.10 \
  nim-serve --enable-auto-tool-choice --tool-call-parser llama3_json

In environments where CLI arguments are not available, such as Kubernetes, use NIM_PASSTHROUGH_ARGS:

export NIM_PASSTHROUGH_ARGS="--enable-auto-tool-choice --tool-call-parser llama3_json"

Argument	Description
`--enable-auto-tool-choice`	Allows the model to choose between generating text or calling a tool. Required for tool calling.
`--tool-call-parser <parser>`	Built-in parser name for extracting tool calls from model output. Must match the model’s tool calling format (for example, `llama3_json` for Llama 3.1 and 3.3).
`--tool-parser-plugin <path>`	Path to a custom tool-call parser file. Accepts an absolute path or a path relative to the model checkpoint directory (the path is used as-is — no basename stripping). Required only when no built-in parser fits your model.

Both --enable-auto-tool-choice and --tool-call-parser are required together. For the full list of built-in parsers, the supported models, and how to provide a custom plugin file, refer to Custom Parsers and Chat Templates. For request/response format and schema details, refer to the vLLM tool calling documentation. For more information on how CLI arguments and NIM_PASSTHROUGH_ARGS work, refer to Advanced Configuration.

Once enabled, use the /v1/chat/completions endpoint with the tools parameter to send tool definitions and receive tool calls. For request/response format, examples, and the tool result loop, refer to the vLLM tool calling documentation.

MCP Integration#

NIM LLM does not connect to MCP servers directly. To use MCP tools, your client application connects to MCP servers, converts tool definitions to the OpenAI tools format, and sends them in the tools array of a Chat Completions request. The model returns tool calls in the response, and your application executes them against the MCP server and returns results to the model.

Schema Compatibility#

MCP tool schemas converted to OpenAI format may include extra fields such as strict or additionalProperties that are not part of the core function definition schema. vLLM accepts these fields without error. For details on schema handling, refer to the vLLM tool calling documentation.

LangChain and LangGraph Integration#

When building agentic applications with LangChain and LangGraph, use create_agent from langchain.agents to implement tool calling with NIM LLM. This agent correctly executes the full tool-calling loop: generating a tool call, executing the tool, feeding the result back to the model, and producing a final response.

Important

Ensure the container is launched with --enable-auto-tool-choice and --tool-call-parser <applicable-parser> in NIM_PASSTHROUGH_ARGS. Without these flags the server rejects "tool_choice": "auto" with a 400 error, and create_agent cannot invoke the tool.

Important

Do not pass a ProviderStrategy to create_agent for tool calling. That pattern bypasses the tool execution loop and causes the model to describe intended tool calls instead of executing them. Constructing create_agent with (llm, tools=[...]) and no strategy uses the correct execution loop.

Install the required packages inside an isolated Python environment. On Ubuntu 23.04+, Debian 12+, and other distributions that ship with a PEP 668 “externally managed” system Python, running pip install directly against the system interpreter fails with an error: externally-managed-environment message. Create a virtual environment first, then install:

python3 -m venv .venv
source .venv/bin/activate

pip install --upgrade pip
pip install "langchain-nvidia-ai-endpoints>=0.3.0" "langchain>=1.0.0" "langchain-core>=1.0.0"

create_react_agent from langgraph.prebuilt is deprecated in LangGraph 1.0 and scheduled for removal in LangGraph 2.0. Use create_agent from langchain.agents for new code. Existing code using from langgraph.prebuilt import create_react_agent still works with a LangGraphDeprecatedSinceV10 warning; migrate by swapping the import.

Define a tool with the @tool decorator so create_agent can bind it, then create the agent and invoke it end-to-end:

from langchain_core.tools import tool
from langchain_nvidia_ai_endpoints import ChatNVIDIA
from langchain.agents import create_agent


@tool
def get_weather(location: str) -> str:
    """Return the current weather for a location."""
    return f"The weather in {location} is 72F and sunny."


llm = ChatNVIDIA(base_url="http://localhost:8000/v1", model="your-model-name")
agent = create_agent(llm, tools=[get_weather])

response = agent.invoke(
    {"messages": [{"role": "user", "content": "What is the weather in Santa Clara, CA?"}]}
)
print(response["messages"][-1].content)

The snippet above pins the LangChain package versions to the API surface it uses: create_agent accepts the model as its first positional argument, tools are @tool-decorated callables, and the response is a dict with a messages list where the final entry is the agent’s answer. For more information on defining tools and building agents, refer to the LangChain agents documentation.

Troubleshooting#

Tool Calls Not Generated#

If the model returns text instead of a tool call, do the following:

Verify that --enable-auto-tool-choice is set.
Verify that --tool-call-parser is set to the correct parser for your model.
Check that the tools array is included in the request.

Error: “auto” Tool Choice Requires Configuration#

If you see "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set, both arguments must be provided when launching the container.