Is this page helpful?

Tool Calling and MCP Integration#

NIM LLM and VLM supports OpenAI-compatible tool calling on both the vLLM and SGLang backends. Models return structured tool calls, and your client executes the requested functions. This supports integration with Model Context Protocol (MCP) servers and clients that use the OpenAI tools format.

Prerequisites#

Before you start, complete the following prerequisites:

  1. Deploy a model that supports tool calling. For deployment instructions, refer to the Quickstart.

  2. Identify the --tool-call-parser name that matches your model and backend. For parser selection and custom vLLM plugins, refer to Custom Parsers and Chat Templates.

Enable Tool Calling#

Configure the parser at container startup. The required arguments depend on the backend:

Backend

Arguments

vLLM

--enable-auto-tool-choice --tool-call-parser <parser>

SGLang

--tool-call-parser <parser>

SGLang does not accept --enable-auto-tool-choice. Parser names also differ between engines, so use a name supported by the backend in your image.

To enable tool calling, complete the following steps:

  1. Pass the appropriate arguments to nim-serve. The following example uses a vLLM image and the llama3_json parser:

    docker run --gpus all \
  -e NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct \
  -p 8000:8000 \
  ${NIM_LLM_MODEL_FREE_IMAGE}:2.1.4 \
  nim-serve --enable-auto-tool-choice --tool-call-parser llama3_json

  2. For a Kubernetes workload, pass the same backend arguments through the container args field. For the NIM Helm chart, use customArgs. For argument precedence and the NIM_PASSTHROUGH_ARGS fallback, refer to Advanced Configuration.

  3. Send tool definitions in the tools array of a /v1/chat/completions request.

  4. Execute the tool calls returned by the model, then send the results back in the conversation.

For request and response examples, refer to vLLM Tool Calling or SGLang Tool Parser.

Use MCP Tools with NIM#

NIM LLM and VLM does not connect to MCP servers directly. Your client application connects to MCP servers, converts tool definitions to the OpenAI tools format, and sends them in the tools array of a Chat Completions request.

To use MCP tools with NIM, complete the following steps:

  1. Connect your client application to the MCP servers that expose the tools you need.

  2. Convert those MCP tool definitions to the OpenAI tools format.

  3. Send the converted definitions in the tools array of a Chat Completions request.

  4. Execute the tool calls from the model response against the MCP server, then return the results to the model.

Use tool definitions accepted by the selected backend. For schema details, refer to vLLM Tool Calling or SGLang Tool Parser.

Integrate LangChain and LangGraph#

When building agentic applications with LangChain and LangGraph, use create_agent from langchain.agents to implement tool calling with NIM LLM and VLM. This agent correctly executes the full tool-calling loop: generating a tool call, executing the tool, feeding the result back to the model, and producing a final response.

To integrate LangChain and LangGraph, complete the following steps:

  1. Configure tool calling for your model and backend as described in Enable Tool Calling.

    Important

    Do not pass a ProviderStrategy to create_agent for tool calling. That pattern bypasses the tool execution loop and causes the model to describe intended tool calls instead of executing them. Constructing create_agent with (llm, tools=[...]) and no strategy uses the correct execution loop.

  2. Install the required packages inside an isolated Python environment.

    Create a virtual environment before installing the packages:

    python3 -m venv .venv
source .venv/bin/activate

pip install --upgrade pip
pip install "langchain-nvidia-ai-endpoints>=0.3.0" "langchain>=1.0.0" "langchain-core>=1.0.0"

  3. Define a tool with the @tool decorator so create_agent can bind it, then create the agent and invoke it end-to-end.

    from langchain_core.tools import tool
from langchain_nvidia_ai_endpoints import ChatNVIDIA
from langchain.agents import create_agent


@tool
def get_weather(location: str) -> str:
    """Return the current weather for a location."""
    return f"The weather in {location} is 72F and sunny."


llm = ChatNVIDIA(base_url="http://localhost:8000/v1", model="your-model-name")
agent = create_agent(llm, tools=[get_weather])

response = agent.invoke(
    {"messages": [{"role": "user", "content": "What is the weather in Santa Clara, CA?"}]}
)
print(response["messages"][-1].content)

    In this example, create_agent accepts the model as its first positional argument, tools are @tool-decorated callables, and the response is a dict with a messages list where the final entry is the agent’s answer. For more information on defining tools and building agents, refer to the LangChain agents documentation.

Troubleshooting#

Use the following topics when the model does not return tool calls or the server rejects "tool_choice": "auto".

Tool Calls Not Generated#

If the model returns text instead of a tool call, do the following:

  • On vLLM, verify that --enable-auto-tool-choice is set.

  • Verify that --tool-call-parser is set to the correct parser for your model.

  • Check that the tools array is included in the request.

Error: “auto” Tool Choice Requires Configuration (vLLM Backend)#

If you see "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set, both arguments must be provided when launching the container.

Next Steps#

After tool calling is enabled, continue with the following topics: