Skip to navigation

Tool Safety Check judges a tool call’s arguments, or a tool result’s content, against safety instructions you write per tool, evaluated by a judge LLM. It is scoped per tool through the same rails.tool_call.per_tool / rails.tool_result.per_tool dispatch mechanism as Regex Detection, so different tools can use different judge models or different instructions.

To use the tool safety check, you should:

  1. Include the desired judge model in the models section of the config.yml file:

    models:
    - type: main
    engine: nim
    model: meta/llama-3.3-70b-instruct
    parameters:
    tools: [...]
    - type: llama_guard
    engine: nim
    model: meta/llama-guard-3-8b

    The type is a unique identifier for the model that will be passed to the tool call and tool result rails as a parameter. tools: [...] above is a placeholder: replace it with run_sql’s actual function declaration and JSON schema, either here (static, applies to every request) or per request: in options.llm_params.tools with the Python API, or in the tools field of a chat completions request to the server, which the server currently accepts only for non-streaming requests on a configuration with passthrough: true. @tool_call_validation blocks a call to a tool that isn’t declared either way, before the judge ever runs. Refer to Tool Calling for the full declaration reference.

  2. Scope the check to a tool in the tool_call and tool_result rails sections of the config.yml file, naming the judge model with $model= and a prompt variant with $variant=:

    rails:
    tool_call:
    per_tool:
    run_sql: ["tool safety check tool call $model=llama_guard $variant=run_sql_check"]
    tool_result:
    per_tool:
    run_sql: ["tool safety check tool result $model=llama_guard $variant=run_sql_check"]

    You must define the models in the models section of the config.yml file before using them here. If a $model= names a model type that you haven’t defined in the models section, the configuration fails to load with an error that names the flow and, for a per_tool flow, the tool. The tool safety check tool call and tool safety check tool result flows check the tool call’s arguments and the tool result’s content, respectively. The $model parameter specifies the judge model, matching the content safety check’s convention. The $variant parameter selects which prompt to render for this invocation, matching self check’s $variant concept: it lets you reuse one prompt across multiple tools, or give the same tool multiple distinct checks, entirely through config.

  3. Specify the prompt for each $model=/$variant= task in the prompts.yml file:

    prompts:
    - task: "tool_safety_check_tool_call $model=llama_guard $variant=run_sql_check"
    content: |
    You are reviewing a call to the `run_sql` tool.
    Arguments: {{ tool_call_arguments }}
    Block any query that reads from the `payroll` or `ssn` tables, or that
    contains a DROP, DELETE, or TRUNCATE statement.
    Provide your safety assessment in exactly one line:
    - If it is safe, respond with "safe" and nothing more.
    - If it is unsafe, respond with "unsafe: <reason>", separating multiple reasons with semicolons.
    output_parser: parse_tool_safety_verdict
    - task: "tool_safety_check_tool_result $model=llama_guard $variant=run_sql_check"
    content: |
    You are reviewing the result of a call to the `run_sql` tool.
    Result: {{ tool_result_content }}
    Block any result that contains a full credit card number or SSN.
    Provide your safety assessment in exactly one line:
    - If it is safe, respond with "safe" and nothing more.
    - If it is unsafe, respond with "unsafe: <reason>", separating multiple reasons with semicolons.
    output_parser: parse_tool_safety_verdict

    If a prompt for the exact $model=/$variant= task name is not defined, or it does not declare an output_parser, an exception will be raised when the configuration is loaded.

    The tool safety check tool call prompt renders {{ tool_name }} and {{ tool_call_arguments }} (the call’s arguments, JSON-encoded). The tool safety check tool result prompt renders {{ tool_name }} and {{ tool_result_content }} (the result’s content, JSON-encoded if it is a list of content blocks).

  4. output_parser is required on every tool safety check prompt. The example above uses the built-in parse_tool_safety_verdict parser, which expects a response of the form “safe” or “unsafe: <reason>”:

    • If the response is just “safe”, the content is considered safe. Surrounding punctuation or markup is ignored, but any other word is not.
    • If the response starts with “unsafe”, the content is considered unsafe. The parser extracts any reasons after it (separated by semicolons), but the rail doesn’t record them in its block, since they could quote sensitive content.
    • Any other response, such as “This violates the safety policy”, is treated as a broken judge response and raises, rather than being interpreted as a safety decision. Keeping the prompt’s output instructions strict, as in the example above, makes this rare.

    You can set output_parser to a different built-in parser instead (for example is_content_safe, whose response format and keyword vocabulary differ); IORails does not support registering custom parser functions.

  5. Optionally, narrow which arguments are sent to the judge on the output direction with $argument=, the same scoping regex detection supports:

    rails:
    tool_call:
    per_tool:
    run_sql: ["tool safety check tool call $model=llama_guard $variant=run_sql_check $argument=query"]

    Only the named argument is sent to the judge, reducing both the data sent to a (possibly third-party) model and the tokens spent per call. $argument= is valid on tool safety check tool call only; tool safety check tool result has no argument structure to scope and does not accept it.

The tool safety check tool call and tool safety check tool result rails execute the tool_safety_check_tool_call and tool_safety_check_tool_result actions respectively.

Complete Example

Here’s a comprehensive example configuration, checking a run_sql tool’s calls and results:

models:
- type: main
engine: nim
model: meta/llama-3.3-70b-instruct
parameters:
tools: [...]
- type: llama_guard
engine: nim
model: meta/llama-guard-3-8b
rails:
tool_call:
per_tool:
run_sql: ["tool safety check tool call $model=llama_guard $variant=run_sql_check $argument=query"]
tool_result:
per_tool:
run_sql: ["tool safety check tool result $model=llama_guard $variant=run_sql_check"]
prompts:
- task: "tool_safety_check_tool_call $model=llama_guard $variant=run_sql_check"
content: |
You are reviewing a call to the `run_sql` tool.
Arguments: {{ tool_call_arguments }}
Block any query that reads from the `payroll` or `ssn` tables, or that
contains a DROP, DELETE, or TRUNCATE statement.
Provide your safety assessment in exactly one line:
- If it is safe, respond with "safe" and nothing more.
- If it is unsafe, respond with "unsafe: <reason>", separating multiple reasons with semicolons.
output_parser: parse_tool_safety_verdict
- task: "tool_safety_check_tool_result $model=llama_guard $variant=run_sql_check"
content: |
You are reviewing the result of a call to the `run_sql` tool.
Result: {{ tool_result_content }}
Block any result that contains a full credit card number or SSN.
Provide your safety assessment in exactly one line:
- If it is safe, respond with "safe" and nothing more.
- If it is unsafe, respond with "unsafe: <reason>", separating multiple reasons with semicolons.
output_parser: parse_tool_safety_verdict

Schema Validation

tool safety check tool call runs behind @tool_call_validation: a call to a tool the model has not declared, or whose arguments violate the tool’s declared JSON schema, is blocked before the judge model is ever called. tool safety check tool result has no such gate, matching regex detection’s asymmetry between the two directions.

Engine Support

FlowLLMRailsIORails
tool safety check tool call✗✓
tool safety check tool result✗✓

Per-tool checks are an IORails-only capability by design. For the full per-rail matrix, refer to Rail Engine Support.