Tool Safety Check
Tool Safety Check judges a tool call’s arguments, or a tool result’s content, against safety instructions you write per tool, evaluated by a judge LLM. It is scoped per tool through the same rails.tool_call.per_tool / rails.tool_result.per_tool dispatch mechanism as Regex Detection, so different tools can use different judge models or different instructions.
To use the tool safety check, you should:
-
Include the desired judge model in the models section of the
config.ymlfile:The
typeis a unique identifier for the model that will be passed to the tool call and tool result rails as a parameter.tools: [...]above is a placeholder: replace it withrun_sql’s actual function declaration and JSON schema, either here (static, applies to every request) or per request: inoptions.llm_params.toolswith the Python API, or in thetoolsfield of a chat completions request to the server, which the server currently accepts only for non-streaming requests on a configuration withpassthrough: true.@tool_call_validationblocks a call to a tool that isn’t declared either way, before the judge ever runs. Refer to Tool Calling for the full declaration reference. -
Scope the check to a tool in the
tool_callandtool_resultrails sections of theconfig.ymlfile, naming the judge model with$model=and a prompt variant with$variant=:You must define the models in the
modelssection of theconfig.ymlfile before using them here. If a$model=names a model type that you haven’t defined in themodelssection, the configuration fails to load with an error that names the flow and, for aper_toolflow, the tool. Thetool safety check tool callandtool safety check tool resultflows check the tool call’s arguments and the tool result’s content, respectively. The$modelparameter specifies the judge model, matching the content safety check’s convention. The$variantparameter selects which prompt to render for this invocation, matching self check’s$variantconcept: it lets you reuse one prompt across multiple tools, or give the same tool multiple distinct checks, entirely through config. -
Specify the prompt for each
$model=/$variant=task in theprompts.ymlfile:If a prompt for the exact
$model=/$variant=task name is not defined, or it does not declare anoutput_parser, an exception will be raised when the configuration is loaded.The
tool safety check tool callprompt renders{{ tool_name }}and{{ tool_call_arguments }}(the call’s arguments, JSON-encoded). Thetool safety check tool resultprompt renders{{ tool_name }}and{{ tool_result_content }}(the result’s content, JSON-encoded if it is a list of content blocks). -
output_parseris required on every tool safety check prompt. The example above uses the built-inparse_tool_safety_verdictparser, which expects a response of the form “safe” or “unsafe:<reason>”:- If the response is just “safe”, the content is considered safe. Surrounding punctuation or markup is ignored, but any other word is not.
- If the response starts with “unsafe”, the content is considered unsafe. The parser extracts any reasons after it (separated by semicolons), but the rail doesn’t record them in its block, since they could quote sensitive content.
- Any other response, such as “This violates the safety policy”, is treated as a broken judge response and raises, rather than being interpreted as a safety decision. Keeping the prompt’s output instructions strict, as in the example above, makes this rare.
You can set
output_parserto a different built-in parser instead (for exampleis_content_safe, whose response format and keyword vocabulary differ); IORails does not support registering custom parser functions. -
Optionally, narrow which arguments are sent to the judge on the output direction with
$argument=, the same scoping regex detection supports:Only the named argument is sent to the judge, reducing both the data sent to a (possibly third-party) model and the tokens spent per call.
$argument=is valid ontool safety check tool callonly;tool safety check tool resulthas no argument structure to scope and does not accept it.
The tool safety check tool call and tool safety check tool result rails execute the tool_safety_check_tool_call and tool_safety_check_tool_result actions respectively.
Complete Example
Here’s a comprehensive example configuration, checking a run_sql tool’s calls and results:
Schema Validation
tool safety check tool call runs behind @tool_call_validation: a call to a tool the model has not declared, or whose arguments violate the tool’s declared JSON schema, is blocked before the judge model is ever called. tool safety check tool result has no such gate, matching regex detection’s asymmetry between the two directions.
Engine Support
Per-tool checks are an IORails-only capability by design. For the full per-rail matrix, refer to Rail Engine Support.