About Building Function-Calling Benchmarks#

This section describes how to build a custom function-calling benchmark as Apache Parquet files with the nemotron steps run byob/bfcl command, and how to evaluate a candidate model against the result.

Unlike the multiple-choice flow in About Building Multiple-Choice Question Benchmarks, generation here does not ask a model to invent content. You supply an Oracle Pack: a tool catalog, conversation templates, an executable backend or an HTTP endpoint, fixtures, and assertions. The pipeline renders conversations from your templates, derives the expected tool calls, then replays every task against the real backend and checks the assertions. A task that the oracle cannot reproduce twice does not reach the benchmark.

That difference is the point of the design. The pack, not a model, is the source of truth about what a correct tool call looks like, so the benchmark can state why each expected answer is correct.

Tip

New to this flow? Follow Getting Started with Building Function-Calling Benchmarks once with the bundled tiny_oracle_pack, then use the grids below to jump to a task guide, a concept, or a field reference.

When to Use#

The nemotron steps run byob/bfcl command enables the following outcomes.

A function-calling benchmark over your own tools, in your own domain, where each expected call is justified by an executable oracle rather than by a model’s opinion.
A repeatable Parquet artifact plus a run_manifest.json that pins the exact pack bytes the benchmark came from, so a published benchmark can always be traced to its source.
Evaluation of one or more candidate models against that benchmark, either by comparing against recorded gold calls or by replaying the candidate’s calls against a live oracle and running the assertions.
Optional translation of a published benchmark into another language, as a separate run.

Pipeline Summary#

At a high level, the step performs the following work.

Prepare: normalize the configuration, load and fingerprint the pack, then validate it. Validation awards a certification tier (gold, silver, or prototype), and a publication-eligible run requires the Gold tier.
Generate: expand templates into task instances, plan conversations, render turns, derive expected traces, validate against the tool schemas, and replay each task against the oracle. Optional stages add surface-quality checks and deduplication or balancing before publication.
Translate, optional: localize a published benchmark and write a new benchmark.parquet.
Evaluate, a separate run: score candidate models against a published benchmark and write a report.

Refer to Pipeline Overview for the stage-by-stage account.

Choose a Way to Get a Pack#

The pack is the hard part. If you have a tool interface, records, and behavior but have not chosen a route yet, start at From Domain Assets to an Oracle Pack. The three flows below converge on the same generation pipeline and the same Gold gate, which means the trust story does not depend on how the pack was written.

Flow	You start from	Guide
Choose a route	Domain assets, no pack yet	From Domain Assets to an Oracle Pack
Manual	Your own knowledge of the domain and its tools	Hand-Author an Oracle Pack
Assisted, conventional source	A Python package or a reviewed HTTP service	Author a Pack with Model Assistance
Assisted, MCP source (experimental)	A running MCP server	Onboard an MCP Server as the Oracle

In the assisted flows a model may propose pack semantics, but it can never award a certification tier, approve its own output, or bypass executable replay. Authoring Flows explains where the human decisions sit and why they are separate.

Documentation Series#

Tutorials

Follow complete journeys for a first benchmark, assisted authoring, and a manual pack lifecycle through evaluation.

hands-on

Function-Calling Benchmark Tutorials

How-To Guides

Author a pack by hand or with model assistance, onboard an MCP server, publish at scale, and evaluate candidates.

task-based

How-To Guides

Concepts

What an Oracle Pack is, how the stages fit together, where the authorization boundaries sit, and how scoring works.

concept-focused

Concepts

Reference

Oracle Pack fields, backend and template contracts, run configuration, output artifacts, and the symptom-to-fix index.

specification

Reference

Source-Tree Documentation#

The rendered pages above are the operator-facing guides. The repository also ships developer documentation and normative contracts beside the BFCL implementation. Start with the byob/bfcl source README for a source-oriented map of the pipeline, then use the byob/references directory for the Oracle Pack, evaluation, assisted-authoring, MCP, and audit contracts.

Some source-tree contracts are runtime inputs and are content-hashed into published artifacts. When an operator guide and a normative contract differ, treat the source-tree contract as authoritative.

All Documentation#

Tutorials

Guide	What you will do
Getting Started with Building Function-Calling Benchmarks	Run `nemotron steps run byob/bfcl` with `tiny.yaml` and inspect the outputs
Run Assisted Authoring End to End	Run source intake, assisted drafting, review, Gold publication, and evaluation without credentials
Publish and Evaluate a Manual Oracle Pack	Validate, publish, and evaluate a reviewed manual Oracle Pack

How-To Guides

Guide	What you will do
From Domain Assets to an Oracle Pack	Choose manual or model-assisted authoring and turn reviewed domain assets into an executable pack
Hand-Author an Oracle Pack	Scaffold, fill in, and validate an Oracle Pack of your own
Author a Pack with Model Assistance	Draft a pack from a Python package or HTTP service with model assistance
Onboard an MCP Server as the Oracle	Onboard a running MCP server as the oracle
Publish a Benchmark Release	Choose a publication budget and produce a released benchmark
Translate a Published Benchmark	Localize a published benchmark without changing oracle truth
Evaluate a Candidate Model	Score candidate models and read the evaluation report

Concepts

Guide	What you will learn
Pipeline Overview	What each stage consumes, transforms, writes, and requires from the operator
Follow One Task Through the Pipeline	How one missing-slot task changes from an abstract template into a published row
The Oracle Pack	Pack layout, certification tiers, and the Gold gate
Authoring Flows	The three authoring flows and the two authorization boundaries
Evaluation	Trace and executable scoring, and the gates that run before inference

Reference

Guide	What you will find
Oracle Pack Inputs	Required pack files, manifest fields, fill order, and cross-file tool lineage
Domain Brief	Human-owned assisted-authoring context, content, and safety rules
Probe Plan	Intake probe cases, A2 coverage, timeout cleanup, and `check_probe_plan`
Manifest Fields	Manifest paths, languages, shared text, confirmation vocabulary, and examples
Tool Catalog and Fixtures	Public tool schemas, deterministic records, and fixture-slot contracts
Python Backend Contract	The four required backend callables and executable-oracle invariants
Task Template Fields	Core template fields, slots, milestones, and turn-policy examples
Assertion Contract	Assertion signatures, exports, capabilities, outcomes, and failures
Validation Case Fields	Probe fields, result classes, coverage, chaining, and failures
Endpoint Configuration	Endpoint identity, credentials, TLS, attestation, and HTTP routes
Held-Out Policy	Fixture and template reservations and leak enforcement
Generation Configuration Reference	Generation YAML fields, block by block
Evaluation Configuration Reference	Evaluation YAML fields and the three envelopes
Output Files	Every path written under `output_dir` / `expt_name`
Troubleshooting	Symptom-to-fix index drawn from the error taxonomy

What You Need#

A Nemotron clone with dependencies installed, including the byob extra from uv sync --extra byob.
An Oracle Pack. To learn the flow first, use the bundled src/nemotron/steps/byob/data/tiny_oracle_pack, which exists to exercise the pipeline quickly.
For evaluation, a candidate model endpoint and its credentials. Generation itself calls no model unless you explicitly enable a model-authored surface role.
For the assisted authoring flows, a model endpoint for drafting and the corresponding feature flag, as described in Author a Pack with Model Assistance.

Quick Start#

Follow Getting Started with Building Function-Calling Benchmarks if you have not run the step yet.
Use From Domain Assets to an Oracle Pack to choose manual or model-assisted authoring for your own tool interface, state, and business behavior.
Open Oracle Pack Inputs for the required files and its standardized reference link for each artifact you are filling.
Read The Oracle Pack for the trust model and Gold tier, then follow the selected authoring guide.
Open Generation Configuration Reference or Evaluation Configuration Reference for run-level YAML fields.

Limitations and Considerations#

Executable replay costs time. Every candidate task is reset and replayed against the oracle, twice, and its assertions are evaluated. A publication-scale run is bounded by your backend’s speed, not by model latency.
Pack admission and publication are separate gates. Generation refuses a pack that is not Gold-eligible. A Gold pack may still run under lineage.policy: smoke_no_publication; that run writes benchmark artifacts for pipeline verification but records that its lineage — the run’s declaration of which model roles influenced the benchmark and whether it may be published — is not eligible for release. See The Oracle Pack and Publish a Benchmark Release.
Pack code executes. The pipeline imports and runs your backend and assertions. It does so in a separate process with a sanitized environment and enforced timeouts, and Gold requires that isolation, but the pack is still code you are choosing to trust.
Model roles are opt-in and pinned. Enabling a model-authored surface role requires a pinned, unambiguous model identity, because a benchmark whose wording came from an unrecorded model cannot be reproduced.
The MCP transport is experimental. Only Mode A is implemented, and it is disabled unless you opt in. Refer to Onboard an MCP Server as the Oracle.