Reference#
Field-level and artifact-level detail for nemotron steps run byob/bfcl with the bfcl family.
Required files, manifest fields, fill order, and how tool names connect the pack.
Required assisted-authoring context, content boundaries, safety rules, and a complete example.
Intake cases, A2 coverage including timeout cleanup, check_probe_plan, and common refusals.
Identity, paths, languages, shared text, confirmation vocabulary, and field examples.
Public function schemas, deterministic reset records, minimal examples, and validation.
The four required callables, deterministic reset, structured errors, and confirmation behavior.
Core fields, slot binding, milestones, turn policies, and complete conversation shapes.
Signatures, exports, capabilities, outcome semantics, examples, and failure reasons.
Case fields, result classes, coverage, chaining, examples, and executable checks.
Pinned identity, credentials, TLS, attestation, HTTP routes, and validation failures.
Reserved fixture and template ids, backend-state policy, validation, and leak checks.
Every accepted key in a generation YAML, block by block, with types and defaults.
Scoring contract, modes, candidate declarations, and the difference between the three bundled envelopes.
Every path written under output_dir / expt_name, and which artifacts are content-addressed.
Symptom-to-fix index derived from the step’s declared error taxonomy.
Bundled Configurations#
The step ships runnable configurations under src/nemotron/steps/byob/bfcl/config/. Copy the one closest to your intent rather than starting from an empty file.
Every number in the example files is a worked example for the pack that file points at, not a framework default. Copying another pack’s task counts and diversity limits is the most common way to make the balancing stage infeasible.
File |
Purpose |
|---|---|
|
Pipeline verification run against the bundled |
|
Annotated template. Its pack path is a placeholder, so it cannot publish an example domain by omission. |
|
Domain-sized verification run. Copy it and repoint it at your own pack. |
|
Publication-scale, template-only Gold profile with a worked budget and balancing targets. |
|
The same executable cases with an opt-in model-authored surface role. |
|
Annotated evaluation template to resolve into your own config. |
|
Direct evaluation envelope. |
|
Launcher evaluation envelope. |
|
Localization of an already published benchmark. |
Command-line Conventions#
These pages use nemotron steps run byob/bfcl -c <CONFIG> stage=<STAGE>. The same code path
is reachable directly as python -m nemotron.steps.byob.scripts.run --config <CONFIG> --stage <STAGE>, which is convenient when the nemotron console script is not on PATH.
Every shipped configuration declares family: bfcl, so the direct form needs no --family;
a configuration that omits the key falls back to the MCQ family, so keep it declared in
configs you write yourself.
The helper commands under nemotron.steps.byob.scripts — the pack validator, the bias auditor, the release archiver, the authoring and MCP release commands — share one exit contract, so a wrapper can branch on the status alone:
Status |
Meaning |
Output |
|---|---|---|
|
The command ran and the answer was yes: the pack is Gold-eligible, the audit passed, the artifact was written. |
The result document on stdout. |
|
The command could not reach an answer. A path was missing, a file would not parse, an invariant was violated. |
A JSON failure envelope on stderr with |
|
The command ran and the answer was no. The pack is not Gold-eligible, the audit found an unexcepted failure, the review packet is blocked. |
The full verdict document on stdout, so you can see which check said no. |
The distinction between 1 and 2 is what makes these commands safe to automate: retry on 1, because a crash may be transient; never retry on 2, because the verdict will not change until a human changes the inputs.
The evaluator is the exception. nemotron steps run byob/bfcl with stage=eval publishes a wider taxonomy — 2 through 7 — because an operator needs to know whether to edit a config, fix a candidate endpoint, or investigate a contamination finding. Refer to Evaluate a Candidate Model.
Normative Contracts#
These pages describe the operator-facing surface. The normative contracts ship with the repository
beside the code that enforces them. Each link displays a path relative to the repository root.
Use the path as the value for configuration fields such as scoring.contract.
The step’s own declared inputs, outputs, and error taxonomy are in src/nemotron/steps/byob/bfcl/step.toml.