Prerequisites#

NGC (NVIDIA GPU Cloud) Account#

Create an account on NGC
Generate an API Key
Log in to the NVIDIA Container Registry, using your NGC API key as password:

docker login nvcr.io --username='$oauthtoken'

Set Up Your NIM Cache#

The NIM needs a directory on your system called the NIM cache, where it can download and then read the model checkpoints on subsequent runs. The container runs as a non-root user (UID 1000), so this directory must be writable by that user.

## Create the NIM cache directory in a location with sufficient storage
mkdir -p ~/.cache/nim

## Set the NIM cache directory permissions to allow all (a) users to read, write, and execute (rwx)
chmod a+rwx ~/.cache/nim

Only the top-level directory needs these permissions. The NIM owns everything it writes beneath it, so after the first run, a recursive chmod -R run as your own user fails on those files.

Refer to Storage for sizing, then continue to Getting Started.

Notes on Sampling Behavior#

Integration step count and the recycle schedule are one setting, not two. Omitting sampling.num_steps, sampling.recycle_strategy and sampling.recycle_schedule runs the checkpoint’s own default preset, recycling included; any of the three pins the trajectory instead. Refer to Recycle Schedules and the API Reference for the full set of fields. Sending num_steps and a recycle_strategy is allowed: the step count wins and the strategy contributes only its recycle shape.
sampling.num_steps is schema-bounded to 1-1000, but that bound belongs to the contract, not to the checkpoint. Flat sampling has a lower floor the loaded weights impose, and values below it are refused, not run; shorter trajectories that are accepted still degrade. Refer to Step Count Floor.
Replicates are batched by default. Omitting sampling.batch_size runs every replicate the request asks for in a single forward pass, which is the fastest way to serve them. Refer to Replicates and Batching for what that costs and when to set it back to 1.

Recycle Schedules#

Recycling runs the trunk more than once inside a single integration step, spending the extra passes where the fold is decided. Because of that, design time scales with trunk passes, not with step count, and all three named strategies are the same 60 steps:

`sampling.recycle_strategy`	Steps	Trunk passes
`thinking_low`	60	110	fastest, lowest quality
`thinking_medium`	60	185	the checkpoint’s default
`thinking_high`	60	295	~1.6x the work of medium

These are the only three names served, and they are spelled exactly as the checkpoint’s own configuration spells them. A recycle_strategy that is not one of them is not rejected: thinking_medium runs instead and the response says so in warnings, naming what was asked for, what ran, and the served names. The schedule names earlier builds of this NIM served are retired and resolve the same way.

Warning

Avoid any schedule shorter than 60 steps. On the shipped checkpoint a 50-step trajectory collapsed to poly-alanine output with no side chains, reproduced with the upstream reference script on the same input and seed. A retired strategy name on its own is safe, since it runs thinking_medium, but a short custom schedule runs as written – and so does a named strategy sent with a num_steps below 60, because num_steps wins and the strategy’s recycle shape is stretched onto it. Any name sent with num_steps: 50 therefore runs a 50-step trajectory, whatever the name resolves to. The response says so in warnings; a client that ignores warnings sees only a normal 200.

Step Count Floor#

The schedule the shipped checkpoint composes samples the tail of the trajectory at higher density, and needs a minimum number of integration steps to lay that tail out. That minimum is a property of the loaded weights – tail_steps x tail_step_multiplier from the checkpoint’s own generation settings – and it is 20 for the shipped checkpoint.

It is enforced, and it applies to both ways of pinning a step count:

a flat sampling.num_steps below the floor is refused with invalid_input (HTTP 422), naming the exact minimum;
a sampling.recycle_schedule whose bands sum to fewer steps than the floor is refused the same way.

A named recycle_strategy always satisfies it: all three are 60 steps. The schema’s 1-1000 bound is deliberately left wide, because the floor moves with the checkpoint and a bound baked into the contract would be wrong for the next one; the error carries the number that actually applies.

Warning

Clearing the floor is not the same as producing a good design. On the shipped checkpoint, trajectories shorter than 60 steps have returned degenerate poly-alanine output, well above the 20-step floor. Treat the floor as the point below which a request is rejected, and 60 steps – any of the three named strategies – as the point below which output is not trustworthy.

Custom Schedules#

sampling.recycle_schedule takes the bands directly, as [[steps, recycle_depth], ...] covering the trajectory in order — the same form the named strategies are built from. A depth of 0 runs the trunk once for those steps; 6 re-runs it six times per step.

{
  "sampling": {
    "recycle_schedule": [[5, 1], [5, 2], [5, 3], [5, 5], [10, 6], [5, 5], [5, 3], [10, 2], [10, 1]]
  }
}

The step counts add up to the number of integration steps, so num_steps can be omitted; sending one that disagrees with an explicit recycle_schedule is a request error. It cannot be combined with recycle_strategy — a named strategy is a schedule, so sending both is refused rather than silently resolved.

A named recycle_strategy sent with a num_steps that disagrees is handled differently, and the asymmetry is deliberate: bands you wrote yourself are taken literally, so a contradicting total is a mistake worth refusing, while a strategy is a shape the service owns and can restate at another length. num_steps wins, the strategy’s depths and their order are preserved, and the band widths are rescaled in proportion to reach that total. The response reports recycle_strategy: "custom" with the resulting bands, and warnings names the strategy it was stretched from. Send num_steps alone for flat sampling, or omit it to run the strategy exactly as the checkpoint defines it. Depths are bounded to 0-32 and a schedule to 64 bands, the summed step count is held to the same floor as a flat num_steps, and total trunk passes are capped per deployment by NIM_MAX_TRUNK_PASSES (reported as limits.max_trunk_passes).

The schedule above is thinking_medium’s own, written out; a schedule whose bands match a named strategy is recognised as that strategy and warns about nothing. Beyond those three nothing has been validated against this checkpoint, so any other schedule returns a warning saying its output quality is unknown, and a schedule under 60 steps additionally warns about the poly-alanine result above. The schedule that actually ran is reported back in full as schedule.recycle_schedule, with schedule.recycle_strategy reading custom.

Replicates and Batching#

sampling.num_replicates asks for several independent designs from one specification. By default they are evaluated together, in one forward pass: sampling.batch_size is optional and defaults to num_replicates. Set it lower to trade that speed back for one of the two things a batch costs.

Replay. A batch draws the noise for every member from a single RNG stream, so which design a given row becomes depends on the batch it was produced in. The seed reported on each design therefore names the batch, not the design: the whole request replays at the same batch_size, but an individual design does not replay on its own. Send sampling.batch_size: 1 when per-design reproducibility matters; replicate i then runs with (seed + i) mod 2**63 and each design replays independently. Every design reports the batch it actually ran in as diagnostics.batch_size, so a replicate count that is not a multiple of batch_size (whose final batch is smaller) is visible in the response rather than inferred.

Replay also requires the same input, weights, execution backend, runtime and batch_size; GPU determinism is not guaranteed in any case.

Memory. GPU memory grows with the batch size, so a replicate count that fits comfortably one at a time can exhaust the GPU in a single batch. num_replicates is deliberately unbounded in the schema and is not capped by any deployment variable, because with batching what a request can ask for is a property of the GPU and the token count rather than a number a schema can know. Too many for the GPU is answered with out_of_memory (HTTP 413) naming the fields to change, not a schema error; refer to Behavior Under Load.

Important

A search block and a batch cannot combine. A beam returns its survivors and any kept scoring rollouts as one flat list per replicate, and a batch leaves no way to attribute an entry to the replicate it came from, so the two are refused together rather than returning designs whose origin and seed are guesses. Because batch_size defaults to num_replicates, a search with num_replicates > 1 must set sampling.batch_size: 1 explicitly.

Troubleshooting#

Container fails to pull. Confirm the tag exists for your host architecture and that docker login nvcr.io succeeded; refer to NGC Account.

docker run reports no GPU / nvidia-smi not found inside the container. Re-run the docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi verification command above. This usually means the NVIDIA Container Toolkit is not installed or --runtime=nvidia/--gpus was omitted.

ManifestDownloadError or permission-denied errors under the NIM cache. The container runs as a non-root UID and needs write access to $LOCAL_NIM_CACHE; refer to Set Up Your NIM Cache.

out_of_memory (HTTP 413) responses. Memory scales with the batch size, with the square of the token count, and with search width together, not with token count alone. Since sampling.batch_size defaults to sampling.num_replicates, lowering it (1 serves the replicates one at a time) is usually the first thing to try, followed by a shorter design or a narrower search; refer to Replicates and Batching, Deployment Limits and Behavior Under Load.

service_busy (HTTP 503) responses. The deployment has no free GPU worker and its queue is full; retry against another deployment, or refer to Admission Control to size queueing explicitly.

A requested metric or pass criterion never appears in the response. Unless a refolding oracle is mounted, every *_refolding-tagged metric and all eleven pass criteria are unavailable; refer to Oracles. A metric tag that the running deployment does not support is rejected at request time rather than silently producing no output.

Refer to the API Reference for the full error-code table.