Skip to navigation

nemo_gym.agents.utils.setup_retry

View as MarkdownOpen in Claude

Retry a setup command that is safe to run twice when the sandbox provider fails to run it.

A sandbox that was just created or restored can refuse a connection for a few seconds even after it first answered, so a seed’s first commands can fail although the sandbox is fine. The provider does not retry a command itself: the error can come after the command started, and running an arbitrary command twice is not safe. A caller whose command is idempotent (a read, a mkdir -p, an install that skips what is present) retries it here.

Module Contents

Functions

NameDescription
exec_idempotentRun command, retrying when sandbox.exec raises; a returned result, any exit code, ends the retries.

Data

LOG

API

nemo_gym.agents.utils.setup_retry.exec_idempotent(
command: str,
what: str,
attempts: int = 3,
delay_s: float = 2.0,
cwd: str | None = None,
env: dict[str, str] | None = None,
timeout_s: int | float | None = None
async

Run command, retrying when sandbox.exec raises; a returned result, any exit code, ends the retries.

cwd and env are passed to sandbox.exec. timeout_s is the budget of all attempts and the waits between them together, including the time a provider waits for capacity before it runs the command, which its own command timeout does not cover. Each attempt gets what is left of the budget. Once it is spent no attempt is started, and one that is running is cancelled: the last error is raised, or, when no attempt raised, the result a provider returns for a command that timed out (error_type="timeout"). The wait doubles after each failed attempt (2 s, then 4 s). The last error is also raised once attempts run out.

Use only for commands that are safe to run twice, and at the same time: the error can arrive after the command started, so a retry may begin while the first attempt is still running in the sandbox.

nemo_gym.agents.utils.setup_retry.LOG = logging.getLogger(__name__)