> This page is for version 0.7.0. > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # ECS Fargate > Run sandboxes as AWS ECS Fargate tasks reached over an SSH sidecar. # ECS Fargate The `ecs_fargate` provider runs each sandbox as an AWS ECS Fargate task reached over an SSH sidecar. It implements the provider-neutral `SandboxProvider` contract, so any sandbox-backed agent or resources server can use it by selecting `ecs_fargate` in its sandbox config. ## Setup ECS Fargate requires an existing sandbox stack and the AWS SDK: ```bash uv pip install boto3 ``` Ask your AWS platform owner for the account, region, SSM project name, runtime permissions, and quota. Use an AWS profile, temporary session, or workload role through the [Boto3 credential chain](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html). Set `AWS_REGION`; temporary environment credentials also need `AWS_SESSION_TOKEN`. The Gym host must reach each task's SSH sidecar on port `52222`. ### Provisioning reference For a new deployment, have the infrastructure owner follow the pinned [NeMo Evaluator Terraform stack instructions](https://github.com/NVIDIA-NeMo/Evaluator/blob/d9c321ad0a9c83918ca9c03cc36396b27990944f/terraform/stacks/ecs-sandbox/README.md). The stack provisions the sandbox infrastructure; deploy Gym and the model endpoint separately. Review the target regions (the default is eleven), project name, optional EFS, and `orchestrator_allowed_cidrs` before applying. > **Warning** > > Provisioning creates billable resources; Terraform state contains credentials > and SSH keys. Protect that state and coordinate provisioning and teardown with > the infrastructure owner. Teardown can delete stored data and secrets. ### Runtime permissions Use the pinned stack's [IAM policies](https://github.com/NVIDIA-NeMo/Evaluator/blob/d9c321ad0a9c83918ca9c03cc36396b27990944f/terraform/stacks/ecs-sandbox/iam.tf) as a starting point, scoped to your deployment. Gym's runtime identity needs ECS task operations, SSM config/cache access, SSH-secret access, S3 staging access, and permission to pass the configured roles. Image mirroring/building also needs ECR and CodeBuild permissions. Use these scoped permissions for runtime access. For staging cleanup, ask the owner to add `s3:DeleteObject` to the pinned orchestrator policy for the staging prefix and configure a lifecycle cleanup rule. Customer-managed encryption keys may also require KMS permissions. ### SSM discovery The stack publishes its [configuration](https://github.com/NVIDIA-NeMo/Evaluator/blob/d9c321ad0a9c83918ca9c03cc36396b27990944f/terraform/modules/ecs-sandbox-region/ssm.tf) at `//ecs-sandbox/config`. Set Gym's `ssm_project` to Terraform's `project`: **the stack defaults to `nel-sandbox`, but Gym defaults to `harbor`**. Set `GYM_SSM_PROJECT` to that project name for the examples below; it is passed explicitly into Gym's config. Before launching, confirm SSM/secret access, SSH connectivity, image access, and Fargate quota with the owner. ## Provider Configuration Load `nemo_gym/sandbox/providers/ecs_fargate/configs/ecs_fargate.yaml` and select it with `sandbox_provider: sandbox`. Set the region and SSM project as below. Explicit YAML values take precedence; setting `cluster` disables SSM discovery and requires the remaining infrastructure fields. Task resources come from `sandbox_spec.resources`. ```yaml sandbox: default_metadata: sandbox-api: ecs-fargate ecs_fargate: region: ${oc.env:AWS_REGION} ssm_project: ${oc.env:GYM_SSM_PROJECT} ``` | Field | Default | Purpose | | ----------------------- | ------------------- | ------------------------------------------------------------------------------------------------ | | `region` | — | AWS region; enables SSM auto-discovery when `cluster` is omitted | | `cpu` / `memory` | `"4096"` / `"8192"` | Fargate task size (CPU units / MiB) when set directly | | `ephemeral_storage_gib` | 20 (implicit) | Task scratch disk; explicit values must be 21–200 | | `auto_mirror` | `true` | Mirror a missing public image into the ECR mirror on demand | | `ssm_project` | `harbor` | SSM namespace for auto-discovery | | `environment_dir` | — | Build the task image from a Dockerfile directory via CodeBuild instead of using a prebuilt image | Cluster, subnets, security groups, roles, ECR mirror, EFS defaults, and the SSH-sidecar key ARNs are filled in from SSM when omitted. ## Provider Options Per-sandbox options go in `SandboxSpec.provider_options`: | Key | Purpose | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `volumes` | EFS mounts: a list of `{"container_path": "/mnt/efs", "efs": true}` entries. Each inherits the provider's `efs_filesystem_id` / `efs_access_point_id` unless it names its own. | | `outside_endpoints` | Host URLs exposed inside the sandbox via an SSH reverse tunnel, as `{"url": ..., "env_var": ...}` (e.g. a model server). | | `environment_dir` | Dockerfile directory to build the task image from via CodeBuild. | Common `SandboxSpec` fields map onto the task as follows: `ttl_s` → sidecar watchdog that stops the task, `ready_timeout_s` → task-startup timeout, `env` / `files` / `workdir` → container environment, seed files, and working directory. ## Resource Mapping and Isolation `SandboxResources` maps onto the Fargate task definition: | `SandboxResources` | Fargate | | ------------------ | --------------------------------------------------------------------------- | | `cpu` (vCPU) | task CPU units (`cpu * 1024`), validated against Fargate's CPU/memory pairs | | `memory_mib` | task memory (MiB) | | `disk_gib` | task ephemeral storage (21–200 GiB) | | `gpu` | unsupported — raises `SandboxCreateError` | Each sandbox is a dedicated Fargate task with its own kernel, network namespace, and microVM boundary — there is no host sharing between sandboxes. The orchestrator holds a per-task SSH connection for exec and file transfer, and TTL is enforced by a sidecar watchdog that stops the task. ## Images and On-Demand Mirroring ECS pulls task images from the account ECR mirror rather than their origin registry. A bare/public image (e.g. `docker.io/swebench/sweb.eval.x86_64.:latest`) resolves to the mirror tag `:`. Resolution order: 1. `environment_dir` set → build the image via CodeBuild and use it. 2. Image is already an ECR reference → use verbatim (never re-mirrored). 3. Bare/public name + `auto_mirror=true` → mirror into ECR on demand (CodeBuild pull → retag → push) during `create`, then launch. The first task for a new image waits on a one-time build (\~1–3 min for typical SWE-bench images); later tasks hit the ECR cache, and concurrent tasks for the same image de-duplicate onto a single build. Set `auto_mirror: false` to require a pre-populated mirror and fail fast on a miss. ## First-Run Example After confirming access and quota, this example starts a billable Fargate task and may trigger a CodeBuild image-mirroring job. The context manager stops the task on exit; shared infrastructure remains. ```python import asyncio import os from nemo_gym.sandbox import AsyncSandbox, SandboxResources, SandboxSpec provider_config = { "ecs_fargate": { "region": os.environ["AWS_REGION"], "ssm_project": os.environ["GYM_SSM_PROJECT"], } } spec = SandboxSpec( image="python:3.13.14-slim", ttl_s=1800, ready_timeout_s=300, resources=SandboxResources(cpu=2, memory_mib=8192), ) async def main() -> None: async with AsyncSandbox(provider_config, spec) as sandbox: await sandbox.start() result = await sandbox.exec("python3 --version", timeout_s=60) print(result.stdout or result.stderr) asyncio.run(main()) ``` For an end-to-end agent run, add this provider config to `mini_swe_agent_2`'s config paths — the agent config is unchanged: ```bash gym env start \ --config responses_api_agents/mini_swe_agent_2/configs/mini_swe_agent_2.yaml \ --config nemo_gym/sandbox/providers/ecs_fargate/configs/ecs_fargate.yaml \ --config "" \ ++sandbox.ecs_fargate.ssm_project="$GYM_SSM_PROJECT" ``` ## Troubleshooting | Symptom | Cause / fix | | ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | `SSM parameter '/…/ecs-sandbox/config' not found` | Check the account, region, and `ssm_project` with the owner before requesting a new stack. The Terraform and Gym project defaults differ. | | `AccessDenied` reading SSM, SSH secrets, or launching tasks | Ask the owner to check the runtime identity's permissions and encryption-key policies. | | Exec/SSH times out after the task reaches RUNNING | The host cannot reach the sidecar port `52222`. Run inside the sandbox VPC or allow the host IP on the sidecar security group. | | `Fargate memory for cpu=… must be …` | The CPU/memory pair is not a supported Fargate combination. Pick a valid pair. | | `ephemeral storage must be between 21 and 200 GiB` | `disk_gib` is out of range. Omit it for the implicit 20 GiB default. | | First task is slow or the image pull fails | The first task mirrors the image via CodeBuild (\~1–3 min); later tasks hit the ECR cache. Set `auto_mirror: false` to require a pre-staged mirror. | | S3 staging objects accumulate | The orchestrator role lacks `s3:DeleteObject`. Grant it, or set a bucket lifecycle policy to reap `*/ecs-sandbox/*` staging artifacts. | | `ECS Fargate does not support GPU sandboxes` | Fargate has no GPU; use a GPU-capable provider instead. | > **Warning** > > The pinned reference requires `orchestrator_allowed_cidrs` and rejects `0.0.0.0/0`. > For older or customized deployments, also limit port `52222` access to the > orchestrator's network.