Sandbox Orchestration

Per-problem isolated execution environments for agent evaluations and code execution.

Overview

NEL provides three levels of isolation:

The per-problem sandbox is essential for:

• Agent evaluations (SWE-bench, OpenHands, terminal-bench): each problem gets an isolated container with its own filesystem, network, and process space.
• Code execution (HumanEval, NeMo Skills math): untrusted code runs in a sandbox with resource limits.
• Multi-turn tool use: agents in sandboxes need to reach the model server while the orchestrator reaches the agent.

Architecture

                    ┌──────────────────────────────────────────┐
                    │          Orchestrator (eval loop)         │
                    │                                          │
                    │  for problem in dataset:                 │
                    │    seed = env.seed(idx)                  │
                    │    sandbox = manager.acquire(spec)       │
                    │    result = solver.solve(task, sandbox)  │
                    │    verify = env.verify(resp, exp, sandbox)│
                    │    manager.release(sandbox)              │
                    └───────────┬──────────────┬───────────────┘
                                │              │
                    ┌───────────▼──┐    ┌──────▼──────────────┐
                    │ Model Server │    │ Sandbox (1/problem)  │
                    │ (vLLM/NIM)   │    │ image: per-task      │
                    │              │◄───┤ bridge network       │
                    │ :8000        │    │ ┌───────────────────┐│
                    └──────────────┘    │ │ Agent or code     ││
                                        │ └───────────────────┘│
                                        └──────────────────────┘

Design Principles

Sandbox is infrastructure-only

The sandbox knows nothing about agents, solvers, or evaluation. Its job:

• Start a container from a given image
• Provide exec(), upload(), download()
• Expose outside endpoints (model URL) inside the container
• Stop and clean up

Whether we exec a Python script or start a multi-turn agent is the solver’s decision.

Async protocol

All sandbox operations are I/O-bound. The eval loop is async. The protocol is fully async to avoid blocking the event loop when running concurrent sandboxes.

Bridge network by default

With --network host, concurrent sandboxes sharing port 3000 collide. Bridge network gives each container its own IP. The orchestrator reaches the container via container_ip.

Verify needs the sandbox

For agent benchmarks, verification (apply patch, run tests) happens inside the same sandbox the agent modified. The sandbox stays alive through: seed → solve(sandbox) → verify(sandbox) → release.

The Per-Task Image Problem

SWE-bench is the critical test case. Every problem has a different Docker image:

• instance_id = "django__django-11099" → swebench/sweb.eval.x86_64.django_1776_django-11099:latest
• instance_id = "scikit-learn__scikit-learn-13779" → swebench/sweb.eval.x86_64.scikit-learn_1776_scikit-learn-13779:latest

NEL handles this via three mechanisms (in priority order):

1. SandboxSpec in SeedResult: The environment returns per-problem sandbox requirements directly.
2. image_template: Config-level template interpolated from metadata: "swebench/sweb.eval.x86_64.{instance_id}:latest".
3. default_image: Fallback for homogeneous benchmarks.

Sandbox Backends

DockerSandbox

Per-problem Docker container on the orchestrator machine.

• Bridge network by default (each container gets its own IP)
• Async operations via asyncio.create_subprocess_exec
• resolve_outside_endpoint() rewrites localhost → host.docker.internal

SlurmSandbox

Per-problem container on SLURM nodes via Pyxis/Enroot.

• Multiplexed: multiple sandboxes per node via --container-name
• Shared filesystem for fast file transfer when available
• SLURM nodes share the job’s network (no URL rewriting needed)

ECSFargateSandbox

Per-problem container on AWS ECS Fargate. No local Docker required.

• SSH sidecar: Each ECS task includes an SSH sidecar container for exec(), upload(), download() via SSH tunnels
• Bidirectional tunnels: Forward tunnel (orchestrator → container exec server) and reverse tunnels (container → host proxy/model endpoint)
• CodeBuild image building: Per-task Docker images built via AWS CodeBuild and pushed to ECR with content-hash tags for caching
• OutsideEndpoint: Host-side services (LiteLLM proxy, model server) are made reachable inside the container via reverse SSH tunnels

1
benchmarks:
2
  - name: harbor://swebench-verified@1.0
3
    solver:
4
      type: harbor
5
      service: model
6
      agent: openhands
7
    sandbox:
8
      type: ecs_fargate
9
      cluster: harbor-cluster
10
      subnets: [subnet-abc123]
11
      security_groups: [sg-def456]
12
      ecr_repository: 123456789.dkr.ecr.us-west-2.amazonaws.com/harbor-swebench
13
      ssh_sidecar:
14
        sshd_port: 2222
15
        public_key_secret_arn: arn:aws:secretsmanager:...
16
        private_key_secret_arn: arn:aws:secretsmanager:...

ApptainerSandbox

Per-problem Apptainer/Singularity container for HPC environments where Docker is unavailable. Supports SLURM integration via salloc/srun.

LocalSandbox

Async subprocess in a temp directory. No container isolation. For development and testing.

Configuration

1
benchmarks:
2
  - name: gym://swebench-server
3
    sandbox:
4
      type: docker
5
      image_template: "swebench/sweb.eval.x86_64.{instance_id}:latest"
6
      concurrency: 8
7
      memory: 4g
8
      network: bridge

SWE-bench on SLURM

1
benchmarks:
2
  - name: gym://swebench-server
3
    sandbox:
4
      type: slurm
5
      image_template: "swebench/sweb.eval.x86_64.{instance_id}:latest"
6
      concurrency: 16
7
      node_pool: sandbox
8
      slots_per_node: 4
9
10
cluster:
11
  type: slurm
12
  node_pools:
13
    sandbox:
14
      partition: batch
15
      nodes: 4

4 sandbox nodes × 4 slots/node = 16 concurrent sandboxes on 4 nodes.

Code execution (HumanEval)

1
benchmarks:
2
  - name: humaneval
3
    sandbox:
4
      type: docker
5
      image: python:3.12-slim
6
      network: none
7
      memory: 256m
8
      cpus: 1
9
      timeout: 30
10
      concurrency: 16

SandboxManager

The SandboxManager provides:

• Concurrency control: semaphore-based limiting of concurrent sandboxes
• Pre-pull: bulk image pulling before eval starts (via env.sandbox_specs())
• Emergency cleanup: atexit and signal handlers to stop leaked containers
• Spec resolution: resolves SandboxSpec from SeedResult, image_template, or default image
• SLURM multiplexing: round-robin slot allocation across nodes

SandboxedAgentSolver

The SandboxedAgentSolver runs agents inside sandboxes. It supports two modes:

• exec-server: no entrypoint → solver execs the agent command via sandbox.exec()
• agent-server: entrypoint starts an HTTP agent → solver talks to it via container_ip

The solver infers the mode from sandbox.spec.entrypoint.

SWE-bench Integration Example

SWE-bench is not a built-in environment. Here’s how to integrate it using the sandbox system:

1
from nemo_evaluator import EvalEnvironment, SeedResult, VerifyResult, benchmark
2
from nemo_evaluator.sandbox import SandboxSpec
3
4
class SWEBenchEnvironment(EvalEnvironment):
5
    name = "swebench"
6
7
    def __init__(self, dataset_path: str):
8
        super().__init__()
9
        import json
10
        with open(dataset_path) as f:
11
            self._dataset = [json.loads(line) for line in f]
12
13
    def _instance_image(self, instance_id: str) -> str:
14
        safe = instance_id.replace("/", "_").replace("__", "_1776_")
15
        return f"swebench/sweb.eval.x86_64.{safe}:latest"
16
17
    async def sandbox_specs(self) -> list[SandboxSpec]:
18
        return [
19
            SandboxSpec(
20
                image=self._instance_image(row["instance_id"]),
21
                workdir="/testbed",
22
            )
23
            for row in self._dataset
24
        ]
25
26
    async def seed(self, idx: int) -> SeedResult:
27
        row = self._dataset[idx]
28
        return SeedResult(
29
            prompt=row["problem_statement"],
30
            expected_answer=row["patch"],
31
            metadata={"instance_id": row["instance_id"]},
32
            sandbox_spec=SandboxSpec(
33
                image=self._instance_image(row["instance_id"]),
34
                workdir="/testbed",
35
            ),
36
        )
37
38
    async def verify(self, response, expected, sandbox=None, **meta):
39
        if sandbox is None:
40
            raise RuntimeError("SWE-bench requires a sandbox for verification")
41
        await sandbox.exec(f"cd /testbed && git apply /tmp/patch.diff")
42
        result = await sandbox.exec(
43
            "cd /testbed && python -m pytest tests/ -x",
44
            timeout_sec=300,
45
        )
46
        passed = result.return_code == 0
47
        return VerifyResult(
48
            reward=1.0 if passed else 0.0,
49
            scoring_details={"method": "swebench_test", "passed": passed},
50
        )

Adding a New Backend

1. Create sandbox/my_backend.py implementing the Sandbox protocol.
2. Add a branch in SandboxManager._create().
3. Add a CustomSandbox variant or extend the SandboxConfig union.
4. Handle emergency cleanup in SandboxManager._sync_cleanup().