Run SWE-bench Verified with Harbor, Pi, and Substrate

Choose an inference endpoint

Use the GLM-5.3-Flash recipe or another model recipe, and complete its smoke test. The supplied Harbor configuration defaults to the SGLang aggregated GB200 recipe in namespace inference.

For an external endpoint, no GPU is required in the sandbox cluster. The endpoint must list the selected model through GET /models under its API base URL and support streaming Chat Completions with tool calls, tool-result continuation, max_tokens, and streaming usage.

Record the API base URL and served model ID for the job configuration. Use an address reachable from the sandbox; your workstation’s localhost port-forward address will not work. For Dynamo recipes, the frontend address is http://<dgd-name>-frontend.<namespace>.svc.cluster.local:8000/v1.

Choose a model with reliable tool calling and enough context for repository files and tool results. The first Pi trial checks tool calls and continuation. Harbor results and sandbox metrics work with either endpoint; the inference trace walkthrough requires a Dynamo endpoint you control.

Configure Harbor access to Substrate

The Harbor runner runs outside the sandboxes. Through the Substrate adapter, it creates and deletes actors, executes guest commands inside them, and transfers task files and results. It needs access to both the Substrate API and the router:

Create the namespace and access policy

This tutorial uses ate-env for both the Kubernetes namespace and the Substrate atespace. These are separate objects. Run the Substrate commands as an administrator with global ownership.

kubectl create namespace ate-env
kubectl-ate create atespace ate-env
kubectl-ate create access-policy \
  -a ate-env -f runner-access-policy.yaml

The policy grants the harbor-runner ServiceAccount the editor role in ate-env. You will create this ServiceAccount when setting up the Harbor runner.

Authorize the Harbor runner at the router

Substrate’s default Envoy router blocks ingress to actors until a client identity is explicitly allowed. Add the Harbor runner so it can execute guest commands and transfer files.

The Harbor runner’s certificate identifies it with this SPIFFE ID, a workload identity derived from its namespace and ServiceAccount:

spiffe://cluster.local/ns/ate-env/sa/harbor-runner

Use your installation’s trust domain if it differs from cluster.local. In this release, the router allowlist grants access to all actors reachable through that router.

Update the atenet-router Deployment’s container arguments:

The default route timeout is five minutes per request, including streamed command output. A 12-minute test command would lose its output stream after five minutes. This timeout does not limit the actor’s lifetime.

Preserve existing identities and other arguments. On a shared installation, have its operator make these changes:

kubectl -n ate-system edit deployment atenet-router
kubectl -n ate-system rollout status deployment/atenet-router --timeout=300s

To preserve these settings across deployments, save the allowlist in substrate-install.yaml and the timeout in your managed router manifest, deps/substrate/manifests/ate-install/atenet-router.yaml. The installer accepts the allowlist as a comma-separated string. For an initially empty allowlist:

ingressAllowedSPIFFEIDs: spiffe://cluster.local/ns/ate-env/sa/harbor-runner

Check the router arguments:

kubectl -n ate-system get deployment atenet-router \
  -o jsonpath='{.spec.template.spec.containers[*].args}' | tr ' ' '\n' \
  | grep -E 'ingress-allowed-spiffe-ids|route-timeout'

--ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/ate-env/sa/harbor-runner
--route-timeout=60m

The allowlist may contain additional identities. If you change the task deadlines, keep the route timeout longer than the longest command.

Create the worker pool

Select a node that runs atelet and has matching gVisor runtime assets. Check its architecture:

kubectl get node your-sandbox-node \
  -o jsonpath='{.metadata.labels.kubernetes\.io/arch}{"\n"}'

Edit the labels and placement in workerpool.yaml. Replace the hostname with your node’s name. For ARM64, change both amd64 values to arm64:

metadata:
  labels:
    demo.harbor.dev/sandbox: gvisor
    demo.harbor.dev/arch: amd64
spec:
  template:
    nodeSelector:
      kubernetes.io/arch: amd64
      kubernetes.io/hostname: your-amd64-sandbox-node

Keep the other fields, including the multi-architecture worker image and gvisor-default SandboxConfig reference. The supplied capacity settings are:

Each task actor uses 1 CPU and 4 GiB, allowing one such actor per worker with capacity headroom. Reserve additional node capacity for the Harbor runner and any Substrate control-plane Pods placed there.

kubectl apply -f workerpool.yaml
kubectl -n ate-env rollout status \
  deployment/harbor-workers --timeout=600s

Wait for the Deployment to roll out successfully. For port-binding failures, follow the installation guide’s low-port permission fix.

Prepare the images for the first task

Each SWE-bench task has its own instance image containing the repository checkout and task environment. This tutorial adds Pi, the Dynamo provider, and dependencies needed by the task verifier. The guest daemon uses a separate image shared across tasks. An ActorTemplate references both images. Each trial creates a fresh actor from the template.

For the first task, prepare the additional tools and dependencies ahead of time to reduce trial startup latency. The planned full-dataset workflow uses the published instance images and installs these additions during trial setup:

The single-task path needs no package-download egress during the trial. Pi still needs access to the inference endpoint. See Run the full dataset for the planned extension and its remaining implementation and validation work.

Build the guest and task images for the worker pool’s architecture. The supplied configuration defaults to amd64; use the arm64 instructions for ARM64 nodes. Use a native builder or configure Docker cross-platform builds. Prepared task images can be large. Reserve node image-cache space before adding tasks.

Build the guest image

ate-env-guest runs commands and moves files inside the actor. Clone its pinned source, then build and publish it for the worker architecture. Replace registry.example.com/harbor-substrate with a repository you can push to and the workers can pull from:

git clone https://github.com/agent-substrate/env deps/env
git -C deps/env checkout 281e86085bb0b6d3090ff3fa82d722cf0d9f2f0b

Build the guest image for your node architecture:

(
	cd deps/env
	umask 022
	KO_DOCKER_REPO=registry.example.com/harbor-substrate/env-guest ko build --bare \
	  --platform=linux/amd64 ./cmd/ate-env-guest
)

Record the digest reference printed by ko build for task registration. The actor mounts this image at /ate. The runner Dockerfile installs the Python client from the same pinned revision.

Build the prepared task image

The Astropy base image contains the repository checkout and Conda test environment. Dockerfile.pi adds Pi 0.72.1, the Dynamo provider, and cached verifier dependencies. Fetch the pinned sources, apply the compatibility patches, and prepare the build context:

git clone https://github.com/ai-dynamo/agent-plugins deps/agent-plugins
git -C deps/agent-plugins checkout 223a0b8823610d042e7479bfff9b93eeac4a23ec
for patch in pi-provider-limits pi-harbor-adapter; do
	git -C deps/agent-plugins apply < "$patch.patch"
done
mkdir -p .build/image
cp -a deps/agent-plugins/pi-plugin .build/image/pi-plugin

Select the sandbox architecture and replace registry.example.com/harbor-substrate with a repository you can push to and the workers can pull from. Each tab pins a task base image for that architecture:

docker build --platform linux/amd64 -f Dockerfile.pi \
  --build-arg TASK_IMAGE=swebench/sweb.eval.x86_64.astropy_1776_astropy-12907@sha256:7485c1e3c8861efd0c6a4a78b952857592e541031039000d25e9481f045dc4a3 \
  -t registry.example.com/harbor-substrate/astropy-pi:prepared-amd64 .build/image
docker push registry.example.com/harbor-substrate/astropy-pi:prepared-amd64

Record the repository and digest printed by docker push, such as registry.example.com/harbor-substrate/astropy-pi@sha256:.... Use that immutable reference at registration. Build time and the first image pull are preparation costs, separate from trial wall time.

Configure and register the task

Register the actor template

Edit actor-template.yaml with both image digests from the builds. Set these fields together and keep the remaining defaults:

The template allocates 1 CPU and 4 GiB per actor. Keep its readiness probe, demo.harbor.dev/sandbox: gvisor selector, and disabled golden snapshots. Register it through the Substrate API:

kubectl-ate create actor-template -f actor-template.yaml
kubectl-ate get actor-template swe-astropy-12907-pi -a ate-env -o yaml

The get command must return the template with both image digests and the selected architecture. ActorTemplates are immutable in this release. For a later change, create a copy with a new metadata.name and update the job’s template_map. Keep the old template while existing trials use it.

Configure the Harbor job and egress

Edit harbor-job.yaml and egress.yaml for the endpoint selected earlier. Substrate denies actor egress by default; the adapter applies egress.yaml to each actor. Match its protocol, hostname, and port to DYNAMO_BASE_URL.

agents:
  - import_path: dynamo_pi:DynamoPi
    # Keep "dynamo/" and match the served model ID from /v1/models.
    model_name: dynamo/RadixArk/GLM-5.3-Flash-NVFP4
    kwargs:
      version: "0.72.1"
      thinking: "off"
    env:
      # SGLang aggregated GB200 recipe in namespace inference. Update for your deployment.
      DYNAMO_BASE_URL: http://glm53-flash-sglang-agg-gb200-frontend.inference.svc.cluster.local:8000/v1
      # Blank means "use the context window the endpoint advertises".
      DYNAMO_CONTEXT_WINDOW: ""
      DYNAMO_MAX_TOKENS: "8192"
      # Enable only for a scheduler that consumes session-final notifications.
      DYN_AGENT_SESSION_FINAL: "0"

rules:
  - http:
      # Hostname only: no scheme or path.
      hostnames:
        - glm53-flash-sglang-agg-gb200-frontend.inference.svc.cluster.local
      ports:
        # Match the endpoint's port.
        numbers: [8000]

Keep DYN_AGENT_SESSION_FINAL: "0" for the recommended recipe and external endpoints. Set it to "1" only for a scheduler that consumes session-final notifications, such as ThunderAgent. It sends an extra request when Pi exits normally so the scheduler can release session state. Tracing does not require this notification.

Keep the job’s supplied task and runtime settings:

Keep all values under agents[].env as strings, along with the supplied CLI and credential paths. The verifier still rebuilds the modified repository, using the image’s cached dependencies without package downloads.

rules:
  - http: # cleartext HTTP, authorized per request
      hostnames: [mirror.example.internal]
      ports:
        numbers: [80]
  - tlsPassthrough: # TLS forwarded by server name, not inspected
      hostnames: [pypi.org, files.pythonhosted.org]
      ports:
        numbers: [443]
  - https: # TLS intercepted by the gateway
      hostnames: [api.example.com]
      ports:
        numbers: [443]

Opening source hosting lets an agent retrieve an upstream fix, which changes what a passing benchmark result means. Keep the default policy for results you intend to compare.

The ActorTemplate belongs to the Substrate API; the Harbor job is consumed by harbor run. Neither file is a manifest for kubectl apply. Each runner Job mounts the Harbor configuration and egress policy from a ConfigMap. Use a new job_name for each run so results on the PVC stay separate.

Set up the Harbor runner

Harbor runs as a Kubernetes Job. Its image contains Harbor, kubectl-ate, the env Python client, and the patched adapters. Pi and the verifier run inside Substrate actors. The runner does not need Docker.

Build the runner image

On a native Linux x86-64 or ARM64 build machine, run these commands from the example directory. Replace the registry path with your own:

docker build -f Dockerfile.harbor \
  -t registry.example.com/harbor-substrate/runner:prepared .
docker push registry.example.com/harbor-substrate/runner:prepared

Dockerfile.harbor fetches the pinned sources, applies the patches, and verifies the architecture-specific CLI download. The adapters live under /opt/harbor, outside the mounted input and results directories. No local deps/ checkout is required for this build.

In harbor-runner.yaml, set the &runnerImage reference to the pushed repository and digest. The main container reuses that reference through *runnerImage. Set spec.template.spec.nodeSelector.kubernetes.io/arch to the image architecture: amd64 for x86-64 or arm64 for ARM64. The runner and sandbox workers may use different architectures.

Prepare credentials and results storage

For an authenticated model endpoint, create the Secret that supplies DYNAMO_API_KEY. Skip this command for an unauthenticated endpoint.

read -rsp 'Model API key: ' MODEL_API_KEY
printf '\n'
printf '%s' "$MODEL_API_KEY" | kubectl -n ate-env \
  create secret generic harbor-model-api --from-file=api-key=/dev/stdin
unset MODEL_API_KEY

Review the StorageClass and size in harbor-results.yaml, then create the runner identity and results PVC:

kubectl apply -f harbor-runner-rbac.yaml
kubectl apply -f harbor-results.yaml

A StorageClass with WaitForFirstConsumer leaves the PVC pending until the first Job is scheduled. The results PVC survives Job deletion.

Review the mounted inputs

Each Job uses these inputs and outputs:

The init container downloads the SWE-bench Verified task definitions from harbor-datasets revision 86723674f04e4209ac479d0fb75d9d9f44b4377e. It preserves the original instructions and verifier; it does not download task container images. A sparse Git checkout fetches only this dataset’s files. Each new Job repeats the download and needs access to GitHub. This network access is separate from actor egress.

The next steps create the ConfigMap and start separate oracle and Pi Jobs from the same manifest. Run one Job at a time with this shared configuration and ReadWriteOnce results PVC. Keep the ConfigMap unchanged while a Job runs. backoffLimit: 0 prevents Kubernetes from repeating a failed evaluation. activeDeadlineSeconds: 10800 limits the entire Job to three hours, including initialization and all trials. Increase it for batches that need more time.

Verify the sandbox and task verifier

Harbor’s oracle agent applies the task’s reference patch instead of calling a model, then runs the verifier. This checks actor creation, guest execution, and offline verification. It also pulls the images onto the selected worker before the Pi trial.

Copy the Harbor configuration on the workstation:

cp harbor-job.yaml oracle-job.yaml

In oracle-job.yaml, set job_name: swe-oracle and replace the agents list:

agents:
  - name: oracle

Create the mounted configuration and submit the Job. The manifest uses generateName, so kubectl create assigns a new Kubernetes Job name:

kubectl -n ate-env create configmap harbor-run-config \
  --from-file=harbor-job.yaml=oracle-job.yaml \
  --from-file=egress.yaml --from-file=kubeconfig=runner-kubeconfig.yaml \
  --dry-run=client -o yaml | kubectl apply -f -
ORACLE_JOB=$(kubectl create -f harbor-runner.yaml -o jsonpath='{.metadata.name}')
kubectl -n ate-env wait --for=jsonpath='{.status.ready}'=1 "job/$ORACLE_JOB" --timeout=600s
kubectl -n ate-env logs -f "job/$ORACLE_JOB" -c runner
kubectl -n ate-env wait --for=condition=Complete "job/$ORACLE_JOB" --timeout=30s

The first image pull can take several minutes. Harbor allows 1,200 seconds for actor readiness and retains the task’s 3,000-second agent and verifier deadlines. The first wait allows the dataset init container to finish before streaming Harbor logs. If startup or completion fails, inspect the Job before deleting it:

kubectl -n ate-env describe job "$ORACLE_JOB"
kubectl -n ate-env logs "job/$ORACLE_JOB" -c dataset
kubectl -n ate-env logs "job/$ORACLE_JOB" -c runner

After the Job completes, delete it to release the PVC, then mount the results in the read-only inspection Pod:

kubectl -n ate-env delete job "$ORACLE_JOB" --wait=true
kubectl apply -f results-reader.yaml
kubectl -n ate-env wait --for=condition=Ready pod/harbor-results --timeout=300s
kubectl -n ate-env exec harbor-results -- \
  sh -c 'cat /work/jobs/swe-oracle/*/verifier/reward.txt'

1

Require a reward of 1 before running Pi. A completed Kubernetes Job alone does not prove that the trial passed. On failure, inspect the saved trial’s result.json, exception_info, and verifier/test-stdout.txt. Repeat this check for every new task image.

Delete the inspection Pod before the next Job so the PVC can attach to its node. The results remain on the PVC:

kubectl -n ate-env delete pod harbor-results --wait=true

Every trial creates a fresh actor. The oracle’s reference patch does not carry into the Pi trial. Cached image layers remain on the worker node; repeat the oracle on each eligible node when validating placement across multiple nodes.

Run one SWE-bench task with Pi

Replace the ConfigMap’s oracle configuration with harbor-job.yaml, then submit another Job using the same runner image:

kubectl -n ate-env create configmap harbor-run-config \
  --from-file=harbor-job.yaml \
  --from-file=egress.yaml --from-file=kubeconfig=runner-kubeconfig.yaml \
  --dry-run=client -o yaml | kubectl apply -f -
PI_JOB=$(kubectl create -f harbor-runner.yaml -o jsonpath='{.metadata.name}')
kubectl -n ate-env wait --for=jsonpath='{.status.ready}'=1 "job/$PI_JOB" --timeout=600s
kubectl -n ate-env logs -f "job/$PI_JOB" -c runner
kubectl -n ate-env wait --for=condition=Complete "job/$PI_JOB" --timeout=30s

If the Job fails, inspect its dataset and runner logs before deleting it. Use a new Harbor job_name when retrying so existing results are preserved.

View and understand the trial

Export the results

After the Job completes, delete it and mount the PVC in the inspection Pod. Copy the results to your workstation so you can inspect them after the Job has been deleted:

kubectl -n ate-env delete job "$PI_JOB" --wait=true
kubectl apply -f results-reader.yaml
kubectl -n ate-env wait --for=condition=Ready pod/harbor-results --timeout=300s
mkdir -p results
# For later runs, replace swe-one with swe-observed, swe-golden, or swe-batch.
export JOB_NAME=swe-one
kubectl -n ate-env cp \
  "harbor-results:/work/jobs/$JOB_NAME" "results/$JOB_NAME"
kubectl -n ate-env delete pod harbor-results --wait=true

The original results remain on the PVC. Delete the inspection Pod before starting another Job; a ReadWriteOnce volume may not mount on two nodes at once. For later runs, PI_JOB records the latest Kubernetes Job name.

Convert the trajectory and open Harbor

From examples/agents/harbor-substrate, create a separate viewer environment, convert the copied Pi logs to Harbor’s ATIF trajectory format, and start the viewer:

python3 -m venv .venv/viewer
.venv/viewer/bin/pip install harbor==0.25.0
.venv/viewer/bin/python pi_trajectory.py results
.venv/viewer/bin/harbor view results --host 127.0.0.1 --port 8080

The converter writes agent/trajectory.json beside each agent/pi.txt. It preserves the raw logs and scores. Existing trajectories, logs with no complete messages, and runs without Pi logs are skipped.

To convert one run, replace results with results/swe-one. A trial directory or an individual agent/pi.txt also works. Add --overwrite to regenerate an existing trajectory.

Keep the viewer running and open http://localhost:8080. Open swe-one, select Rollouts, then select the trial. If an Overview link opens an empty Rollouts list, select Clear all to remove the agent filter.

Read the trial summary

Check the reward and timing at the top of the trial page. The example below completed in about 3 minutes 18 seconds with a reward of 0. Your result and timings may differ.

Read the timing bar as four phases:

These are recorded phase durations. The viewer labels setup segments scaled, so their widths are not proportional to elapsed time. Trial wall time also includes work between phases and cleanup. Kubernetes scheduling, runner startup, and the init container’s dataset download happen before the trial timings.

Follow a tool call

In Trajectory, read the task prompt and expand a tool call to inspect its arguments and returned output. In this example, Pi uses bash to locate separable.py. The OBSERVATIONS entry shows the path returned by the sandbox:

Continue through the following steps to see how Pi uses that output. The trajectory contains recorded messages and tool results. Interrupted recordings retain complete messages and flag unfinished calls. Differences between event timestamps are not isolated inference or tool durations.

Check the verifier outcome

Open Verifier to inspect the test output and reward. A reward of 1 means the task passed verification. A reward of 0 with no execution exception means the run completed but the solution did not pass verification. This example has 13 passing tests and 2 failing tests. Select report.json to see which tests failed:

If Harbor reports an execution error, open Exception before interpreting the reward. A failed solution and a failed sandbox or model request require different fixes. Keep the copied results with the job configuration, templates, policies, and dataset revision so you can reproduce the run.