Evaluate Agentic UI Generation
The ui_generation_agentic environment asks a coding harness (OpenCode, Claude Code, Hermes or Codex) to write one front-end UI file from scratch inside a sandbox that holds a ready-made project for one of three stacks: Next.js, Vite or a static nginx page. After the agent session ends, the resources server stops what the agent left running, snapshots the file, builds it in a fresh copy of the project template and asks an LLM judge to grade it against the prompt. The reward is build × score / 4.
The guard and runtime reference (G1 to G6, guard modes, rollout fields) is in the server README.
What This Tutorial Runs
Prerequisites
- Complete Installation.
- A sandbox provider. With OpenSandbox, set
OPENSANDBOX_DOMAINandOPENSANDBOX_API_KEY; restricted egress (agent_network_mode: restricted, the default) needs OpenSandbox’s egress sidecar. - A policy model and a judge model from different families, in
env.yaml:
Build and Publish the Images
No images are published for you. Build the three agent images from the Dockerfiles in the server’s docker/ folder (base images are pinned by digest, and the Next and Vite npm trees by committed lockfiles), push them to a registry your sandbox provider can pull from, and export the pushed digests:
The images bake in OpenCode 1.17.11 and ripgrep, checked by SHA-256. For the Codex composition, build them with WITH_CODEX=1, which also bakes in the Codex runtime; the default build leaves it out. Optionally pin each image’s manifest with UI_GEN_{NEXT,VITE,STATIC}_MANIFEST_SHA256; a seed whose image does not match is refused.
Make the Policy Reachable from the Sandbox
The harness calls the policy model server from inside the sandbox. With a self-hosted OpenSandbox server on the same machine (Docker runtime), bind only the policy model server to the Docker bridge in an overlay:
Otherwise pass ++use_absolute_ip=true '++absolute_ip_server_types=[responses_api_models]'. Restricted egress opens every port of the policy model server’s host, so the judge model server keeps host: 127.0.0.1 and stays out of the sandbox’s reach.
Run the Example Smoke Test
End-to-end collection does not take --split example, so collate the examples into task envelopes (taskset ui_generation_agentic:example), serve the composition, then collect against the running servers:
Each rollout records build_passed, final_code, judge_quality, judge_prompt_adherence, failure_reason and the guard settings. A sandbox, provider or judge failure masks the sample (mask_sample: true) instead of scoring it 0; the per-stack masked metric counts them.
Prepare Your Own Tasks
No task data ships with the environment. environments/ui_generation_agentic/prepare.py turns a JSONL of prompts ({"id", "prompt" | "messages", "stack"}) into task rows, writing to a path you choose outside the repository. Rows it cannot convert are skipped and counted in the output’s manifest.
Then add tasks.jsonl as a train dataset with taskset ui_generation_agentic:train through a config overlay that replaces the datasets list, and run gym eval run with --split train (the README has the full overlay).
Choose a Harness
Swap opencode.yaml for another composition:
In sandbox mode, the Hermes run’s result file is writable by commands the model runs, so the Hermes README advises against using these rollouts as RL training data.