> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/aerial/aodt/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/aerial/aodt/_mcp/server.

# Development Workflow

The development workflow is for engineers who need to change AODT source code, rebuild worker binaries, or produce custom Docker images. It complements the [Runtime Workflow](/runtime-workflow), which uses prebuilt images and binaries.

## Development vs runtime

|                    | Runtime workflow                       | Development workflow                            |
| ------------------ | -------------------------------------- | ----------------------------------------------- |
| **Worker startup** | Started by `./worker/up.sh`            | Started manually inside a development container |
| **Binaries**       | Prebuilt Docker images and executables | Built from source inside the container          |
| **Use case**       | Production-like runs and quickstarts   | Source changes, debugging, custom images        |

In the development workflow, the worker is started independently inside the container. The client connects over gRPC after the worker process is running.

### Container types

**Runtime container** — A lightweight image without build tools. Use it to run simulations after code is built and tested.

**Development container** — Includes compilers, CMake, and other development tools. Use it to modify and rebuild source code.

### Development cycle

1. Download the worker source code from NGC.
2. Build a development container image (once per development cycle).
3. Open an interactive shell in the development container.
4. Modify source code inside the shell.
5. Build and test the changes.
6. When the code is stable, build a runtime container image.
7. Run simulations using the runtime container.

The sections below walk through each step.

### Download the worker from NGC

**From the NGC website:**

[AODT 1.5.0 on NVIDIA NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/aerial/resources/aodt-sim-source?version=1.5.0)

**Using the NGC CLI:**

1. Download and install the [NGC CLI](https://org.ngc.nvidia.com/setup/installers/cli).
2. Download the worker using the NGC CLI command:

```
ngc registry resource download-version "nvidia/aerial/aodt-sim-source:1.5.0"
```

### Build the development container image

Run once at the start of a development cycle, or after container recipe changes:

```bash
./container/build.sh
```

Devel and MATLAB images are built from [HPCCM](https://github.com/NVIDIA/hpc-container-maker) recipes in `container/`.

## Get the source

After cloning, install Git LFS and initialize submodules:

```bash
# Install Git LFS (once per machine)
sudo apt-get install git-lfs
git lfs install

# Download LFS objects (prebuilt .so binaries shipped with the repo)
git lfs pull
```

Initialize the MatX submodule (GPU-accelerated tensor library used by the EM solver):

```bash
git submodule update --init --recursive
```

## Start the development container

From the AODT package root on the worker host, launch an interactive development shell alongside the full infrastructure stack (ClickHouse, MinIO, Nessie):

```bash
GPU=<gpu_num> ./docker/up.sh --devel --ports
```

| Flag / variable | Description                                                              |
| --------------- | ------------------------------------------------------------------------ |
| `GPU=<gpu_num>` | GPU index for the worker on multi-GPU systems                            |
| `--devel`       | Starts the container in development mode with source access              |
| `--ports`       | Exposes ports so worker, Nessie, and MinIO are reachable from the client |

Continue building and testing inside this shell.

## Build the source

Inside the development container, configure and build the worker:

```bash
cmake -B build -GNinja \
  -DCMAKE_CUDA_ARCHITECTURES="80;86;89;90" \
  -DPROFILING_ENABLED=ON -DNVTX_ENABLED=ON -DENABLE_NVTX=ON \
  -DBUILD_ASIM_TESTS=ON -DENABLE_CCACHE=OFF \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo

cmake --build build
```

Adjust `CMAKE_CUDA_ARCHITECTURES` to match the GPUs in your system.

## Run the worker

Still inside the container, start the Digital Twin server:

```bash
cd src_py
PYTHONPATH=$PYTHONPATH:/opt/nvidia/packman/lib/python:/opt/nvidia/aodt_sim/build/src_py/aodt \
  python -m aodt.app.sim --dt-server --dt-server-port 50051 --log=debug
```

The worker listens on port `50051` and waits for the client to connect and start a simulation.

### Connect the client

* **Colocated** — Run client examples from the same machine or container network.
* **Non-colocated** — Run client examples from a separate machine and pass the worker IP, S3 endpoint, and Iceberg catalog URI. See [Worker Installation](/worker-installation) for remote connection arguments.

For end-to-end examples, continue to the [Quickstart](/quickstart) guide.

## Build a runtime container image

After code is built and tested, package it into a lighter runtime image:

```bash
./docker/build.sh
```

## Run the runtime container

Start an interactive runtime shell:

```bash
./docker/up.sh
```

Stop and remove containers when finished:

```bash
./docker/down.sh
```

### Additional information

Use locally built development or MATLAB images:

```bash
DEVEL_IMAGE=aodt-sim-devel:$USER GPU=<gpu_num> ./docker/up.sh --devel
DEVEL_IMAGE=aodt-sim-matlab:$USER GPU=<gpu_num> ./docker/up.sh --devel
```

### MATLAB runtime image

Build a MATLAB image (layers MATLAB Runtime on top of the development image):

```bash
./container/build.sh --matlab
```