> This page is for VISTA-3D, version 1.0.0 (Latest) (default).
> For other versions, use one of these documentation indexes:
> - 1.0.0 (Latest) (default): https://docs.nvidia.com/nim/medical/vista3d/1.0.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nim/medical/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nim/medical/_mcp/server.

# Getting Started

## Core Workflows of VISTA-3D

1. Segment everything: This feature enables comprehensive exploration of the whole body, which is vital for understanding complex diseases that affect multiple organs and devising holistic treatment plans.
2. Segment using classes: It provides detailed sectional views based on specific classes, key for focused disease analysis or organ mapping, such as pinpointing tumors in critical organs.
3. Interactive Point-Prompt Segmentation: By allowing user-directed, click-based selection, this workflow significantly improves segmentation precision, facilitating the rapid creation of accurate ground-truth data for medical imaging analysis.

Playground Demonstration: [https://build.nvidia.com/nvidia/vista-3d](https://build.nvidia.com/nvidia/vista-3d)

![3D Slicer Rendering Example 4](/nim/medical/_fern-img/f00fcfe24ece47c2a7fc9afe5f069007fcd2df68bc592165baa32684cc0f88ce.webp)

## Setup and Prerequisites

### Hardware Requirements

- Minimum GPU memory (GB): 48 GB
- CPU x86-64 >= 8 core (Recommended)
- Memory >= 32 GB (Recommended)
- Minimum Storage: 20 GB (8 GB container size)
- Validated on A100, H100, and L40

### Software Requirements

- Minimum NVIDIA Driver Version: 470
- Install [Docker](https://docs.docker.com/engine/install/)
- Install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installing-the-nvidia-container-toolkit)

### NGC (NVIDIA GPU Cloud) Account Setup

1. [Create an account on NGC](https://catalog.ngc.nvidia.com/)
2. [Generate an API Key](https://org.ngc.nvidia.com/setup/api-key)
3. Set `NGC_API_KEY`:

    ```bash
    export NGC_API_KEY=<Your NGC API Key>
    ```

4. Docker log in with your NGC API key using `docker login nvcr.io --username='$oauthtoken' --password=$NGC_API_KEY`

### NGC CLI Tool Setup

1. Download the [NGC CLI tool](https://org.ngc.nvidia.com/setup/installers/cli) for your OS.
2. Set up your NGC CLI Tool locally (You'll need your API key for this!):

   ```bash
   ngc config set
   ```

   After you enter your API key, you may see multiple options for the org and team. Select as desired or hit enter to accept the default.

## Usage Instructions

### Running NIM

1. Pull the NIM container:

   ```bash
   docker pull nvcr.io/nim/nvidia/vista3d:1.0.0
   ```
   

2. Run the NIM container:

   ```bash
   docker run --rm -it --name vista3d \
      --runtime=nvidia -e CUDA_VISIBLE_DEVICES=0 \
      --shm-size=8G \
      -p 8000:8000 \
      -e NGC_API_KEY=$NGC_API_KEY \
      nvcr.io/nim/nvidia/vista3d:1.0.0
   ```
   

3. Monitor progress with docker logs:

   ```bash
   docker logs --follow vista3d
   ```

### Health and Liveness Checks

- Readiness check:

  ```bash
  curl localhost:8000/v1/health/ready
  ```

- Liveness check:

  ```bash
  curl localhost:8000/v1/health/live
  ```

- Wait for the health check to return status code `200`:

   Bash:

   ```bash
   base_url="http://localhost:8000"
   endpoint="/v1/health/ready"

   response=$(curl -s -o /dev/null -w "%{http_code}" "$base_url$endpoint")

   while [ "$response" -ne 200 ]; do
      echo "Waiting for the service to be ready..."
      sleep 5
      response=$(curl -s -o /dev/null -w "%{http_code}" "$base_url$endpoint")
   done

   echo "Service is ready!"
   ```

   Python:

   ```Python
   import requests
   import time

   base_url = "http://localhost:8000"

   while True:
       try:
           response = requests.get(f"{base_url}/v1/health/ready")
           if response.status_code == 200:
               print("Server is ready")
               break
       except requests.exceptions.ConnectionError:
           print("Server is not ready yet, waiting...")
       time.sleep(5)
   ```

### Running Inference

1. Prepare your NIfTI images or use the sample image.
2. (optional) Download and install [3D Slicer](https://www.slicer.org/) for viewing results.

<Note>
For a more detailed set of instructions for setting up your data, refer to our blog [Running VISTA-3D with your data](https://developer.nvidia.com/blog/computed-tomography-organ-and-disease-segmentation-using-the-nvidia-vista-3d-nim/#running_vista-3d_with_your_data%C2%A0)
</Note>

<Note>
For region of interest, typically user's data doesn't cover all components that contains all anatomies, classes can be empty if the image doesn't contain the class or even wrong class. For example, "brain" and "liver" may not exist in the same time, providing these two class prompts can result in wrong predictions. The reference image (example-1.nii.gz) is an abdomen CT scan, which contains most abdominal organs, vessels, and vertebrae classes.
</Note>

#### Method 1: Segment All Labels

Bash:

```bash
LOCAL_URL='http://localhost:8000/v1/vista3d/inference'
DATA=$(cat <<EOF
{
    "image": "https://assets.ngc.nvidia.com/products/api-catalog/vista3d/example-1.nii.gz"
}
EOF
)
response=$(curl -s -o output.zip  -w "%{http_code}" -X POST -H "Content-Type: application/json" -d "$DATA" $LOCAL_URL)
if [ "$response" -eq 200 ]; then
  echo "Response Success, save inference results into folder: output"
  unzip -o "output.zip" -d "output"
else
  echo "Request failed with status $response"
fi
```

Python:

```Python
import requests
import zipfile

base_url = "http://localhost:8000"

data = {
    "image": "https://assets.ngc.nvidia.com/products/api-catalog/vista3d/example-1.nii.gz",
}

def unzip_file(zip_filepath, dest_dir):
    with zipfile.ZipFile(zip_filepath, 'r') as zip_ref:
        zip_ref.extractall(dest_dir)

response = requests.post(f"{base_url}/v1/vista3d/inference", json=data)
if response.status_code == 200:
    output_folder = "output"
    output_zip_name = "output.zip"

    with open(output_zip_name, "wb") as f:
        f.write(response.content)

    unzip_file(output_zip_name, output_folder)
```

#### Method 2: Segment Specific Organs

Bash:

```bash
LOCAL_URL='http://localhost:8000/v1/vista3d/inference'
DATA=$(cat <<EOF
{
    "image": "https://assets.ngc.nvidia.com/products/api-catalog/vista3d/example-1.nii.gz",
    "prompts": {"classes": ["spleen", "liver"]}
}
EOF
)
response=$(curl -s -o output.zip -w "%{http_code}" -X POST -H "Content-Type: application/json" -d "$DATA" $LOCAL_URL)

if [ "$response" -eq 200 ]; then
  echo "Response Success, save inference results into folder: output"
  unzip -o "output.zip" -d "output"
else
  echo "Request failed with status $response"
fi
```

Python:

```Python
import requests
import zipfile

base_url = "http://localhost:8000"

data = {
    "image": "https://assets.ngc.nvidia.com/products/api-catalog/vista3d/example-1.nii.gz",
    "prompts": {"classes": ["spleen", "liver"]}
}

def unzip_file(zip_filepath, dest_dir):
    with zipfile.ZipFile(zip_filepath, 'r') as zip_ref:
        zip_ref.extractall(dest_dir)

response = requests.post(f"{base_url}/v1/vista3d/inference", json=data)
if response.status_code == 200:
    output_folder = "output"
    output_zip_name = "output.zip"

    with open(output_zip_name, "wb") as f:
        f.write(response.content)

    unzip_file(output_zip_name, output_folder)
```

#### Method 3: Single Organ with User Points

Bash:

```bash
LOCAL_URL='http://localhost:8000/v1/vista3d/inference'
DATA=$(cat <<EOF
{
    "image": "https://assets.ngc.nvidia.com/products/api-catalog/vista3d/example-1.nii.gz",
    "prompts": {"classes": ["spleen"], "points": {"spleen": [[97, 249, 87]]}}
}
EOF
)
response=$(curl -s -o output.zip  -w "%{http_code}" -X POST -H "Content-Type: application/json" -d "$DATA" $LOCAL_URL)
if [ "$response" -eq 200 ]; then
  echo "Response Success, save inference results into folder: output"
  unzip -o "output.zip" -d "output"
else
  echo "Request failed with status $response"
fi
```

Python:

```Python
import requests
import zipfile

base_url = "http://localhost:8000"

data = {
    "image": "https://assets.ngc.nvidia.com/products/api-catalog/vista3d/example-1.nii.gz",
    "prompts": {"classes": ["spleen"], "points": {"spleen": [[97, 249, 87]]}}
}

def unzip_file(zip_filepath, dest_dir):
    with zipfile.ZipFile(zip_filepath, 'r') as zip_ref:
        zip_ref.extractall(dest_dir)

response = requests.post(f"{base_url}/v1/vista3d/inference", json=data)
if response.status_code == 200:
    output_folder = "output"
    output_zip_name = "output.zip"

    with open(output_zip_name, "wb") as f:
        f.write(response.content)

    unzip_file(output_zip_name, output_folder)
```

### Stopping the Container

To stop the container, run:

```bash
docker stop vista3d
```

### Visualization with 3D Slicer

Users can visualize the VISTA3D NIM generated segmentation results by viewers such as 3D Slicer, in this document, we provide simple tutorial of installing 3D Slicer and rendering of the VISTA3D segmentation.

#### Installation

To begin with 3D Slicer, you'll need to download and install stable and preview versions. For more information on installing 3D Slicer, check out the [3D Slicer Documentation](https://download.slicer.org/).

- Extract the compressed files, and double click "3D Slicer" application, follow necessary installation steps on different operation systems.

#### Visualization and Rendering

- Open the 3D Slicer app, under launchpad viewer:
![3D Slicer Rendering Example 1](/nim/medical/_fern-img/a89f57036f3eda2e27a62e60c4a5dc998de1302ef3a84f6bb58dca762ac096af.webp)
- Open the VISTA3D NIM generated segmentation file, it can be either NRRD or NIFTI files. Drag the file or import from the menu. In the pop-up window, if you would like to load the file alone, go with the volume option. Or you can select "segmentation" mode if you would like to edit the segments.
![3D Slicer Rendering Example 2](/nim/medical/_fern-img/7efd9bc179ad113c2cc418a2cfa1175c6cd9bf0d43e0de0559b93a2254c37e97.webp)
- Visualize the labels generated, such as liver and spleen:
![3D Slicer Rendering Example 3](/nim/medical/_fern-img/fabd14556f6e53e892b640bd034bce5b003eaef9536023b1661408fdc1838aed.webp)
- Visualize the all the anatomies generated:
![3D Slicer Rendering Example 4](/nim/medical/_fern-img/6224a17ba21fe9c4e73e664dd5d4f6512cdc75e8269e0c75de38865f982119c5.webp)
- Visualize both the CT image and its VISTA3D inference labels:
![3D Slicer Rendering Example 4](/nim/medical/_fern-img/6c48700fe3c971f0be2f3c7355266de756226c2acc1ccdc39cd91f5934cf6c4c.webp)
- Visualize the point prompt patch: If you would like to use 3D Slicer to see the point prompt inference results, which are patch-based predictions with NRRD or NIFTI format. You can load it as "volume" mode in Slicer, then click "volume" panel on the top 3D Slicer menu. Then adjust the constrast and threshhold around 0-3, as the prediction values from point propt ranges from 0-1. Then you can see the results. 

![3D Slicer Rendering Example 4](/nim/medical/_fern-img/3945757e30e2fdb6af83931da2fe9283610ce5a1b5c38300dc1ac8c4a217ca96.webp)