> This page is for AIPerf, version dev.
> For other versions, use one of these documentation indexes:
> - Latest (v0.13.0) (default): https://docs.nvidia.com/aiperf/latest/llms.txt
> - dev: https://docs.nvidia.com/aiperf/dev/llms.txt
> - v0.13.0: https://docs.nvidia.com/aiperf/v0.13.0/llms.txt
> - v0.12.0: https://docs.nvidia.com/aiperf/v0.12.0/llms.txt
> - v0.11.0: https://docs.nvidia.com/aiperf/v0.11.0/llms.txt
> - v0.9.0: https://docs.nvidia.com/aiperf/v0.9.0/llms.txt
> - v0.8.0: https://docs.nvidia.com/aiperf/v0.8.0/llms.txt
> - v0.7.0: https://docs.nvidia.com/aiperf/v0.7.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/aiperf/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/aiperf/_mcp/server.

# Benchmarking AWS Endpoints

This guide walks you through benchmarking inference endpoints protected by AWS IAM authentication. AIPerf signs every request with your AWS credentials automatically -- you just need to tell it your AWS region and service name.

`--auth-type sigv4` signs every HTTP request AIPerf sends and works for any SigV4-protected endpoint.

## What's Supported

| Scenario | Non-Streaming | Streaming | Notes |
|----------|:---:|:---:|-------|
| API Gateway + vLLM/TGI/NIM | Yes | Yes | Standard HTTP + SSE |
| SageMaker + vLLM/LMI container | Yes | Yes | Use the SageMaker transport (`--sagemaker-endpoint-name`), which decodes SageMaker's eventstream framing. See [AWS SageMaker](/aiperf/dev/tutorials/model-endpoint-guides/aws-sage-maker). |
| Bedrock Converse / InvokeModel | No | No | Different request/response schema -- not OpenAI-compatible |

Endpoints that send `multipart/form-data` (`image_edit`, `video_generation`) are
not supported: signing a multipart body is not implemented, so AIPerf rejects
`--auth-type sigv4` for them at startup rather than sending unauthenticated
requests. `--aws-region` is required with `--auth-type sigv4`. `--aws-service` is
required too, unless the transport supplies the signing name itself -- the SageMaker
transport does. The `--aws-*` flags are rejected if `--auth-type sigv4` is not set,
since they would otherwise be silently ignored.

## Before You Start

1. Install the AWS extra (this pulls in `botocore` for credential handling):

```bash
uv pip install 'aiperf[aws]'
```

2. Make sure your AWS credentials are working:

```bash
aws sts get-caller-identity
```

If that prints your account and role info, you're good to go. If not, see [Setting Up Credentials](#setting-up-credentials).

## Quick Start

The key flags are `--auth-type sigv4`, `--aws-region`, and `--aws-service`. Add these to any `aiperf profile` command and AIPerf will sign every request automatically.

### API Gateway with IAM Auth

Your API Gateway fronts an OpenAI-compatible server and has IAM authorization enabled. Both streaming and non-streaming work:

```bash
aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-east-1.amazonaws.com/prod/v1 \
    --endpoint-type chat \
    --streaming \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service execute-api \
    --request-count 100
```

If your API Gateway maps a custom path to the backend, use `--endpoint` to set it:

```bash
aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-east-1.amazonaws.com \
    --endpoint /prod/inference/v1/chat/completions \
    --endpoint-type chat \
    --streaming \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service execute-api \
    --request-count 100
```

### SageMaker with vLLM or LMI

Use the SageMaker transport: `--sagemaker-endpoint-name` derives the URL, the
invocation path, the signing name and the streaming operation, and supports
`--streaming`. See [AWS SageMaker](/aiperf/dev/tutorials/model-endpoint-guides/aws-sage-maker).

The plain HTTP transport can also reach a SageMaker endpoint through an explicit
`--endpoint` path:

```bash
aiperf profile \
    --model my-model \
    --url https://runtime.sagemaker.us-east-1.amazonaws.com \
    --endpoint /endpoints/my-endpoint/invocations \
    --endpoint-type chat \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service sagemaker \
    --request-count 100
```

This `/invocations` route is non-streaming. To stream, add `--streaming` and
change `--endpoint` to `/endpoints/my-endpoint/invocations-response-stream`;
AIPerf decodes the AWS eventstream response on either transport. Keep the two
in step by hand: `--streaming` against `/invocations` gets one buffered
response, so TTFT equals total latency.

## Figuring Out Your Region and Service Name

The `--aws-region` should match the region in your endpoint URL:

```text
https://abc123.execute-api.us-east-1.amazonaws.com/...
                           ^^^^^^^^^
                           this is your --aws-region
```

The `--aws-service` depends on which AWS service handles your traffic:

| If your traffic goes through... | Use `--aws-service` |
|--------------------------------|---------------------|
| API Gateway | `execute-api` |
| SageMaker Runtime | `sagemaker` |

A common gotcha: the service name isn't always what you'd guess. For example, it's `sagemaker` not `sagemaker-runtime`. If you get a "SignatureDoesNotMatch" error, the service name is the first thing to double-check.

## Setting Up Credentials

If `aws sts get-caller-identity` already works, you can skip this section -- AIPerf will pick up the same credentials automatically.

### Environment Variables (simplest)

Good for quick local testing:

```bash
export AWS_ACCESS_KEY_ID="AKIA..."
export AWS_SECRET_ACCESS_KEY="wJal..."
export AWS_SESSION_TOKEN="FwoG..."  # only if using temporary credentials

aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-east-1.amazonaws.com/prod/v1 \
    --endpoint-type chat \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service execute-api \
    --request-count 100
```

### Named Profiles (multiple accounts)

If you work with more than one AWS account, you probably already have profiles set up in `~/.aws/credentials`. Point AIPerf at the right one with `--aws-profile`:

```bash
aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-west-2.amazonaws.com/prod/v1 \
    --endpoint-type chat \
    --auth-type sigv4 \
    --aws-region us-west-2 \
    --aws-service execute-api \
    --aws-profile staging \
    --request-count 100
```

Without `--aws-profile`, AIPerf uses whichever credentials the AWS CLI would use by default (environment variables first, then `[default]` profile, then IAM roles).

### SSO

If your team uses AWS IAM Identity Center (SSO), log in first, then pass the profile:

```bash
aws sso login --profile my-sso-profile

aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-east-1.amazonaws.com/prod/v1 \
    --endpoint-type chat \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service execute-api \
    --aws-profile my-sso-profile \
    --request-count 100
```

### Kubernetes (EKS)

On EKS, credentials are typically injected into your pod automatically via IRSA or Pod Identity. You don't need `--aws-profile` -- just make sure your pod's service account has the right IAM role attached:

```bash
aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-east-1.amazonaws.com/prod/v1 \
    --endpoint-type chat \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service execute-api \
    --request-count 1000
```

One thing to watch: if your pod has `AWS_ACCESS_KEY_ID` set as an environment variable (e.g., from a Kubernetes Secret), that takes priority over IRSA/Pod Identity. If you're hitting the wrong account, check for stale env vars.

## Long-Running Benchmarks

AIPerf refreshes AWS credentials automatically before each request. This means temporary credentials (from SSO, assumed roles, or IRSA) won't expire mid-benchmark. If you're running a long benchmark with thousands of requests, you don't need to do anything special.

The one exception: if your SSO session itself expires (they typically last 8-12 hours), you'll need to re-run `aws sso login` and restart the benchmark.

## Requests Other Than the Benchmark Itself

A benchmark run makes more than just inference requests, and all of them are
signed with the same credential chain, region, and service:

| Request | When it fires | Flags |
|---|---|---|
| Readiness probe (`/v1/models`, then a minimal inference request) | Before the run starts | `--wait-for-model-mode`, `--wait-for-model-timeout`, `--wait-for-model-interval` |
| KV-cache reset hook | Before each run | `--reset-kv-cache`, `--reset-kv-cache-path` |
| Server profiler start/stop hooks | At profiling phase start and end | `--server-profiler`, `--server-profiler-start-path`, `--server-profiler-stop-path` |

Two details matter behind an IAM-protected endpoint:

- **Each attempt is signed fresh.** A SigV4 signature embeds `x-amz-date` and
  AWS rejects it outside a five-minute skew window. Readiness polling and
  control-hook retry backoff both routinely run longer than that, so signing
  once per URL would start failing partway through the wait.
- **`--api-key` is suppressed on these paths when `--auth-type sigv4` is set**,
  exactly as it is on the request path. The signer owns the `Authorization`
  header; a stray `Authorization: Bearer ...` or `x-api-key` would overwrite or
  conflict with it. AIPerf logs a warning at startup when both are set, so a
  key that is never sent does not go unnoticed.

If the probe were unsigned, API Gateway would answer `403`, and the readiness
rule treats any status below `500` as "the server is up" -- so the run would
start against an endpoint that rejects every request. Signing the probe
closes that gap: when a signer is configured, a `401`/`403` response is
never treated as "ready". Preflight stops there and reports the likely
causes, because none of them are fixed by retrying.

A rejection is not proof that the signature is wrong. An IAM policy denial, a
WAF rule, or a path the endpoint does not route all return `401`/`403` on a
perfectly valid signature -- API Gateway in particular answers an unrouted
path with `403 Missing Authentication Token`, which AIPerf names explicitly so
you check the URL rather than the credentials.

## Examples

### High-Throughput API Gateway with Warmup

```bash
aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-east-1.amazonaws.com/prod/v1 \
    --endpoint-type chat \
    --streaming \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service execute-api \
    --request-rate 50 \
    --request-count 1000 \
    --warmup-request-count 20
```

### Multiple API Gateway Endpoints

Distribute load across two endpoints in the same region:

```bash
aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-east-1.amazonaws.com/prod/v1 \
    --url https://def456.execute-api.us-east-1.amazonaws.com/prod/v1 \
    --endpoint-type chat \
    --streaming \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service execute-api \
    --request-count 500
```

### SageMaker with Custom Dataset

```bash
aiperf profile \
    --model my-model \
    --url https://runtime.sagemaker.us-west-2.amazonaws.com \
    --endpoint /endpoints/my-endpoint/invocations \
    --endpoint-type chat \
    --auth-type sigv4 \
    --aws-region us-west-2 \
    --aws-service sagemaker \
    --dataset prompts.jsonl \
    --dataset-type single_turn
```

## Troubleshooting

### "SignatureDoesNotMatch"

This is the most common error. Check these in order:

1. **Is `--aws-region` correct?** It must match the region in the URL.
2. **Is `--aws-service` correct?** See the [service name table](#figuring-out-your-region-and-service-name) above. The names aren't always obvious.
3. **Is your system clock accurate?** AWS rejects signatures that are more than 5 minutes off. Docker containers and VMs are especially prone to clock drift. Run `date -u` and compare to actual UTC.

### "The security token included in the request is expired"

Your temporary credentials have expired. Re-authenticate:

```bash
# For SSO
aws sso login --profile my-profile

# For assumed roles, this usually resolves itself --
# botocore refreshes automatically if the source credentials are still valid
```

### "No AWS credentials found"

AIPerf can't find any credentials. Verify with:

```bash
aws sts get-caller-identity
```

If that also fails, you need to set up credentials -- see [Setting Up Credentials](#setting-up-credentials).

### "the optional botocore dependency is not installed"

The full message starts with what needed botocore, for example:

```text
SigV4 request signing is enabled but the optional botocore dependency is not
installed.
```

Install the AWS extra:

```bash
uv pip install 'aiperf[aws]'
```