Benchmarking AWS Endpoints

This guide walks you through benchmarking inference endpoints protected by AWS IAM authentication. AIPerf signs every request with your AWS credentials automatically — you just need to tell it your AWS region and service name.

--auth-type sigv4 signs every HTTP request AIPerf sends and works for any SigV4-protected endpoint.

What’s Supported

Endpoints that send multipart/form-data (image_edit, video_generation) are not supported: signing a multipart body is not implemented, so AIPerf rejects --auth-type sigv4 for them at startup rather than sending unauthenticated requests. --aws-region is required with --auth-type sigv4. --aws-service is required too, unless the transport supplies the signing name itself — the SageMaker transport does. The --aws-* flags are rejected if --auth-type sigv4 is not set, since they would otherwise be silently ignored.

Before You Start

1. Install the AWS extra (this pulls in botocore for credential handling):

uv pip install 'aiperf[aws]'

2. Make sure your AWS credentials are working:

aws sts get-caller-identity

If that prints your account and role info, you’re good to go. If not, see Setting Up Credentials.

Quick Start

The key flags are --auth-type sigv4, --aws-region, and --aws-service. Add these to any aiperf profile command and AIPerf will sign every request automatically.

API Gateway with IAM Auth

Your API Gateway fronts an OpenAI-compatible server and has IAM authorization enabled. Both streaming and non-streaming work:

aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-east-1.amazonaws.com/prod/v1 \
    --endpoint-type chat \
    --streaming \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service execute-api \
    --request-count 100

If your API Gateway maps a custom path to the backend, use --endpoint to set it:

aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-east-1.amazonaws.com \
    --endpoint /prod/inference/v1/chat/completions \
    --endpoint-type chat \
    --streaming \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service execute-api \
    --request-count 100

SageMaker with vLLM or LMI

Use the SageMaker transport: --sagemaker-endpoint-name derives the URL, the invocation path, the signing name and the streaming operation, and supports --streaming. See AWS SageMaker.

The plain HTTP transport can also reach a SageMaker endpoint through an explicit --endpoint path:

aiperf profile \
    --model my-model \
    --url https://runtime.sagemaker.us-east-1.amazonaws.com \
    --endpoint /endpoints/my-endpoint/invocations \
    --endpoint-type chat \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service sagemaker \
    --request-count 100

This /invocations route is non-streaming. To stream, add --streaming and change --endpoint to /endpoints/my-endpoint/invocations-response-stream; AIPerf decodes the AWS eventstream response on either transport. Keep the two in step by hand: --streaming against /invocations gets one buffered response, so TTFT equals total latency.

Figuring Out Your Region and Service Name

The --aws-region should match the region in your endpoint URL:

https://abc123.execute-api.us-east-1.amazonaws.com/...
                           ^^^^^^^^^
                           this is your --aws-region

The --aws-service depends on which AWS service handles your traffic:

A common gotcha: the service name isn’t always what you’d guess. For example, it’s sagemaker not sagemaker-runtime. If you get a “SignatureDoesNotMatch” error, the service name is the first thing to double-check.

Setting Up Credentials

If aws sts get-caller-identity already works, you can skip this section — AIPerf will pick up the same credentials automatically.

Environment Variables (simplest)

Good for quick local testing:

export AWS_ACCESS_KEY_ID="AKIA..."
export AWS_SECRET_ACCESS_KEY="wJal..."
export AWS_SESSION_TOKEN="FwoG..."  # only if using temporary credentials
aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-east-1.amazonaws.com/prod/v1 \
    --endpoint-type chat \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service execute-api \
    --request-count 100

Named Profiles (multiple accounts)

If you work with more than one AWS account, you probably already have profiles set up in ~/.aws/credentials. Point AIPerf at the right one with --aws-profile:

aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-west-2.amazonaws.com/prod/v1 \
    --endpoint-type chat \
    --auth-type sigv4 \
    --aws-region us-west-2 \
    --aws-service execute-api \
    --aws-profile staging \
    --request-count 100

Without --aws-profile, AIPerf uses whichever credentials the AWS CLI would use by default (environment variables first, then [default] profile, then IAM roles).

SSO

If your team uses AWS IAM Identity Center (SSO), log in first, then pass the profile:

aws sso login --profile my-sso-profile
aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-east-1.amazonaws.com/prod/v1 \
    --endpoint-type chat \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service execute-api \
    --aws-profile my-sso-profile \
    --request-count 100

Kubernetes (EKS)

On EKS, credentials are typically injected into your pod automatically via IRSA or Pod Identity. You don’t need --aws-profile — just make sure your pod’s service account has the right IAM role attached:

aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-east-1.amazonaws.com/prod/v1 \
    --endpoint-type chat \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service execute-api \
    --request-count 1000

One thing to watch: if your pod has AWS_ACCESS_KEY_ID set as an environment variable (e.g., from a Kubernetes Secret), that takes priority over IRSA/Pod Identity. If you’re hitting the wrong account, check for stale env vars.

Long-Running Benchmarks

AIPerf refreshes AWS credentials automatically before each request. This means temporary credentials (from SSO, assumed roles, or IRSA) won’t expire mid-benchmark. If you’re running a long benchmark with thousands of requests, you don’t need to do anything special.

The one exception: if your SSO session itself expires (they typically last 8-12 hours), you’ll need to re-run aws sso login and restart the benchmark.

Requests Other Than the Benchmark Itself

A benchmark run makes more than just inference requests, and all of them are signed with the same credential chain, region, and service:

Two details matter behind an IAM-protected endpoint:

• Each attempt is signed fresh. A SigV4 signature embeds x-amz-date and AWS rejects it outside a five-minute skew window. Readiness polling and control-hook retry backoff both routinely run longer than that, so signing once per URL would start failing partway through the wait.
• --api-key is suppressed on these paths when --auth-type sigv4 is set, exactly as it is on the request path. The signer owns the Authorization header; a stray Authorization: Bearer ... or x-api-key would overwrite or conflict with it. AIPerf logs a warning at startup when both are set, so a key that is never sent does not go unnoticed.

If the probe were unsigned, API Gateway would answer 403, and the readiness rule treats any status below 500 as “the server is up” — so the run would start against an endpoint that rejects every request. Signing the probe closes that gap: when a signer is configured, a 401/403 response is never treated as “ready”. Preflight stops there and reports the likely causes, because none of them are fixed by retrying.

A rejection is not proof that the signature is wrong. An IAM policy denial, a WAF rule, or a path the endpoint does not route all return 401/403 on a perfectly valid signature — API Gateway in particular answers an unrouted path with 403 Missing Authentication Token, which AIPerf names explicitly so you check the URL rather than the credentials.

Examples

High-Throughput API Gateway with Warmup

aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-east-1.amazonaws.com/prod/v1 \
    --endpoint-type chat \
    --streaming \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service execute-api \
    --request-rate 50 \
    --request-count 1000 \
    --warmup-request-count 20

Multiple API Gateway Endpoints

Distribute load across two endpoints in the same region:

aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-east-1.amazonaws.com/prod/v1 \
    --url https://def456.execute-api.us-east-1.amazonaws.com/prod/v1 \
    --endpoint-type chat \
    --streaming \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service execute-api \
    --request-count 500

SageMaker with Custom Dataset

aiperf profile \
    --model my-model \
    --url https://runtime.sagemaker.us-west-2.amazonaws.com \
    --endpoint /endpoints/my-endpoint/invocations \
    --endpoint-type chat \
    --auth-type sigv4 \
    --aws-region us-west-2 \
    --aws-service sagemaker \
    --dataset prompts.jsonl \
    --dataset-type single_turn

Troubleshooting

”SignatureDoesNotMatch”

This is the most common error. Check these in order:

1. Is --aws-region correct? It must match the region in the URL.
2. Is --aws-service correct? See the service name table above. The names aren’t always obvious.
3. Is your system clock accurate? AWS rejects signatures that are more than 5 minutes off. Docker containers and VMs are especially prone to clock drift. Run date -u and compare to actual UTC.

”The security token included in the request is expired”

Your temporary credentials have expired. Re-authenticate:

# For SSO
aws sso login --profile my-profile
# For assumed roles, this usually resolves itself --
# botocore refreshes automatically if the source credentials are still valid

“No AWS credentials found”

AIPerf can’t find any credentials. Verify with:

aws sts get-caller-identity

If that also fails, you need to set up credentials — see Setting Up Credentials.

”the optional botocore dependency is not installed”

The full message starts with what needed botocore, for example:

SigV4 request signing is enabled but the optional botocore dependency is not
installed.

Install the AWS extra:

uv pip install 'aiperf[aws]'