Benchmarking an AWS SageMaker Endpoint

AIPerf can drive a model hosted behind a SageMaker Runtime endpoint directly — no translation shim in front of it.

SageMaker is not an OpenAI-compatible HTTP server. Requests go to POST /endpoints/{name}/invocations, must be SigV4-signed, and streaming responses come back as binary application/vnd.amazon.eventstream frames rather than SSE. AIPerf handles all three.

What it does not change is the request body. If you are running vLLM, TGI, or an LMI container behind the endpoint, that container still speaks OpenAI, so --endpoint-type chat (or completions, or embeddings) works exactly as it does against a local server.

Quick Start

aiperf profile \
    -m my-model \
    --endpoint-type chat \
    --streaming \
    --sagemaker-endpoint-name my-vllm-endpoint \
    --aws-region us-west-2

That is the whole thing. --sagemaker-endpoint-name implies the rest:

Install the AWS extra first if you have not already:

uv pip install 'aiperf[aws]'

The One Surprising Part

--endpoint-type chat still applies, even though /v1/chat/completions never appears in the URL.

This trips up nearly everyone the first time. SageMaker routes on the request body, not the path: every endpoint type is invoked through the same /endpoints/{name}/invocations path, and the container behind it decides what to do with the JSON you sent.

So --endpoint-type still selects the payload shape and the response parser — it just no longer selects a URL path. Use chat for a chat-completions container, completions for a text-completions one, embeddings for an embeddings model. If you pass --endpoint-type chat and see no /v1/chat/completions in the logs, that is correct.

Streaming

--streaming switches AIPerf to InvokeEndpointWithResponseStream, which is a genuinely different SageMaker operation on a different path — not a content-negotiation flag. AIPerf picks the right one for you.

Getting this wrong manually is quiet rather than loud: calling the non-streaming path with "stream": true in the body returns one buffered response, and your TTFT and ITL numbers silently become meaningless. Deriving it from --streaming is why you should not hand-write the path.

Interpreting ITL

The eventstream reader stamps one timestamp per network read and shares it across every message decoded from that read. This is exactly what AIPerf already does for SSE, and it is deliberate: frames that arrived in the same TCP segment genuinely arrived together, so timestamping them individually would only add decode latency to your measurements and make SageMaker ITL non-comparable to SSE ITL from the same model.

In other words, ITL from a SageMaker run and ITL from the same vLLM behind plain HTTP are directly comparable.

Credentials

AIPerf uses botocore’s standard credential chain. If aws sts get-caller-identity works, AIPerf will authenticate.

# A named profile
aiperf profile -m my-model --sagemaker-endpoint-name my-ep \
    --aws-region us-west-2 --aws-profile my-profile

Environment variables, ~/.aws/credentials, SSO, EC2/ECS instance roles, and EKS IRSA all work. See the SigV4 tutorial for the full walkthrough, including credential rotation during long benchmarks.

Minimum IAM Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sagemaker:InvokeEndpoint",
                "sagemaker:InvokeEndpointWithResponseStream"
            ],
            "Resource": "arn:aws:sagemaker:us-west-2:123456789012:endpoint/my-vllm-endpoint"
        }
    ]
}

InvokeEndpointWithResponseStream is a separate action from InvokeEndpoint. Granting only the first works until you add --streaming, then fails with AccessDeniedException.

Multi-Model and Multi-Component Endpoints

# Multi-Model Endpoint: pick which model artifact to invoke
aiperf profile -m my-model --sagemaker-endpoint-name my-mme \
    --aws-region us-west-2 --sagemaker-target-model my-model.tar.gz
# Inference components (SageMaker's newer hosting model)
aiperf profile -m my-model --sagemaker-endpoint-name my-ep \
    --aws-region us-west-2 --sagemaker-inference-component-name my-component
# Pin one production variant, bypassing the endpoint's traffic split
aiperf profile -m my-model --sagemaker-endpoint-name my-ep \
    --aws-region us-west-2 --sagemaker-target-variant variant-b

Set --sagemaker-target-model only for a Multi-Model Endpoint, and give it the model artifact’s path relative to the endpoint’s model location (for example model-a.tar.gz), not the -m name. AIPerf never fills it in for you: SageMaker rejects every request that carries TargetModel against a single-model endpoint with “Endpoint … is not a multi-model endpoint and does not support target model header”. AWS also does not accept TargetModel on the streaming operation, so AIPerf omits it when --streaming is set.

--sagemaker-target-variant is useful for A/B deployments: it makes every request land on one variant, so you are benchmarking that variant rather than the blend.

All three options need the SageMaker transport, which --sagemaker-endpoint-name or --transport sagemaker selects. AIPerf rejects them under any other transport, which would otherwise ignore them without a word and benchmark the endpoint’s default routing.

VPC / PrivateLink and Custom Domains

An explicit --url always wins over the derived hostname, and --sagemaker-endpoint-name still supplies the path:

aiperf profile -m my-model \
    --sagemaker-endpoint-name my-ep \
    --aws-region us-west-2 \
    --url https://vpce-0abc123-xyz.sagemaker.us-west-2.vpce.amazonaws.com

--aws-region is still required in this case: it is the SigV4 credential scope, not just a hostname component.

What Is Not Supported

Some AIPerf features cannot work against a SageMaker endpoint. AIPerf refuses these at startup rather than failing partway into a run:

Also out of scope: SageMaker Asynchronous Inference and Serverless Inference endpoints, which exchange payloads through S3 rather than in the HTTP request body.

Troubleshooting

403 with no detail, or AccessDeniedException

Your request went out unsigned or signed wrong. Check, in order:

1. Is --aws-region correct? It must match the endpoint’s region. It sets both the hostname and the signing scope.
2. Do your credentials have sagemaker:InvokeEndpoint? Run aws sagemaker-runtime invoke-endpoint --endpoint-name my-ep ... to confirm independently of AIPerf.
3. Are you streaming? That needs the separate sagemaker:InvokeEndpointWithResponseStream action.
4. Is your system clock accurate? AWS rejects signatures more than 5 minutes out. Containers and VMs drift; compare date -u to real UTC.

ValidationError / ModelError from the endpoint

The container rejected your payload. This is almost always a payload-shape mismatch rather than a SageMaker problem — check that --endpoint-type matches what the container actually serves.

”An AWS eventstream response was received but the optional botocore dependency is not installed”

uv pip install 'aiperf[aws]'

Streaming “works” but TTFT equals total latency

You are hitting the non-streaming operation, so the whole response arrives at once. Check that --streaming is set: with the SageMaker transport it picks the operation, even when --url already ends in /endpoints/<name>/invocations or /invocations-response-stream for the same endpoint.

A path is used exactly as written only when you set it with --endpoint, or when you use the plain HTTP transport (see the SigV4 tutorial). There, a streaming run must point at /endpoints/<name>/invocations-response-stream.

404 from the endpoint

The endpoint name is wrong, or the endpoint is in a different region than --aws-region. Confirm with:

aws sagemaker describe-endpoint --endpoint-name my-ep --region us-west-2

See Also

• AWS SigV4 Authentication — signing against any SigV4-protected endpoint, including API Gateway.
• SageMaker Data Capture — correlating AIPerf records with SageMaker’s captured inference data. AIPerf sends its request ID as X-Amzn-SageMaker-Inference-Id on every request, which is the join key.