CertSanctuary

MLA-C02 · AIF-C01 · SAA-C03

SageMaker AI inference options: real-time, serverless, async and batch

How to choose between Amazon SageMaker AI real-time endpoints, Serverless Inference, Asynchronous Inference and batch transform based on latency, payload size and traffic.

Four ways to get predictions

Once a model is trained in Amazon SageMaker AI, you need a way to send it data and get predictions back. SageMaker AI offers four main deployment options. They differ in how fast responses come back, how big the inputs can be, and whether you pay for idle time.

The exam tests these by describing traffic and latency needs, so it pays to link each option to a pattern.

Real-time inference

A real-time endpoint is a persistent HTTPS endpoint backed by instances you choose, including GPU instances. Requests return predictions in milliseconds, and the endpoint stays up between requests.

It fits steady traffic with strict latency needs. A product recommendation model called on every page view, or a fraud check that must answer before a card payment completes, are typical cases.

You can attach auto scaling so the instance count grows with traffic. You pay for the instances as long as the endpoint runs, even when no requests arrive. Multi-model endpoints let many models share one endpoint, which saves money when you have lots of small, rarely used models.

Serverless Inference

Serverless Inference removes instance management. You pick a memory size and a concurrency limit, and SageMaker AI provisions compute when requests come in and scales to zero when they stop. You pay for the compute time used to process requests.

The trade-off is cold starts. After an idle period, the first request waits while compute spins up. Serverless Inference also does not support GPUs.

It fits intermittent or unpredictable traffic that can tolerate an occasional slower response, such as an internal tool used a few times an hour.

Asynchronous Inference

Asynchronous Inference puts incoming requests in a queue. The client sends a request pointing to input data in Amazon S3 and gets back an output location right away. When processing finishes, the result is written to S3 and SageMaker AI can send an Amazon SNS notification.

It fits large inputs and long processing times, such as transcribing hour-long audio files or running a computer vision model on high-resolution images. The endpoint can scale down to zero instances when the queue is empty, so you avoid paying for idle capacity.

Batch transform

Batch transform runs predictions on a whole dataset at once. You point a job at data in S3, SageMaker AI starts instances, processes every record, writes results to S3 and shuts the instances down. There is no endpoint.

It fits offline work where nobody waits on a single answer: scoring all customers for churn every night, or generating predictions for a backlog of historical records. It is also a common way to preprocess data or run inference on a test set during evaluation.

Choosing quickly

Match the clues in a scenario to an option:

Common traps

"Must respond in milliseconds" rules out batch transform and Asynchronous Inference. "Requires GPU" rules out Serverless Inference. "Payloads are very large" or "processing takes several minutes" points away from real-time and serverless toward Asynchronous Inference.

Cost wording matters too. If a question stresses that the model is idle most of the day, a real-time endpoint is usually the expensive choice, because its instances keep running. Serverless and Asynchronous Inference both avoid paying for idle capacity, and batch transform only runs while the job does.

Practice questions

  1. An internal tool calls a model hosted on SageMaker AI Serverless Inference. Traffic is near zero overnight, but during business hours users complain that some requests take several seconds longer because of cold starts. The company wants to keep the serverless endpoint. What should the ML engineer do?

    1. Configure provisioned concurrency on the serverless endpoint and scale it up for business hours.
    2. Raise MaxConcurrency on the serverless endpoint so that more requests can run in parallel.
    3. Raise the memory size of the serverless endpoint so that the container can never cold start.
    4. Add a second production variant to the serverless endpoint to split the traffic evenly.
    Answer

    A. Provisioned concurrency keeps serverless compute initialized so requests avoid cold starts, and it can be scaled with Application Auto Scaling around business hours. MaxConcurrency only caps parallel requests and does not keep capacity warm. More memory can shorten initialization but does not eliminate cold starts.

    MLA-C02 question 100

  2. A SageMaker AI batch transform job scores millions of records stored as many CSV files in Amazon S3, but it runs too slowly to finish within its nightly window. Which two changes will reduce the job duration? (Choose two.)

    1. Enable data capture on the transform job
    2. Switch the job to a real-time endpoint called once per record
    3. Use the MultiRecord batch strategy with Line splitting and raise the maximum concurrent transforms per instance
    4. Combine all input into one large file before starting the job
    5. Increase the instance count so that the input files are distributed across more instances
    Answer

    E, C. More instances let batch transform process different S3 objects in parallel, and MultiRecord batching with a higher number of concurrent transforms sends mini-batches of records to every model worker on each instance instead of one record or one request at a time. Merging everything into one file prevents the work from being spread across instances, and a per-record real-time loop adds request overhead.

    MLA-C02 question 11

  3. An ML platform team wants to define its SageMaker AI endpoints, IAM roles and auto scaling in Python, using loops and reusable classes so that 20 similar model deployments share one definition. The resources must still be provisioned through AWS CloudFormation. Which tool should the team use?

    1. AWS Config
    2. AWS CodeDeploy
    3. AWS CodeArtifact
    4. AWS Cloud Development Kit (AWS CDK)
    Answer

    D. The AWS CDK lets teams define infrastructure in languages such as Python with reusable constructs, and it synthesizes CloudFormation templates for deployment. CodeArtifact stores software packages, CodeDeploy releases application revisions, and AWS Config records and evaluates resource configurations.

    MLA-C02 question 336

  4. A document-processing model runs on a SageMaker AI asynchronous inference endpoint. Requests arrive only a few times per day, and the company wants to pay for no instances while the queue is empty. The endpoint must still start processing when a new request arrives. Which TWO actions should the ML engineer take? (Choose two.)

    1. Configure provisioned concurrency of 0 on the asynchronous inference endpoint.
    2. Add a target tracking policy on SageMakerVariantInvocationsPerInstance with a target of 0.
    3. Set the scale-in cooldown to 0 seconds on the existing target tracking policy.
    4. Add a step scaling policy driven by an alarm on the HasBacklogWithoutCapacity metric.
    5. Register the variant as a scalable target with a minimum capacity of 0 instances.
    Answer

    E, D. Asynchronous endpoints can scale to zero when the scalable target's minimum capacity is 0. Because a target tracking policy based on backlog per instance cannot act when there are no instances, a step scaling policy on HasBacklogWithoutCapacity is used to add the first instance when requests are queued. Provisioned concurrency is a serverless inference setting, and a cooldown change does not let the endpoint scale out from zero.

    MLA-C02 question 246

Practice MLA-C02 Practice AIF-C01 Practice SAA-C03 All topics