Configure & tune

Most deployments need no tuning — the defaults are production-ready. When you do need to adjust behaviour (for example, to raise concurrency on a larger instance), you set container environment variables at model-creation time.

How configuration works

Pass an Environment map when you create the model. SageMaker injects these into the model container, so they apply to whatever endpoint you build from that model.

1
sm.create_model(
2
    ModelName=endpoint_name,
3
    PrimaryContainer={
4
        "ModelPackageName": model_package_arn,
5
        "Environment": {
6
            # container tuning variables go here
7
        },
8
    },
9
    ExecutionRoleArn=role,
10
    EnableNetworkIsolation=True,
11
)

To change a value later, update the model (or create a new one) and point a new endpoint configuration at it.

Environment variables

Defaults are auto-derived per instance, so most endpoints need no tuning — override only with a reason. The variables below are for the Sarvam Vision container.

Sync vs async profiles

The same container serves both modes — it can’t detect which it was deployed into, and its auto-derived concurrency targets throughput, not AWS’s 60-second sync deadline. Pin a profile per mode in the Model Environment.

Sync — fit every request inside 60 s by capping pages and adding the admission valve:

$
# small, single-GPU instance (ml.g6.xlarge / ml.g6e.xlarge)
$
SM_MAX_CONCURRENT_JOBS=1
$
SM_MAX_INFLIGHT=2
$
SM_MAX_PAGES=5            # keep sync documents small so they finish inside 60 s
$
SM_JOB_WAIT_TIMEOUT_SECONDS=75
$
$
# larger / multi-GPU instance (a document fans out across GPUs)
$
SM_MAX_INFLIGHT=<2 x concurrent docs>   # e.g. 8 on ml.g6e.2xlarge
$
SM_MAX_PAGES=5
$
SM_JOB_WAIT_TIMEOUT_SECONDS=75

Async — long jobs and large payloads; keep the container defaults but raise the wait budget:

$
SM_MAX_PAGES=500
$
SM_JOB_WAIT_TIMEOUT_SECONDS=3300         # inside AWS's async ceiling

and set InvocationTimeoutSeconds=3600 per request for large documents.

Text-to-Speech (Bulbul) concurrency

The Bulbul v3 container admits work through a fixed door and sheds excess immediately with 503 service_overloaded (plus a body retry_after) rather than queueing it. Two ceilings matter:

• HTTP serve ceiling = WEB_CONCURRENCY × MAX_CONCURRENT_HTTP_REQUESTS — concurrent real-time / SSE requests admitted.
• WebSocket cap = MAX_CONCURRENT_WS_SESSIONS — concurrent bidirectional sessions.

With these defaults the HTTP serve ceiling is 5 (WEB_CONCURRENCY × MAX_CONCURRENT_HTTP_REQUESTS) and the WebSocket cap is ~28. Scale throughput by adding instances or raising WEB_CONCURRENCY — not by choosing a larger single-GPU instance (xlarge ≈ 4xlarge for throughput; vCPU/RAM don’t move the needle).

Sizing vs tuning

Tuning changes how a single instance behaves; sizing and autoscaling change how many instances you run and how big they are. For throughput and cost, start with Operations.