LiveKit in Production

LiveKit in Production

A tuning and operations guide for teams shipping cascading voice agents (Sarvam STT, then an LLM, then Sarvam TTS) on the LiveKit Agents SDK.

Everything here is derived from the current livekit-plugins-sarvam plugin source and the livekit-agents runtime. New to LiveKit and Sarvam? Start with Build Your First Voice Agent using LiveKit first; come back here before you ship.

Which STT class to use

saaras:v3-realtime gives you true partial transcripts (barge-in no longer waits for a final), a simpler VAD (four millisecond-valued parameters instead of ten frame counters), live reconfiguration without a reconnect, and a structured event and close-code protocol. This guide documents STTRealtime throughout; the legacy class is covered only in Section 11.

Contents

1. Reference config
2. Where latency hides
3. Skip LiveKit’s Silero VAD
4. STT configuration
5. Tuning Sarvam’s VAD
6. Turn handling and endpointing
7. Barge-in and interruptions
8. TTS configuration
9. Measuring latency
10. Capturing request IDs
11. The legacy class and migration
12. Reliability and failure modes
13. Tuning playbook
14. Copy-paste configs

1. Reference config

from livekit.agents import AgentSession
from livekit.plugins import sarvam
session = AgentSession(
    # STT: Sarvam's own server-side VAD drives turn-taking
    stt=sarvam.STTRealtime(
        language="hi-IN",
        stream_type="fast",         # was "balanced": partials every 500 ms instead of 1000 ms
        mode="transcribe",
        endpointing="vad",
        sample_rate=16000,
        vad_min_silence_ms=400,     # Sarvam's end-of-turn silence window
    ),
    llm=sarvam.LLM(model="sarvam-105b"),   # or any other LLM plugin
    tts=sarvam.TTS(
        model="bulbul:v4-flash",
        speaker="ritu_hi_customer",
        target_language_code="hi-IN",
        min_buffer_size=30,         # was 50, lowers TTFB
        max_chunk_length=150,
        output_audio_codec="linear16",   # skip the mp3 decode hop
        speech_sample_rate=24000,
    ),
    # No LiveKit VAD: Sarvam STT runs its own server-side (Section 3 covers the trade-off)
    vad=None,
    turn_handling={
        "turn_detection": "stt",             # trust Sarvam's vad.speech_end
        "endpointing": {
            "mode": "fixed",
            "min_delay": 0.25,               # was 0.5 (or 0.3 streaming default)
            "max_delay": 2.0,
        },
        "interruption": {
            "enabled": True,
            "mode": "vad",                   # the adaptive detector can't work here, see Section 7
            "min_words": 1,                  # the real noise filter when vad=None
            "false_interruption_timeout": 1.5,
            "resume_false_interruption": True,
        },
        "preemptive_generation": {"enabled": True},
    },
    # other defaults worth revisiting
    aec_warmup_duration=None,        # None for inbound SIP (outbound SIP already defaults to None); keep 3.0 for WebRTC + mic
    user_away_timeout=None,          # or a value that matches your product
    min_consecutive_speech_delay=0.0,
)

Every override above is explained below. If you change nothing else, change vad, turn_detection, stream_type, and endpointing.min_delay.

2. Where latency hides

The user-perceived latency is the sum of the whole chain, not any one model:

Three things worth knowing:

• stream_type sets how soon partial transcripts arrive. Sarvam’s server feeds the streaming model audio in 500 ms windows on "fast" and 1000 ms windows on the default "balanced", so a partial can’t arrive sooner than one window. Partials drive barge-in (Section 7) and live captions. End-of-speech isn’t delayed: the server’s VAD runs on the raw audio, not on the windows. Set stream_type="fast".
• vad_min_silence_ms and min_delay stack. They don’t overlap. 500 ms of Sarvam silence plus a 0.5 s min_delay is a full second of dead air. Tune both together.
• TTS time-to-first-byte is dominated by connection state, not model speed. A cold Sarvam TTS websocket costs a full TLS handshake on the critical path. Call prewarm() (Section 8).

Measure this instrumented, don’t guess: Section 9.

3. Skip LiveKit’s Silero VAD

We recommend passing vad=None and letting Sarvam’s server-side VAD drive turn-taking. It’s a trade-off, not a requirement: LiveKit’s own pattern for turn_detection="stt" keeps its bundled VAD for interruption detection, and a VAD is what enables the features in the table below.

STTRealtime runs with endpointing="vad" by default, so the server emits vad.speech_start and vad.speech_end on every session, and the plugin maps those onto the framework’s start/end-of-speech events. That VAD already runs server-side on the exact audio the recognizer sees, is tunable through four explicit parameters trained on Indic speech (Section 5), and adds no local inference cost. Running Silero alongside it means two VADs disagreeing about turn boundaries: double-triggered interruptions, warnings in your logs, and endpointing metrics anchored on the wrong timestamp.

Passing vad=None does disable five framework features gated on a VAD instance. Be deliberate about the trade:

The trade is a good one on saaras:v3-realtime specifically: because it emits real partial transcripts and signals end-of-speech before the final transcript, barge-in fires on partials (Section 7) and preemptive generation actually engages (Section 6), neither of which worked without a VAD on the legacy plugin.

LiveKit’s turn detector is a weak fit for most Indic languages regardless. The default inference.TurnDetector is an audio end-of-turn model in two versions: v1, served on LiveKit Inference, and v1-mini, which runs locally on CPU anywhere. It’s calibrated for 14 languages, including Hindi but not Tamil, Telugu, Bengali or other Indic languages; any other language uses its default calibration. Sarvam’s own VAD plus a tuned min_delay is the more predictable choice for Indic languages.

Keep a VAD instead when you need transcription_timeout, or LiveKit’s turn detector for Hindi (v1-mini doesn’t need LiveKit Cloud). If you do keep one, its min_silence_duration must be at least 0.25 s, or AgentSession raises a ValueError.

4. STT configuration

sarvam.STTRealtime(
    language="hi-IN",            # default: "en-IN"; "auto" detects it
    stream_type="fast",          # default: "balanced"
    mode="transcribe",           # default: "transcribe"
    endpointing="vad",           # default: "vad"
    encoding="linear16",         # or "linear32", "mulaw", "alaw"
    sample_rate=16000,           # default: 16000 (8000 or 16000 only)
    prompt=None,
    return_timestamps=False,
    api_key=None,                # falls back to SARVAM_API_KEY
    # server-side VAD tuning, see Section 5
    vad_sot_threshold=None,
    vad_min_speech_ms=None,
    vad_min_silence_ms=None,
    vad_prefix_padding_ms=None,  # connection-only
)

Endpoint: wss://api.sarvam.ai/speech-to-text-realtime/ws. Model is fixed at saaras:v3-realtime; any invalid parameter raises ValueError at construction.

stream_type sets how often Sarvam produces partial transcripts. The server feeds the streaming model 500 ms audio windows on fast and 1000 ms windows on balanced (the default); simulated produces no partials, only finals. The plugin itself always sends 50 ms frames, and the server’s VAD runs on the raw audio, so stream_type doesn’t delay end-of-speech detection. Set "fast" for any interactive agent, because partials are what drive barge-in.

endpointing: vad (default) has the server emit speech-start and speech-end events, and pairs with turn_detection="stt". manual has the plugin send events itself on the first audio frame and on flush; pair with the framework’s turn_detection="manual" for push-to-talk or IVR flows. In manual mode the VAD tuning parameters aren’t sent at all.

mode:

codemix is worth A/B-testing against transcribe for consumer-facing Indian deployments.

language="auto" has Sarvam detect the language, and it reports the detected language on every partial and final. Use it when callers switch languages unpredictably. When your app already knows the caller switched, set the language with update_options(language=...) instead (below).

prompt carries domain vocabulary: product names, city names, SKUs, agent names. It’s the cheapest accuracy win available and frequently left empty. The realtime plugin sends it as a URL query parameter on the websocket handshake (the legacy class doesn’t send prompt at all for saaras:v3 or saaras:v4), so keep it short (a compact term list) or a long prompt risks a rejected handshake; use a TTS pronunciation dictionary (dict_id) for output-side fixes instead.

stt = sarvam.STTRealtime(
    language="hi-IN",
    stream_type="fast",
    prompt="Insurance policy renewal. Terms: premium, NCB, IDV, Bajaj Allianz, endorsement.",
)

Live reconfiguration. STTRealtime.update_options(...) sends a config update over the live websocket, no reconnect, no dropped audio, and the plugin defers any endpointing switch until the current utterance completes. Updatable in place: language, stream_type, mode, prompt and endpointing, which apply from the next utterance, and the three VAD tuning parameters, which apply immediately. sample_rate, return_timestamps and vad_prefix_padding_ms are connection-only: changing them on a live stream is logged and ignored, so set them at construction.

# e.g. the caller switched to Tamil: no reconnect, no dropped audio
session.stt.update_options(language="ta-IN")

sample_rate and encoding: sample_rate is 8000 or 16000 only; encoding is linear16 (default), linear32, mulaw or alaw. Use 16000 for WebRTC; 8000 matches PSTN, but upsampling to 16 kHz is usually cheaper than degrading the recognizer’s input.

Event mapping:

END_OF_SPEECH precedes FINAL_TRANSCRIPT, the opposite of the legacy plugin, and it’s why preemptive generation works here (Section 6): the framework marks the turn committed at end-of-speech, then the arriving final triggers speculative LLM work and starts the endpointing hold.

5. Tuning Sarvam’s VAD

Four parameters, sent only when endpointing="vad". All default to None, which means Sarvam’s server-side default (below) applies; set them explicitly whenever you need reproducible behavior across environments.

Lowering vad_min_silence_ms ends turns faster (snappier, more mid-sentence cutoffs); raising it holds through natural pauses (no cutoffs, more dead air). It stacks with endpointing.min_delay (Section 6): budget them together, since tuning one down while leaving the other conservative won’t move the needle.

On noisy telephony, turn on noise cancellation first (below). Then raise vad_sot_threshold, the cleanest VAD-side defense against a VAD that keeps triggering on ambient audio. Raise vad_min_speech_ms to ignore clicks, coughs, keyboard noise, and DTMF beeps; lower it to catch very short words.

Recommended starting points by channel:

Tune the VAD before you tune min_delay. They compensate for each other, so changing both at once means you can’t attribute a regression.

Noise cancellation. Removing background voices and noise before audio reaches Sarvam cuts false turns and false interruptions more than any VAD threshold can. LiveKit’s Krisp plugin does this on the agent side: krisp.voice_isolation() for app and WebRTC callers, krisp.voice_isolation_telephony() for SIP callers. Install livekit-plugins-krisp and pass the filter through room_options when you start the session; Section 14 shows both. It authenticates through LiveKit Cloud by default. On self-hosted LiveKit you need your own Krisp license, passed as auth_provider=krisp.auth.krisp_license(...).

from livekit.agents import room_io
from livekit.plugins import krisp
await session.start(
    agent=agent,
    room=ctx.room,
    room_options=room_io.RoomOptions(
        audio_input=room_io.AudioInputOptions(
            noise_cancellation=krisp.voice_isolation_telephony(),  # krisp.voice_isolation() for app callers
        ),
    ),
)

6. Turn handling and endpointing

min_endpointing_delay, allow_interruptions, turn_detection, preemptive_generation, and similar flat AgentSession kwargs are deprecated in favor of a nested turn_handling dict:

AgentSession(
    turn_handling={
        "turn_detection": "stt",
        "endpointing":           {"mode": ..., "min_delay": ..., "max_delay": ..., "alpha": ...},
        "interruption":          {"enabled": ..., "mode": ..., "min_duration": ..., "min_words": ...,
                                  "false_interruption_timeout": ..., "resume_false_interruption": ...},
        "preemptive_generation": {"enabled": ..., "preemptive_tts": ..., "max_speech_duration": ..., "max_retries": ...},
        "user_turn_limit":       {"max_words": ..., "max_duration": ...},
    },
)

turn_detection: set it to "stt". Omitting it defaults to LiveKit’s audio end-of-turn model, which needs a VAD and is calibrated for only 14 languages; "vad" needs a LiveKit VAD; "stt" trusts Sarvam’s own vad.speech_end and is the right choice here.

Endpointing defaults, and why to change them:

min_delay is a hard sleep after the turn is detected and before it’s committed: sleep(min_delay - time_since_last_speech). It’s a grace period, not pure waste; a new vad.speech_start during the hold cancels the pending commit. Because Sarvam’s VAD already confirmed silence server-side, the default 0.5 s is redundant margin stacked on margin. Dropping it to 0.25 s typically removes about 250 ms of dead air per turn with no increase in cutoffs, provided vad_min_silence_ms isn’t itself set aggressively low.

In "fixed" mode, max_delay only applies when the end-of-turn model returns a low probability. With turn_detection="stt" or vad=None, no prediction is made, so the hold is exactly min_delay. In "dynamic" mode, max_delay also caps the learned delay (below).

Preemptive generation should stay enabled. Because saaras:v3-realtime emits end-of-speech before the final transcript, the framework has already marked the turn committed by the time the final arrives, which is the condition it needs to start speculative LLM work: a genuine improvement over the legacy plugin, where the reverse event order kept speculation inert in "stt" mode.

mode: "dynamic" learns min_delay per session from the user’s own pausing rhythm, clamped between your min_delay (the floor) and max_delay (the ceiling). Worth trying for long consultative calls with a stable speaker; skip for short transactional calls, which never accumulate enough samples to learn anything. With turn_detection="stt" and vad=None there are fewer observations available, so expect it to stay near the floor; "fixed" is the honest default.

Other defaults worth revisiting:

To change endpointing mid-call, call session.update_options(endpointing_opts=...). For example, use a longer hold while a caller reads out a card number, then set it back:

session.update_options(endpointing_opts={"min_delay": 0.8})   # caller is reading digits
# ...
session.update_options(endpointing_opts={"min_delay": 0.25})  # back to normal

7. Barge-in and interruptions

Interruption is triggered from three places in the runtime; only two are reachable with vad=None: an interim-transcript hook, since Sarvam’s transcript.partial lands there, and a final-transcript hook as backstop. VAD-driven interruption never fires. So barge-in latency equals the time for Sarvam to return the first partial of the interrupting speech, one 500 ms audio window plus model and network time on stream_type="fast" (1000 ms on "balanced"), not a completed utterance or local VAD energy detection. It’s also more robust than local-VAD barge-in: a cough, a door slam, or TV audio can’t interrupt the agent, because none of those produce a transcript.

Since min_duration is inert, min_words is your noise filter:

"interruption": {"enabled": True, "mode": "vad", "min_words": 1}

0 (default) lets any transcript interrupt; occasionally over-eager on ASR artifacts. 1 is a good default, filtering empty or whitespace-only artifacts. 2 suits noisy channels, but while the agent is speaking, a user turn shorter than min_words doesn’t end the turn: it’s held and joined to the user’s next utterance, so a single-word acknowledgement mid-agent-speech is merged into whatever the user says next. Don’t exceed 2, and don’t go above 1 at all if single-word confirmations matter in your flow. Word counting is whitespace-delimited, so Devanagari, Tamil, Telugu, Bengali, and similar scripts are counted correctly with no adjustment needed.

False-interruption recovery. When something interrupts the agent and no real user turn materializes, the agent’s speech is paused, not cancelled; after false_interruption_timeout seconds of silence it resumes and fires an agent_false_interruption event. Lower the timeout to 1.2 to 1.5 s (the 2.0 s default is a long, awkward silence on a call). Instrument the event; a high false-interruption rate means the VAD is too eager, and the fix is at the VAD (raise vad_sot_threshold or vad_min_speech_ms), not here.

@session.on("agent_false_interruption")
def _on_false_interruption(ev):
    logger.warning("false interruption", extra={"resumed": ev.resumed})

For confirmations, legal disclaimers, or OTP readback, disable interruptions per turn rather than for the whole session (which makes the agent feel like an IVR):

handle = session.say("Your OTP is 4 7 2 9.", allow_interruptions=False)
await handle

While uninterruptible speech plays, the STT receives silence (discard_audio_if_uninterruptible defaults to True), so anything the caller says during the readback isn’t transcribed. Keep these turns short.

8. TTS configuration

sarvam.TTS(
    model="bulbul:v4-flash",                 # default: "bulbul:v3"
    speaker="ritu_hi_customer",        # v4 Flash persona ID; always set it
    target_language_code="hi-IN",      # default: "en-IN"
    speech_sample_rate=24000,          # default: 22050
    pace=1.0,
    output_audio_codec="linear16",     # default: "mp3"
    output_audio_bitrate="128k",
    min_buffer_size=30,                # default: 50
    max_chunk_length=150,              # default: 150
    dict_id=None,                      # v3 and v4 Flash
    send_completion_event=True,        # leave True
)

Use bulbul:v4-flash (latest) or bulbul:v3. dict_id works on both (not on bulbul:v2). temperature applies to bulbul:v3 only: v4 Flash ignores it, so leave it out. v4 Flash takes persona speaker IDs such as ritu_hi_customer, not v3 short names such as priya; see Voices for the full list.

Validation ranges. The plugin checks these client-side. Out-of-range values raise ValueError, except pitch, which is clamped to the nearest bound with a warning. bulbul:v4-flash has tighter limits than v2 and v3:

bulbul:v3 doesn’t use pitch or loudness, so the plugin doesn’t send them for v3. For every model, min_buffer_size is 30 to 200, max_chunk_length is 50 to 500, output_audio_bitrate is one of 32k/64k/96k/128k/192k, and output_audio_codec is one of mp3/opus/flac/aac/wav/linear16/mulaw/alaw.

Most of these checks run at construction, so a bad value fails the worker at startup. Speaker and model compatibility is checked there too. The v4 Flash sample-rate check is the exception: it runs when the websocket opens, so a bad speech_sample_rate fails on the first utterance instead.

Two more v4 Flash limits:

• No Malayalam or Odia voices yet. v4 Flash accepts ml-IN and od-IN, but has no personas for them. Keep Malayalam and Odia traffic on bulbul:v3.
• Text normalisation is always on. v3 and v4 Flash always normalise numbers, dates and mixed-script text, so enable_preprocessing=False has no effect on them.

Latency levers, in order of impact:

prewarm() in practice:

from livekit.agents import AgentServer, JobProcess
def setup(proc: JobProcess) -> None:
    proc.userdata["tts"] = sarvam.TTS(model="bulbul:v4-flash", speaker="ritu_hi_customer",
                                      target_language_code="hi-IN")
    proc.userdata["tts"].prewarm()
server = AgentServer(setup_fnc=setup)

Verify it worked: TTSMetrics.connection_reused should read True and acquire_time near zero from the second turn on. If it’s False every turn, you’re likely constructing a new sarvam.TTS() per session instead of reusing one.

Changing model or send_completion_event through tts.update_options(...) also retires the pooled sockets, because both are fixed when a socket opens. The next utterance pays the connection cost again, so set them at construction.

Leave send_completion_event=True; setting it False means the receive loop never sees a completion signal and blocks until it times out as APITimeoutError.

Voice quality. pace at 1.0 often reads slightly fast for support flows; 0.9 to 0.95 measurably helps comprehension, and intelligibility degrades above 1.15. On bulbul:v3, temperature at 0.3 to 0.5 gives consistent delivery for regulated content (disclaimers, amounts, OTPs); 0.8 and up is more expressive, at the cost of occasional mispronunciation. v4 Flash ignores temperature; pick a persona whose style fits the content instead. dict_id (v3 and v4 Flash) is worth setting for brand names and place names that generic TTS mangles. AgentSession filters markdown and emoji from TTS input by default; keep both on, and instruct your LLM to produce properly punctuated sentences, since the plugin tokenizes on sentence boundaries.

9. Measuring latency

Track end-to-end latency first, then decompose. ChatMessage.metrics is the primary surface. User and assistant messages carry different fields:

@session.on("conversation_item_added")
def _on_item(ev):
    m = getattr(ev.item, "metrics", None)
    role = getattr(ev.item, "role", None)
    if not m:
        return
    if role == "assistant":
        logger.info("turn_latency", extra={"e2e_latency": m.get("e2e_latency"),
                                            "llm_ttft": m.get("llm_node_ttft"),
                                            "tts_ttfb": m.get("tts_node_ttfb")})
    elif role == "user":
        logger.info("user_turn", extra={"transcription_delay": m.get("transcription_delay"),
                                         "end_of_turn_delay": m.get("end_of_turn_delay")})

Report the p50 and p95, not just the average. (p50 is the median: half your turns are faster than this. p95 is the value 95% of turns beat, so it captures the slow tail that an average hides.) A p50 of 800 ms with a p95 of 4 s feels broken to users even though the average looks fine.

For per-request metrics, listen on the plugin instances, not the session: stt.on("metrics_collected", ...) and tts.on("metrics_collected", ...) deliver STTMetrics and TTSMetrics, including request IDs and connection_reused. The session-level metrics_collected event has been deprecated since livekit-agents 1.5.0 and logs a warning; usage and cost tracking is on session_usage_updated. TTSMetrics.cancelled=True counts barge-ins that killed a synthesis in flight, a good proxy for interruption health. For realtime STT, billed audio duration comes from the server’s session.end when available, falling back to a local estimate if the session dies first; reconcile against Sarvam’s dashboard rather than assuming either source is complete.

The SDK also emits OpenTelemetry spans, among them user_turn, eou_wait and eou_detection (the end-of-turn wait and its predictions), llm_node, llm_request_run, tts_node and tts_request_run. Speech-to-text has no span of its own; its timing is on user_turn as lk.transcription_delay. Since livekit-agents 1.8.0, spans follow the OpenTelemetry GenAI semantic conventions (gen_ai.* attributes), so dashboards built on the older span format need updating. Wire up an OTel exporter in production; it’s the only view that shows where a slow turn was actually slow, and it’s where Sarvam’s TTS request IDs land (Section 10).

Suggested SLOs for a cascading Sarvam pipeline:

10. Capturing request IDs

Sarvam’s server-side request_id is what their support team needs to investigate a bad transcript or mispronunciation. Coverage differs by surface:

STTMetrics.request_id is the real Sarvam ID; log it and you’re done for STT. On the streaming TTS path, TTSMetrics.request_id is a locally generated ID assigned before Sarvam replies, so Sarvam’s actual ID only reaches the OTel span attribute lk.provider_request_ids; enable tracing if you need it there. ChatMessage.metrics["provider_request_ids"] is populated only for realtime (speech-to-speech) models, so it stays empty in any cascading STT-LLM-TTS pipeline. LLM calls also carry the provider’s response ID in lk.provider_request_ids, on their llm_request_run span.

A minimal production log line should carry session_id, turn_id, speech_id, the Sarvam STT request ID, the Sarvam TTS request IDs, and e2e_latency. With those six fields you can escalate any complaint to Sarvam support with evidence. speech_id ties one utterance’s LLM and TTS calls together; segment_id identifies an individual synthesis segment within an utterance. Keep the IDs logged for as long as your support SLA requires, and don’t store transcript text alongside them unless that store meets your PII policy: Indian consumer call transcripts routinely contain names, addresses, and financial details.

11. The legacy class and migration

The legacy sarvam.STT (saaras:v4 by default, or saaras:v3) is the only path that supports batch recognize() (STTRealtime raises NotImplementedError for it). If you’re maintaining an existing deployment on it, the differences that matter:

New builds should use STTRealtime. Migrating is mostly a constructor swap plus replacing the VAD parameters per Section 5, but re-tune min_delay afterward, since the event ordering and chunking both change your latency profile.

12. Reliability and failure modes

Retries. Neither Sarvam STT class retries. The plugin forces max_retry=0 because Sarvam bills per connection, and a realtime stream never reconnects after a socket failure. The connection timeout still applies, and the framework default of 10.0 s is too long for voice; a caller will have hung up. Tighten the timeouts:

from livekit.agents import APIConnectOptions
from livekit.agents.voice.agent_session import SessionConnectOptions
conn_options = SessionConnectOptions(
    stt_conn_options=APIConnectOptions(timeout=5.0),  # Sarvam STT ignores max_retry
    tts_conn_options=APIConnectOptions(max_retry=1, timeout=5.0),
    llm_conn_options=APIConnectOptions(max_retry=1, timeout=8.0),
    max_unrecoverable_errors=3,
)

For calls you can’t afford to drop, wrap TTS in a FallbackAdapter; a failed TTS is a silent agent, which is worse than a wrong one. Sarvam STT never reconnects on its own, but AgentSession recreates the STT stream 0.5 s after an unrecoverable error. That opens a new connection, which Sarvam bills separately. Each failure counts toward max_unrecoverable_errors and any transcript resets the count, so the session closes only after more than that many failures in a row (3 by default). Use stt.FallbackAdapter when you want to switch to another STT provider rather than reconnect to Sarvam. Listen to the session’s error event to log and alert on failures.

STT websocket close codes are the most useful operational signal available. This table covers the realtime STT socket; the TTS socket is covered below it:

“Transient” means the failure may clear on its own. The plugin tags 1013 as retryable, but it never retries (max_retry=0); recovery comes from AgentSession recreating the stream, as described above.

Non-fatal error events (is_fatal: false) log a warning and let the session continue; watch their rate as an early degradation signal, since it’s otherwise invisible. Fatal errors raise APIStatusError. ValueError on any parameter fires at construction time, so instantiate sarvam.STTRealtime and sarvam.TTS in setup_fnc, not the entrypoint, so config errors fail the worker at startup rather than the caller’s session.

Sarvam closes idle TTS websockets after 60 s. The plugin defends on two layers, a transport-level ping every 20 s and an application-level ping every 30 s, so stale pooled connections are detected and replaced transparently. Size your key’s rate limits against peak concurrent calls times two (one STT socket plus one TTS socket per call), not average volume, and alert on close code 1003.

TTS websocket closes don’t follow the STT table. Any close other than 1000 or 1001 raises APIStatusError, with the close code as its status_code, so log that code. The plugin’s connection pool also retires each TTS socket after one hour (3600 s) and opens a fresh one on the next utterance.

13. Tuning playbook

Change one thing at a time and measure between steps.

1. Instrument first. Wire up Section 9 and Section 10 before touching any parameter, and collect a baseline over at least 30 real calls.
2. Get the architecture right. stream_type="fast", vad=None, turn_detection="stt", model="bulbul:v4-flash". Confirm there’s no TurnDetector requires a VAD model warning at startup; if you see it, vad=None took effect but turn_detection="stt" didn’t, so check that it sits inside turn_handling.
3. Clean the input, then tune Sarvam’s VAD. On noisy channels, turn on Krisp noise cancellation first (Section 5). Then raise vad_min_silence_ms if users get cut off mid-sentence; lower it if there’s dead air after they finish. Raise vad_sot_threshold, then vad_min_speech_ms, if noise or crosstalk is triggering turns.
4. Trim endpointing.min_delay. Step down from 0.5 in 0.1 s increments (0.4, 0.3, 0.25, 0.2), stopping one step before cutoffs reappear. Most Sarvam pipelines land at 0.2 to 0.3 s.
5. Fix barge-in. Raise interruption.min_words (0, then 1, then 2) until noise-triggered interruptions stop, watching that real single-word replies survive. Lower false_interruption_timeout to 1.2 to 1.5 s. For inbound telephony, set aec_warmup_duration=None (outbound SIP already defaults to it).
6. Cut TTS time-to-first-byte. min_buffer_size=30, output_audio_codec="linear16" (or mulaw for 8 kHz), and prewarm().
7. Cut LLM time-to-first-token. Shrink the system prompt, scope the tool list to the current conversation phase, cap conversation history, and keep max_tool_steps low. Then check reasoning and the model:
   • Reasoning. sarvam-105b has thinking mode on by default, so it generates reasoning tokens before the first word of the reply. Sarvam’s /v1 chat API turns thinking off with reasoning_effort=None. The LiveKit plugin, though, sends sarvam-105b to /v2, whose spec doesn’t document reasoning_effort for sarvam-105b. So sarvam.LLM(model="sarvam-105b", reasoning_effort=None) may not turn thinking off; compare llm_node_ttft before and after rather than assuming it did.
   • The conversational model. sarvam-105b-conversations is a variant post-trained for voice agents and real-time dialogue, at the same price as sarvam-105b but with a 32K context window instead of 128K. Long sessions need history capped to fit. The plugin calls it on /v1 and doesn’t send reasoning_effort for it.
   • Model names. For voice, use sarvam-105b-conversations, sarvam-105b or gemma4. From plugin 1.7.1, sarvam-30b, sarvam-30b-16k, sarvam-105b-32k and sarvam-m raise ValueError, so update older configs that use them.
8. Polish the voice. pace 0.9 to 0.95 for support flows, a calmer persona style (or a lower temperature on bulbul:v3) for regulated content, and a dict_id dictionary for brand and place names.
9. Re-baseline. Re-measure over 30-plus calls and set alerts on p95 e2e_latency, false-interruption rate, close code 1003, and non-fatal STT error rate.

14. Copy-paste configs

WebRTC or app: lowest latency, clean audio

import logging
from livekit.agents import (
    Agent, AgentServer, AgentSession, JobContext, JobProcess,
    TurnHandlingOptions, cli, metrics, room_io,
)
from livekit.plugins import krisp, sarvam
logger = logging.getLogger("sarvam-agent")
def setup(proc: JobProcess) -> None:
    # Construct here so config errors fail the worker, not the caller's session.
    proc.userdata["stt"] = sarvam.STTRealtime(
        language="hi-IN",
        stream_type="fast",
        mode="codemix",
        endpointing="vad",
        sample_rate=16000,
        vad_min_silence_ms=300,
        vad_min_speech_ms=80,
        vad_sot_threshold=0.5,
        prompt="Customer support for a fintech app. Terms: UPI, KYC, NEFT, IMPS, mandate.",
    )
    proc.userdata["tts"] = sarvam.TTS(
        model="bulbul:v4-flash",
        speaker="ritu_hi_customer",
        target_language_code="hi-IN",
        speech_sample_rate=24000,
        output_audio_codec="linear16",
        min_buffer_size=30,
        max_chunk_length=150,
        pace=0.95,
    )
    proc.userdata["tts"].prewarm()
    # Per-request metrics (with Sarvam request IDs) come from the plugins. They're shared by every
    # session in this process, so attach the listeners once, here, rather than per session.
    proc.userdata["stt"].on("metrics_collected", metrics.log_metrics)
    proc.userdata["tts"].on("metrics_collected", metrics.log_metrics)
server = AgentServer(setup_fnc=setup)
@server.rtc_session()
async def entrypoint(ctx: JobContext) -> None:
    ctx.log_context_fields = {"room": ctx.room.name}
    session = AgentSession(
        stt=ctx.proc.userdata["stt"],
        llm=sarvam.LLM(model="sarvam-105b"),
        tts=ctx.proc.userdata["tts"],
        vad=None,
        turn_handling=TurnHandlingOptions(
            turn_detection="stt",
            endpointing={"mode": "fixed", "min_delay": 0.22, "max_delay": 2.0},
            interruption={
                "enabled": True,
                "mode": "vad",
                "min_words": 1,
                "false_interruption_timeout": 1.3,
                "resume_false_interruption": True,
            },
            preemptive_generation={"enabled": True},
        ),
        aec_warmup_duration=3.0,     # keep for open mic + speaker
        user_away_timeout=None,
        max_tool_steps=2,
    )
    @session.on("agent_false_interruption")
    def _on_false(ev):
        logger.warning("false_interruption", extra={"resumed": ev.resumed})
    await session.start(
        agent=Agent(instructions="You are a concise Hindi-speaking support agent. "
                                 "Reply in one or two short sentences. Use proper punctuation."),
        room=ctx.room,
        room_options=room_io.RoomOptions(
            audio_input=room_io.AudioInputOptions(noise_cancellation=krisp.voice_isolation()),
        ),
    )
if __name__ == "__main__":
    cli.run_app(server)

Telephony or SIP: noisy, echo handled by the carrier

This uses the same imports, logger and server as the WebRTC config. In production, build stt and tts in setup() as shown there. The agent_name turns on explicit dispatch, so calls reach this agent only through a dispatch rule that names it.

stt = sarvam.STTRealtime(
    language="hi-IN",
    stream_type="fast",
    mode="transcribe",
    endpointing="vad",
    sample_rate=16000,             # upsample; usually cheaper than degrading the recognizer
    vad_min_silence_ms=500,
    vad_min_speech_ms=200,         # ignore line clicks, ringback, DTMF
    vad_sot_threshold=0.7,         # reject TV / crosstalk / ambient speech
)
tts = sarvam.TTS(
    model="bulbul:v4-flash",
    speaker="simran_enhi_customer",
    target_language_code="hi-IN",
    speech_sample_rate=8000,       # match the carrier
    output_audio_codec="mulaw",    # decoded in-process, no resample
    output_audio_bitrate="64k",
    min_buffer_size=30,
    pace=0.92,                     # slower reads better on narrowband
)
@server.rtc_session(agent_name="sarvam-telephony")  # your SIP dispatch rule routes calls to this name
async def telephony_entrypoint(ctx: JobContext) -> None:
    session = AgentSession(
        stt=stt, llm=sarvam.LLM(model="sarvam-105b"), tts=tts,
        vad=None,
        turn_handling={
            "turn_detection": "stt",
            "endpointing": {"mode": "fixed", "min_delay": 0.3, "max_delay": 2.5},
            "interruption": {
                "enabled": True,
                "mode": "vad",
                "min_words": 2,        # noisy line: stricter filter
                "false_interruption_timeout": 1.5,
                "resume_false_interruption": True,
            },
            "preemptive_generation": {"enabled": True},
            "user_turn_limit": {"max_duration": 45.0},   # catch a stuck call
        },
        aec_warmup_duration=None,      # important for inbound telephony; outbound SIP defaults to None
        user_away_timeout=20.0,
    )
    @session.on("user_turn_exceeded")
    def _on_turn_exceeded(ev):
        logger.warning("user_turn_exceeded", extra={"duration": ev.duration,
                                                    "words": ev.accumulated_word_count})
    await session.start(
        agent=Agent(instructions="You are a concise Hindi-speaking phone agent. "
                                 "Reply in one or two short sentences. Use proper punctuation."),
        room=ctx.room,
        room_options=room_io.RoomOptions(
            audio_input=room_io.AudioInputOptions(
                noise_cancellation=krisp.voice_isolation_telephony(),
            ),
        ),
    )

For compliance and audit flows, use mode="verbatim", return_timestamps=True, a higher vad_min_silence_ms (around 900), a lower temperature (around 0.3) if you use bulbul:v3, and disable preemptive_generation to avoid speculative spend; disable interruptions per-turn for disclosures. For push-to-talk or IVR, set endpointing="manual" on the STT and turn_detection="manual" on the session, then call session.commit_user_turn() yourself on a DTMF key or button release.

Quick reference