LiveKit in Production

LiveKit in Production

A tuning and operations guide for teams shipping cascading voice agents (Sarvam STT, then an LLM, then Sarvam TTS) on the LiveKit Agents SDK.

Everything here is derived from the current livekit-plugins-sarvam plugin source and the livekit-agents runtime. New to LiveKit and Sarvam? Start with Build Your First Voice Agent using LiveKit first; come back here before you ship.

Which STT class to use

saaras:v3-realtime gives you true partial transcripts (barge-in no longer waits for a final), a simpler VAD (three millisecond-valued parameters instead of ten frame counters), live reconfiguration without a reconnect, and a structured event and close-code protocol. This guide documents STTStreaming throughout; the legacy class is covered only in Section 11.

Contents

1. Reference config
2. Where latency hides
3. Skip LiveKit’s Silero VAD
4. STT configuration
5. Tuning Sarvam’s VAD
6. Turn handling and endpointing
7. Barge-in and interruptions
8. TTS configuration
9. Measuring latency
10. Capturing request IDs
11. The legacy class and migration
12. Reliability and failure modes
13. Tuning playbook
14. Copy-paste configs

1. Reference config

1
from livekit.agents import AgentSession
2
from livekit.plugins import sarvam
3
4
session = AgentSession(
5
    # STT: Sarvam's own server-side VAD drives turn-taking
6
    stt=sarvam.STTStreaming(
7
        language="hi-IN",
8
        stream_type="fast",         # was "balanced": 500 ms chunks instead of 1000 ms
9
        mode="transcribe",
10
        endpointing="vad",
11
        sample_rate=16000,
12
        vad_min_silence_ms=400,     # Sarvam's end-of-turn silence window
13
    ),
14
15
    llm=sarvam.LLM(model="sarvam-105b"),   # or any other LLM plugin
16
17
    tts=sarvam.TTS(
18
        model="bulbul:v3",
19
        speaker="priya",
20
        target_language_code="hi-IN",
21
        min_buffer_size=30,         # was 50, lowers TTFB
22
        max_chunk_length=150,
23
        output_audio_codec="linear16",   # skip the mp3 decode hop
24
        speech_sample_rate=24000,
25
    ),
26
27
    # Critical: no LiveKit VAD. Sarvam STT already has one.
28
    vad=None,
29
30
    turn_handling={
31
        "turn_detection": "stt",             # trust Sarvam's vad.speech_end
32
        "endpointing": {
33
            "mode": "fixed",
34
            "min_delay": 0.25,               # was 0.5 (or 0.3 streaming default)
35
            "max_delay": 2.0,
36
        },
37
        "interruption": {
38
            "enabled": True,
39
            "mode": "vad",                   # the adaptive detector can't work here, see Section 7
40
            "min_words": 1,                  # the real noise filter when vad=None
41
            "false_interruption_timeout": 1.5,
42
            "resume_false_interruption": True,
43
        },
44
        "preemptive_generation": {"enabled": True},
45
    },
46
47
    # other defaults worth revisiting
48
    aec_warmup_duration=None,        # None for telephony/SIP, keep 3.0 for WebRTC + mic
49
    user_away_timeout=None,          # or a value that matches your product
50
    min_consecutive_speech_delay=0.0,
51
)

Every override above is explained below. If you change nothing else, change vad, turn_detection, stream_type, and endpointing.min_delay.

2. Where latency hides

The user-perceived latency is the sum of the whole chain, not any one model:

Three things worth knowing:

• The first hop is a hard floor most people never look at. The plugin buffers audio into fixed-size chunks before sending: 1000 ms on the default stream_type="balanced", 500 ms on "fast". Sarvam’s VAD can’t detect end-of-speech until the chunk containing that silence arrives, so balanced adds up to a full second in front of every turn. Set stream_type="fast".
• vad_min_silence_ms and min_delay stack, they don’t overlap. 500 ms of Sarvam silence plus a 0.5 s min_delay is a full second of dead air. Tune both together.
• TTS time-to-first-byte is dominated by connection state, not model speed. A cold Sarvam TTS websocket costs a full TLS handshake on the critical path. Call prewarm() (Section 8).

Measure this instrumented, don’t guess: Section 9.

3. Skip LiveKit’s Silero VAD

Pass vad=None and let Sarvam’s server-side VAD drive turn-taking.

STTStreaming runs with endpointing="vad" by default, so the server emits vad.speech_start and vad.speech_end on every session, and the plugin maps those onto the framework’s start/end-of-speech events. That VAD already runs server-side on the exact audio the recognizer sees, is tunable through three explicit parameters trained on Indic speech (Section 5), and adds no local inference cost. Running Silero alongside it means two VADs disagreeing about turn boundaries: double-triggered interruptions, warnings in your logs, and endpointing metrics anchored on the wrong timestamp.

Passing vad=None does disable four framework features gated on a VAD instance. Be deliberate about the trade:

The trade is a good one on saaras:v3-realtime specifically: because it emits real partial transcripts and signals end-of-speech before the final transcript, barge-in fires on partials (Section 7) and preemptive generation actually engages (Section 6), neither of which worked without a VAD on the legacy plugin.

The semantic turn detector is a weak fit for Sarvam regardless: outside LiveKit Cloud it resolves to a local model whose per-language thresholds cover 14 languages (Hindi included), falling back to an English-calibrated threshold for everything else. Sarvam’s own VAD plus a tuned min_delay is the more predictable choice for Indic languages.

If you do keep a VAD (for example, on LiveKit Cloud, to get the EOT model for Hindi), its min_silence_duration must be at least 0.25 s, or AgentSession raises a ValueError.

4. STT configuration

1
sarvam.STTStreaming(
2
    language="hi-IN",            # default: "en-IN"
3
    stream_type="fast",          # default: "balanced"
4
    mode="transcribe",           # default: "transcribe"
5
    endpointing="vad",           # default: "vad"
6
    encoding="linear16",         # only value accepted
7
    sample_rate=16000,           # default: 16000 (8000 or 16000 only)
8
    prompt=None,
9
    return_timestamps=False,
10
    api_key=None,                # falls back to SARVAM_API_KEY
11
    # server-side VAD tuning, see Section 5
12
    vad_sot_threshold=None,
13
    vad_min_speech_ms=None,
14
    vad_min_silence_ms=None,
15
)

Endpoint: wss://api.sarvam.ai/speech-to-text-realtime/ws. Model is fixed at saaras:v3-realtime; any invalid parameter raises ValueError at construction.

stream_type is the single biggest latency knob. fast buffers audio in 500 ms chunks before sending; balanced (the default) and simulated use 1000 ms. Set "fast" for any interactive agent; the default is the single most impactful bad setting for voice.

endpointing: vad (default) has the server emit speech-start and speech-end events, and pairs with turn_detection="stt". manual has the plugin send events itself on the first audio frame and on flush; pair with the framework’s turn_detection="manual" for push-to-talk or IVR flows. In manual mode the VAD tuning parameters aren’t sent at all.

mode:

codemix is worth A/B-testing against transcribe for consumer-facing Indian deployments.

prompt carries domain vocabulary: product names, city names, SKUs, agent names. It’s the cheapest accuracy win available and frequently left empty. Unlike the legacy class, the realtime plugin sends it as a URL query parameter on the websocket handshake, so keep it short (a compact term list) or a long prompt risks a rejected handshake; use a TTS pronunciation dictionary (dict_id) for output-side fixes instead.

1
stt = sarvam.STTStreaming(
2
    language="hi-IN",
3
    stream_type="fast",
4
    prompt="Insurance policy renewal. Terms: premium, NCB, IDV, Bajaj Allianz, endorsement.",
5
)

Live reconfiguration. STTStreaming.update_options(...) sends a config update over the live websocket, no reconnect, no dropped audio, and the plugin defers any endpointing switch until the current utterance completes. Updatable in place: language, stream_type, mode, prompt, endpointing, and all three VAD parameters. sample_rate and return_timestamps are connection-only: changing them on a live stream is logged and ignored, so set them at construction.

1
# e.g. the caller switched to Tamil: no reconnect, no dropped audio
2
session.stt.update_options(language="ta-IN")

sample_rate and encoding: 8000 or 16000 only, and linear16 only. Use 16000 for WebRTC; 8000 matches PSTN, but upsampling to 16 kHz is usually cheaper than degrading the recognizer’s input.

Event mapping:

END_OF_SPEECH precedes FINAL_TRANSCRIPT, the opposite of the legacy plugin, and it’s why preemptive generation works here (Section 6): the framework marks the turn committed at end-of-speech, then the arriving final triggers speculative LLM work and starts the endpointing hold. If the final never arrives, the plugin emits END_OF_SPEECH anyway after a 2.0 s fallback.

5. Tuning Sarvam’s VAD

Three parameters, sent only when endpointing="vad". All default to None (Sarvam’s server-side default applies); set them explicitly whenever you need reproducible behavior across environments.

Lowering vad_min_silence_ms ends turns faster (snappier, more mid-sentence cutoffs); raising it holds through natural pauses (no cutoffs, more dead air). It stacks with endpointing.min_delay (Section 6): budget them together, since tuning one down while leaving the other conservative won’t move the needle.

Raise vad_sot_threshold first on noisy telephony, before touching anything else; it’s the cleanest defense against a VAD that keeps triggering on ambient audio. Raise vad_min_speech_ms to ignore clicks, coughs, keyboard noise, and DTMF beeps; lower it to catch very short words.

Recommended starting points by channel:

Tune the VAD before you tune min_delay. They compensate for each other, so changing both at once means you can’t attribute a regression.

6. Turn handling and endpointing

min_endpointing_delay, allow_interruptions, turn_detection, preemptive_generation, and similar flat AgentSession kwargs are deprecated in favor of a nested turn_handling dict:

1
AgentSession(
2
    turn_handling={
3
        "turn_detection": "stt",
4
        "endpointing":           {"mode": ..., "min_delay": ..., "max_delay": ..., "alpha": ...},
5
        "interruption":          {"enabled": ..., "mode": ..., "min_duration": ..., "min_words": ...,
6
                                  "false_interruption_timeout": ..., "resume_false_interruption": ...},
7
        "preemptive_generation": {"enabled": ..., "preemptive_tts": ..., "max_speech_duration": ..., "max_retries": ...},
8
        "user_turn_limit":       {"max_words": ..., "max_duration": ...},
9
    },
10
)

turn_detection: set it to "stt". Omitting it defaults to a semantic EOT model that needs a VAD and is English-calibrated outside 14 languages; "vad" needs a LiveKit VAD; "stt" trusts Sarvam’s own vad.speech_end and is the right choice here.

Endpointing defaults, and why to change them:

min_delay is a hard sleep after the turn is detected and before it’s committed: sleep(min_delay - time_since_last_speech). It’s a grace period, not pure waste; a new vad.speech_start during the hold cancels the pending commit. Because Sarvam’s VAD already confirmed silence server-side, the default 0.5 s is redundant margin stacked on margin. Dropping it to 0.25 s typically removes about 250 ms of dead air per turn with no increase in cutoffs, provided vad_min_silence_ms isn’t itself set aggressively low.

max_delay only applies when a semantic EOT model returns a low probability. With turn_detection="stt" or vad=None, no prediction is ever made, so the hold is always exactly min_delay. Keep max_delay as a documented ceiling, but don’t tune it expecting an effect.

Preemptive generation should stay enabled. Because saaras:v3-realtime emits end-of-speech before the final transcript, the framework has already marked the turn committed by the time the final arrives, which is the condition it needs to start speculative LLM work: a genuine improvement over the legacy plugin, where the reverse event order kept speculation inert in "stt" mode.

mode: "dynamic" learns min_delay per session from the user’s own pausing rhythm, clamped so your configured value is a floor. Worth trying for long consultative calls with a stable speaker; skip for short transactional calls, which never accumulate enough samples to learn anything. With turn_detection="stt" and vad=None there are fewer observations available, so expect it to stay near the floor; "fixed" is the honest default.

Other defaults worth revisiting:

7. Barge-in and interruptions

Interruption is triggered from three places in the runtime; only two are reachable with vad=None: an interim-transcript hook, since Sarvam’s transcript.partial lands there, and a final-transcript hook as backstop. VAD-driven interruption never fires. So barge-in latency equals the time for Sarvam to return the first partial of the interrupting speech, a single 500 ms chunk plus model and network time on stream_type="fast", not a completed utterance or local VAD energy detection. It’s also more robust than local-VAD barge-in: a cough, a door slam, or TV audio can’t interrupt the agent, because none of those produce a transcript.

Since min_duration is inert, min_words is your noise filter:

1
"interruption": {"enabled": True, "mode": "vad", "min_words": 1}

0 (default) lets any transcript interrupt; occasionally over-eager on ASR artifacts. 1 is a good default, filtering empty or whitespace-only artifacts. 2 suits noisy channels, but while the agent is speaking, a user turn shorter than min_words is dropped entirely rather than merely failing to interrupt, so single-word acknowledgements get discarded mid-agent-speech. Don’t exceed 2, and don’t go above 1 at all if single-word confirmations matter in your flow. Word counting is whitespace-delimited, so Devanagari, Tamil, Telugu, Bengali, and similar scripts are counted correctly with no adjustment needed.

False-interruption recovery. When something interrupts the agent and no real user turn materializes, the agent’s speech is paused, not cancelled; after false_interruption_timeout seconds of silence it resumes and fires an agent_false_interruption event. Lower the timeout to 1.2 to 1.5 s (the 2.0 s default is a long, awkward silence on a call). Instrument the event; a high false-interruption rate means the VAD is too eager, and the fix is at the VAD (raise vad_sot_threshold or vad_min_speech_ms), not here.

1
@session.on("agent_false_interruption")
2
def _on_false_interruption(ev):
3
    logger.warning("false interruption", extra={"resumed": ev.resumed})

For confirmations, legal disclaimers, or OTP readback, disable interruptions per turn rather than for the whole session (which makes the agent feel like an IVR):

1
handle = session.say("Your OTP is 4 7 2 9.", allow_interruptions=False)
2
await handle

8. TTS configuration

1
sarvam.TTS(
2
    model="bulbul:v3",                 # default: "bulbul:v3"
3
    speaker="priya",                   # default: "shubh"
4
    target_language_code="hi-IN",      # default: "en-IN"
5
    speech_sample_rate=24000,          # default: 22050
6
    pace=1.0,
7
    temperature=0.6,                   # v3 only
8
    output_audio_codec="linear16",     # default: "mp3"
9
    output_audio_bitrate="128k",
10
    min_buffer_size=30,                # default: 50
11
    max_chunk_length=150,              # default: 150
12
    dict_id=None,                      # v3 only
13
    send_completion_event=True,        # leave True
14
)

Use bulbul:v3; temperature and dict_id are only supported on this model.

Validation ranges (enforced client-side, ValueError unless noted): pace 0.3 to 3.0, loudness 0.5 to 2.0, temperature 0.01 to 2.0, min_buffer_size 30 to 200, max_chunk_length 50 to 500, output_audio_bitrate one of 32k/64k/96k/128k/192k, speech_sample_rate one of 8000/16000/22050/24000/32000/44100/48000, output_audio_codec one of mp3/opus/flac/aac/wav/linear16/mulaw/alaw. pitch (-0.75 to 0.75) is the one exception: out-of-range values are clamped with a warning instead of raising. Speaker and model compatibility is validated at construction, so an unsupported voice fails at startup rather than mid-call.

Latency levers, in order of impact:

prewarm() in practice:

1
from livekit.agents import AgentServer, JobProcess
2
3
def setup(proc: JobProcess) -> None:
4
    proc.userdata["tts"] = sarvam.TTS(model="bulbul:v3", speaker="priya",
5
                                      target_language_code="hi-IN")
6
    proc.userdata["tts"].prewarm()
7
8
server = AgentServer(setup_fnc=setup)

Verify it worked: TTSMetrics.connection_reused should read True and acquire_time near zero from the second turn on. If it’s False every turn, you’re likely constructing a new sarvam.TTS() per session instead of reusing one.

Leave send_completion_event=True; setting it False means the receive loop never sees a completion signal and blocks until it times out as APITimeoutError.

Voice quality. pace at 1.0 often reads slightly fast for support flows; 0.9 to 0.95 measurably helps comprehension, and intelligibility degrades above 1.15. temperature at 0.3 to 0.5 gives consistent delivery for regulated content (disclaimers, amounts, OTPs); 0.8 and up is more expressive, at the cost of occasional mispronunciation. dict_id (v3 only) is worth setting for brand names and place names that generic TTS mangles. AgentSession filters markdown and emoji from TTS input by default; keep both on, and instruct your LLM to produce properly punctuated sentences, since the plugin tokenizes on sentence boundaries.

9. Measuring latency

Track end-to-end latency first, then decompose. ChatMessage.metrics is the primary surface; every committed turn carries these fields:

1
@session.on("conversation_item_added")
2
def _on_item(ev):
3
    m = getattr(ev.item, "metrics", None)
4
    if m:
5
        logger.info("turn_latency", extra={"e2e_latency": m.get("e2e_latency"),
6
                                            "llm_ttft": m.get("llm_node_ttft"),
7
                                            "tts_ttfb": m.get("tts_node_ttfb")})

Report the p50 and p95, not just the average. (p50 is the median: half your turns are faster than this. p95 is the value 95% of turns beat, so it captures the slow tail that an average hides.) A p50 of 800 ms with a p95 of 4 s feels broken to users even though the average looks fine.

metrics_collected remains the per-request latency and request-ID feed (usage and cost tracking has moved to session_usage_updated). TTSMetrics.cancelled=True counts barge-ins that killed a synthesis in flight, a good proxy for interruption health. For realtime STT, billed audio duration comes from the server’s session.end when available, falling back to a local estimate if the session dies first; reconcile against Sarvam’s dashboard rather than assuming either source is complete.

The SDK also emits OpenTelemetry spans (user_turn, eou_detection, stt_node, llm_node, tts_node, tts_request_run). Wire up an OTel exporter in production; it’s the only view that shows where a slow turn was actually slow, and it’s where Sarvam’s TTS request IDs land (Section 10).

Suggested SLOs for a cascading Sarvam pipeline:

10. Capturing request IDs

Sarvam’s server-side request_id is what their support team needs to investigate a bad transcript or mispronunciation. Coverage differs by surface:

STTMetrics.request_id is the real Sarvam ID; log it and you’re done for STT. On the streaming TTS path, TTSMetrics.request_id is a locally generated ID assigned before Sarvam replies, so Sarvam’s actual ID only reaches the OTel span attribute lk.provider_request_ids; enable tracing if you need it there. ChatMessage.metrics["provider_request_ids"] is populated only for realtime (speech-to-speech) models, so it stays empty in any cascading STT-LLM-TTS pipeline.

A minimal production log line should carry session_id, turn_id, speech_id, the Sarvam STT request ID, the Sarvam TTS request IDs, and e2e_latency. With those six fields you can escalate any complaint to Sarvam support with evidence. speech_id ties one utterance’s LLM and TTS calls together; segment_id identifies an individual synthesis segment within an utterance. Keep the IDs logged for as long as your support SLA requires, and don’t store transcript text alongside them unless that store meets your PII policy: Indian consumer call transcripts routinely contain names, addresses, and financial details.

11. The legacy class and migration

sarvam.STT with model="saaras:v3" is the only path that supports batch recognize() (STTStreaming raises NotImplementedError for it). If you’re maintaining an existing deployment on it, the differences that matter:

New builds should use STTStreaming. Migrating is mostly a constructor swap plus replacing the VAD parameters per Section 5, but re-tune min_delay afterward, since the event ordering and chunking both change your latency profile.

12. Reliability and failure modes

Retries. STTStreaming uses the framework default: 3 retries, 2.0 s interval, 10.0 s timeout. That timeout is too long for voice; a caller will have hung up. Tighten it:

1
from livekit.agents import APIConnectOptions
2
from livekit.agents.voice.agent_session import SessionConnectOptions
3
4
conn_options = SessionConnectOptions(
5
    stt_conn_options=APIConnectOptions(max_retry=2, timeout=5.0),
6
    tts_conn_options=APIConnectOptions(max_retry=1, timeout=5.0),
7
    llm_conn_options=APIConnectOptions(max_retry=1, timeout=8.0),
8
    max_unrecoverable_errors=3,
9
)

For calls you can’t afford to drop, wrap STT and TTS in a FallbackAdapter; a failed TTS is a silent agent, which is worse than a wrong one.

Websocket close codes are the most useful operational signal available:

Non-fatal error events (is_fatal: false) log a warning and let the session continue; watch their rate as an early degradation signal, since it’s otherwise invisible. Fatal errors raise APIStatusError. ValueError on any parameter fires at construction time, so instantiate sarvam.STTStreaming and sarvam.TTS in setup_fnc, not the entrypoint, so config errors fail the worker at startup rather than the caller’s session.

Sarvam closes idle TTS websockets after 60 s. The plugin defends on two layers, a transport-level ping every 20 s and an application-level ping every 30 s, so stale pooled connections are detected and replaced transparently. Size your key’s rate limits against peak concurrent calls times two (one STT socket plus one TTS socket per call), not average volume, and alert on close code 1003.

13. Tuning playbook

Change one thing at a time and measure between steps.

1. Instrument first. Wire up Section 9 and Section 10 before touching any parameter, and collect a baseline over at least 30 real calls.
2. Get the architecture right. stream_type="fast", vad=None, turn_detection="stt", model="bulbul:v3". Confirm there’s no TurnDetector requires a VAD model warning at startup; if you see it, vad=None didn’t take effect.
3. Tune Sarvam’s VAD. Raise vad_min_silence_ms if users get cut off mid-sentence; lower it if there’s dead air after they finish. Raise vad_sot_threshold, then vad_min_speech_ms, if noise or crosstalk is triggering turns.
4. Trim endpointing.min_delay. Step down from 0.5 in 0.1 s increments (0.4, 0.3, 0.25, 0.2), stopping one step before cutoffs reappear. Most Sarvam pipelines land at 0.2 to 0.3 s.
5. Fix barge-in. Raise interruption.min_words (0, then 1, then 2) until noise-triggered interruptions stop, watching that real single-word replies survive. Lower false_interruption_timeout to 1.2 to 1.5 s. For telephony, set aec_warmup_duration=None.
6. Cut TTS time-to-first-byte. min_buffer_size=30, output_audio_codec="linear16" (or mulaw for 8 kHz), and prewarm().
7. Cut LLM time-to-first-token. Shrink the system prompt, scope the tool list to the current conversation phase, cap conversation history, and keep max_tool_steps low.
8. Polish the voice. pace 0.9 to 0.95 for support flows, lower temperature for regulated content, and a dict_id dictionary for brand and place names.
9. Re-baseline. Re-measure over 30-plus calls and set alerts on p95 e2e_latency, false-interruption rate, close code 1003, and non-fatal STT error rate.

14. Copy-paste configs

WebRTC or app: lowest latency, clean audio

1
import logging
2
3
from livekit.agents import (
4
    Agent, AgentServer, AgentSession, JobContext, JobProcess,
5
    TurnHandlingOptions, cli, metrics,
6
)
7
from livekit.plugins import sarvam
8
9
logger = logging.getLogger("sarvam-agent")
10
11
12
def setup(proc: JobProcess) -> None:
13
    # Construct here so config errors fail the worker, not the caller's session.
14
    proc.userdata["stt"] = sarvam.STTStreaming(
15
        language="hi-IN",
16
        stream_type="fast",
17
        mode="codemix",
18
        endpointing="vad",
19
        sample_rate=16000,
20
        vad_min_silence_ms=300,
21
        vad_min_speech_ms=80,
22
        vad_sot_threshold=0.5,
23
        prompt="Customer support for a fintech app. Terms: UPI, KYC, NEFT, IMPS, mandate.",
24
    )
25
    proc.userdata["tts"] = sarvam.TTS(
26
        model="bulbul:v3",
27
        speaker="priya",
28
        target_language_code="hi-IN",
29
        speech_sample_rate=24000,
30
        output_audio_codec="linear16",
31
        min_buffer_size=30,
32
        max_chunk_length=150,
33
        pace=0.95,
34
        temperature=0.5,
35
    )
36
    proc.userdata["tts"].prewarm()
37
38
39
server = AgentServer(setup_fnc=setup)
40
41
42
@server.rtc_session()
43
async def entrypoint(ctx: JobContext) -> None:
44
    ctx.log_context_fields = {"room": ctx.room.name}
45
46
    session = AgentSession(
47
        stt=ctx.proc.userdata["stt"],
48
        llm=sarvam.LLM(model="sarvam-105b"),
49
        tts=ctx.proc.userdata["tts"],
50
        vad=None,
51
        turn_handling=TurnHandlingOptions(
52
            turn_detection="stt",
53
            endpointing={"mode": "fixed", "min_delay": 0.22, "max_delay": 2.0},
54
            interruption={
55
                "enabled": True,
56
                "mode": "vad",
57
                "min_words": 1,
58
                "false_interruption_timeout": 1.3,
59
                "resume_false_interruption": True,
60
            },
61
            preemptive_generation={"enabled": True},
62
        ),
63
        aec_warmup_duration=3.0,     # keep for open mic + speaker
64
        user_away_timeout=None,
65
        max_tool_steps=2,
66
    )
67
68
    @session.on("metrics_collected")
69
    def _on_metrics(ev):
70
        metrics.log_metrics(ev.metrics)
71
72
    @session.on("agent_false_interruption")
73
    def _on_false(ev):
74
        logger.warning("false_interruption", extra={"resumed": ev.resumed})
75
76
    await session.start(
77
        agent=Agent(instructions="You are a concise Hindi-speaking support agent. "
78
                                 "Reply in one or two short sentences. Use proper punctuation."),
79
        room=ctx.room,
80
    )
81
82
83
if __name__ == "__main__":
84
    cli.run_app(server)

Telephony or SIP: noisy, echo handled by the carrier

1
stt = sarvam.STTStreaming(
2
    language="hi-IN",
3
    stream_type="fast",
4
    mode="transcribe",
5
    endpointing="vad",
6
    sample_rate=16000,             # upsample; usually cheaper than degrading the recognizer
7
    vad_min_silence_ms=500,
8
    vad_min_speech_ms=200,         # ignore line clicks, ringback, DTMF
9
    vad_sot_threshold=0.7,         # reject TV / crosstalk / ambient speech
10
)
11
12
tts = sarvam.TTS(
13
    model="bulbul:v3",
14
    speaker="shubh",
15
    target_language_code="hi-IN",
16
    speech_sample_rate=8000,       # match the carrier
17
    output_audio_codec="mulaw",    # decoded in-process, no resample
18
    output_audio_bitrate="64k",
19
    min_buffer_size=30,
20
    pace=0.92,                     # slower reads better on narrowband
21
    temperature=0.5,
22
)
23
24
session = AgentSession(
25
    stt=stt, llm=sarvam.LLM(model="sarvam-105b"), tts=tts,
26
    vad=None,
27
    turn_handling={
28
        "turn_detection": "stt",
29
        "endpointing": {"mode": "fixed", "min_delay": 0.3, "max_delay": 2.5},
30
        "interruption": {
31
            "enabled": True,
32
            "mode": "vad",
33
            "min_words": 2,        # noisy line: stricter filter
34
            "false_interruption_timeout": 1.5,
35
            "resume_false_interruption": True,
36
        },
37
        "preemptive_generation": {"enabled": True},
38
        "user_turn_limit": {"max_duration": 45.0},   # catch a stuck call
39
    },
40
    aec_warmup_duration=None,      # important for telephony
41
    user_away_timeout=20.0,
42
)

For compliance and audit flows, use mode="verbatim", return_timestamps=True, a higher vad_min_silence_ms (around 900), a lower temperature (around 0.3), and disable preemptive_generation to avoid speculative spend; disable interruptions per-turn for disclosures. For push-to-talk or IVR, set endpointing="manual" on the STT and turn_detection="manual" on the session, then call session.commit_user_turn() yourself on a DTMF key or button release.

Quick reference