Realtime Speech-to-Text API

Overview

saaras:v3-realtime is Sarvam’s WebSocket streaming model for voice agents and live transcription: real interim (partial) transcripts as the user speaks, plus millisecond-based VAD tuning.

How this differs from the legacy Streaming API

Connection parameters

Sent as query parameters on the WebSocket URL (or the matching keyword argument on connect() in the Python SDK).

Stream types

stream_type controls the tradeoff between partial-transcript latency and accuracy:

Getting started

1
import asyncio
2
import base64
3
from sarvamai import AsyncSarvamAI, RealtimeAudioInput, RealtimeEnd
4
5
API_KEY = "YOUR_SARVAM_API_KEY"
6
7
async def transcribe(audio_chunks):
8
    """audio_chunks: an async iterator yielding raw linear16 PCM bytes."""
9
    client = AsyncSarvamAI(api_subscription_key=API_KEY)
10
11
    async with client.speech_to_text_realtime_streaming.connect(
12
        language_code="hi-IN",
13
        stream_type="fast",
14
    ) as ws:
15
16
        async def send_audio():
17
            async for chunk in audio_chunks:
18
                await ws.send_realtime_audio_input(
19
                    RealtimeAudioInput(audio=base64.b64encode(chunk).decode("utf-8"))
20
                )
21
            await ws.send_realtime_end(RealtimeEnd())
22
23
        async def receive_events():
24
            async for message in ws:
25
                if message.event == "transcript.partial":
26
                    print(f"partial: {message.text}")
27
                elif message.event == "transcript.final":
28
                    print(f"final: {message.text}")
29
                    return  # one-shot script: stop after the first utterance's final
30
                elif message.event == "error":
31
                    print(f"error ({message.code}): {message.message}")
32
                    if message.is_fatal:
33
                        return
34
35
        await asyncio.gather(send_audio(), receive_events())
36
37
38
async def pcm_chunks_from_file(path, chunk_size=3200):
39
    """Yields raw linear16 PCM chunks (~100ms each at 16kHz mono 16-bit)."""
40
    with open(path, "rb") as f:
41
        while chunk := f.read(chunk_size):
42
            yield chunk
43
            await asyncio.sleep(0.1)  # pace it like real-time audio
44
45
46
if __name__ == "__main__":
47
    asyncio.run(transcribe(pcm_chunks_from_file("path/to/audio.pcm")))

Turn detection

• vad (default): the server runs its own VAD and emits vad.speech_start/vad.speech_end automatically. Tune with threshold, silence_duration_ms, min_speech_duration_ms.
• manual: the client delimits turns itself by sending {"event": "speech_start"} and {"event": "speech_end"}. VAD parameters have no effect. A {"event": "flush"} message force-finalizes buffered audio without waiting for speech_end.

Language support

language_code is required. 24 values are accepted, including adaptive auto-detection:

With language_code="auto", transcript.partial and transcript.final include a detected language field (transcript.final also adds language_confidence). With a specific language_code, neither field is present.

Message reference

Client to server: audio_input ({"event": "audio_input", "audio": "<base64>"}), speech_start / speech_end / flush (manual mode only), config.update (change settings mid-call, see below), end (graceful close), ping (keepalive).

Server to client: session.begin (sent on connect), vad.speech_start / vad.speech_end, transcript.partial, transcript.final, config.updated, pong, session.end (includes audio_duration_s, the billed audio), error (code, is_fatal, message).

Live config updates

Change settings mid-call without reconnecting:

1
from sarvamai import RealtimeConfigUpdate
2
3
await ws.send_realtime_config_update(
4
    RealtimeConfigUpdate(mode="translate", prompt="medical terminology")
5
)

language_code, prompt, mode, stream_type, and endpointing apply at the next utterance boundary. threshold, silence_duration_ms, and min_speech_duration_ms apply immediately (vad mode only). encoding, sample_rate, and return_timestamps are connection-only.

Error handling

Best practices

• Use stream_type="fast" for conversational agents; reach for simulated only if you truly don’t need partials.
• Drive barge-in off vad.speech_start or early partials, not transcript.final.
• Reconfigure with config.update instead of reconnecting when you can.
• Reconcile billing against session.end.audio_duration_s, the server-authoritative value.