> For clean Markdown of any page, append `.md` to the page URL. > For a complete documentation index, see https://docs.sarvam.ai/llms.txt. > For full documentation content in one file, see https://docs.sarvam.ai/llms-full.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.sarvam.ai/_mcp/server. # Realtime Streaming Speech-to-Text API > True partial transcripts, live mid-stream reconfiguration, and millisecond-based VAD tuning with saaras:v3-realtime and saaras:v4, Sarvam's next-generation WebSocket streaming API for voice agents and live transcription. ## Overview Sarvam's WebSocket streaming models for voice agents and live transcription: real interim (partial) transcripts as the user speaks, plus millisecond-based VAD tuning. `saaras:v3-realtime` is the default; `saaras:v4` is also available on this endpoint and accepts the same connection parameters and message protocol described below. ### How this differs from the legacy Streaming API | | Realtime Streaming (`saaras:v3-realtime` / `saaras:v4`) | Streaming API (Legacy, `saaras:v3`) | | ------------------------ | -------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | | Endpoint | `GET /speech-to-text-realtime/ws` | `GET /speech-to-text/ws` | | Interim results | Real `transcript.partial` events throughout the utterance | None; only a final transcript per utterance | | VAD tuning | 3 parameters, in milliseconds (`threshold`, `silence_duration_ms`, `min_speech_duration_ms`) | 10+ parameters, in frame counts | | Mid-call reconfiguration | Yes, via `config.update` over the open socket | No, reconnect required to change parameters | | Manual turn control | `endpointing="manual"` with client-sent `speech_start` / `speech_end` / `flush` | `flush_signal` only | | Audio message format | `{"event": "audio_input", "audio": ""}` | `{"audio": {"data": ..., "sample_rate": ..., "encoding": ...}}` | | Language detection | `language_code="auto"`, detected language on every partial/final | `language_code="unknown"`, `language_probability` on final only | | Errors | Structured `error` events (`code`, `is_fatal`, `message`) + documented close codes | Close code only, no structured error event | ## Connection parameters Sent as query parameters on the WebSocket URL (or the matching keyword argument on `connect()` in the Python SDK). | Parameter | Type | Default | Description | | ------------------------ | ------------------- | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `language_code` | string | **required** | BCP-47 code of the input audio, or `auto` for adaptive language detection. | | `model` | string | `saaras:v3-realtime` | `saaras:v3-realtime` (default) or `saaras:v4`. | | `stream_type` | string | `balanced` | Latency/accuracy tradeoff for partials. See [Stream types](#stream-types) below. | | `mode` | string | `transcribe` | Task applied to the **final** transcript only: `transcribe`, `translate`, `verbatim`, `translit`, `codemix`. Partials are always plain transcription. | | `endpointing` | string | `vad` | `vad` (server auto-detects turn boundaries) or `manual` (client sends `speech_start`/`speech_end`). | | `encoding` | string | `linear16` | `linear16`, `linear32`, `mulaw`, or `alaw`. Mono only. | | `sample_rate` | integer | `16000` | `8000` or `16000` only; any other value closes the connection (code `4000`). | | `threshold` | float | `0.3` | VAD sensitivity (0.0-1.0). Only applies when `endpointing=vad`. | | `silence_duration_ms` | integer | `500` | Silence (ms) marking end-of-turn. Only applies when `endpointing=vad`. | | `min_speech_duration_ms` | integer | `250` | Minimum speech duration (ms) to count as an utterance. Only applies when `endpointing=vad`. | | `return_timestamps` | boolean | `false` | Adds `start_s`/`end_s` to `transcript.final` when `true`. | | `prompt` | string | *(none)* | Optional terminology hint applied to the final transcript. | | `keyterms` | string (JSON array) | *(none)* | JSON-encoded list of up to 50 keyterms (64 characters each) to bias recognition. **`saaras:v4` only.** | ### Stream types `stream_type` controls the tradeoff between partial-transcript latency and accuracy: | Value | Partials | Use when | | -------------------- | ------------------------------ | -------------------------------------------------------------- | | `fast` | Lowest latency | Conversational voice agents where snappy barge-in matters most | | `balanced` (default) | Slightly slower, more accurate | General live transcription and captioning | | `simulated` | None emitted at all | You only need the final transcript per utterance | ## Getting started #### Python ```python import asyncio import base64 from sarvamai import AsyncSarvamAI, RealtimeAudioInput, RealtimeEnd API_KEY = "YOUR_SARVAM_API_KEY" async def transcribe(audio_chunks): """audio_chunks: an async iterator yielding raw linear16 PCM bytes.""" client = AsyncSarvamAI(api_subscription_key=API_KEY) async with client.speech_to_text_realtime_streaming.connect( language_code="hi-IN", stream_type="fast", ) as ws: async def send_audio(): async for chunk in audio_chunks: await ws.send_realtime_audio_input( RealtimeAudioInput(audio=base64.b64encode(chunk).decode("utf-8")) ) await ws.send_realtime_end(RealtimeEnd()) async def receive_events(): async for message in ws: if message.event == "transcript.partial": print(f"partial: {message.text}") elif message.event == "transcript.final": print(f"final: {message.text}") return # one-shot script: stop after the first utterance's final elif message.event == "error": print(f"error ({message.code}): {message.message}") if message.is_fatal: return await asyncio.gather(send_audio(), receive_events()) async def pcm_chunks_from_file(path, chunk_size=3200): """Yields raw linear16 PCM chunks (~100ms each at 16kHz mono 16-bit).""" with open(path, "rb") as f: while chunk := f.read(chunk_size): yield chunk await asyncio.sleep(0.1) # pace it like real-time audio if __name__ == "__main__": asyncio.run(transcribe(pcm_chunks_from_file("path/to/audio.pcm"))) ``` #### JavaScript ```javascript import { SarvamAIClient } from "sarvamai"; import * as fs from "fs"; const API_KEY = "YOUR_SARVAM_API_KEY"; async function transcribe(audioPath) { const client = new SarvamAIClient({ apiSubscriptionKey: API_KEY }); const socket = await client.speechToTextRealtimeStreaming.connect({ language_code: "hi-IN", stream_type: "fast", "Api-Subscription-Key": API_KEY, }); socket.on("open", () => { const audio = fs.readFileSync(audioPath); const chunkSize = 3200; // ~100ms of 16kHz mono 16-bit audio let offset = 0; const interval = setInterval(() => { if (offset >= audio.length) { clearInterval(interval); socket.sendRealtimeEnd({ event: "end" }); return; } socket.sendRealtimeAudioInput({ event: "audio_input", audio: audio.subarray(offset, offset + chunkSize).toString("base64"), }); offset += chunkSize; }, 100); }); socket.on("message", (message) => { if (message.event === "transcript.partial") { console.log(`partial: ${message.text}`); } else if (message.event === "transcript.final") { console.log(`final: ${message.text}`); socket.socket.close(1000); // one-shot script: stop after the first utterance's final } else if (message.event === "error") { console.error(`error (${message.code}): ${message.message}`); } }); } transcribe("path/to/audio.pcm"); ``` #### cURL ```bash # curl has no way to send/receive WebSocket frames, so this only tests the # handshake: it confirms your API key and query parameters are accepted. # Use the Python or JavaScript client above for the actual audio streaming. curl -i -N \ -H "api-subscription-key: YOUR_SARVAM_API_KEY" \ -H "Connection: Upgrade" \ -H "Upgrade: websocket" \ -H "Sec-WebSocket-Version: 13" \ -H "Sec-WebSocket-Key: $(openssl rand -base64 16)" \ --http1.1 \ "https://api.sarvam.ai/speech-to-text-realtime/ws?language_code=hi-IN&stream_type=fast" # A successful handshake returns: HTTP/1.1 101 Switching Protocols ``` > **Tip** > > To try the latest model, pass `model="saaras:v4"` when connecting. Connection parameters and message events are identical to `saaras:v3-realtime`, and `model="saaras:v4"` additionally supports `keyterms` — see [Keyterm Prompting](/api/api-guides-tutorials/speech-to-text/how-to/keyterms). ## Turn detection * **`vad`** (default): the server runs its own VAD and emits `vad.speech_start`/`vad.speech_end` automatically. Tune with `threshold`, `silence_duration_ms`, `min_speech_duration_ms`. * **`manual`**: the client delimits turns itself by sending `{"event": "speech_start"}` and `{"event": "speech_end"}`. VAD parameters have no effect. A `{"event": "flush"}` message force-finalizes buffered audio without waiting for `speech_end`. ## Language support `language_code` is **required**. 24 values are accepted, including adaptive auto-detection: | Language | Code | | Language | Code | | ----------- | ------- | - | -------- | -------- | | Auto-detect | `auto` | | Urdu | `ur-IN` | | English | `en-IN` | | Nepali | `ne-IN` | | Hindi | `hi-IN` | | Konkani | `kok-IN` | | Bengali | `bn-IN` | | Kashmiri | `ks-IN` | | Kannada | `kn-IN` | | Sindhi | `sd-IN` | | Malayalam | `ml-IN` | | Sanskrit | `sa-IN` | | Marathi | `mr-IN` | | Santali | `sat-IN` | | Odia | `or-IN` | | Manipuri | `mni-IN` | | Punjabi | `pa-IN` | | Bodo | `brx-IN` | | Tamil | `ta-IN` | | Maithili | `mai-IN` | | Telugu | `te-IN` | | Dogri | `doi-IN` | | Gujarati | `gu-IN` | | Assamese | `as-IN` | > **Warning** > > **Odia's code changed.** The legacy Streaming API uses `od-IN` for Odia; this endpoint uses `or-IN`. With `language_code="auto"`, `transcript.partial` and `transcript.final` include a detected `language` field (`transcript.final` also adds `language_confidence`). With a specific `language_code`, neither field is present. ## Message reference **Client to server:** `audio_input` (`{"event": "audio_input", "audio": ""}`), `speech_start` / `speech_end` / `flush` (manual mode only), `config.update` (change settings mid-call, see below), `end` (graceful close), `ping` (keepalive). **Server to client:** `session.begin` (sent on connect), `vad.speech_start` / `vad.speech_end`, `transcript.partial`, `transcript.final`, `config.updated`, `pong`, `session.end` (includes `audio_duration_s`, the billed audio), `error` (`code`, `is_fatal`, `message`). ### Live config updates Change settings mid-call without reconnecting: ```python from sarvamai import RealtimeConfigUpdate await ws.send_realtime_config_update( RealtimeConfigUpdate(mode="translate", prompt="medical terminology") ) ``` `language_code`, `prompt`, `mode`, `stream_type`, and `endpointing` apply at the next utterance boundary. `threshold`, `silence_duration_ms`, and `min_speech_duration_ms` apply immediately (`vad` mode only). `encoding`, `sample_rate`, and `return_timestamps` are connection-only. ## Error handling | Close code | Meaning | | ---------- | ----------------------------------------------------------------- | | `1003` | Rate limit, quota exceeded, or invalid subscription key | | `1008` | Inactivity timeout, send periodic `ping`s to avoid this | | `1011` | Internal server error, retry with backoff | | `4000` | Invalid `model`/`language_code`/parameter, or account not enabled | ## Best practices * Use `stream_type="fast"` for conversational agents; reach for `simulated` only if you truly don't need partials. * Drive barge-in off `vad.speech_start` or early partials, not `transcript.final`. * Reconfigure with `config.update` instead of reconnecting when you can. * Reconcile billing against `session.end.audio_duration_s`, the server-authoritative value. > **Note** > > Full endpoint reference: [Realtime Streaming](/api-reference/speech-to-text/transcribe/realtime/ws) API reference. > True partial transcripts, live mid-stream reconfiguration, and millisecond-based VAD tuning with saaras:v3-realtime and saaras:v4, Sarvam's next-generation WebSocket streaming API for voice agents and live transcription.