Skip to navigation

Realtime Streaming Speech-to-Text API

View as Markdown

Overview

Sarvam’s WebSocket streaming models for voice agents and live transcription: real interim (partial) transcripts as the user speaks, plus millisecond-based VAD tuning. saaras:v4 is the default; saaras:v3-realtime is also available on this endpoint and accepts the same connection parameters and message protocol described below, except that keyterms is saaras:v4 only.

How this differs from the legacy Streaming API

Realtime Streaming (saaras:v4 / saaras:v3-realtime)Streaming API (Legacy, saaras:v3)
EndpointGET /speech-to-text-realtime/wsGET /speech-to-text/ws
Interim resultsReal transcript.partial events throughout the utteranceNone; only a final transcript per utterance
VAD tuning3 parameters, in milliseconds (threshold, silence_duration_ms, min_speech_duration_ms)10+ parameters, in frame counts
Mid-call reconfigurationYes, via config.update over the open socketNo, reconnect required to change parameters
Manual turn controlendpointing="manual" with client-sent speech_start / speech_end / flushflush_signal only
Audio message format{"event": "audio_input", "audio": "<base64>"}{"audio": {"data": ..., "sample_rate": ..., "encoding": ...}}
Language detectionlanguage_code="auto", detected language on every partial/finallanguage_code="unknown", language_probability on final only
ErrorsStructured error events (code, is_fatal, message) + documented close codesAn error message with only a message field (no code or is_fatal), then the socket closes. Authentication failures are rejected with HTTP 403 before the connection opens

Connection parameters

Sent as query parameters on the WebSocket URL (or the matching keyword argument on connect() in the Python SDK).

ParameterTypeDefaultDescription
language_codestringrequiredBCP-47 code of the input audio, or auto for adaptive language detection.
modelstringsaaras:v4saaras:v4 (default) or saaras:v3-realtime.
stream_typestringbalancedLatency/accuracy tradeoff for partials. See Stream types below.
modestringtranscribeTask applied to the final transcript only: transcribe, translate, verbatim, translit, codemix. Partials are always plain transcription.
endpointingstringvadvad (server auto-detects turn boundaries) or manual (client sends speech_start/speech_end).
encodingstringlinear16linear16, linear32, mulaw, or alaw. Mono only.
sample_rateinteger160008000 or 16000 only; any other value closes the connection (code 4000).
thresholdfloat0.3VAD sensitivity (0.0-1.0). Only applies when endpointing=vad.
silence_duration_msinteger1000Silence (ms) marking end-of-turn. Only applies when endpointing=vad.
prefix_padding_msinteger300Audio (ms) kept from before the detected start of speech. Only applies when endpointing=vad.
min_speech_duration_msinteger250Minimum speech duration (ms) to count as an utterance. Only applies when endpointing=vad.
return_timestampsbooleanfalseAdds start_s/end_s to transcript.final when true.
promptstring(none)Optional terminology hint applied to the final transcript.
keytermsstring (JSON array)(none)JSON-encoded list of up to 50 keyterms (64 characters each) to bias recognition. saaras:v4 only.

Stream types

stream_type controls the tradeoff between partial-transcript latency and accuracy:

ValuePartialsUse when
fastLowest latencyConversational voice agents where snappy barge-in matters most
balanced (default)Slightly slower, more accurateGeneral live transcription and captioning
simulatedNone emitted at allYou only need the final transcript per utterance

Getting started

import asyncio
import base64
from sarvamai import AsyncSarvamAI, RealtimeAudioInput, RealtimeEnd
API_KEY = "YOUR_SARVAM_API_KEY"
async def transcribe(audio_chunks):
"""audio_chunks: an async iterator yielding raw linear16 PCM bytes."""
client = AsyncSarvamAI(api_subscription_key=API_KEY)
async with client.speech_to_text_realtime_streaming.connect(
language_code="hi-IN",
stream_type="fast",
) as ws:
async def send_audio():
async for chunk in audio_chunks:
await ws.send_realtime_audio_input(
RealtimeAudioInput(audio=base64.b64encode(chunk).decode("utf-8"))
)
await ws.send_realtime_end(RealtimeEnd())
async def receive_events():
async for message in ws:
if message.event == "transcript.partial":
print(f"partial: {message.text}")
elif message.event == "transcript.final":
print(f"final: {message.text}")
return # one-shot script: stop after the first utterance's final
elif message.event == "error":
print(f"error ({message.code}): {message.message}")
if message.is_fatal:
return
await asyncio.gather(send_audio(), receive_events())
async def pcm_chunks_from_file(path, chunk_size=3200):
"""Yields raw linear16 PCM chunks (~100ms each at 16kHz mono 16-bit)."""
with open(path, "rb") as f:
while chunk := f.read(chunk_size):
yield chunk
await asyncio.sleep(0.1) # pace it like real-time audio
if __name__ == "__main__":
asyncio.run(transcribe(pcm_chunks_from_file("path/to/audio.pcm")))

saaras:v4 is used when you don’t pass model. To use the earlier model, pass model="saaras:v3-realtime". Connection parameters and message events are identical, and saaras:v4 additionally supports keyterms, which you can also change mid-call with config.update. See Keyterm Prompting.

Turn detection

  • vad (default): the server runs its own VAD and emits vad.speech_start/vad.speech_end automatically. Tune with threshold, silence_duration_ms, min_speech_duration_ms.
  • manual: the client delimits turns itself by sending {"event": "speech_start"} and {"event": "speech_end"}. VAD parameters have no effect. A {"event": "flush"} message force-finalizes buffered audio without waiting for speech_end.

Language support

language_code is required. 24 values are accepted, including adaptive auto-detection:

LanguageCodeLanguageCode
Auto-detectautoUrduur-IN
Englishen-INNepaline-IN
Hindihi-INKonkanikok-IN
Bengalibn-INKashmiriks-IN
Kannadakn-INSindhisd-IN
Malayalamml-INSanskritsa-IN
Marathimr-INSantalisat-IN
Odiaor-INManipurimni-IN
Punjabipa-INBodobrx-IN
Tamilta-INMaithilimai-IN
Telugute-INDogridoi-IN
Gujaratigu-INAssameseas-IN

Odia’s code changed. The legacy Streaming API uses od-IN for Odia; this endpoint uses or-IN.

With language_code="auto", transcript.partial and transcript.final include a detected language field (transcript.final also adds language_confidence). With a specific language_code, neither field is present.

Message reference

Client to server: audio_input ({"event": "audio_input", "audio": "<base64>"}), speech_start / speech_end / flush (manual mode only), config.update (change settings mid-call, see below), end (graceful close), ping (keepalive).

Server to client: session.begin (sent on connect), vad.speech_start / vad.speech_end, transcript.partial, transcript.final, config.updated, pong, session.end (includes audio_duration_s, the billed audio), error (code, is_fatal, message).

Live config updates

Change settings mid-call without reconnecting:

from sarvamai import RealtimeConfigUpdate
await ws.send_realtime_config_update(
RealtimeConfigUpdate(mode="translate", prompt="medical terminology")
)

language_code, prompt, mode, stream_type, and endpointing apply at the next utterance boundary. threshold, silence_duration_ms, and min_speech_duration_ms apply immediately (vad mode only). keyterms can also be updated mid-call (saaras:v4 only). encoding, sample_rate, and return_timestamps are connection-only.

Connection parameters are range-checked when you connect (for example, threshold outside 0.0-1.0 closes the connection with 4000). Values sent in config.update are not range-checked, so keep threshold between 0.0 and 1.0 and the millisecond values at 0 or above.

Error handling

Failures arrive as an error event (code, message, is_fatal, status_code) followed, for fatal errors, by a close frame.

Close codeMeaning
1003Invalid subscription key (invalid_subscription_key). Rate-limit and quota failures also close with 1003.
1008No audio received for 60 seconds (inactivity_timeout). Send audio or periodic pings to avoid this.
1011Internal server error, retry with backoff.
4000Invalid model, language_code, or other parameter (invalid_request, invalid_model), or account not enabled.
error.codeFatalMeaning
invalid_requestYesA missing or invalid connection parameter, such as no language_code, sample_rate other than 8000/16000, an unsupported encoding, an out-of-range threshold, or keyterms on saaras:v3-realtime.
invalid_modelYesmodel is not saaras:v4 or saaras:v3-realtime.
invalid_subscription_keyYesThe API key is missing or invalid (status_code 401).
inactivity_timeoutYesNo audio received for 60 seconds (status_code 408).
chunk_too_largeNoOne audio_input frame exceeds the per-frame cap (500 ms of audio, 16,000 bytes at 16 kHz mono, on stream_type="fast"). The session stays open; send smaller chunks of about 100 ms.

Best practices

  • Use stream_type="fast" for conversational agents; reach for simulated only if you truly don’t need partials.
  • Drive barge-in off vad.speech_start or early partials, not transcript.final.
  • Reconfigure with config.update instead of reconnecting when you can.
  • Reconcile billing against session.end.audio_duration_s, the server-authoritative value.

Full endpoint reference: Realtime Streaming API reference.