Live Video Transcription

Overview

This cookbook builds a real-time transcription and translation demo. Play or upload any video with speech in the browser, and watch a live transcript and English translation appear as it plays, powered by Sarvam’s Streaming Speech-to-Text API over a persistent WebSocket connection.

What you’ll build:

• A browser page that captures a playing video’s audio and streams it to a server, no file upload round-trip required
• A Flask-SocketIO server that keeps one persistent Sarvam streaming connection open per client per mode
• Live transcription in the source language, running concurrently with live translation to English
• Results pushed back to the browser over WebSocket as each speech segment finalizes

Business Value

• Live captions for lectures, webinars, and internal recordings
• Real-time subtitling for video content in a viewer’s own language
• Accessibility support for hearing-impaired viewers watching pre-recorded or live video

How It Works

Transcription and translation run as two independent streaming connections, so you can start one, both, or switch between them without interrupting the other.

1. Prerequisites

• Python 3.8 or higher
• A Sarvam API key, sign up on the Sarvam AI Dashboard to get one
• A modern browser (Web Audio API and HTMLMediaElement.captureStream() support)
• A video file with speech to try it on

2. Clone and Install

$
git clone https://github.com/sarvamai/sarvam-ai-cookbook.git
$
cd sarvam-ai-cookbook/examples/Live_Video_Transcription
$
python -m venv venv
$
source venv/bin/activate
$
pip install -r requirements.txt

3. Configure Environment Variables

Create a .env file in the project root:

SARVAM_API_KEY=sk_xxxxxxxxxxxxxxxxxxxxxxxx

config.py reads this key and also holds a few settings worth knowing about before you run the app:

4. Run It

$
python app.py

Open http://localhost:5001, upload a video with speech, then click Start Transcription and/or Start Translation. Text appears in the corresponding panel as each segment finalizes.

5. Pipeline Walkthrough

Step 1: Capture and encode audio in the browser

templates/index.html pulls the audio track off the <video> element with captureStream(), buffers raw samples from a ScriptProcessorNode, and every second resamples the buffer to 16kHz and encodes it as WAV before sending it over Socket.IO:

1
const stream = video.captureStream();
2
mediaStreamSource = audioContext.createMediaStreamSource(stream);
3
processor = audioContext.createScriptProcessor(4096, 1, 1);
4
5
processor.onaudioprocess = function(event) {
6
    const audioData = event.inputBuffer.getChannelData(0);
7
    const int16Array = new Int16Array(audioData.length);
8
    for (let i = 0; i < audioData.length; i++) {
9
        int16Array[i] = audioData[i] * 32767;
10
    }
11
    transcriptionChunks.push(int16Array);
12
    // ...every ~1s: resample to 16kHz, encode as WAV, base64, and emit
13
};

Step 2: One persistent connection per mode

app.py defines a StreamingSession class that opens a Sarvam streaming connection and keeps it alive in a dedicated background thread with its own asyncio event loop, one instance per (client, mode) pair:

1
async def _session(self):
2
    client = AsyncSarvamAI(api_subscription_key=SARVAM_API_KEY)
3
4
    conn = client.speech_to_text_streaming.connect(
5
        language_code=config.API_LANGUAGE,
6
        model="saaras:v3",
7
        mode="transcribe" if self.mode == "transcribe" else "translate",
8
    )
9
10
    async with conn as ws:
11
        receiver = asyncio.create_task(self._receive(ws))
12
        try:
13
            while True:
14
                audio_b64 = await self.queue.get()
15
                if audio_b64 is None:
16
                    break
17
                await ws.transcribe(audio=audio_b64)
18
            await ws.flush()
19
            await asyncio.sleep(2)  # let the receiver drain remaining transcripts
20
        finally:
21
            receiver.cancel()

Audio frames arrive from the browser on Flask-SocketIO’s own thread and get handed off to the session’s asyncio loop through a queue (asyncio.run_coroutine_threadsafe), since the WebSocket connection itself must be driven from a single event loop.

_session() runs inside _run(), the background thread’s entry point, which wraps the whole coroutine in a try/except. Any failure, a rejected connection, an auth error, a dropped socket, is caught, logged, and relayed to the browser as an error event so the client isn’t left waiting on a connection that’s already dead:

1
def _run(self):
2
    asyncio.set_event_loop(self.loop)
3
    try:
4
        self.loop.run_until_complete(self._session())
5
    except Exception as e:
6
        logger.error(f"[{self.mode}] session loop error: {e}")
7
        self._stopped = True
8
        streaming_sessions.pop((self.sid, self.mode), None)
9
        with app.app_context():
10
            socketio.emit(
11
                "error",
12
                {"message": f"Failed to start {self.mode} stream: {e}"},
13
                to=self.sid,
14
            )
15
    finally:
16
        self.loop.close()

Sarvam’s role: This is where the actual transcription and translation happen. The AsyncSarvamAI client opens a live WebSocket to Sarvam’s Streaming Speech-to-Text API (Saaras v3), and everything downstream of that, voice activity detection, segmenting continuous audio into utterances, language handling, and the transcription/translation inference itself, runs on Sarvam’s side. The app’s only job here is to keep the connection open and feed it audio frames as they arrive.

Step 3: Relay results back to the browser

A second task on the same connection continuously calls ws.recv() and emits each finalized result back to the originating client:

1
async def _receive(self, ws):
2
    try:
3
        while True:
4
            resp = await ws.recv()
5
            text = _extract_text(resp)
6
            if text and text.strip():
7
                emit_streaming_result(self.sid, self.mode, text.strip())
8
    except asyncio.CancelledError:
9
        pass
10
    except Exception as e:
11
        logger.info(f"[{self.mode}] receiver ended: {e}")

Sarvam’s role: Sarvam produces every resp this loop reads. Each one is a finalized transcript or translation segment that Sarvam decided was complete, based on its own voice-activity detection and segmentation, not something the app computes. This step only relays that already-finished text back over Socket.IO.

Step 4: Socket.IO event wiring

Client and server agree on a small set of events to open, feed, and tear down each streaming session:

Available Options

Source Language

API_LANGUAGE in config.py accepts any Saaras v3 BCP-47 language code, or "unknown" to auto-detect. See Specify Language Codes for the full list of supported codes.

Output Modes

This demo only wires up transcribe and translate, but the underlying Streaming API also supports verbatim, translit, and codemix. See Streaming Speech-to-Text for what each mode returns.

Customization

Tune voice activity detection

Pass high_vad_sensitivity=True when opening the connection in StreamingSession._session for a shorter silence boundary (0.5s instead of 1s), which makes segments finalize faster at a small cost to context:

1
conn = client.speech_to_text_streaming.connect(
2
    language_code=config.API_LANGUAGE,
3
    model="saaras:v3",
4
    high_vad_sensitivity=True,
5
)

Add speech start/end signals

Pass vad_signals=True to receive START_SPEECH / END_SPEECH events alongside transcripts, useful if you want to show a “listening” indicator in the UI rather than only finalized text. See Streaming Speech-to-Text for the message shapes.

Transcribe a live microphone instead of a video file

Swap video.captureStream() in templates/index.html for navigator.mediaDevices.getUserMedia({ audio: true }). The rest of the pipeline, resampling, WAV encoding, and the Socket.IO events, stays the same, since it already operates on a generic MediaStream.

Change the chunk interval

The browser buffers and sends audio roughly once a second (currentTime - lastChunkTime >= 1000 in templates/index.html). Shorter intervals lower latency at the cost of more frequent small requests; longer intervals do the opposite.

Tips & Best Practices

• Keep the sample rate consistent: The demo resamples to 16kHz before encoding, matching the rate it passes to connect(). If you change one, change the other, mismatched sample rates degrade transcription quality.
• One connection per mode, not per chunk: StreamingSession opens a single long-lived connection and streams every frame into it. Opening a new connection per audio chunk would add connection-setup latency to every single frame.
• Independent buffers per mode: Transcription and translation each keep their own audioChunks buffer in the browser, so running both at once doesn’t mean one mode steals frames meant for the other.
• Handle disconnects: This demo doesn’t automatically reconnect on a dropped WebSocket. For production use, add reconnect-with-backoff on close codes 1006/1011, see Handling Disconnects.
• Restrict CORS before deploying: CORS_ALLOWED_ORIGINS = "*" in config.py is fine for local testing only.

Error Handling

You may encounter these errors while using the underlying API:

• 403 Forbidden (invalid_api_key_error), invalid API key. Use a valid key from the Sarvam AI Dashboard.
• 429 Too Many Requests (rate_limit_exceeded_error), too many concurrent streaming connections or requests. Check Credits & Rate Limits for streaming concurrency limits.
• WebSocket close 1006 / 1011, abnormal closure or server error. Reconnect with exponential backoff rather than retrying immediately, see Handling Disconnects.
• WebSocket close 4xxx, application-specific, usually auth or quota. Read the close reason before retrying; blind-retrying an auth failure will just fail again.

For the full error-code table and SDK exception reference, see Errors & Troubleshooting.

Additional Resources

• Documentation: docs.sarvam.ai
• Related APIs: Streaming Speech-to-Text · Specify Language Codes
• Full source: Live_Video_Transcription on GitHub
• Community: Join the Discord Community

Final Notes

• Keep your API key secure, prefer environment variables over hardcoding.
• This demo keeps one persistent connection per client per mode rather than reconnecting per audio chunk, that’s what keeps latency low.
• Add reconnect logic and restrict CORS before taking this beyond local experimentation.

Keep Building! 🚀