> For clean Markdown of any page, append `.md` to the page URL.
> For a complete documentation index, see https://docs.sarvam.ai/llms.txt.
> For full documentation content in one file, see https://docs.sarvam.ai/llms-full.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.sarvam.ai/_mcp/server.

# Live Video Transcription

> Build a real-time transcription and translation demo with Sarvam AI's Streaming Speech-to-Text API, Flask-SocketIO, and the browser's Web Audio API.

## **Overview**

This cookbook builds a real-time transcription and translation demo. Play or upload any video with speech in the browser, and watch a live transcript and English translation appear as it plays, powered by Sarvam's **Streaming Speech-to-Text API** over a persistent WebSocket connection.

**What you'll build:**

* A browser page that captures a playing video's audio and streams it to a server, no file upload round-trip required
* A Flask-SocketIO server that keeps one persistent Sarvam streaming connection open per client per mode
* Live transcription in the source language, running concurrently with live translation to English
* Results pushed back to the browser over WebSocket as each speech segment finalizes

#### [View on GitHub](https://github.com/sarvamai/sarvam-ai-cookbook/tree/main/examples/Live_Video_Transcription)

Browse the full source for this app in the sarvam-ai-cookbook repo.

#### [Run Code Locally](#2-clone-and-install)

Jump to Clone and Install to set this demo up on your machine.

### Business Value

* Live captions for lectures, webinars, and internal recordings
* Real-time subtitling for video content in a viewer's own language
* Accessibility support for hearing-impaired viewers watching pre-recorded or live video

#### EdTech

Caption recorded lectures live as students watch, with an English translation running alongside the original-language transcript.

#### Media & Broadcast

Add live subtitles to a video feed without waiting for the whole file to be transcribed first.

#### Enterprise

Transcribe town halls, training videos, or internal recordings for search and accessibility as they're played back.

## **How It Works**

#### Capture audio in the browser

The browser reads the video element's audio track via the Web Audio API, resamples it to 16kHz mono, and encodes it as WAV, entirely client-side, no server upload of the raw video file.

#### Stream \~1 second frames over Socket.IO

Every second, the buffered audio is packaged into one WAV chunk, base64-encoded, and emitted to the Flask server over a Socket.IO event.

#### Forward into a persistent Sarvam connection

For each client, the server opens one long-lived Sarvam streaming connection per active mode (transcribe and/or translate) and forwards incoming frames into it as they arrive.

#### Relay results back live

Sarvam emits a finalized transcript or translation as each speech segment completes. The server relays it back to the same browser tab over Socket.IO, and it's appended to the transcript panel immediately.

Transcription and translation run as two independent streaming connections, so you can start one, both, or switch between them without interrupting the other.

## **1. Prerequisites**

* Python 3.8 or higher
* A Sarvam API key, sign up on the [Sarvam AI Dashboard](https://dashboard.sarvam.ai/) to get one
* A modern browser (Web Audio API and `HTMLMediaElement.captureStream()` support)
* A video file with speech to try it on

## **2. Clone and Install**

#### macOS/Linux

```bash
git clone https://github.com/sarvamai/sarvam-ai-cookbook.git
cd sarvam-ai-cookbook/examples/Live_Video_Transcription
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
```

#### Windows

```bash
git clone https://github.com/sarvamai/sarvam-ai-cookbook.git
cd sarvam-ai-cookbook/examples/Live_Video_Transcription
python -m venv venv
venv\Scripts\activate
pip install -r requirements.txt
```

## **3. Configure Environment Variables**

Create a `.env` file in the project root:

```env
SARVAM_API_KEY=sk_xxxxxxxxxxxxxxxxxxxxxxxx
```

`config.py` reads this key and also holds a few settings worth knowing about before you run the app:

| Setting                | Default              | Meaning                                                                                                                                                                |
| ---------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `HOST` / `PORT`        | `127.0.0.1` / `5001` | Where the Flask server binds                                                                                                                                           |
| `API_LANGUAGE`         | `"unknown"`          | Source language passed to the transcription stream; `"unknown"` auto-detects. Set to a specific code (e.g. `"hi-IN"`) for more consistent accuracy on a known language |
| `CORS_ALLOWED_ORIGINS` | `"*"`                | Fine for local testing; restrict this before deploying anywhere public                                                                                                 |

## **4. Run It**

```bash
python app.py
```

Open [http://localhost:5001](http://localhost:5001), upload a video with speech, then click **Start Transcription** and/or **Start Translation**. Text appears in the corresponding panel as each segment finalizes.

## **5. Pipeline Walkthrough**

### Step 1: Capture and encode audio in the browser

`templates/index.html` pulls the audio track off the `<video>` element with `captureStream()`, buffers raw samples from a `ScriptProcessorNode`, and every second resamples the buffer to 16kHz and encodes it as WAV before sending it over Socket.IO:

```javascript
const stream = video.captureStream();
mediaStreamSource = audioContext.createMediaStreamSource(stream);
processor = audioContext.createScriptProcessor(4096, 1, 1);

processor.onaudioprocess = function(event) {
    const audioData = event.inputBuffer.getChannelData(0);
    const int16Array = new Int16Array(audioData.length);
    for (let i = 0; i < audioData.length; i++) {
        int16Array[i] = audioData[i] * 32767;
    }
    transcriptionChunks.push(int16Array);
    // ...every ~1s: resample to 16kHz, encode as WAV, base64, and emit
};
```

> **Note**
>
> Streaming endpoints only accept WAV or raw PCM audio, not MP3, AAC, or other compressed formats. Encoding to WAV client-side, as this demo does, is what makes the captured browser audio acceptable to the API.

### Step 2: One persistent connection per mode

`app.py` defines a `StreamingSession` class that opens a Sarvam streaming connection and keeps it alive in a dedicated background thread with its own asyncio event loop, one instance per `(client, mode)` pair:

```python
async def _session(self):
    client = AsyncSarvamAI(api_subscription_key=SARVAM_API_KEY)

    conn = client.speech_to_text_streaming.connect(
        language_code=config.API_LANGUAGE,
        model="saaras:v4",
        mode="transcribe" if self.mode == "transcribe" else "translate",
    )

    async with conn as ws:
        receiver = asyncio.create_task(self._receive(ws))
        try:
            while True:
                audio_b64 = await self.queue.get()
                if audio_b64 is None:
                    break
                await ws.transcribe(audio=audio_b64)
            await ws.flush()
            await asyncio.sleep(2)  # let the receiver drain remaining transcripts
        finally:
            receiver.cancel()
```

> **Tip**
>
> Both modes go through the same unified `speech_to_text_streaming` endpoint here, the `mode` parameter (`"transcribe"` vs `"translate"`) picks the behavior, and both still call `ws.transcribe()` to send audio. See [Streaming Speech-to-Text](/api/api-guides-tutorials/speech-to-text/streaming-api#key-differences-stt-vs-sttt) for the full comparison.

Audio frames arrive from the browser on Flask-SocketIO's own thread and get handed off to the session's asyncio loop through a queue (`asyncio.run_coroutine_threadsafe`), since the WebSocket connection itself must be driven from a single event loop.

`_session()` runs inside `_run()`, the background thread's entry point, which wraps the whole coroutine in a try/except. Any failure, a rejected connection, an auth error, a dropped socket, is caught, logged, and relayed to the browser as an `error` event so the client isn't left waiting on a connection that's already dead:

```python
def _run(self):
    asyncio.set_event_loop(self.loop)
    try:
        self.loop.run_until_complete(self._session())
    except Exception as e:
        logger.error(f"[{self.mode}] session loop error: {e}")
        self._stopped = True
        streaming_sessions.pop((self.sid, self.mode), None)
        with app.app_context():
            socketio.emit(
                "error",
                {"message": f"Failed to start {self.mode} stream: {e}"},
                to=self.sid,
            )
    finally:
        self.loop.close()
```

Sarvam's role: This is where the actual transcription and translation happen. The `AsyncSarvamAI` client opens a live WebSocket to Sarvam's Streaming Speech-to-Text API (Saaras v3), and everything downstream of that, voice activity detection, segmenting continuous audio into utterances, language handling, and the transcription/translation inference itself, runs on Sarvam's side. The app's only job here is to keep the connection open and feed it audio frames as they arrive.

### Step 3: Relay results back to the browser

A second task on the same connection continuously calls `ws.recv()` and emits each finalized result back to the originating client:

```python
async def _receive(self, ws):
    try:
        while True:
            resp = await ws.recv()
            text = _extract_text(resp)
            if text and text.strip():
                emit_streaming_result(self.sid, self.mode, text.strip())
    except asyncio.CancelledError:
        pass
    except Exception as e:
        logger.info(f"[{self.mode}] receiver ended: {e}")
```

Sarvam's role: Sarvam produces every `resp` this loop reads. Each one is a finalized transcript or translation segment that Sarvam decided was complete, based on its own voice-activity detection and segmentation, not something the app computes. This step only relays that already-finished text back over Socket.IO.

### Step 4: Socket.IO event wiring

Client and server agree on a small set of events to open, feed, and tear down each streaming session:

| Event                                         | Direction       | Purpose                                                                                  |
| --------------------------------------------- | --------------- | ---------------------------------------------------------------------------------------- |
| `start_stream` / `stop_stream`                | Client → Server | Open or close a persistent streaming session for `{ mode: "transcribe" \| "translate" }` |
| `audio_chunk` / `translation_chunk`           | Client → Server | Forward one WAV audio frame, `{ audio: "<base64>" }`                                     |
| `transcription_result` / `translation_result` | Server → Client | A finalized piece of text, `{ text: "..." }`                                             |
| `status` / `error`                            | Server → Client | Connection status and error messages                                                     |

## **Available Options**

### Source Language

`API_LANGUAGE` in `config.py` accepts any Saaras v3 BCP-47 language code, or `"unknown"` to auto-detect. See [Specify Language Codes](/api/api-guides-tutorials/speech-to-text/how-to/specify-language-codes) for the full list of supported codes.

### Output Modes

This demo only wires up `transcribe` and `translate`, but the underlying Streaming API also supports `verbatim`, `translit`, and `codemix`. See [Streaming Speech-to-Text](/api/api-guides-tutorials/speech-to-text/streaming-api#supported-modes-saaras-v3) for what each mode returns.

## **Customization**

### Tune voice activity detection

Pass `high_vad_sensitivity=True` when opening the connection in `StreamingSession._session` for a shorter silence boundary (\~64ms instead of 1s at 16kHz), which makes segments finalize faster at a small cost to context:

```python
conn = client.speech_to_text_streaming.connect(
    language_code=config.API_LANGUAGE,
    model="saaras:v4",
    high_vad_sensitivity=True,
)
```

### Add speech start/end signals

Pass `vad_signals=True` to receive `START_SPEECH` / `END_SPEECH` events alongside transcripts, useful if you want to show a "listening" indicator in the UI rather than only finalized text. See [Streaming Speech-to-Text](/api/api-guides-tutorials/speech-to-text/streaming-api#response-types) for the message shapes.

### Transcribe a live microphone instead of a video file

Swap `video.captureStream()` in `templates/index.html` for `navigator.mediaDevices.getUserMedia({ audio: true })`. The rest of the pipeline, resampling, WAV encoding, and the Socket.IO events, stays the same, since it already operates on a generic `MediaStream`.

### Change the chunk interval

The browser buffers and sends audio roughly once a second (`currentTime - lastChunkTime >= 1000` in `templates/index.html`). Shorter intervals lower latency at the cost of more frequent small requests; longer intervals do the opposite.

## **Tips & Best Practices**

* **Keep the sample rate consistent**: The demo resamples to 16kHz before encoding, matching the rate it passes to `connect()`. If you change one, change the other, mismatched sample rates degrade transcription quality.
* **One connection per mode, not per chunk**: `StreamingSession` opens a single long-lived connection and streams every frame into it. Opening a new connection per audio chunk would add connection-setup latency to every single frame.
* **Independent buffers per mode**: Transcription and translation each keep their own `audioChunks` buffer in the browser, so running both at once doesn't mean one mode steals frames meant for the other.
* **Handle disconnects**: This demo doesn't automatically reconnect on a dropped WebSocket. For production use, add reconnect-with-backoff on close codes `1006`/`1011`, see [Handling Disconnects](/api/api-guides-tutorials/speech-to-text/streaming-api#handling-disconnects).
* **Restrict CORS before deploying**: `CORS_ALLOWED_ORIGINS = "*"` in `config.py` is fine for local testing only.

## **Error Handling**

You may encounter these errors while using the underlying API:

* **403 Forbidden** (`invalid_api_key_error`), invalid API key. Use a valid key from the [Sarvam AI Dashboard](https://dashboard.sarvam.ai/).
* **429 Too Many Requests** (`rate_limit_exceeded_error`), too many concurrent streaming connections or requests. Check [Credits & Rate Limits](https://docs.sarvam.ai/api/getting-started/ratelimits) for streaming concurrency limits.
* **WebSocket close `1006` / `1011`**, abnormal closure or server error. Reconnect with exponential backoff rather than retrying immediately, see [Handling Disconnects](/api/api-guides-tutorials/speech-to-text/streaming-api#handling-disconnects).
* **WebSocket close `4xxx`**, application-specific, usually auth or quota. Read the close reason before retrying; blind-retrying an auth failure will just fail again.

For the full error-code table and SDK exception reference, see [Errors & Troubleshooting](https://docs.sarvam.ai/api/getting-started/errors-troubleshooting).

## **Additional Resources**

* **Documentation**: [docs.sarvam.ai](https://docs.sarvam.ai)
* **Related APIs**: [Streaming Speech-to-Text](/api/api-guides-tutorials/speech-to-text/streaming-api) · [Specify Language Codes](/api/api-guides-tutorials/speech-to-text/how-to/specify-language-codes)
* **Full source**: [Live\_Video\_Transcription on GitHub](https://github.com/sarvamai/sarvam-ai-cookbook/tree/main/examples/Live_Video_Transcription)
* **Community**: [Join the Discord Community](https://discord.com/invite/5rAsykttcs)

## **Final Notes**

* Keep your API key secure, prefer environment variables over hardcoding.
* This demo keeps one persistent connection per client per mode rather than reconnecting per audio chunk. That's what keeps latency low.
* Add reconnect logic and restrict CORS before taking this beyond local experimentation.

**Keep Building!** 🚀

> Build a real-time transcription and translation demo with Sarvam AI's Streaming Speech-to-Text API, Flask-SocketIO, and the browser's Web Audio API.