Build a LiveKit voice agent with Sarvam

Want a production voice agent without assembling the stack yourself? Sarvam Voice Agents is a managed platform with speech recognition, reasoning, Indian-language voices, telephony, testing, and analytics included, from ₹3 per minute. This guide shows how to build your own pipeline on the APIs instead.

Overview

This guide demonstrates how to build a real-time voice agent that can listen, understand, and respond naturally using LiveKit for real-time communication and Sarvam AI for speech processing. Perfect for building voice assistants, customer support bots, and conversational AI applications for Indian languages.

Use the Sarvam voice-agents skill to generate correct LiveKit integration code from your AI coding assistant:

npx skills add sarvamai/skills --skill voice-agents

See Agent Skills for the full list.

What You’ll Build

A voice agent that can:

Listen to users speaking (in multiple Indian languages!)
Understand and process their requests
Respond back in natural-sounding voices

Quick Overview

Get API keys (LiveKit, Sarvam)
Install packages: pip install "livekit-agents[sarvam]>=1.8.6,<2" python-dotenv
Create .env file with your API keys
Write ~40 lines of Python code
Run: python agent.py dev
Test: python agent.py console

Quick Start

1. Prerequisites

Python 3.10 or higher (required by livekit-agents 1.8.6 and later)
API keys from:
- LiveKit Cloud (free account)
- Sarvam AI (get API key from dashboard)

2. Install Dependencies

macOS/Linux

Windows

pip install "livekit-agents[sarvam]>=1.8.6,<2" python-dotenv

3. Create Environment File

Create a file named .env in your project folder and add your API keys:

LIVEKIT_URL=wss://your-project-xxxxx.livekit.cloud
LIVEKIT_API_KEY=APIxxxxxxxxxxxxx
LIVEKIT_API_SECRET=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
SARVAM_API_KEY=sk_xxxxxxxxxxxxxxxxxxxxxxxx

Replace the values with your actual API keys.

4. Write Your Agent

Create agent.py:

import logging
from dotenv import load_dotenv
from livekit.agents import JobContext, WorkerOptions, cli
from livekit.agents.voice import Agent, AgentSession
from livekit.plugins import sarvam
# Load environment variables
load_dotenv()
# Set up logging
logger = logging.getLogger("voice-agent")
logger.setLevel(logging.INFO)
class VoiceAgent(Agent):
    def __init__(self) -> None:
        super().__init__(
            # Your agent's personality and instructions
            instructions="""
                You are a helpful voice assistant.
                Be friendly, concise, and conversational.
                Speak naturally as if you're having a real conversation.
            """,
            
            # Saaras v4 STT - Converts speech to text
            stt=sarvam.STT(
                language="unknown",  # Auto-detect language, or use "en-IN", "hi-IN", etc.
                model="saaras:v4",
                mode="transcribe"
            ),
            
            # Sarvam LLM - The "brain" that processes and generates responses
            llm=sarvam.LLM(model="sarvam-105b-conversations"),
            
            # Bulbul TTS - Converts text to speech
            tts=sarvam.TTS(
                target_language_code="en-IN",
                model="bulbul:v4-flash",
                speaker="simran_en_customer"  # v4 Flash persona ID; see the Voices page for the full list
            ),
        )
    
    async def on_enter(self):
        """Called when user joins - agent starts the conversation"""
        self.session.generate_reply()
async def entrypoint(ctx: JobContext):
    """Main entry point - LiveKit calls this when a user connects"""
    logger.info(f"User connected to room: {ctx.room.name}")
    
    # Create and start the agent session
    session = AgentSession(
        vad=None,  # Sarvam STT runs its own VAD; leaving vad out installs LiveKit's Silero VAD
        turn_handling={"turn_detection": "stt"},
    )
    await session.start(
        agent=VoiceAgent(),
        room=ctx.room
    )
if __name__ == "__main__":
    # Run the agent
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

5. Run Your Agent

python agent.py dev

6. Test Your Agent

In a new terminal, run:

python agent.py console

That’s it! You’ve built your first voice agent!

Customization Examples

Example 1: Hindi Voice Agent

stt=sarvam.STT(
    language="hi-IN",  # Hindi
    model="saaras:v4",
    mode="transcribe"
),
tts=sarvam.TTS(
    target_language_code="hi-IN",
    model="bulbul:v4-flash",
    speaker="simran_hi_conversation"  # Or another Hindi persona, e.g. ritu_hi_customer, aditya_hi_conversational
)

Example 2: Tamil Voice Agent

stt=sarvam.STT(language="ta-IN", model="saaras:v4", mode="transcribe"),
tts=sarvam.TTS(
    target_language_code="ta-IN",
    model="bulbul:v4-flash",
    speaker="gokul_ta_narration"
)

Example 3: Multilingual Agent (Auto-detect)

stt=sarvam.STT(language="unknown", model="saaras:v4", mode="transcribe"),  # Auto-detects language
tts=sarvam.TTS(target_language_code="en-IN", model="bulbul:v4-flash", speaker="deven_en_conversation")

Example 4: Speech-to-English Agent (Saaras)

Difference: Saaras v4 handles both transcription (same-language output) and translation (English output) via the mode parameter. Use mode="translate" when user speaks Indian languages but you want to process/respond in English.

# User speaks Hindi → Saaras converts to English → LLM processes → Responds in English
stt=sarvam.STT(model="saaras:v4", mode="translate"),  # Speech-to-English translation
llm=sarvam.LLM(model="sarvam-105b-conversations"),
tts=sarvam.TTS(target_language_code="en-IN", model="bulbul:v4-flash", speaker="dev_en_conversational")

Note: Saaras v4 with mode="translate" automatically detects the source language (Hindi, Tamil, etc.) and translates spoken content directly to English text, making Indian language speech comprehensible to English-based LLMs.

Available Options

Language Codes

Language	Code
English (India)	`en-IN`
Hindi	`hi-IN`
Bengali	`bn-IN`
Tamil	`ta-IN`
Telugu	`te-IN`
Gujarati	`gu-IN`
Kannada	`kn-IN`
Malayalam	`ml-IN`
Marathi	`mr-IN`
Punjabi	`pa-IN`
Odia	`od-IN`
Auto-detect	`unknown`

Speaker Voices (Bulbul v4 Flash)

bulbul:v4-flash takes persona IDs in the form name_language_style, for example simran_hi_conversation or dev_en_conversational. Pick a persona whose language matches target_language_code. v3 short names such as priya or aditya are rejected. Listen to every persona on the Voices page.

To use the short-name v3 voices instead, set model="bulbul:v3".

Pro Tips

Use language="unknown" to automatically detect the language. Great for multilingual scenarios!
Sarvam’s models understand code-mixing - your agent can naturally handle Hinglish, Tanglish, and other mixed languages.

Best Practices

When using Sarvam AI plugins with LiveKit, follow these recommendations for optimal performance:

1. Pass `vad=None` to AgentSession

Sarvam STT runs voice activity detection on the server, so LiveKit doesn’t need its own. Pass vad=None explicitly: on livekit-agents 1.6.1 and later, leaving vad out installs LiveKit’s bundled Silero VAD, and two VADs can disagree about where a turn ends.

# ❌ Avoid both of these
session = AgentSession(vad=silero.VAD.load())
session = AgentSession()  # installs LiveKit's bundled Silero VAD
# ✅ Do this instead
session = AgentSession(vad=None)

2. Leave Out `flush_signal`

You don’t need flush_signal=True. Sarvam’s server ends each turn with its own VAD. flush_signal=True only sends a flush message when your code flushes the stream itself, and AgentSession never flushes Sarvam STT, so inside an agent the setting has no effect. (Before livekit-plugins-sarvam 1.8.9, the plugin also sent a flush at every end of speech; it no longer does.)

stt=sarvam.STT(
    language="unknown",
    model="saaras:v4",
    mode="transcribe",
)

3. Set Turn Detection to STT

Set turn_detection to "stt" inside turn_handling. This ensures turn detection is handled by the Sarvam plugin, which emits start and end of speech signals. The flat turn_detection= keyword still works but is deprecated.

session = AgentSession(
    vad=None,
    turn_handling={"turn_detection": "stt"},
)

4. Configure Min Endpointing Delay

Set the endpointing min_delay to 0.07 in turn_handling (it replaces the deprecated min_endpointing_delay keyword). The Sarvam STT plugin has a processing latency of approximately 70ms. This setting ensures the agent transitions to the next pipeline step (LLM) as soon as STT finishes processing, minimizing response delay.

session = AgentSession(
    vad=None,
    turn_handling={
        "turn_detection": "stt",
        "endpointing": {"min_delay": 0.07},
    },
)

Complete Optimized Example

Here’s a complete example incorporating all best practices:

# STT: the plugin handles speech events and flushing itself
stt=sarvam.STT(
    language="unknown",
    model="saaras:v4",
    mode="transcribe",
)
# AgentSession with optimized settings (Sarvam STT handles VAD)
session = AgentSession(
    vad=None,
    turn_handling={
        "turn_detection": "stt",
        "endpointing": {"min_delay": 0.07},
    },
)

Troubleshooting

API key errors: Check that all keys are in your .env file and the file is in the same directory as your script.

Module not found: Run the installation command again based on your operating system (see Step 2 above).

Poor transcription: Try language="unknown" for auto-detection, or specify the correct language code (en-IN, hi-IN, etc.).

Going to Production?

This guide gets a demo agent talking. Before you ship it, see LiveKit in Production for the defaults you need to change: VAD and turn detection, endpointing delays, barge-in tuning, latency measurement, and a full caveats list derived from the Sarvam LiveKit plugin source.

Additional Resources

Need Help?

Sarvam Support: support@sarvam.ai
Community: Join the Discord Community

Happy Building!