Skip to navigation

Build a Pipecat voice agent with Sarvam

View as Markdown

Want a production voice agent without assembling the stack yourself? Sarvam Voice Agents is a managed platform with speech recognition, reasoning, Indian-language voices, telephony, testing, and analytics included, from ₹3 per minute. This guide shows how to build your own pipeline on the APIs instead.

Overview

This guide demonstrates how to build a real-time voice agent that can listen, understand, and respond naturally using Pipecat for real-time communication and Sarvam AI for speech processing. Perfect for building voice assistants, customer support bots, and conversational AI applications for Indian languages.

Use the Sarvam voice-agents skill to generate correct Pipecat integration code from your AI coding assistant:

npx skills add sarvamai/skills --skill voice-agents

See Agent Skills for the full list.

What You’ll Build

A voice agent that can:

  • Listen to users speaking (in multiple Indian languages!)
  • Understand and process their requests
  • Respond back in natural-sounding voices

Quick Overview

  1. Get API keys (Sarvam)
  2. Install packages: pip install "pipecat-ai[daily,sarvam,runner]>=1.14" python-dotenv
  3. Create .env file with your API keys
  4. Write ~80 lines of Python code
  5. Run with appropriate transport

Quick Start

1. Prerequisites

  • Python 3.11 or higher (required by pipecat-ai)
  • API keys from:

2. Install Dependencies

pip install "pipecat-ai[daily,sarvam,runner]>=1.14" python-dotenv loguru

3. Create Environment File

Create a file named .env in your project folder and add your API keys:

SARVAM_API_KEY=sk_xxxxxxxxxxxxxxxxxxxxxxxx

Replace the values with your actual API keys.

4. Write Your Agent

Create agent.py:

import os
from dotenv import load_dotenv
from loguru import logger
from pipecat.frames.frames import LLMRunFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.worker import PipelineWorker
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
LLMContextAggregatorPair,
)
from pipecat.runner.types import RunnerArguments
from pipecat.runner.utils import create_transport
from pipecat.services.sarvam.stt import SarvamSTTService
from pipecat.services.sarvam.tts import SarvamTTSService
from pipecat.services.sarvam.llm import SarvamLLMService
from pipecat.transports.base_transport import TransportParams
from pipecat.transports.daily.transport import DailyParams
from pipecat.workers.runner import WorkerRunner
load_dotenv(override=True)
async def bot(runner_args: RunnerArguments):
"""Main bot entry point."""
# Create transport (supports both Daily and WebRTC)
transport = await create_transport(
runner_args,
{
"daily": lambda: DailyParams(audio_in_enabled=True, audio_out_enabled=True),
"webrtc": lambda: TransportParams(
audio_in_enabled=True, audio_out_enabled=True
),
},
)
# Initialize AI services
stt = SarvamSTTService(
api_key=os.getenv("SARVAM_API_KEY"),
settings=SarvamSTTService.Settings(vad_signals=True), # Sarvam detects when the user stops speaking
)
tts = SarvamTTSService(
api_key=os.getenv("SARVAM_API_KEY"),
settings=SarvamTTSService.Settings(
model="bulbul:v4-flash", # latest; omit model in raw API → bulbul:v3
voice="simran_en_customer", # v4 persona ID — see Voices guide
language="en-IN",
),
)
llm = SarvamLLMService(
api_key=os.getenv("SARVAM_API_KEY"),
settings=SarvamLLMService.Settings(model="sarvam-105b"),
)
# Set up conversation context
messages = [
{
"role": "system",
"content": "You are a friendly AI assistant. Keep your responses brief and conversational.",
},
]
context = LLMContext(messages)
context_aggregator = LLMContextAggregatorPair(context)
# Build pipeline
pipeline = Pipeline(
[
transport.input(),
stt,
context_aggregator.user(),
llm,
tts,
transport.output(),
context_aggregator.assistant(),
]
)
worker = PipelineWorker(pipeline)
runner = WorkerRunner(handle_sigint=runner_args.handle_sigint)
await runner.add_workers(worker)
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
logger.info("Client connected")
messages.append(
{"role": "system", "content": "Say hello and briefly introduce yourself."}
)
await worker.queue_frames([LLMRunFrame()])
@transport.event_handler("on_client_disconnected")
async def on_client_disconnected(transport, client):
logger.info("Client disconnected")
await runner.cancel()
await runner.run()
if __name__ == "__main__":
from pipecat.runner.run import main
main()

5. Run Your Agent

For Daily transport:

python agent.py

The agent will create a Daily room and provide you with a URL to join.

6. Test Your Agent

Open the provided Daily room URL in your browser and start speaking. Your voice agent will listen and respond!


Customization Examples

Example 1: Hindi Voice Agent

# Initialize AI services with Hindi support
stt = SarvamSTTService(
api_key=os.getenv("SARVAM_API_KEY"),
settings=SarvamSTTService.Settings(
model="saaras:v4",
vad_signals=True,
language="hi-IN", # Hindi
),
mode="transcribe",
)
tts = SarvamTTSService(
api_key=os.getenv("SARVAM_API_KEY"),
settings=SarvamTTSService.Settings(
model="bulbul:v4-flash",
language="hi-IN",
voice="ishita_hi_devotional", # v4 persona; v3 short names need model="bulbul:v3"
),
)
llm = SarvamLLMService(
api_key=os.getenv("SARVAM_API_KEY"),
settings=SarvamLLMService.Settings(model="sarvam-105b"),
)

Example 2: Tamil Voice Agent

stt = SarvamSTTService(
api_key=os.getenv("SARVAM_API_KEY"),
settings=SarvamSTTService.Settings(
model="saaras:v4",
vad_signals=True,
language="ta-IN",
),
mode="transcribe",
)
tts = SarvamTTSService(
api_key=os.getenv("SARVAM_API_KEY"),
settings=SarvamTTSService.Settings(
model="bulbul:v4-flash",
language="ta-IN",
voice="gokul_ta_narration",
),
)
llm = SarvamLLMService(
api_key=os.getenv("SARVAM_API_KEY"),
settings=SarvamLLMService.Settings(model="sarvam-105b"),
)

Example 3: Multilingual Agent (Auto-detect)

# Auto-detect the user's language
stt = SarvamSTTService(
api_key=os.getenv("SARVAM_API_KEY"),
settings=SarvamSTTService.Settings(
model="saaras:v4",
vad_signals=True,
language="unknown", # Auto-detects language
),
mode="transcribe",
)
tts = SarvamTTSService(
api_key=os.getenv("SARVAM_API_KEY"),
settings=SarvamTTSService.Settings(
model="bulbul:v4-flash",
language="en-IN",
voice="simran_en_customer",
),
)
llm = SarvamLLMService(
api_key=os.getenv("SARVAM_API_KEY"),
settings=SarvamLLMService.Settings(model="sarvam-105b"),
)

Example 4: Speech-to-English Agent (Saaras)

Difference: mode="transcribe" converts speech to text in the same language, while mode="translate" translates speech directly to English text. Use translate mode when the user speaks Indian languages but you want to process/respond in English.

# User speaks Hindi → Saaras converts to English → LLM processes → Responds in English
stt = SarvamSTTService(
api_key=os.getenv("SARVAM_API_KEY"),
settings=SarvamSTTService.Settings(model="saaras:v4", vad_signals=True),
mode="translate", # Speech-to-English translation
)
tts = SarvamTTSService(
api_key=os.getenv("SARVAM_API_KEY"),
settings=SarvamTTSService.Settings(
model="bulbul:v4-flash",
language="en-IN",
voice="shubh_en_narration",
),
)
llm = SarvamLLMService(
api_key=os.getenv("SARVAM_API_KEY"),
settings=SarvamLLMService.Settings(model="sarvam-105b"),
)

Note: Saaras automatically detects the source language (Hindi, Tamil, etc.) and translates spoken content directly to English text, making Indian language speech comprehensible to English-based LLMs.


Available Options

Language Codes

LanguageCode
English (India)en-IN
Hindihi-IN
Bengalibn-IN
Tamilta-IN
Telugute-IN
Gujaratigu-IN
Kannadakn-IN
Malayalamml-IN
Marathimr-IN
Punjabipa-IN
Odiaod-IN
Auto-detectunknown

TTS model and voices

Recommended: model="bulbul:v4-flash" (latest, needs pipecat-ai 1.13.0 or later) with a persona ID in Settings.voice (for example simran_en_customer, ishita_hi_devotional). Browse the Bulbul v4 Flash catalog. If you omit model, Pipecat uses bulbul:v3; if you pick v4 Flash and omit voice, it uses shubh_en_narration_gentle.

Bulbul v3: use model="bulbul:v3" with short names such as shubh, priya, anand — see Voices for the full v3 list.

On current Pipecat versions, language and voice belong in SarvamTTSService.Settings. As bare constructor kwargs they are silently ignored, so the agent falls back to the default voice. See the Pipecat Production Guide for latency tuning.

TTS additional parameters

tts = SarvamTTSService(
api_key=os.getenv("SARVAM_API_KEY"),
settings=SarvamTTSService.Settings(
model="bulbul:v4-flash",
language="en-IN",
voice="simran_en_customer",
pace=1.0, # 0.5–2.0
pitch=0.0, # -0.5–0.5 (v4 Flash)
loudness=1.0, # 0.1–2.5 (v4 Flash)
min_buffer_size=30, # the minimum Sarvam accepts; lower = faster first audio (default 50)
),
sample_rate=24000, # match transport; streaming supports up to 24000 Hz
)

temperature (0.01–1.0) applies to bulbul:v3 only. bulbul:v4-flash ignores it, so Pipecat doesn’t send it and you can leave it out.


Understanding the Pipeline

Pipecat uses a pipeline architecture where data flows through a series of processors:

User Audio → STT → Context Aggregator → LLM → TTS → Audio Output
  1. Transport Input: Receives audio from the user
  2. STT (Speech-to-Text): Converts audio to text using Sarvam’s Saaras
  3. Context Aggregator (User): Adds user message to conversation context
  4. LLM: Generates response using Sarvam
  5. TTS (Text-to-Speech): Converts response to audio using Sarvam’s Bulbul v4 Flash (or v3)
  6. Transport Output: Sends audio back to the user
  7. Context Aggregator (Assistant): Saves assistant’s response to context

Pro Tips

  • Use language="unknown" to automatically detect the language. Great for multilingual scenarios!
  • Sarvam’s models understand code-mixing - your agent can naturally handle Hinglish, Tanglish, and other mixed languages.
  • Adjust pace to customize the voice delivery speed.
  • Use sarvam-105b for the LLM step.

Troubleshooting

API key errors: Check that all keys are in your .env file and the file is in the same directory as your script.

Module not found: Run the installation command again based on your operating system (see Step 2 above).

Poor transcription: Try language="unknown" for auto-detection, or specify the correct language code (en-IN, hi-IN, etc.).

Connection issues: Ensure you have a stable internet connection and the transport is properly configured.


Going to Production?

This guide gets a demo agent talking. Before you ship it, see the Pipecat Production Guide for the defaults you need to change: turn-taking strategies, latency budgets, barge-in tuning, turning off LLM reasoning for voice, reliability, and a full caveats checklist verified against the Sarvam Pipecat plugin source.


Additional Resources


Need Help?


Happy Building!