Build Your First Voice Agent using Exotel

Overview

This guide shows you how to build a real-time voice agent that answers phone calls. It uses Exotel for telephony, Pipecat to orchestrate the real-time audio pipeline, and Sarvam AI for speech-to-text, the LLM, and text-to-speech. It’s a great starting point for IVR replacements, customer support lines, and conversational phone agents in Indian languages.

What You’ll Build

A voice agent that can:

• Answer inbound phone calls on an Exotel number
• Listen to callers speaking, in multiple Indian languages
• Understand and process what they say
• Respond back in a natural-sounding voice, over the phone

Quick Overview

1. Get an API key (Sarvam). Exotel needs no API credentials for this flow.
2. Install packages: pip install "pipecat-ai[websocket,sarvam]" python-dotenv
3. Create a .env file with your API key
4. Write about 80 lines of Python code
5. Point an Exotel Voicebot Applet at your agent’s WebSocket URL
6. Call your Exotel number

Quick Start

1. Prerequisites

• Python 3.9 or higher
• ngrok, to expose your local server during development
• An Exotel account with voice streaming enabled and a provisioned ExoPhone
• A Sarvam AI API key from your dashboard

2. Install Dependencies

$
pip install "pipecat-ai[websocket,sarvam]" python-dotenv loguru

3. Create Environment File

Create a file named .env in your project folder:

SARVAM_API_KEY=sk_xxxxxxxxxxxxxxxxxxxxxxxx

Replace the placeholder with your real key from the Sarvam dashboard.

4. Write Your Agent

Create agent.py:

1
import os
2
from dotenv import load_dotenv
3
from loguru import logger
4
from pipecat.frames.frames import LLMRunFrame
5
from pipecat.pipeline.pipeline import Pipeline
6
from pipecat.pipeline.runner import PipelineRunner
7
from pipecat.pipeline.task import PipelineParams, PipelineTask
8
from pipecat.processors.aggregators.llm_context import LLMContext
9
from pipecat.processors.aggregators.llm_response_universal import (
10
    LLMContextAggregatorPair,
11
)
12
from pipecat.runner.types import RunnerArguments
13
from pipecat.runner.utils import create_transport
14
from pipecat.services.sarvam.stt import SarvamSTTService
15
from pipecat.services.sarvam.tts import SarvamTTSService
16
from pipecat.services.sarvam.llm import SarvamLLMService
17
from pipecat.transports.websocket.fastapi import FastAPIWebsocketParams
18
19
load_dotenv(override=True)
20
21
async def bot(runner_args: RunnerArguments):
22
    """Main bot entry point."""
23
24
    # create_transport auto-detects Exotel's WebSocket handshake and builds
25
    # the matching ExotelFrameSerializer using the stream and call identifiers
26
    # Exotel sends when it connects. No Exotel credentials are needed here.
27
    transport = await create_transport(
28
        runner_args,
29
        {
30
            "exotel": lambda: FastAPIWebsocketParams(
31
                audio_in_enabled=True, audio_out_enabled=True
32
            ),
33
        },
34
    )
35
36
    # Initialize AI services
37
    stt = SarvamSTTService(api_key=os.getenv("SARVAM_API_KEY"))
38
    tts = SarvamTTSService(api_key=os.getenv("SARVAM_API_KEY"))
39
    llm = SarvamLLMService(
40
        api_key=os.getenv("SARVAM_API_KEY"),
41
        settings=SarvamLLMService.Settings(model="sarvam-105b"),
42
    )
43
44
    # Set up conversation context
45
    messages = [
46
        {
47
            "role": "system",
48
            "content": "You are a friendly AI phone assistant. Keep your responses brief and conversational.",
49
        },
50
    ]
51
    context = LLMContext(messages)
52
    context_aggregator = LLMContextAggregatorPair(context)
53
54
    # Build pipeline
55
    pipeline = Pipeline(
56
        [
57
            transport.input(),
58
            stt,
59
            context_aggregator.user(),
60
            llm,
61
            tts,
62
            transport.output(),
63
            context_aggregator.assistant(),
64
        ]
65
    )
66
67
    # Exotel Voice Streaming sends and receives 8kHz mono audio
68
    task = PipelineTask(
69
        pipeline,
70
        params=PipelineParams(
71
            audio_in_sample_rate=8000,
72
            audio_out_sample_rate=8000,
73
        ),
74
    )
75
76
    @transport.event_handler("on_client_connected")
77
    async def on_client_connected(transport, client):
78
        logger.info("Caller connected")
79
        messages.append(
80
            {"role": "system", "content": "Greet the caller and briefly introduce yourself."}
81
        )
82
        await task.queue_frames([LLMRunFrame()])
83
84
    @transport.event_handler("on_client_disconnected")
85
    async def on_client_disconnected(transport, client):
86
        logger.info("Caller disconnected")
87
        await task.cancel()
88
89
    runner = PipelineRunner(handle_sigint=runner_args.handle_sigint)
90
    await runner.run(task)
91
92
if __name__ == "__main__":
93
    from pipecat.runner.run import main
94
    main()

5. Configure Exotel to Reach Your Agent

Exotel is different from Twilio here: it doesn’t fetch an XML webhook at call time. Instead, you paste your WebSocket URL directly into a Voicebot Applet inside a call flow, and Exotel opens that connection itself.

Start ngrok:

$
ngrok http 7860

ngrok prints a forwarding URL that looks like https://xxxx.ngrok-free.app. Turn it into a wss:// URL for the next step.

Build the App Bazaar flow:

1. In the Exotel dashboard, go to App Bazaar
2. Create a new custom app (or edit an existing one)
3. Add a Voicebot applet (not Stream or Passthru), and set its URL field to wss://xxxx.ngrok-free.app/ws
4. Add a Hangup applet right after it, so the flow is: Call Start → Voicebot Applet → Hangup Applet
5. Save the flow

Assign the flow to your number:

1. Go to ExoPhones, and select your number
2. Under Installed App, choose the flow you just built
3. Confirm the change when prompted

6. Run Your Agent

$
python agent.py --transport exotel

No --proxy flag is needed here since Exotel doesn’t fetch a webhook, it just connects straight to /ws.

7. Test Your Agent

Call your Exotel number from any phone. Your voice agent will pick up and start the conversation.

Customization Examples

Example 1: Hindi Voice Agent

1
stt = SarvamSTTService(
2
    api_key=os.getenv("SARVAM_API_KEY"),
3
    settings=SarvamSTTService.Settings(
4
        model="saaras:v3",
5
        language="hi-IN",  # Hindi
6
    ),
7
    mode="transcribe",
8
)
9
10
tts = SarvamTTSService(
11
    api_key=os.getenv("SARVAM_API_KEY"),
12
    settings=SarvamTTSService.Settings(
13
        model="bulbul:v3",
14
        voice="simran",  # Or: priya, ishita, kavya, aditya, anand, rohan
15
        target_language_code="hi-IN",
16
    ),
17
)
18
19
llm = SarvamLLMService(
20
    api_key=os.getenv("SARVAM_API_KEY"),
21
    settings=SarvamLLMService.Settings(model="sarvam-105b"),
22
)

Example 2: Tamil Voice Agent

1
stt = SarvamSTTService(
2
    api_key=os.getenv("SARVAM_API_KEY"),
3
    settings=SarvamSTTService.Settings(
4
        model="saaras:v3",
5
        language="ta-IN",
6
    ),
7
    mode="transcribe",
8
)
9
10
tts = SarvamTTSService(
11
    api_key=os.getenv("SARVAM_API_KEY"),
12
    settings=SarvamTTSService.Settings(
13
        model="bulbul:v3",
14
        voice="shubh",
15
        target_language_code="ta-IN",
16
    ),
17
)
18
19
llm = SarvamLLMService(
20
    api_key=os.getenv("SARVAM_API_KEY"),
21
    settings=SarvamLLMService.Settings(model="sarvam-105b"),
22
)

Example 3: Multilingual Agent (Auto-detect)

1
# Auto-detect the caller's language
2
stt = SarvamSTTService(
3
    api_key=os.getenv("SARVAM_API_KEY"),
4
    settings=SarvamSTTService.Settings(
5
        model="saaras:v3",
6
        language="unknown",  # Auto-detects language
7
    ),
8
    mode="transcribe",
9
)
10
11
tts = SarvamTTSService(
12
    api_key=os.getenv("SARVAM_API_KEY"),
13
    settings=SarvamTTSService.Settings(
14
        model="bulbul:v3",
15
        voice="anand",
16
        target_language_code="en-IN",
17
    ),
18
)
19
20
llm = SarvamLLMService(
21
    api_key=os.getenv("SARVAM_API_KEY"),
22
    settings=SarvamLLMService.Settings(model="sarvam-105b"),
23
)

Example 4: Speech-to-English Agent (Saaras)

Saarika transcribes speech to text in the same language. Saaras translates speech directly to English text. Use Saaras when the caller speaks an Indian language but you want to process and respond in English.

1
# Caller speaks Hindi, Saaras converts it to English, the LLM processes it,
2
# and the agent responds in English.
3
4
stt = SarvamSTTService(
5
    api_key=os.getenv("SARVAM_API_KEY"),
6
    settings=SarvamSTTService.Settings(model="saaras:v3"),
7
    mode="translate",  # Speech-to-English translation
8
)
9
10
tts = SarvamTTSService(
11
    api_key=os.getenv("SARVAM_API_KEY"),
12
    settings=SarvamTTSService.Settings(
13
        model="bulbul:v3",
14
        voice="aditya",
15
        target_language_code="en-IN",
16
    ),
17
)
18
19
llm = SarvamLLMService(
20
    api_key=os.getenv("SARVAM_API_KEY"),
21
    settings=SarvamLLMService.Settings(model="sarvam-105b"),
22
)

Available Options

Language Codes

Speaker Voices (Bulbul v3)

Male (23): Shubh (default), Aditya, Rahul, Rohan, Amit, Dev, Ratan, Varun, Manan, Sumit, Kabir, Aayan, Ashutosh, Advait, Anand, Tarun, Sunny, Mani, Gokul, Vijay, Mohit, Rehan, Soham

Female (14): Ritu, Priya, Neha, Pooja, Simran, Kavya, Ishita, Shreya, Roopa, Tanya, Shruti, Suhani, Kavitha, Rupali

TTS Additional Parameters

You can customize the TTS service with additional parameters:

1
tts = SarvamTTSService(
2
    api_key=os.getenv("SARVAM_API_KEY"),
3
    settings=SarvamTTSService.Settings(
4
        model="bulbul:v3",
5
        voice="shubh",
6
        target_language_code="en-IN",
7
        pace=1.0,  # Range: 0.5 to 2.0
8
    ),
9
)

Understanding the Call Flow

1. Caller dials your Exotel number. The App Bazaar flow reaches the Voicebot Applet, and Exotel opens a WebSocket directly to your agent — there’s no webhook fetch in between.
2. Transport Input: Your Pipecat transport receives the caller’s audio over the WebSocket, and hands it to STT.
3. STT (Speech-to-Text): Converts audio to text using Sarvam’s Saarika or Saaras, and the context aggregator adds it to the conversation context.
4. LLM: Generates a response using Sarvam.
5. TTS (Text-to-Speech): Converts the response to audio using Sarvam’s Bulbul.
6. Transport Output: Streams the audio back over the WebSocket, and Exotel plays it to the caller, while the context aggregator saves the assistant’s response to context. The Hangup Applet then ends the call once your agent closes the WebSocket.

Pro Tips

• Use language="unknown" to automatically detect the caller’s language. This works well for multilingual support lines.
• Sarvam’s models understand code-mixing, so your agent can naturally handle Hinglish, Tanglish, and other mixed languages, which is common on real support calls.
• Use sarvam-30b for faster responses, or sarvam-105b for more complex conversations.

Troubleshooting

Call connects but there’s no audio. Confirm the Voicebot Applet’s URL uses wss://, not ws:// or https://, and that it points at your current ngrok URL. Free ngrok URLs change on every restart.

Call doesn’t reach your agent at all. Double-check the flow order in App Bazaar: Call Start → Voicebot Applet → Hangup Applet. A missing or misordered applet is the most common cause.

API key errors. Make sure SARVAM_API_KEY is in your .env file and that the file sits in the same directory as agent.py.

Module not found. Re-run the install command for your operating system (see Step 2 above).

Poor transcription. Try language="unknown" for auto-detection, or set the exact language code (en-IN, hi-IN, and so on) if you already know it.

Choppy or robotic audio. Make sure audio_in_sample_rate and audio_out_sample_rate in PipelineParams match the rate Exotel is actually streaming: 8000 by default, or whatever you set via ?sample-rate= on the Voicebot Applet’s WebSocket URL (Exotel also supports 16000 and 24000). A mismatch causes distorted playback.

Call disconnects a few seconds after connecting. Exotel expects your WebSocket server to accept the connection within about 10 seconds. Make sure ngrok and agent.py are both already running before you place the call.

Long call cuts off unexpectedly. Exotel caps a single streaming session at around 60 minutes (this can vary by plan) — for calls that need to run longer, design your agent to end the conversation before that limit.

Additional Resources

• Sarvam AI Documentation
• Pipecat Documentation
• Pipecat Sarvam LLM Service
• Exotel Stream and Voicebot Applet Guide
• Exotel Updated Extension Guide (timeouts, session limits, Passthru)
• Exotel Dashboard

Need Help?

• Sarvam Support: developer@sarvam.ai
• Community: Join the Discord Community

Happy Building!