Build a Voice Agent or WhatsApp Bot using Twilio

Overview

This guide shows you how to build a real-time conversational agent with Twilio and Sarvam AI, on either of two channels: a phone voice agent that answers inbound calls, or a WhatsApp bot that replies to text and voice messages. Both use Sarvam AI for speech-to-text, the LLM, and text-to-speech, and are a great starting point for IVR replacements, customer support lines, and conversational agents in Indian languages.

What You’ll Build

Pick the channel that matches your use case:

Part 1: Phone Call Agent (Telephony)

A voice agent that can:

• Answer inbound phone calls on a Twilio number
• Listen to callers speaking, in multiple Indian languages
• Understand and process what they say
• Respond back in a natural-sounding voice, over the phone

Quick Overview

1. Get API keys (Twilio, Sarvam)
2. Install packages: pip install "pipecat-ai[websocket,sarvam]" python-dotenv
3. Create a .env file with your API keys
4. Write about 80 lines of Python code
5. Point a Twilio phone number at your agent using TwiML
6. Call your Twilio number

Quick Start

1. Prerequisites

• Python 3.9 or higher
• ngrok, to expose your local server during development
• Accounts and keys from:
  • Twilio: an Account SID, an Auth Token, and a phone number with voice capability
  • Sarvam AI: an API key from your dashboard

2. Install Dependencies

$
pip install "pipecat-ai[websocket,sarvam]" python-dotenv loguru

3. Create Environment File

Create a file named .env in your project folder:

SARVAM_API_KEY=sk_xxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_AUTH_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Then replace each placeholder with your real credentials from the Sarvam dashboard and the Twilio Console.

4. Write Your Agent

Create agent.py:

1
import os
2
from dotenv import load_dotenv
3
from loguru import logger
4
from pipecat.frames.frames import LLMRunFrame
5
from pipecat.pipeline.pipeline import Pipeline
6
from pipecat.pipeline.runner import PipelineRunner
7
from pipecat.pipeline.task import PipelineParams, PipelineTask
8
from pipecat.processors.aggregators.llm_context import LLMContext
9
from pipecat.processors.aggregators.llm_response_universal import (
10
    LLMContextAggregatorPair,
11
)
12
from pipecat.runner.types import RunnerArguments
13
from pipecat.runner.utils import create_transport
14
from pipecat.services.sarvam.stt import SarvamSTTService
15
from pipecat.services.sarvam.tts import SarvamTTSService
16
from pipecat.services.sarvam.llm import SarvamLLMService
17
from pipecat.transports.websocket.fastapi import FastAPIWebsocketParams
18
19
load_dotenv(override=True)
20
21
async def bot(runner_args: RunnerArguments):
22
    """Main bot entry point."""
23
24
    # create_transport auto-detects Twilio's WebSocket handshake and builds
25
    # the matching TwilioFrameSerializer, using TWILIO_ACCOUNT_SID and
26
    # TWILIO_AUTH_TOKEN from the environment.
27
    transport = await create_transport(
28
        runner_args,
29
        {
30
            "twilio": lambda: FastAPIWebsocketParams(
31
                audio_in_enabled=True, audio_out_enabled=True
32
            ),
33
        },
34
    )
35
36
    # Initialize AI services
37
    stt = SarvamSTTService(api_key=os.getenv("SARVAM_API_KEY"))
38
    tts = SarvamTTSService(api_key=os.getenv("SARVAM_API_KEY"))
39
    llm = SarvamLLMService(
40
        api_key=os.getenv("SARVAM_API_KEY"),
41
        settings=SarvamLLMService.Settings(model="sarvam-105b"),
42
    )
43
44
    # Set up conversation context
45
    messages = [
46
        {
47
            "role": "system",
48
            "content": (
49
                "You are Sarvam AI Agent, a multilingual voice assistant powered by Sarvam AI. "
50
                "You can understand and respond fluently in English and 10+ Indian languages. "
51
                "Keep your responses brief and conversational."
52
            ),
53
        },
54
    ]
55
    context = LLMContext(messages)
56
    context_aggregator = LLMContextAggregatorPair(context)
57
58
    # Build pipeline
59
    pipeline = Pipeline(
60
        [
61
            transport.input(),
62
            stt,
63
            context_aggregator.user(),
64
            llm,
65
            tts,
66
            transport.output(),
67
            context_aggregator.assistant(),
68
        ]
69
    )
70
71
    # Twilio Media Streams send and receive 8kHz mono audio
72
    task = PipelineTask(
73
        pipeline,
74
        params=PipelineParams(
75
            audio_in_sample_rate=8000,
76
            audio_out_sample_rate=8000,
77
        ),
78
    )
79
80
    @transport.event_handler("on_client_connected")
81
    async def on_client_connected(transport, client):
82
        logger.info("Caller connected")
83
        messages.append(
84
            {"role": "system", "content": "Greet the caller and briefly introduce yourself."}
85
        )
86
        await task.queue_frames([LLMRunFrame()])
87
88
    @transport.event_handler("on_client_disconnected")
89
    async def on_client_disconnected(transport, client):
90
        logger.info("Caller disconnected")
91
        await task.cancel()
92
93
    runner = PipelineRunner(handle_sigint=runner_args.handle_sigint)
94
    await runner.run(task)
95
96
if __name__ == "__main__":
97
    from pipecat.runner.run import main
98
    main()

5. Configure Twilio to Reach Your Agent

Twilio needs a public wss:// URL to stream call audio to. For local development, expose your agent with ngrok.

Start ngrok:

$
ngrok http 7860

ngrok prints a forwarding URL that looks like https://xxxx.ngrok-free.app. You’ll turn this into a wss:// URL in the next step.

Create a TwiML Bin:

1. Open the Twilio Console and search for TwiML Bins using the search bar at the top (or find it under Developer Tools in the sidebar)
2. Click Create new TwiML Bin, give it a friendly name, and paste the following, replacing the domain with your own ngrok URL:

1
<?xml version="1.0" encoding="UTF-8"?>
2
<Response>
3
  <Connect>
4
    <Stream url="wss://xxxx.ngrok-free.app/ws" />
5
  </Connect>
6
</Response>

3. Click Create to save the bin

Assign it to your phone number:

1. Go to Numbers & Senders, then Phone Numbers, and select your number
2. Open its configuration/voice settings tab
3. Find the setting for what happens when a call comes in, and set it to use your TwiML Bin (Twilio has labeled this option TwiML, TwiML Bin, or similar depending on your console version)
4. Save

6. Run Your Agent

$
python agent.py --transport twilio

This starts a local FastAPI server, via Pipecat’s development runner, that accepts Twilio’s Media Streams WebSocket connection at /ws.

7. Test Your Agent

Call your Twilio phone number from any phone. Your voice agent will pick up and start the conversation.

Customization Examples

Example 1: Hindi Voice Agent

1
stt = SarvamSTTService(
2
    api_key=os.getenv("SARVAM_API_KEY"),
3
    settings=SarvamSTTService.Settings(
4
        model="saaras:v3",
5
        language="hi-IN",  # Hindi
6
    ),
7
    mode="transcribe",
8
)
9
10
tts = SarvamTTSService(
11
    api_key=os.getenv("SARVAM_API_KEY"),
12
    settings=SarvamTTSService.Settings(
13
        model="bulbul:v3",
14
        voice="simran",  # Or: priya, ishita, kavya, aditya, anand, rohan
15
        language_code="hi-IN",
16
    ),
17
)
18
19
llm = SarvamLLMService(
20
    api_key=os.getenv("SARVAM_API_KEY"),
21
    settings=SarvamLLMService.Settings(model="sarvam-105b"),
22
)

Example 2: Tamil Voice Agent

1
stt = SarvamSTTService(
2
    api_key=os.getenv("SARVAM_API_KEY"),
3
    settings=SarvamSTTService.Settings(
4
        model="saaras:v3",
5
        language="ta-IN",
6
    ),
7
    mode="transcribe",
8
)
9
10
tts = SarvamTTSService(
11
    api_key=os.getenv("SARVAM_API_KEY"),
12
    settings=SarvamTTSService.Settings(
13
        model="bulbul:v3",
14
        voice="shubh",
15
        language_code="ta-IN",
16
    ),
17
)
18
19
llm = SarvamLLMService(
20
    api_key=os.getenv("SARVAM_API_KEY"),
21
    settings=SarvamLLMService.Settings(model="sarvam-105b"),
22
)

Example 3: Multilingual Agent (Auto-detect)

1
# Auto-detect the caller's language
2
stt = SarvamSTTService(
3
    api_key=os.getenv("SARVAM_API_KEY"),
4
    settings=SarvamSTTService.Settings(
5
        model="saaras:v3",
6
        language="unknown",  # Auto-detects language
7
    ),
8
    mode="transcribe",
9
)
10
11
tts = SarvamTTSService(
12
    api_key=os.getenv("SARVAM_API_KEY"),
13
    settings=SarvamTTSService.Settings(
14
        model="bulbul:v3",
15
        voice="anand",
16
        language_code="en-IN",
17
    ),
18
)
19
20
llm = SarvamLLMService(
21
    api_key=os.getenv("SARVAM_API_KEY"),
22
    settings=SarvamLLMService.Settings(model="sarvam-105b"),
23
)

Example 4: Speech-to-English Agent (Saaras)

Saarika transcribes speech to text in the same language. Saaras translates speech directly to English text. Use Saaras when the caller speaks an Indian language but you want to process and respond in English.

1
# Caller speaks Hindi, Saaras converts it to English, the LLM processes it,
2
# and the agent responds in English.
3
4
stt = SarvamSTTService(
5
    api_key=os.getenv("SARVAM_API_KEY"),
6
    settings=SarvamSTTService.Settings(model="saaras:v3"),
7
    mode="translate",  # Speech-to-English translation
8
)
9
10
tts = SarvamTTSService(
11
    api_key=os.getenv("SARVAM_API_KEY"),
12
    settings=SarvamTTSService.Settings(
13
        model="bulbul:v3",
14
        voice="aditya",
15
        language_code="en-IN",
16
    ),
17
)
18
19
llm = SarvamLLMService(
20
    api_key=os.getenv("SARVAM_API_KEY"),
21
    settings=SarvamLLMService.Settings(model="sarvam-105b"),
22
)

Available Options

Language Codes

Speaker Voices (Bulbul v3)

Male (23): Shubh (default), Aditya, Rahul, Rohan, Amit, Dev, Ratan, Varun, Manan, Sumit, Kabir, Aayan, Ashutosh, Advait, Anand, Tarun, Sunny, Mani, Gokul, Vijay, Mohit, Rehan, Soham

Female (14): Ritu, Priya, Neha, Pooja, Simran, Kavya, Ishita, Shreya, Roopa, Tanya, Shruti, Suhani, Kavitha, Rupali

TTS Additional Parameters

You can customize the TTS service with additional parameters:

1
tts = SarvamTTSService(
2
    api_key=os.getenv("SARVAM_API_KEY"),
3
    settings=SarvamTTSService.Settings(
4
        model="bulbul:v3",
5
        voice="shubh",
6
        language_code="en-IN",
7
        pace=1.0,  # Range: 0.5 to 2.0
8
    ),
9
)

Understanding the Call Flow

1. Caller dials your Twilio number. Twilio answers using the TwiML Bin and opens a WebSocket (Media Stream) to your agent.
2. Transport Input: Your Pipecat transport receives the caller’s audio over the WebSocket, and hands it to STT.
3. STT (Speech-to-Text): Converts audio to text using Sarvam’s Saarika or Saaras, and the context aggregator adds it to the conversation context.
4. LLM: Generates a response using Sarvam.
5. TTS (Text-to-Speech): Converts the response to audio using Sarvam’s Bulbul.
6. Transport Output: Streams the audio back over the WebSocket, and Twilio plays it to the caller, while the context aggregator saves the assistant’s response to context.

Pro Tips

• Use language="unknown" to automatically detect the caller’s language. This works well for multilingual support lines, but transcription accuracy is slightly lower than when you specify the exact language code, so prefer an explicit code whenever you already know which language the caller will use.
• Sarvam’s models understand code-mixing, so your agent can naturally handle Hinglish, Tanglish, and other mixed languages, which is common on real support calls.
• Use sarvam-105b for the LLM step.

Troubleshooting

Call connects but there’s no audio. Confirm the TwiML <Stream url> uses wss://, not wss:/ or https://, and that it points at your current ngrok URL. Free ngrok URLs change on every restart.

Call drops immediately. Check that TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN are set correctly in .env. The Twilio serializer needs both to manage the call.

API key errors. Make sure all keys are in your .env file and that the file sits in the same directory as agent.py.

Module not found. Re-run the install command for your operating system (see Step 2 above).

Poor transcription. Try language="unknown" for auto-detection, or set the exact language code (en-IN, hi-IN, and so on) if you already know it.

Choppy or robotic audio. Make sure audio_in_sample_rate and audio_out_sample_rate are both set to 8000 in PipelineParams. Twilio Media Streams don’t support other rates, and a mismatch causes distorted playback.

Part 2: WhatsApp Bot

A WhatsApp bot that can:

• Receive inbound WhatsApp messages, text or voice notes, on a Twilio WhatsApp number
• Transcribe voice notes in multiple Indian languages
• Understand and process what the user said or typed
• Reply with text, or optionally, a spoken voice note

Unlike the phone agent, this path doesn’t use Pipecat. WhatsApp messages arrive as one-off HTTP webhook requests rather than a continuous audio stream, so a small web server (Flask) that calls Sarvam’s APIs directly is all you need.

WhatsApp Bot: Quick Overview

1. Get API keys (Twilio, Sarvam)
2. Install packages: pip install flask twilio sarvamai python-dotenv requests
3. Create a .env file with your API keys
4. Write about 50 lines of Python code
5. From WhatsApp, send the Sandbox’s join code to opt your number in
6. Point the Sandbox’s webhook at your agent
7. Message your Twilio WhatsApp number

WhatsApp Bot: Quick Start

Step 1: Prerequisites

• Python 3.9 or higher
• ngrok, to expose your local server during development
• Accounts and keys from:
  • Twilio: an Account SID and an Auth Token
  • Sarvam AI: an API key from your dashboard

Step 2: Install Dependencies

$
pip install flask twilio sarvamai python-dotenv requests

Step 3: Create Environment File

Create a file named .env in your project folder:

SARVAM_API_KEY=sk_xxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_AUTH_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Then replace each placeholder with your real credentials from the Sarvam dashboard and the Twilio Console.

Step 4: Write Your Agent

Create whatsapp_agent.py:

1
import logging
2
import os
3
from pathlib import Path
4
5
import requests
6
from dotenv import load_dotenv
7
from flask import Flask, Response, request
8
from sarvamai import SarvamAI
9
from twilio.twiml.messaging_response import MessagingResponse
10
11
load_dotenv(override=True)
12
13
logging.basicConfig(level=logging.INFO)
14
logger = logging.getLogger(__name__)
15
16
app = Flask(__name__)
17
sarvam = SarvamAI(api_subscription_key=os.getenv("SARVAM_API_KEY"))
18
19
SYSTEM_PROMPT = (
20
    "You are Sarvam AI Agent, a multilingual assistant powered by Sarvam AI, "
21
    "chatting over WhatsApp. You can understand and respond fluently in "
22
    "English and 10+ Indian languages. Keep replies short (2-3 sentences) "
23
    "and conversational."
24
)
25
26
27
@app.route("/whatsapp", methods=["POST"])
28
def whatsapp_webhook():
29
    num_media = int(request.form.get("NumMedia", 0))
30
    twiml = MessagingResponse()
31
32
    try:
33
        if num_media > 0 and "audio" in request.form.get("MediaContentType0", ""):
34
            # Sender sent a voice note - download and transcribe it.
35
            # Twilio media URLs require your account credentials to fetch.
36
            media_url = request.form["MediaUrl0"]
37
            media_response = requests.get(
38
                media_url,
39
                auth=(os.getenv("TWILIO_ACCOUNT_SID"), os.getenv("TWILIO_AUTH_TOKEN")),
40
                timeout=10,
41
            )
42
            media_response.raise_for_status()
43
44
            audio_file = Path("/tmp/voice_note.ogg")
45
            audio_file.write_bytes(media_response.content)
46
47
            transcript = sarvam.speech_to_text.transcribe(
48
                file=open(audio_file, "rb"),
49
                model="saaras:v3",
50
                language_code="unknown",  # auto-detects the sender's language
51
            )
52
            user_text = transcript.transcript
53
        else:
54
            user_text = request.form.get("Body", "")
55
56
        completion = sarvam.chat.completions(
57
            model="sarvam-105b",
58
            messages=[
59
                {"role": "system", "content": SYSTEM_PROMPT},
60
                {"role": "user", "content": user_text},
61
            ],
62
        )
63
        reply_text = completion.choices[0].message.content
64
    except Exception:
65
        # Network hiccups, a failed download, or an API error shouldn't crash
66
        # the webhook - Twilio would just get a 500 and the sender gets silence.
67
        logger.exception("Failed to process incoming WhatsApp message")
68
        reply_text = "Sorry, I couldn't process that. Please try again in a moment."
69
70
    twiml.message(reply_text)
71
    return Response(str(twiml), mimetype="application/xml")
72
73
74
if __name__ == "__main__":
75
    app.run(port=7861)

Step 5: Configure Twilio to Reach Your Agent

Start ngrok:

$
ngrok http 7861

ngrok prints a forwarding URL that looks like https://xxxx.ngrok-free.app.

5a. Register your WhatsApp number with the Sandbox (required first)

The Sandbox is a shared Twilio number (+1 XXX XXX XXXX, shown on the console page below), used by every developer testing WhatsApp. Twilio only forwards messages from phone numbers that have explicitly opted in to your sandbox — nothing reaches your webhook until this step is done, no matter how correctly your code or webhook URL is set up.

1. Open the Twilio Console and go to Messaging → Try it out → Send a WhatsApp message (or search for “WhatsApp sandbox” in the console search bar). This page shows the sandbox number and your account’s unique join code, for example join happy-elephant.
2. From the WhatsApp app on your phone, send that exact join <your-code> message as a normal WhatsApp text to the sandbox number shown on that page (or scan the QR code Twilio shows, which pre-fills this message for you).
3. Twilio replies confirming you’re connected. Your number is now opted in to your sandbox.

5b. Point the Sandbox at your webhook

1. Still on the Sandbox settings page, find When a message comes in
2. Set it to your ngrok URL plus /whatsapp (for example, https://xxxx.ngrok-free.app/whatsapp), with the method set to HTTP POST
3. Save

Step 6: Run and Test Your Agent

$
python whatsapp_agent.py

Send a text message, or a voice note, to your Twilio WhatsApp number from any phone. Your bot will transcribe and reply within a few seconds.

Reply with a Voice Note (Optional)

To have your bot speak its reply back as a voice note instead of (or alongside) text, synthesize the reply with Sarvam’s text-to-speech API, serve the audio file from your own Flask app, and attach it to the TwiML response as media:

1
import uuid
2
3
from flask import send_from_directory
4
from sarvamai.play import save
5
6
AUDIO_DIR = Path("audio_replies")
7
AUDIO_DIR.mkdir(exist_ok=True)
8
9
10
@app.route("/audio/<filename>")
11
def serve_audio(filename):
12
    return send_from_directory(AUDIO_DIR, filename, mimetype="audio/wav")
13
14
15
# Inside whatsapp_webhook(), replace the final `twiml.message(reply_text)` with:
16
message = twiml.message(reply_text)
17
18
try:
19
    audio = sarvam.text_to_speech.convert(
20
        text=reply_text,
21
        language_code="en-IN",
22
        model="bulbul:v3",
23
        speaker="shubh",
24
    )
25
    filename = f"{uuid.uuid4().hex}.wav"
26
    save(audio, AUDIO_DIR / filename)
27
    message.media(f"{request.host_url}audio/{filename}")
28
except Exception:
29
    # If TTS fails, the sender still gets the text reply above.
30
    logger.exception("Text-to-speech failed, sending text-only reply")
31
32
return Response(str(twiml), mimetype="application/xml")

You can use the same language codes and speaker voices listed above for the phone agent when customizing STT and TTS for your WhatsApp bot.

Understanding the Message Flow

1. User sends a message. Text or a voice note, to your Twilio WhatsApp number. Twilio POSTs it to your /whatsapp webhook.
2. STT (Speech-to-Text): If the message is a voice note, your agent downloads it and converts it to text using Sarvam’s Saaras.
3. LLM: Generates a reply using Sarvam.
4. TTS (Text-to-Speech, optional): Converts the reply to audio using Sarvam’s Bulbul, if you’ve added voice-note replies.
5. Response: Your agent returns TwiML, and Twilio delivers the reply, text and/or audio, back to the user on WhatsApp.

WhatsApp Bot: Pro Tips

• Use language_code="unknown" in STT to auto-detect whichever language the sender types or speaks in.
• Keep system prompts explicit about reply length (as in the example) — WhatsApp messages are capped at 1600 characters by Twilio, and short replies also read better in a chat window.
• For production traffic, move off the Sandbox to a verified WhatsApp Sender, and validate Twilio’s request signature on every webhook call.

WhatsApp Bot: Troubleshooting

Webhook never gets called. First confirm the sending number actually joined the sandbox (Step 5a) — Twilio silently drops messages from numbers that haven’t opted in, and this is the most common reason nothing reaches your server. Then check that the sandbox’s “When a message comes in” URL matches your current ngrok URL exactly, ends in /whatsapp, and uses HTTPS.

Works for a few days, then stops. The sandbox opt-in expires 3 days after joining (or after the last message exchanged). Re-send the join <code> message from that WhatsApp number to reconnect.

No response outside a fresh conversation. WhatsApp’s 24-hour session-messaging window has closed; you’ll need a pre-approved template message to reach the user again.

Voice note transcription fails or comes back empty. Confirm MediaContentType0 starts with audio/, and that you’re passing your Twilio Account SID and Auth Token as Basic Auth when downloading MediaUrl0 (Twilio media URLs return 401 without it).

Voice notes over 30 seconds fail. The synchronous speech-to-text endpoint caps input at 30 seconds. Switch to the Batch STT API for longer voice notes.

Module not found. Re-run the install command from Step 2 above.

Additional Resources

• Sarvam AI Documentation
• Pipecat Documentation
• Pipecat Sarvam LLM Service
• Pipecat Twilio Serializer Reference
• Twilio Media Streams Documentation
• Twilio Getting Started with TwiML Bins
• Twilio Sandbox for WhatsApp
• Twilio WhatsApp API Overview
• Twilio Python Helper Library
• Twilio Console

Need Help?

• Sarvam Support: developer@sarvam.ai
• Community: Join the Discord Community

Happy Building!