> For clean Markdown of any page, append `.md` to the page URL.
> For a complete documentation index, see https://docs.sarvam.ai/llms.txt.
> For full documentation content in one file, see https://docs.sarvam.ai/llms-full.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.sarvam.ai/_mcp/server.

# Build Your First Voice Agent using Twilio

> A beginner-friendly guide to building a real-time phone voice agent using Twilio, Pipecat, and Sarvam AI. Support for 11 languages (10 Indian + English) with natural voices and multilingual conversations.

## Overview

This guide shows you how to build a **real-time voice agent that answers phone calls**. It uses **Twilio** for telephony, **Pipecat** to orchestrate the real-time audio pipeline, and **Sarvam AI** for speech-to-text, the LLM, and text-to-speech. It's a great starting point for IVR replacements, customer support lines, and conversational phone agents in Indian languages.

## What You'll Build

A voice agent that can:

* Answer inbound phone calls on a Twilio number
* Listen to callers speaking, in multiple Indian languages
* Understand and process what they say
* Respond back in a natural-sounding voice, over the phone

## Quick Overview

1. Get API keys (Twilio, Sarvam)
2. Install packages: `pip install "pipecat-ai[websocket,sarvam]" python-dotenv`
3. Create a `.env` file with your API keys
4. Write about 80 lines of Python code
5. Point a Twilio phone number at your agent using TwiML
6. Call your Twilio number

***

## Quick Start

### 1. Prerequisites

* Python 3.9 or higher
* [ngrok](https://ngrok.com/download), to expose your local server during development
* Accounts and keys from:
  * [Twilio](https://console.twilio.com): an Account SID, an Auth Token, and a phone number with voice capability
  * [Sarvam AI](https://dashboard.sarvam.ai): an API key from your dashboard

### 2. Install Dependencies

```bash
pip install "pipecat-ai[websocket,sarvam]" python-dotenv loguru
```

```bash
pip install pipecat-ai[websocket,sarvam] python-dotenv loguru
```

### 3. Create Environment File

Create a file named `.env` in your project folder:

```env
SARVAM_API_KEY=sk_xxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_AUTH_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
```

Then replace each placeholder with your real credentials from the [Sarvam dashboard](https://dashboard.sarvam.ai) and the [Twilio Console](https://console.twilio.com).

`TWILIO_ACCOUNT_SID` and `TWILIO_AUTH_TOKEN` aren't optional here. Pipecat's Twilio transport uses them to authenticate with Twilio's REST API and to hang up the call cleanly when your agent ends the conversation.

### 4. Write Your Agent

Create `agent.py`:

```python
import os
from dotenv import load_dotenv
from loguru import logger
from pipecat.frames.frames import LLMRunFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
)
from pipecat.runner.types import RunnerArguments
from pipecat.runner.utils import create_transport
from pipecat.services.sarvam.stt import SarvamSTTService
from pipecat.services.sarvam.tts import SarvamTTSService
from pipecat.services.sarvam.llm import SarvamLLMService
from pipecat.transports.websocket.fastapi import FastAPIWebsocketParams

load_dotenv(override=True)

async def bot(runner_args: RunnerArguments):
    """Main bot entry point."""

    # create_transport auto-detects Twilio's WebSocket handshake and builds
    # the matching TwilioFrameSerializer, using TWILIO_ACCOUNT_SID and
    # TWILIO_AUTH_TOKEN from the environment.
    transport = await create_transport(
        runner_args,
        {
            "twilio": lambda: FastAPIWebsocketParams(
                audio_in_enabled=True, audio_out_enabled=True
            ),
        },
    )

    # Initialize AI services
    stt = SarvamSTTService(api_key=os.getenv("SARVAM_API_KEY"))
    tts = SarvamTTSService(api_key=os.getenv("SARVAM_API_KEY"))
    llm = SarvamLLMService(
        api_key=os.getenv("SARVAM_API_KEY"),
        settings=SarvamLLMService.Settings(model="sarvam-105b"),
    )

    # Set up conversation context
    messages = [
        {
            "role": "system",
            "content": "You are a friendly AI phone assistant. Keep your responses brief and conversational.",
        },
    ]
    context = LLMContext(messages)
    context_aggregator = LLMContextAggregatorPair(context)

    # Build pipeline
    pipeline = Pipeline(
        [
            transport.input(),
            stt,
            context_aggregator.user(),
            llm,
            tts,
            transport.output(),
            context_aggregator.assistant(),
        ]
    )

    # Twilio Media Streams send and receive 8kHz mono audio
    task = PipelineTask(
        pipeline,
        params=PipelineParams(
            audio_in_sample_rate=8000,
            audio_out_sample_rate=8000,
        ),
    )

    @transport.event_handler("on_client_connected")
    async def on_client_connected(transport, client):
        logger.info("Caller connected")
        messages.append(
            {"role": "system", "content": "Greet the caller and briefly introduce yourself."}
        )
        await task.queue_frames([LLMRunFrame()])

    @transport.event_handler("on_client_disconnected")
    async def on_client_disconnected(transport, client):
        logger.info("Caller disconnected")
        await task.cancel()

    runner = PipelineRunner(handle_sigint=runner_args.handle_sigint)
    await runner.run(task)

if __name__ == "__main__":
    from pipecat.runner.run import main
    main()
```

### 5. Configure Twilio to Reach Your Agent

Twilio needs a public `wss://` URL to stream call audio to. For local development, expose your agent with ngrok.

**Start ngrok:**

```bash
ngrok http 7860
```

ngrok prints a forwarding URL that looks like `https://xxxx.ngrok-free.app`. You'll turn this into a `wss://` URL in the next step.

The domain suffix ngrok assigns varies per account — you may see `ngrok-free.app`, `ngrok-free.dev`, or the older `ngrok.io`, regardless of whether you're on macOS, Windows, or Linux. Always use whatever URL ngrok actually prints, not the placeholder shown here.

Free ngrok URLs change every time you restart ngrok. If your agent stops receiving calls, check whether the URL in your TwiML Bin still matches the one ngrok is currently printing.

**Create a TwiML Bin:**

TwiML Bins aren't in the Twilio Console's left sidebar by default. The fastest way to reach them is to click the search bar at the top of the console and type "TwiML Bins".

1. Open the [Twilio Console](https://console.twilio.com) and search for **TwiML Bins** using the search bar at the top (or find it under **Developer Tools** in the sidebar)
2. Click **Create new TwiML Bin**, give it a friendly name, and paste the following, replacing the domain with your own ngrok URL:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="wss://xxxx.ngrok-free.app/ws" />
  </Connect>
</Response>
```

3. Click **Create** to save the bin

**Assign it to your phone number:**

1. Go to **Numbers & Senders**, then **Phone Numbers**, and select your number
2. Open its configuration/voice settings tab
3. Find the setting for what happens when **a call comes in**, and set it to use your TwiML Bin (Twilio has labeled this option **TwiML**, **TwiML Bin**, or similar depending on your console version)
4. Save

Twilio has redesigned this page's tabs and section names more than once (you may see **Configuration**, **Configure**, or **Configuration Details**; **Voice & Fax**, **Voice Configuration**, or **Voice and emergency calling**). The setting you want is always the one controlling what happens when a call comes in. If you can't find it, Twilio's own [Getting Started with TwiML Bins](https://www.twilio.com/docs/serverless/twiml-bins/getting-started) guide has current screenshots.

### 6. Run Your Agent

```bash
python agent.py --transport twilio
```

This starts a local FastAPI server, via Pipecat's development runner, that accepts Twilio's Media Streams WebSocket connection at `/ws`.

### 7. Test Your Agent

Call your Twilio phone number from any phone. Your voice agent will pick up and start the conversation.

***

## Customization Examples

### Example 1: Hindi Voice Agent

```python
stt = SarvamSTTService(
    api_key=os.getenv("SARVAM_API_KEY"),
    settings=SarvamSTTService.Settings(
        model="saaras:v3",
        language="hi-IN",  # Hindi
    ),
    mode="transcribe",
)

tts = SarvamTTSService(
    api_key=os.getenv("SARVAM_API_KEY"),
    settings=SarvamTTSService.Settings(
        model="bulbul:v3",
        voice="simran",  # Or: priya, ishita, kavya, aditya, anand, rohan
        target_language_code="hi-IN",
    ),
)

llm = SarvamLLMService(
    api_key=os.getenv("SARVAM_API_KEY"),
    settings=SarvamLLMService.Settings(model="sarvam-105b"),
)
```

`language`, `voice`, and `target_language_code` are `Settings` fields, not constructor keyword arguments. Passing them directly to `SarvamSTTService(...)` or `SarvamTTSService(...)` raises a `TypeError` on current versions of `pipecat-ai`. Only `mode` (STT) and `api_key` stay outside `settings`.

### Example 2: Tamil Voice Agent

```python
stt = SarvamSTTService(
    api_key=os.getenv("SARVAM_API_KEY"),
    settings=SarvamSTTService.Settings(
        model="saaras:v3",
        language="ta-IN",
    ),
    mode="transcribe",
)

tts = SarvamTTSService(
    api_key=os.getenv("SARVAM_API_KEY"),
    settings=SarvamTTSService.Settings(
        model="bulbul:v3",
        voice="shubh",
        target_language_code="ta-IN",
    ),
)

llm = SarvamLLMService(
    api_key=os.getenv("SARVAM_API_KEY"),
    settings=SarvamLLMService.Settings(model="sarvam-105b"),
)
```

### Example 3: Multilingual Agent (Auto-detect)

```python
# Auto-detect the caller's language
stt = SarvamSTTService(
    api_key=os.getenv("SARVAM_API_KEY"),
    settings=SarvamSTTService.Settings(
        model="saaras:v3",
        language="unknown",  # Auto-detects language
    ),
    mode="transcribe",
)

tts = SarvamTTSService(
    api_key=os.getenv("SARVAM_API_KEY"),
    settings=SarvamTTSService.Settings(
        model="bulbul:v3",
        voice="anand",
        target_language_code="en-IN",
    ),
)

llm = SarvamLLMService(
    api_key=os.getenv("SARVAM_API_KEY"),
    settings=SarvamLLMService.Settings(model="sarvam-105b"),
)
```

Use `language="unknown"` for support lines where callers might speak any of several languages. Sarvam auto-detects the spoken language per utterance, so the same agent can handle a Hindi caller followed by a Tamil caller without any code changes.

### Example 4: Speech-to-English Agent (Saaras)

Saarika transcribes speech to text in the same language. Saaras translates speech directly to English text. Use Saaras when the caller speaks an Indian language but you want to process and respond in English.

```python
# Caller speaks Hindi, Saaras converts it to English, the LLM processes it,
# and the agent responds in English.

stt = SarvamSTTService(
    api_key=os.getenv("SARVAM_API_KEY"),
    settings=SarvamSTTService.Settings(model="saaras:v3"),
    mode="translate",  # Speech-to-English translation
)

tts = SarvamTTSService(
    api_key=os.getenv("SARVAM_API_KEY"),
    settings=SarvamTTSService.Settings(
        model="bulbul:v3",
        voice="aditya",
        target_language_code="en-IN",
    ),
)

llm = SarvamLLMService(
    api_key=os.getenv("SARVAM_API_KEY"),
    settings=SarvamLLMService.Settings(model="sarvam-105b"),
)
```

Saaras auto-detects the source language (Hindi, Tamil, and so on) and translates spoken content directly to English text, which makes Indian-language speech usable by English-based LLMs.

***

## Available Options

### Language Codes

| Language        | Code      |
| --------------- | --------- |
| English (India) | `en-IN`   |
| Hindi           | `hi-IN`   |
| Bengali         | `bn-IN`   |
| Tamil           | `ta-IN`   |
| Telugu          | `te-IN`   |
| Gujarati        | `gu-IN`   |
| Kannada         | `kn-IN`   |
| Malayalam       | `ml-IN`   |
| Marathi         | `mr-IN`   |
| Punjabi         | `pa-IN`   |
| Odia            | `od-IN`   |
| Auto-detect     | `unknown` |

### Speaker Voices (Bulbul v3)

**Male (23):** Shubh (default), Aditya, Rahul, Rohan, Amit, Dev, Ratan, Varun, Manan, Sumit, Kabir, Aayan, Ashutosh, Advait, Anand, Tarun, Sunny, Mani, Gokul, Vijay, Mohit, Rehan, Soham

**Female (14):** Ritu, Priya, Neha, Pooja, Simran, Kavya, Ishita, Shreya, Roopa, Tanya, Shruti, Suhani, Kavitha, Rupali

### TTS Additional Parameters

You can customize the TTS service with additional parameters:

```python
tts = SarvamTTSService(
    api_key=os.getenv("SARVAM_API_KEY"),
    settings=SarvamTTSService.Settings(
        model="bulbul:v3",
        voice="shubh",
        target_language_code="en-IN",
        pace=1.0,  # Range: 0.5 to 2.0
    ),
)
```

Twilio Media Streams always carry 8kHz mono audio. You don't need to touch `sample_rate` on `SarvamTTSService` for this: Pipecat resamples the TTS output to match the `audio_out_sample_rate` set in `PipelineParams` automatically.

***

## Understanding the Call Flow

```mermaid
flowchart LR
    caller(["📞 Caller"]) <--> twilio["Twilio"]
    twilio <-->|"wss://<br />audio"| pipeline

    subgraph pipeline["Your Agent Server (Pipecat)"]
        direction LR
        stt["STT"] --> llm["LLM"] --> tts["TTS"]
    end

    pipeline -.->|HTTPS| sarvam["Sarvam AI<br />Saarika/Saaras · sarvam-30b/105b · Bulbul"]
```

1. **Caller dials your Twilio number.** Twilio answers using the TwiML Bin and opens a WebSocket (Media Stream) to your agent.
2. **Transport Input:** Your Pipecat transport receives the caller's audio over the WebSocket, and hands it to STT.
3. **STT (Speech-to-Text):** Converts audio to text using Sarvam's Saarika or Saaras, and the context aggregator adds it to the conversation context.
4. **LLM:** Generates a response using Sarvam.
5. **TTS (Text-to-Speech):** Converts the response to audio using Sarvam's Bulbul.
6. **Transport Output:** Streams the audio back over the WebSocket, and Twilio plays it to the caller, while the context aggregator saves the assistant's response to context.

***

## Pro Tips

* Use `language="unknown"` to automatically detect the caller's language. This works well for multilingual support lines, but transcription accuracy is slightly lower than when you specify the exact language code, so prefer an explicit code whenever you already know which language the caller will use.
* Sarvam's models understand code-mixing, so your agent can naturally handle Hinglish, Tanglish, and other mixed languages, which is common on real support calls.
* Use `sarvam-30b` for faster responses, or `sarvam-105b` for more complex conversations.

Log the transcript from `context_aggregator` if you want post-call analytics. Writing it to a file or database is cheap and won't slow down the live pipeline.

***

## Troubleshooting

**Call connects but there's no audio.** Confirm the TwiML `<Stream url>` uses `wss://`, not `wss:/` or `https://`, and that it points at your current ngrok URL. Free ngrok URLs change on every restart.

**Call drops immediately.** Check that `TWILIO_ACCOUNT_SID` and `TWILIO_AUTH_TOKEN` are set correctly in `.env`. The Twilio serializer needs both to manage the call.

**API key errors.** Make sure all keys are in your `.env` file and that the file sits in the same directory as `agent.py`.

**Module not found.** Re-run the install command for your operating system (see Step 2 above).

**Poor transcription.** Try `language="unknown"` for auto-detection, or set the exact language code (`en-IN`, `hi-IN`, and so on) if you already know it.

**Choppy or robotic audio.** Make sure `audio_in_sample_rate` and `audio_out_sample_rate` are both set to `8000` in `PipelineParams`. Twilio Media Streams don't support other rates, and a mismatch causes distorted playback.

***

## Additional Resources

* [Sarvam AI Documentation](https://docs.sarvam.ai)
* [Pipecat Documentation](https://docs.pipecat.ai)
* [Pipecat Sarvam LLM Service](https://docs.pipecat.ai/api-reference/server/services/llm/sarvam)
* [Pipecat Twilio Serializer Reference](https://reference-server.pipecat.ai/en/latest/api/pipecat.serializers.twilio.html)
* [Twilio Media Streams Documentation](https://www.twilio.com/docs/voice/twiml/stream)
* [Twilio Getting Started with TwiML Bins](https://www.twilio.com/docs/serverless/twiml-bins/getting-started)
* [Twilio Console](https://console.twilio.com)

***

## Need Help?

* Sarvam Support: [developer@sarvam.ai](mailto:developer@sarvam.ai)
* Community: [Join the Discord Community](https://discord.com/invite/5rAsykttcs)

***

**Happy Building!**