> For clean Markdown of any page, append `.md` to the page URL. > For a complete documentation index, see https://docs.sarvam.ai/llms.txt. > For full documentation content in one file, see https://docs.sarvam.ai/llms-full.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.sarvam.ai/_mcp/server. # WhatsApp Bot > Build a WhatsApp bot that replies to text messages and voice notes using Sarvam AI's speech-to-text, chat, and text-to-speech APIs, in 11 languages (10 Indian + English). ## **Overview** This cookbook builds a WhatsApp bot that replies to inbound text messages and voice notes on a WhatsApp number. It chains together three Sarvam AI APIs, **Saaras STT** to transcribe voice notes, a **Sarvam chat model** to generate a reply, and **Bulbul TTS** to optionally speak that reply back, behind a small Flask webhook. **What you'll build:** ### Business Value #### Customer Support Answer common questions or triage issues directly inside the WhatsApp conversation customers already use, in their own language. #### Field & Agent Networks Let field agents or delivery partners report issues by voice note instead of filling out a form, especially useful where typing is slower than speaking. #### Citizen Services Offer scheme information or grievance redressal over WhatsApp in the citizen's preferred Indian language. ## **How It Works** #### Receive the message Twilio POSTs every inbound WhatsApp message, text or voice note, to your Flask webhook as a standard form-encoded request. #### Transcribe voice notes If the message is a voice note, download it from Twilio (it arrives as OGG/Opus, which Sarvam's STT REST API accepts directly, no transcoding needed) and transcribe it with **Saaras v3**, auto-detecting the sender's language. #### Generate a reply Send the transcribed or typed text to a **Sarvam chat model**, along with a system prompt describing the bot's persona. #### Reply, optionally as a voice note Return the reply as WhatsApp text, and optionally synthesize it with **Bulbul TTS** and attach it as a voice-note reply. ## **1. Prerequisites** This cookbook uses the free **Twilio Sandbox for WhatsApp** for testing. Moving to production requires registering a WhatsApp Sender, which involves Meta Business verification and can take a few days. See [Twilio's WhatsApp docs](https://www.twilio.com/docs/whatsapp) when you're ready to go live. ## **2. Install the SDK and Dependencies** ```bash pip install flask twilio sarvamai python-dotenv requests ``` ## **3. Configure Twilio for WhatsApp** Set your keys as environment variables: ```bash export SARVAM_API_KEY="your-sarvam-api-key" export TWILIO_ACCOUNT_SID="ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" export TWILIO_AUTH_TOKEN="your-twilio-auth-token" ``` Loading these from environment variables, instead of hardcoding them, keeps credentials out of source control. The script below reads them via `os.environ`. Twilio's Sandbox is a shared number (`+1 XXX XXX XXXX`, shown on the console page below), used by every developer testing WhatsApp. Twilio only forwards messages from phone numbers that have explicitly opted in to *your* sandbox, so before writing any code, connect your own number: 1. Open the [Twilio Console](https://console.twilio.com) and go to **Messaging** → **Try it out** → **Send a WhatsApp message** (or search for "WhatsApp sandbox" in the console search bar). This page shows the sandbox number and your account's unique join code, for example `join happy-elephant`. 2. From the WhatsApp app on your phone, send that exact `join ` message to the sandbox number shown on that page (or scan the QR code Twilio shows, which pre-fills this message for you). 3. Twilio replies confirming you're connected. This opt-in is per phone number, not per account. Anyone else who wants to message your bot (a teammate, a demo user) must send the same `join ` message from their own WhatsApp number first, there's no way around this for the Sandbox. The opt-in also expires 3 days after joining (or after the last message exchanged); if messages stop arriving, re-send the join code. Once your webhook is running (Step 4 below) and exposed via ngrok, come back to the Sandbox settings page, find **When a message comes in**, and set it to your ngrok URL plus `/whatsapp` (for example, `https://xxxx.ngrok-free.app/whatsapp`), method **HTTP POST**. ## **4. The Full Bot Script** Everything below, imports, the Flask webhook, and the transcribe → chat → reply flow, lives in one script. Copy it into `whatsapp_bot.py`: ```python import logging import os from pathlib import Path import requests from dotenv import load_dotenv from flask import Flask, Response, request from sarvamai import SarvamAI from twilio.twiml.messaging_response import MessagingResponse load_dotenv(override=True) logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) app = Flask(__name__) sarvam = SarvamAI(api_subscription_key=os.getenv("SARVAM_API_KEY")) SYSTEM_PROMPT = ( "You are Sarvam AI Agent, a multilingual assistant powered by Sarvam AI, " "chatting over WhatsApp. You can understand and respond fluently in " "English and 10+ Indian languages. Keep replies short (2-3 sentences) " "and conversational." ) @app.route("/whatsapp", methods=["POST"]) def whatsapp_webhook(): num_media = int(request.form.get("NumMedia", 0)) twiml = MessagingResponse() try: if num_media > 0 and "audio" in request.form.get("MediaContentType0", ""): # Sender sent a voice note - download and transcribe it. # Twilio media URLs require your account credentials to fetch. media_url = request.form["MediaUrl0"] media_response = requests.get( media_url, auth=(os.getenv("TWILIO_ACCOUNT_SID"), os.getenv("TWILIO_AUTH_TOKEN")), timeout=10, ) media_response.raise_for_status() audio_file = Path("/tmp/voice_note.ogg") audio_file.write_bytes(media_response.content) transcript = sarvam.speech_to_text.transcribe( file=open(audio_file, "rb"), model="saaras:v3", language_code="unknown", # auto-detects the sender's language ) user_text = transcript.transcript else: user_text = request.form.get("Body", "") completion = sarvam.chat.completions( model="sarvam-105b", messages=[ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": user_text}, ], ) reply_text = completion.choices[0].message.content except Exception: # Network hiccups, a failed download, or an API error shouldn't crash # the webhook - Twilio would just get a 500 and the sender gets silence. logger.exception("Failed to process incoming WhatsApp message") reply_text = "Sorry, I couldn't process that. Please try again in a moment." twiml.message(reply_text) return Response(str(twiml), mimetype="application/xml") if __name__ == "__main__": app.run(port=7861) ``` ### How it works * **`whatsapp_webhook`** is the single entry point Twilio calls for every inbound message. `NumMedia` and `MediaContentType0` tell you whether the message carries a voice note; `Body` holds typed text. * If a voice note is present, it's downloaded with HTTP Basic Auth (Twilio media URLs require your Account SID and Auth Token) and handed straight to **Saaras v3** with `language_code="unknown"`, so the bot works for a Hindi caller followed by a Tamil caller without any code changes. * The (transcribed or typed) text is sent to `sarvam-105b` along with a system prompt that sets the bot's persona and instructs it to keep replies short, since WhatsApp is a chat surface, not a document. * The `try`/`except` around the whole body exists because this endpoint is a system boundary, it calls two external services (Twilio's media download, Sarvam's API) over the network, either of which can fail independently of your code. Replying with a short apology keeps the conversation alive instead of leaving the sender with no response at all. Voice notes longer than 30 seconds will fail on this synchronous STT endpoint. For longer audio, switch to the [Batch STT API](/api/api-guides-tutorials/speech-to-text/batch-api). This script trusts every request to `/whatsapp`. Before going to production, validate the `X-Twilio-Signature` header on each request (using `twilio.request_validator.RequestValidator`) so requests can't be spoofed as coming from Twilio. Run it, then expose it with ngrok: ```bash python whatsapp_bot.py ``` ```bash ngrok http 7861 ``` Send a text message, or a voice note, to your Twilio WhatsApp number from any phone. Your bot will transcribe and reply within a few seconds. ## **5. Reply with a Voice Note (Optional)** To have your bot speak its reply back instead of (or alongside) text, synthesize it with Bulbul TTS, serve the audio file from your own Flask app, and attach it to the TwiML response as media: ```python import uuid from flask import send_from_directory from sarvamai.play import save AUDIO_DIR = Path("audio_replies") AUDIO_DIR.mkdir(exist_ok=True) @app.route("/audio/") def serve_audio(filename): return send_from_directory(AUDIO_DIR, filename, mimetype="audio/wav") # Inside whatsapp_webhook(), replace the final `twiml.message(reply_text)` with: message = twiml.message(reply_text) try: audio = sarvam.text_to_speech.convert( text=reply_text, target_language_code="en-IN", model="bulbul:v3", speaker="shubh", ) filename = f"{uuid.uuid4().hex}.wav" save(audio, AUDIO_DIR / filename) message.media(f"{request.host_url}audio/{filename}") except Exception: # If TTS fails, the sender still gets the text reply above. logger.exception("Text-to-speech failed, sending text-only reply") return Response(str(twiml), mimetype="application/xml") ``` The TTS call has its own nested `try`/`except`, separate from the outer one, so a synthesis failure only drops the audio attachment, the sender still gets the text reply instead of the generic apology. `request.host_url` resolves to your current ngrok URL automatically, since ngrok forwards the original `Host` header. WhatsApp only renders a native, waveform voice-note bubble for OGG/Opus audio. A `.wav` reply still plays back fine, just as a regular audio attachment rather than a voice-note bubble. ## **6. Example Conversation** Here's what a typical exchange looks like once the bot is running, a Hindi voice note in, a text reply out: ``` User (voice note, Hindi): "मुझे अपने ऑर्डर का स्टेटस चेक करना है" Bot: "बिल्कुल! कृपया अपना ऑर्डर नंबर बताएं, मैं अभी स्टेटस चेक करता हूं।" User (text): "Can you also reply in English?" Bot: "Of course! I can switch to English anytime, just let me know what you need." ``` ## **Available Options** ### Supported Languages | Language | Code | | --------------- | --------- | | English (India) | `en-IN` | | Hindi | `hi-IN` | | Bengali | `bn-IN` | | Tamil | `ta-IN` | | Telugu | `te-IN` | | Gujarati | `gu-IN` | | Kannada | `kn-IN` | | Malayalam | `ml-IN` | | Marathi | `mr-IN` | | Punjabi | `pa-IN` | | Odia | `od-IN` | | Auto-detect | `unknown` | ### Speaker Voices (Bulbul v3) **Male:** Shubh (default), Aditya, Rahul, Rohan, Amit, Anand, Karun, and more **Female:** Ritu, Priya, Neha, Pooja, Simran, Kavya, Ishita, and more See [Text-to-Speech](/api/api-guides-tutorials/text-to-speech/overview) for the full voice list. ## **Customization** ### Change the bot's persona `SYSTEM_PROMPT` in the script controls tone and behavior. Swap it for something domain-specific, for example a support bot that only answers questions about order status, and stays on-topic otherwise. ### Reply in the sender's own language `target_language_code` in the TTS call is hardcoded to `en-IN` in the example above. Pair it with the language Saaras detected (`transcript.language_code`) so a Hindi voice note gets a Hindi voice-note reply, not an English one. ### Move off the Sandbox Before real users message your bot, register a WhatsApp Sender under **Messaging** → **Senders** → **WhatsApp senders** in the Twilio Console. This requires Meta Business verification and removes both the shared-number and 3-day opt-in limitations of the Sandbox. ## **Tips & Best Practices** * **Language auto-detection**: Use `language_code="unknown"` in STT to auto-detect whichever language the sender types or speaks in, rather than hardcoding one. * **Reply length**: Keep system prompts explicit about reply length. WhatsApp message bodies are capped at 1600 characters by Twilio, and short replies also read better in a chat window. * **The 24-hour session window**: WhatsApp only allows free-form replies within 24 hours of the user's last message. Outside that window, you need a pre-approved template message, this is a WhatsApp Business Platform policy that neither Sarvam nor Twilio can bypass. * **Sandbox opt-in**: If messages never reach your webhook, check that the sending number actually joined the Sandbox (Step 3), Twilio silently drops messages from numbers that haven't opted in, and this is the most common reason nothing arrives. * **API key security**: Load `SARVAM_API_KEY`, `TWILIO_ACCOUNT_SID`, and `TWILIO_AUTH_TOKEN` from environment variables rather than hardcoding them, especially outside local experimentation. ## **Error Handling** You may encounter these errors while using the underlying APIs: * **403 Forbidden** (`invalid_api_key_error`), invalid Sarvam API key. Use a valid key from the [Sarvam AI Dashboard](https://dashboard.sarvam.ai/). * **429 Too Many Requests** (`insufficient_quota_error` / `rate_limit_exceeded_error`), Sarvam credits exhausted or rate limit hit. Retry with exponential backoff. * **500 Internal Server Error** (`internal_server_error`), issue on Sarvam's servers. Try again later; contact support if persistent. * **Twilio error 63015**, "recipient has not joined the Sandbox." The sending number needs to send the `join ` message first (or its 3-day opt-in has expired), see Step 3 above. For the full Sarvam error-code table and SDK exception reference, see [Errors & Troubleshooting](https://docs.sarvam.ai/api/getting-started/errors-troubleshooting). ## **Additional Resources** * **Documentation**: [docs.sarvam.ai](https://docs.sarvam.ai) * **Related APIs**: [Speech-to-Text](/api/api-guides-tutorials/speech-to-text/overview) · [Chat Completion](/api/api-guides-tutorials/chat-completion/overview) · [Text-to-Speech](/api/api-guides-tutorials/text-to-speech/overview) * **Twilio docs**: [Sandbox for WhatsApp](https://www.twilio.com/docs/whatsapp/sandbox) · [WhatsApp API Overview](https://www.twilio.com/docs/whatsapp) · [Python Helper Library](https://www.twilio.com/docs/libraries/python) * **Community**: [Join the Discord Community](https://discord.com/invite/5rAsykttcs) ## **Final Notes** * Keep your API keys secure, prefer environment variables over hardcoding. * The Sandbox is for development only, each recipient must opt in, and the connection expires after 3 days of inactivity. * The webhook degrades gracefully: a failed transcription or chat call still gets the sender a reply, and a failed voice-note synthesis still gets them the text. **Keep Building!** 🚀