Using OpenAI Realtime API to build a Twilio Voice AI assistant with Python

NOTE: OpenAI Realtime is now out of beta. We published an update video here:    • How to use OpenAI Realtime API with Twilio...  

We're so excited for our friends at OpenAI, who recently launched their Realtime API. This tutorial is built using this exciting API. The API opens up Speech to Speech (S2S) capabilities for their GPT-4o multimodal model, which supports direct audio input and output – avoiding translating back and forth from text with a speech-to-text (SST) or text-to-speech (TTS) step. We originally built this in Node.js but heard you loud and clear that you wanted a Python version.

Chapters:
0:49 Prerequisites
1:48 Code starts here
2:40 Configuring the application
3:42 Handling the incoming call
4:26 Setting up websockets
5:10 Handling Twilio Media Stream events
5:53 Handling OpenAI Realtime API events
7:11 Configuring send_session object
8:16 Starting the application
8:55 Configuring our Twilio phone number
9:20 Testing the application

This video will help you build an AI assistant using Twilio Voice and OpenAI's Realtime API. Here's what you'll need to build it:
A Twilio account - you can get one at https://twilio.com/try-twilio?utm_sou...
A Twilio phone number - learn more about phone numbers here: https://www.twilio.com/docs/phone-num...
An OpenAI API Key with Realtime API access (https://platform.openai.com/docs/over...)

Blog post: https://www.twilio.com/en-us/blog/voi...

Github repo: https://github.com/twilio-samples/spe...

Resources:
Twilio Media Streams: https://www.twilio.com/docs/voice/med...
OpenAI GPT-4o: https://openai.com/index/hello-gpt-4o/
Ngrok: https://ngrok.com
CodeExchange example: https://www.twilio.com/code-exchange/...