How to: Process Twilio voice transcriptions with AI

Goal

Classify, route and further process transcribed voice calls by integrating a Twilio-enabled telephone number into a processing workflow.

Combining the power of Twilio’s built-in transcriptions of voice messages and the Hero Platform_’s connectivity and automation capabilities will help you achieve great results.

In this how-to article, we’ll be creating a simple processing flow that uses Twilio as an input, processes the information available through the Twilio REST API, and then makes a decision based on the content of a call. This example can be the basis for more powerful automations that integrate AI models and other complex decision making.

Requirements

1. A twilio trial account
2. A Hero Platform_ trial account

Get Set Up

We will create a Twilio trial account and provide a basic set up for a telephone number that can receive voice messages and transcribe them. We’ll set up a Hero Platfom_ trial account and connect this to our newly created Twilio trial account. Finally, we’ll set up a processing flow to analyze transcribed voice messages and use an AI model to extract appointment dates.

Create a Twilio trial account

1. Login to the Twilio console and select Create New Account
2. Provide an account name
3. Verify your email and phone number
4. Provide the requested information around what you are looking to build and which tools you’ll be using.
5. From the dashboard click on Get a Trial Number
   1. Twilio will suggest a trial number for receiving voice, sms and mms messages.
   2. Select a number that is able to receive voice messages, we won’t be using SMS, MMS or Fax functionalities below.
6. Click on Explore products and find TwiML Bins
   1. create a new TwiML Bin to say “goodbye”, so the user knows that the call has been recorded.
      
      

      <?xml version="1.0" encoding="UTF-8"?> <Response> <Say>Thanks, Goodbye!</Say> </Response>

   2. Collect the SID of the above TwiML Bin (click on the name from the list)
   3. Create a TwiML Bin called call handler referencing the SID at the end of the action (line 4):
      

      <?xml version="1.0" encoding="UTF-8"?> <Response> <Say>Please leave a message after the beep then wait 3 seconds.</Say> <Record transcribe="true" playBeep="true" timeout="3" action="https://handler.twilio.com/twiml/EHXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"/> <Say>We didn't record any input. Goodbye!</Say> </Response> 

7. Under ¹Phone Numbers > Manage > Active Numbers select your trial number
8. Under Voice where it says A CALL COMES IN, select TwiML Bin, and then select your call handler TwiML Bin
9. Save your changes and test out the transcription functionality by calling the number and leaving a message, requesting an appointment on a certain date.

¹Phone numbers for a Twilio trial account are all US based.

As we now have a working Twilio phone number, let’s dive into Automation Hero’s Hero Platform_ and start bringing in data from Twilio’s REST API.

Create a REST connection to your Twilio account

We’ll create two connections. One to the Hero Platform_’s internal Data Store to save results, and one to Twilio.

1. Create a trial account and log in to Hero Platform_.
2. Under Integration > Connections click Create New Connection
3. Provide a name for the internal Data Store connection, e.g. twilio_appointments
4. Under Connection Type choose Tuple Store
5. Click Create New Connection again
6. Provide a name e.g. twilio_connection
7. Choose REST as the Connection Type
8. We are using Basic Auth in this example
9. The user name is the ACCOUNT SID (this can be found in the Twilio console account page.)
10. Use the AUTH TOKEN instead of a password
11. The Base URL is built up from the standard Twilio Base URL
    

    https://api.twilio.com/2010-04-01 + /Accounts/ + <your ACCOUNT SID> + / e.g. 

12. Under Test endpoint use Recordings.json and click on Test connection to verify that the end point is properly set up.
13. Press OK, and we have a connection to connect the Hero Platform_ to Twilio

Create a Flow

After getting data into the Hero Platform_ we can use a Flow to start automating how we route or use the data.

1. Under Automations > Flows click Create New Flow
2. Give the flow a name e.g. twilio flow
3. In the element browser on the right, hit the plus button to create a new input
4. Give it a name e.g. twilio input and choose the twilio_connection created previously
5. Under Endpoint enter Recordings.json
6. Our Root path is recordings
7. Click on the button to fetch the Fields mapping table
8. Finally open the Input section in the element browser on the right hand side of the screen, and drag-and-drop your input onto the flow canvas.

We now have one element in our flow. We can click on the preview icon (square with a magnifying glass) to see what kind of data is returned by this connection. Clicking on chevron will show us information around which outputs this connection provides (e.g. column names and data types).

For this example we’re really only interested in sid column, a unique identifier for the various call recordings.

1. Double click on the Input node, and choose Edit output fields
2. At the top, use the toggle to deselect all columns and select sid
3. Run the preview, by clicking on the preview icon (square with a magnifying glass), and copy one of the returned sid values.

RE0183922e9348ec1befc658a9f9991200

Add a REST API lookup

We want to get the actual transcripts of the calls. For this we need to create another REST lookup. In the element browser, we can see various categories of operators that we can add to our flow. There is also an option to search for a particular operator.

1. In the search bar on the right search for REST to bring up a REST Lookup.
2. Drag the REST operator onto the canvas and pull a cable from the REST input to the new node.
3. Enter sid as the argument.
4. We’ll reuse the twilio_connection
5. Select GET as the HTTP method. As the endpoint varies according to the sid, we can parameterize the endpoint, and use a different endpoint per record in Twilio. Enter Recordings/${sid}/Transcriptions.json as the endpoint.
6. The root path of the the returned JSON is transcriptions
7. Click on the Fields mapping table button, paste in the sid value from above when prompted.
   1. This pulls in all fields from the REST API
   2. Notice that sid is also a field pulled in and is marked as red, as there is already a column named sid
   3. As we already have the sid of every recording, we’ll hit the x, to delete this column.
8. Click ok, and the node is added and finalized on the canvas.

Analyze the Transcribed Text and Further Automate

Hero Platform_ provides many options for how to analyze the text contained within the transcriptions from Twilio – from simple contains functions to advanced AI models for classification or extraction of important information. For this example, we want to use a simple AI model to obtain appointment dates from a call.

1. From the element browser, under Output drag and drop an Automatic Output onto the canvas
2. Connect this node by dragging a cable from the REST Lookup
3. Provide a name for this output, e.g. twilio_transcriptions_output
4. Choose twilio_tuple_store as the connection

We have just brought data in from Twilio and pushed it to the Hero Platform_’s Data Store. This is where processed data can be stored before being processed further.

Create a Simple AI Model

We need to configure an input to pull data from the Data Store and provide it to an AI model.

1. In the element browser, click the plus button to create a new input
2. Provide a name, e.g. twilio_transcriptions_input
3. Choose twilio_tuple_store as the connection
4. On the left hand side, navigate to AI > AI Models
5. Click on Create Text Extractor
6. Give the new AI model a name, e.g. twilio_appointment_date
7. Click Next
8. We can use the newly created twilio_transcriptions_input as our source
9. Select transcription_text from the list of possible fieldnames.
10. Click Save
11. While the Extraction Engine warms up, browse the transcribed text from Twilio.
12. Click on Add Field and provide a name of what you want to extract, e.g. “appointment_date”
13. Write a question in standard language, “On which date is the suggested appointment?”
14. Select the check box, “Always answer”
15. Click OK

The extraction engine will start to process, and provide an answer to the question. Beside the answer is a small bar that shows how much confidence the model has in its answer. The more full the bar is, the more confident the model is.

Finally click Save and return to the flow that we have created.

Add the AI Model to a Flow & Route Information According to a Confidence Score

We have all the pieces in place to start making decisions based on what information is in a transcribed voice message.

1. From the element browser under AI text extraction function, drag your text extraction to the canvas
2. As before, connect by dragging a cable to the new node
3. Verify that we are using the transcription_text field as an argument, and click OK

The text extractor returns a tuple of data with two elements

1. The extracted text
2. A confidence value from 0 (not confident) to 1 (extremely confident)

We can use the confidence values to route only answers above a certain confidence value automatically, and send the lower values for further processing (either automated or by a human).

1. In the element browser, drag the IsGreaterThanOrEqual operator to the canvas
2. Connect with a cable
3. Choose the field appointment_date.confidence as the argument
4. Enter 0.5 as the right side argument, and click ok

In this example, we’ll route answers with a confidence of 50% to another automatic output.

1. From the element browser under output, drag an automatic output to the canvas
2. Provide a name, e.g. twilio_appointments
3. Save the Flow and click Run Now

We see the Flow overview screen, listing all flows and other flow-related information. When the flow has stopped running, navigate to Storage > Data Store and click on twilio_appointments. Here we will see all of the calls that were transcribed along with the appointment dates which were extracted at above 50% confidence.

This is Just the Beginning

In our example, we connected the Hero Platform_ to another service, brought data into the platform, ran an AI model against the data, and used a confidence score to route our data. Connections in Hero Platform_ operate both ways. Instead of finishing with our data in the platform, we could easily push this to another system, like a CRM or a calendar. Flows can be simple or extremely complex.

To see the results in action, check out our short demo video.

How did it go? Do you have any feedback? Send it to hi@automationhero.ai with the subject line: Automation Hero and Twilio Integration Feedback.