GSP967

Overview

Conversational Agents provides a simple, visual bot building approach to virtual agent design. For a full voice experience, your Conversational Agents can be integrated with various conversational platforms, including telephony providers. In this lab, you'll explore these Interactive Voice Response (IVR) features as well as two additional features - conversation repair and Speech Synthesis Markup Language (SSML) - that help end users feel as though they're having a natural, interactive, and cooperative conversation.

This lab will show you how to enable various IVR features, but you will only be able to test some of them with the Conversational Agents Phone Gateway. Features like DTMF (Dual-Tone Multi-Frequency) and Barge-in (where the user can interrupt the bot) are not supported in Conversational Agent Telephony and can only be tested with your telephony provider.

In this lab you will continue building a conversational agent, exploring and adding the IVR features that Conversational Agents provides.

Voice and telephony features such as DTMF, Barge-in, and End of speech sensitivity (so the bot can accommodate for pauses in a phrase, such as a number or ID) can all be configured in Conversational Agents.

Conversation repair is the practice of fixing misunderstandings, mishearings, and misarticulations to resume a conversation. Repairing a conversation can help build a user's trust by showing that the voice agent is listening to their request. Situations where conversations might fail are handled in a more graceful manner, such as when a voice agent cannot find an intent using the NoMatch event, or when the agent detects no verbal response using the NoInput event feature. You'll configure these events to rephrase a prompt up to a maximum of 3 times and then escalate to a live agent to avoid trapping users in a loop of handling errors.

SSML - Speech Synthesis Markup Language helps make the Text-to-Speech voice interaction sound more natural.

To do this work efficiently you'll restore a provided agent. This agent will have 2 new pages and an additional intent that will jumpstart your exploration of the new conversational features.

In this lab you will do the following

• Enable and configure IVR features
• Add in NoMatch and NoInput handling scenarios to escalate to an Agent
• Add in rich voice responses with SSML

Prerequisites

Students should be generally familiar with the basic concepts of conversational AI. Before you start the lab, read through these Sample Transcripts to get an idea of what client transcripts might look like. Often the first step in creating an agent is to read through client transcripts and/or other contextual data to understand the use case and specific business requirements.

This Lab builds upon the basic Flight Booker agent developed in Conversational Agents: Bot Building Basics. Instructions to restore the previously built Flight Booker agent are described in the next section.

Setup and requirements

Before you click the Start Lab button

Read these instructions. Labs are timed and you cannot pause them. The timer, which starts when you click Start Lab, shows how long Google Cloud resources are made available to you.

This hands-on lab lets you do the lab activities in a real cloud environment, not in a simulation or demo environment. It does so by giving you new, temporary credentials you use to sign in and access Google Cloud for the duration of the lab.

To complete this lab, you need:

• Access to a standard internet browser (Chrome browser recommended).

Note: Use an Incognito (recommended) or private browser window to run this lab. This prevents conflicts between your personal account and the student account, which may cause extra charges incurred to your personal account.

• Time to complete the lab—remember, once you start, you cannot pause a lab.

Note: Use only the student account for this lab. If you use a different Google Cloud account, you may incur charges to that account.

How to start your lab and sign in to the Google Cloud console

1. Click the Start Lab button. If you need to pay for the lab, a dialog opens for you to select your payment method. On the left is the Lab Details pane with the following:

   • The Open Google Cloud console button
   • Time remaining
   • The temporary credentials that you must use for this lab
   • Other information, if needed, to step through this lab

2. Click Open Google Cloud console (or right-click and select Open Link in Incognito Window if you are running the Chrome browser).

   The lab spins up resources, and then opens another tab that shows the Sign in page.

   Tip: Arrange the tabs in separate windows, side-by-side.

   Note: If you see the Choose an account dialog, click Use Another Account.

3. If necessary, copy the Username below and paste it into the Sign in dialog.

   {{{user_0.username | "Username"}}}

   You can also find the Username in the Lab Details pane.

4. Click Next.

5. Copy the Password below and paste it into the Welcome dialog.

   {{{user_0.password | "Password"}}}

   You can also find the Password in the Lab Details pane.

6. Click Next.

   Important: You must use the credentials the lab provides you. Do not use your Google Cloud account credentials. Note: Using your own Google Cloud account for this lab may incur extra charges.

7. Click through the subsequent pages:

   • Accept the terms and conditions.
   • Do not add recovery options or two-factor authentication (because this is a temporary account).
   • Do not sign up for free trials.

After a few moments, the Google Cloud console opens in this tab.

Note: To access Google Cloud products and services, click the Navigation menu or type the service or product name in the Search field.

Task 1. Get started with Conversational Agents

In this task, you log in to Conversational Agents and create the new virtual agent.

Assumption: You've already logged into Google Cloud before continuing with the steps below.

1. In a new incognito window, go to the Conversational Agents page.

2. In the Select Project dialog, click the All tab.

3. Click your Project ID, .

4. In the Google Conversational Agent API dialog, — click Enable API to proceed.

5. In the Agents window, click Create agent.

6. Click Build your own.

7. In the Create agent dialog, set the following fields:

   Field	Value
   Display name	Flight Booker
   Location	global
   Conversation Start	Select Flow

8. Click Create.

   Once you create the agent, the Default start flow window opens.

9. After creating the agent, navigate to Settings > General > Logging settings and click Enable Cloud Logging and Enable conversation history. This generates logs for this agent.

10. Click Save.

Click Check my progress to verify the objective. Create the agent

Task 2. Restore the base Flight booker agent

1. Download the sample virtual agent, gsp967-start-agent, to your local hard drive.

2. On the Agent Overview page, click the Export/Restore button.

3. Select Restore Agent from the expanded menu options.

4. Select the Upload radio button.

5. Click on select file.

6. Navigate to and select the exported_agent_skillbadge_lab3_start-baseimport.blob that you downloaded to your hard drive.

7. Click Open.

8. Click Restore.

Refer to the Conversational Agents 'Export and restore' documentation as needed.

Click Check my progress to verify the objective. Restore the agent

Task 3. Review the base Agent

The base Agent you restored has a few items to jumpstart your IVR exploration. Take a quick review of what the agent can do before enabling IVR and other features.

Look up Flight intent

An intent called main.look_up_flight will be used to check for an existing reservation.

• To view this Intent, select the Manage tab, then the Intents list on the left menu. This Intent has a few training phrases to indicate that the user wants to look up existing Flight information.

This Intent will direct to the Find paid ticket page.

Find paid ticket page

1. Navigate to the Build tab. The Visual Builder shows the Start page connected to the Find paid ticket page.

2. Select the Start page to see that a new Route has been added using the main.look_up_flight Intent.

3. Select the Route main.look_up_flight to see that this Intent routes to Find paid ticket page in the Transition section.

4. Explore the configuration of the Find paid ticket page - it will look up a confirmation number by collecting a required confirmation number and then provide a mock response of the flight details.

To do this, a parameter has been added, as well as a route, to validate that all the parameters have been captured.

Confirmation Number Parameter

1. Select the Find paid ticket page and notice that the base agent already has a parameter named confirmation_number.

2. Select the confirmation_number parameter to see that its configuration has the following:

   • Display name: confirmation_number
   • Entity type: @sys.number
   • The Required checkbox checked
   • In the Fulfillment section, the Agent asks What is your flight confirmation number?

3. Now look how the parameter is handled once it's been gathered - the response with a mock reservation number.

Route on collected parameter

1. From the Find paid ticket page, select the new route named $page.params.status = "FINAL"

2. In the Condition section:

• Match At Least One rule option is selected.

For this lab, in the Fulfillment section of this Route, the Agent says will only have two responses to be returned.

First, a response that thanks the user for the confirmation number. Second, a mock response of the result of a flight information lookup.

In a production scenario where there is an existing datasource of flights, you'd configure a Webhook in Conversational Agents to connect to that backend system which stores flight information to retrieve the appropriate data. Creating a webhook and configuring it for this Page is outside the scope of this lab.

Now, test this agent to see what the user should expect so far.

3. To test the agent, click on Toggle Simulator in the upper-right corner to open it if it's not already open.

4. In the Talk to agent box, enter I'd like to look up my flight and then a number, such as 12345.

You should receive two sentences, a thank you and the mock flight lookup.

Note: If you typed "lookup", you will not see the correct flow! To properly train your agent, you'll need to anticipate how users will interact with it. Bookmark this link to Training phrases to learn more.

Great! Now you're all set to configure IVR and other features!

Task 4. Enable IVR features

The Speech and IVR Settings in Conversational Agent can be enabled at three different levels: Agent level, Flow level and Page level.

To enable the settings at different levels, the Agent level settings have to be enabled first. You'll do this in the next section.

The Agent level settings have three features to point out for Speech and IVR:

• End of speech sensitivity: helps determine how sensitive the speech endpointing should be when the caller finishes speaking, and it uses Google Cloud Speech.

• Barge-in: allows a speaker to interrupt the agent. This is helpful so that the caller doesn't have to wait to hear the full response from the agent in order to speak. Another way to understand this is, if Barge-in is disabled, the agent will only begin to listen to the user once the agent itself has finished playing its response.

• Google Cloud Storage URI: enables you to reference a path to access pre-recorded audio for playback.

After enabling IVR settings at the Agent level, you can customize IVR settings on a per Flow level and at the Page level. Enabling settings at the Flow level overrides the default Agent level IVR settings. This is important if you want to have different settings per Flow in a multi-flow Agent.

How you use Flow and Page level IVR settings depends on your use case. For example, if the user is dictating a number, the user will benefit from extending the timeout period so they're not cut off by the Agent at the Page level where this input occurs. If the user stays silent, because they may be searching for their flight number, this will allow the agent to help the user further by prompting on where to get that account number.

Agent level Speech and IVR settings are in the Agent Settings section in the Speech and IVR tab.

To access Flow level IVR settings, in the left pane, hover over Default Start Flow, and then click the context menu () > Flow settings.

Page level IVR settings are available from the Pages list on the left of the console, also via the Page name's context menu.

Start with enabling the Agent level Speech and IVR settings first in the Agent Settings.

Enable Agent Settings

1. Click Agent Settings at the top right.

2. Click the Speech and IVR tab.

3. Check the box to enable each of the following:

   • Enable auto speech adaptation

   • Enable advanced speech settings

   • Enable barge-in

4. Click Save.

Enable DTMF

Dual-Tone Multi-Frequency (DTMF) allows users to use the keypad on their phone to provide a response instead of using their voice. There are two types of DTMF implementations: Single-digit DTMF and Multi-digit DTMF. The Single-digit DTMF collects only one digit from a user response, while Multi-digit collects multiple digits in a response. In this lab, you will learn how to implement a multi-digit DTMF for collecting flight confirmation numbers.

1. Go to Find paid ticket Page and click the Parameter confirmation_number.

2. Scroll down to the DTMF settings and check the box to Enable DTMF. For now, assume the confirmation number is usually a 5 digit number such as "12345".

   Set the Max digits to 5. You will not set a finish digit for this lab, but you should consider setting a finish digit for a production virtual agent as a sign that the user has finished entering digits - silence does not always mean that the user has finished typing. Also, a webhook is usually implemented to validate the numbers to make sure the user input matches the expected value in the customer database.

3. Click Save.

4. Close the Find paid ticket page.

You will not be able to test this feature in the simulator.

Knowledge check

Enable/disable Barge-in

Barge-in allows users to interrupt an agent in the middle of a response. This helps the user to move along the flow faster, if they are not interested in the content that the agent is providing.

Barge-in can be enabled through the Advanced Settings on the agent level settings, flow level settings and page level settings. Since you already enabled Barge-in at the agent level, you do not need to enable barge-in everywhere else in the agent. For this lab, you'll disable the barge-in on the Confirm trip page.

1. Click the Confirm trip page, then click the entry fulfillment.

2. On the Fulfillment page, scroll down to Advanced settings and find Barge-in. Since this fulfillment provides the key information for the passenger who books the flight ticket, you do not want users to exit the flow in the middle of this response. Deselect the Enable barge-in option here by clicking Customize, which overrides the flow level barge-in setting.

3. Click Save.

4. Close the Confirm trip page.

Knowledge Check

Task 5. Handling error scenarios

To improve the user experience for callers with a voice agent, it's important to have a graceful way to handle conditions where the agent may have misunderstood, misheard, or is unable to collect the expected information - this is called conversation repair. Repairing a conversation can help build trust with the user by showing that the voice agent is listening to the request and attempting to understand.

Common conversation failures include inability to detect a verbal response from the user and the inability to match the intent of the user. In these cases, you'll implement the built-in events NoInput and NoMatch, respectively, to handle these, and make sure you're not stuck in an error loop by escalating to an Agent Handoff page after 3 errors.

Conversational Agents has built-in events for NoInput and NoMatch that are available at the flow, page, and parameter levels. Additionally, there are up to 6 numerically ordered events, such as sys.no-match-1, sys.no-match-2, etc., where you will be able to decide on the number of attempts that users can make for each type of event and create customized agent responses.

• Flow-level event handlers apply to a whole Flow and are useful in the case where there are broad event requirements that need to be fulfilled when using a Flow, such as transitioning from one Flow to another.
• Page-level event handlers apply when there are unexpected end-user inputs or other errors in transitioning between Pages.
• Parameter-level event handlers are useful within a Page when capturing a single or a series of parameters that are needed.

This lab will focus on Parameter-level events around capturing the ticket confirmation number.

Handling silence/noise with NoInput

Sometimes, especially in a voice scenario, the end user might not say a confirmation number quickly enough, or there is a long silence. Also, background noise or static that is not recognized as any text is considered as no-input instead of no-match.

When the Agent registers this as a NoInput event, the built-in feature of Conversational Agent can gracefully handle this and keep the user engaged and move along the flow.

State Handler events, such as NoInput and NoMatch can be added at a variety of levels, at the Flow, on the Page itself, and also on specific Parameters.

For this exercise, you'll add the No-Input and No-Match events to the ticket confirmation number parameter.

1. Open the Find paid ticket page and open the Parameter confirmation_number.

2. Scroll to the Reprompt Event handlers section of the Parameter and click the Add event handler link.

3. In the Event dropdown menu, select No-input 1.

4. In the Fulfillment section, in the Agent response, add the text Sorry, I didn't get that. Please enter or say your ticket confirmation number.

5. Click Save to save this Event.

6. Repeat above steps with the following to set up two more events.

   Event	Fulfillment	Transition
   No-input 2	To look up your flight information, we would need your ticket confirmation number. Please say or enter the confirmation number.
   No-input 3	You have not provided a confirmation number yet. Let me transfer you to a live agent to further assist you.	Page > Agent handoff

7. On the third, No-input 3, scroll down to the Transition section and configure the Transition to a new Page called Agent handoff.

8. Click Save. Close the Event handler and Find paid ticket pages.

   Your agent should now look like this

   

9. Test this by opening up the simulator by clicking Test Agent located at the upper right of the Conversational Agents console.

10. Type in a request for flight information, such as "I'd like to look up my flight information". When the agent asks for a confirmation number, instead of entering anything, press Enter a few times to simulate no input by the user.

Knowledge Check

Select if the following statement is true or false:

Handling unrecognized input with No-Match

Add NoMatch events in the case that there's input received, but the agent is unable to match the confirmation_number parameter.

1. From the Find paid ticket's Page, in the Parameter confirmation_number's panel, add 3 new events, with the third No-match 3 transitioning to the Agent handoff page.

   Event	Fulfillment	Transition
   No-match 1	Sorry, I didn't get that. Can you rephrase that?
   No-match 2	I'm still having trouble. Can you try again?
   No-match 3	Let me transfer you to someone else who can help.	Agent handoff

2. Test this by opening the simulator (click Test Agent) and asking the Agent to look up flight information. Since you defined the flight number as a @sys.number, the Agent will be expecting all numbers. If you respond to the Agent with letters, for example abcd, that'll be registered as a NoMatch. After three attempts, you'll see the NoMatch handler transition to the Agent handoff page.

Knowledge Check

Select if the following statement is true or false:

A note on Agent Handoff

For this example we've added a page called Agent handoff which is used as the destination for the expected transfer to a live agent in the case this virtual agent is unable to detect a response or determine the correct confirmation number.

For this lab, while there is no target destination for a live agent, you can see where this would be configured. Conversational Agents provides a Fulfillment type that can be used to signal to the telephony or chat client to perform the required transfer.

1. To see where this is located, open the Agent handoff page and select the Entry fulfillment.

2. Select Live agent handoff from the Add dialogue option dropdown.

This results in an area to provide a custom JSON message.

Every target live agent system is different. Refer to the system's documentation as to what message format will be necessary to add to provide the proper communication parameters.

As an example, if you're using Business Messages, the format that you'd enter here would look something like this:

{ "userStatus": { "requestedLiveAgent": true } }

Please check the Business Messages documentation for the precise JSON message.

For more information on Business Messages live agent handoff formats, see Handoff from bot to live agent.

Task 6. Add SSML support to your agent

Speech Synthesis Markup Language (SSML) enables you to customize your audio responses by providing details on pauses, audio formatting for numbers, dates, or text. This lets your agent have a more natural conversation.

<speak>is the root element of SSML response. Without this element, your text cannot talk. Implement them on the Find paid ticket page.

1. From the Find paid ticket page, click the $page.params.status="FINAL" route and scroll down to the fulfillment.

2. Add the <speak> element to the entire text of the second fulfillment. Remember to close the text with </speak>.

   

   Now, increase the pause after "Here is the flight information". You can use the empty element <break time> to control the pausing between words or sentences. The break time length can be seconds or milliseconds.

3. Add a pause after "Here is the flight information" with the following:

<break time = "1s"/>

This is what it will look like after you add the break time.

4. You can also adjust the speed of the response by using the <prosody> element.

5. Add the following before "Here is the flight information" so that this response will be rendered at a slower speech rate to allow users to take notes of their flight details:

<prosody rate="slow">

When you finish adding the above mentioned SSML, this is what your fulfillment will look like.

In the next section you'll have an opportunity to test some of the settings you've created for your Agent. If you're running out of time, go as far as you can. The next section will not be part of your lab's score.

Task 7. Optional: Testing the agent with Conversational Agents phone gateway

Conversational Agents can be integrated with various conversation and telephony providers either directly through the 1-click Integrations in the Conversational Agent Console or via the Conversational Agents API.

Conversational Agents includes a preview feature called the Conversational Agents Phone Gateway that provides a telephone interface to your agent. For this lab, you'll use the Conversational Agents Phone Gateway.

Please note that this feature has limited functionality. Current limitations are as follows:

• Agents must be in the global region
• Only US phone numbers are supported
• Features NOT supported relevant to this lab are: DTMF

Task 8. Set up a phone gateway

1. From the Main Menu tab in the Conversational Agents console, select Integrations.

2. In the Conversational Phone Gateway section, click Manage and then Create new.

3. Set Country Code to United States.

4. Enter an Area code of your choice, or leave it blank.

5. Click Request.

6. Select a phone number option, then add a display name.

7. Press Save.

You've reserved a number!

Click Check my progress to verify the objective. Set up a phone number for agent

Test your agent

• On your personal phone, call the phone number you created and follow the voice prompts. Try out the no-match, no-input, and barge-in features you enabled during this lab. (The Conversational Agent Gateway does not currently support DTMF.)

If the agent reaches the end session state, the call ends.

Task 9. (Optional) Exporting your agent

When you build an agent for one project, you can export it to use in a different project. You can export your agent to continue building upon it in your own personal project!

1. In the Agent drop down at the top of the Conversational Agents console, click View all agents.
2. On the Agent list screen, click the context menu () next to your agent and then click Export.
3. On the Export Agent screen, choose Download to local file, then click Export.

Congratulations!

You've made your Conversational Agents Agent IVR-ready! In this lab, you have learned how to configure Speech and IVR Settings and implement IVR features such as DTMF and Barge-in. You have also learned to use No-Input and No-Match to handle conversational repair and to use SSML to help create a more natural speech.

Next steps / Learn more

• Built-in events and their levels: https://cloud.google.com/dialogflow/cx/docs/concept/handler#event-built-in
• To learn more about SSML: https://cloud.google.com/text-to-speech/docs/ssml
• See more about the Conversational Agents Phone Gateway here: https://cloud.google.com/dialogflow/cx/docs/concept/integration/phone-gateway
• Read more about Conversational Agents in the Conversational Agents Documentation

Manual Last Updated May 28, 2025

Lab Last Tested May 26, 2025

Copyright 2025 Google LLC. All rights reserved. Google and the Google logo are trademarks of Google LLC. All other company and product names may be trademarks of the respective companies with which they are associated.