Evaluate bot performance

Understand overall bot performance with the Experience and Automation page.

The Experience and Automation page provides you with a high-level summary of how your bot is doing. Metrics, such as Bot Automation Score (BAS) and Bot Experience Score (BES), offer insights into the overall automation and experience the bot is providing by taking into account a variety of signals.

Bot Experience Score (BES)

Bot experience score is an all encompassing metric used to measure the experience of the conversation a user had with a chat bot or voice bot. It takes into consideration the following negative signals to arrive at the experience score.

Bot repetition - the bot repeats itself for any reason during a conversation
User paraphrase - the customer uses a similar query twice or more in a conversation
Abandonment - the customer leaves the conversation in the middle without reaching a legitimate end response configured on the bot
Negative sentiment - AI-based sentiment model of conversation
Negative feedback - explicit negative feedback received in the conversation
Profanity - profanity present in the conversation
Request to escalate multiple times - the customer used the word “agent” or a similar word more than once in a conversation. Note that using the word "agent” once and being directly escalated is not generally a bad experience.

Bot Automation Score (BAS)

An all encompassing metric used to measure the effectiveness of the conversation that a user had with a chat or voice. Essentially, it shows how often the bot can satisfy the customer's needs without escalation to a live agent. Calabrio ONE takes into consideration the following negative signals to arrive at the automation score.

Live agent escalation
Escalation requested but not connected
Negative feedback - explicit negative feedback received in the conversation
Abandonment - the customer leaves the conversation in the middle without reaching a legitimate end response configured on the bot
False positive - the customer received an unrelated response to their question

Prerequisites

You have a Bot Analytics license.
You have the View Experience Automation permission.

Page location

Bot Analytics > Home > Experience and Automation

Procedures

Improve overall BAS and BES scores

The Conversation Topics section at the bottom of the page shows individual BES and BAS scores for each conversation topic.

NOTE Scroll horizontally to view BES scores, BAS scores, and other metrics.

Identify conversation topics that are below the overall BES/BAS scores and focus on improving these topics.
Navigate to the Metric Analysis tab (Bot Analytics > Conversation Analytics > Conversation Topics > Metric Analysis tab). The Metric Analysis page provides a breakdown of the negative signals impacting BES and BAS scores for that topic.
Utilize transcripts to identify why these negative signals are appearing and strategize ways to minimize them. Note, focusing on large volume conversation topics will have a larger impact on the overall BES/BAS scores.

Understanding overall bot performance with the Virtual Agent Analytics (VAA) Overview page

The VAA Overview page provides insights into how the bot is performing through a detailed metric breakdown.

The page shows highlights of key drivers of virtual agents overall performance. It includes natural language understanding (NLU), containment, and customer satisfaction.

Analyze the insights on the VAA Overview page, which allows bot management teams to have a high-level view of under-performing areas.
View metrics such as targets, NLU training, corpus, contained conversations, and handed off conversations.

Improve Natural Language Understanding (NLU)

The VAA Overview page identifies three important metrics at the top of the page: Natural Language Understanding, Virtual Agent Conversations Contained, and Negative Feedback Score.

Select a metric you would like to improve and navigate to the corresponding section on the page. This section provides more context and metrics that influence the overall score. For example, NLU training shows metrics such as false positive rates, undertrained intents, and similar intents that all have an impact on overall NLU score. Use these more detailed metrics to make an impact on overall performance.