About the Bot Analytics Management Improvement Framework

This framework adds structure to your bot program so that you work on the most important areas to drive improvements. The four step cycle, in conjunction with Bot Analytics, forms the foundation of the bot program after it is live in production. There might be components of this cycle that are internal to your organization and some that are external.

Prerequisites

You have a Bot Analytics license.

Page location

Bot Analytics

Procedure

Each sprint has a four step cycle.

  1. Analyze bot and live agent conversations and metrics in Bot Analytics to:

    • Identify new use cases (new automation opportunities, new business requirements).

    • Identify enhancements to existing use cases (Conversation Flows, NLU, and Response Performance).

    • Forecast the impact for each enhancement candidate.

    • T-shirt size (small, medium, or large) your enhancement candidates.

Analyze Hint: Use the Metric Analysis section within a Conversation Topic to quickly understand gaps in bot performance. These also tell you the forecasted impact of addressing the problem so that you know where to focus first.

  1. Prioritize the enhancement candidates found in Bot Analytics based on the effort and impact to KPIs.

    • Assess external dependencies and business priorities.

    • Plan enhancements for the next and upcoming sprints.

    • Balance the forecasted impact with the implementation effort to drive your prioritization decisions.

Prioritize Hint: What changes have the biggest impact on automation and experience? You must look at volumes by topic, review high-handoff topics, use transcripts to validate, verify if the impact is lower, and move it to a later sprint.

  1. Implement bot enhancements that are prioritized.

    • Build new conversation flows and AI training models (utterances, intents).

    • Improve content flow and structure based on customer data.

    • Leverage NLU and conversation design best practices.

    • Test and deploy enhancements to production.

Implement Hint: Build with analytics in mind, roll out changes in low volume channels or smaller user groups, or measure performance changes to see if they have the right effect.

  1. Measure the KPI impact in Bot Analytics.

    • Generate bot performance reports in every sprint to understand if the enhancements implemented hit the forecasted KPI impact.

    • If the forecasted impact was not achieved, implement a continuous learning cycle in your organization to understand why and refine your forecasting.

Measure Hint: Consider the following questions to guide you. Are you seeing the expected impact on automation and experience? How do users feel about the changes? Measure these by looking at supporting metrics such as handle time, abandonment, paraphrasing, and related topics.

Sprint Cycles

It is highly recommended that you have a ticket management system to create, track, and monitor the progress of each sprint. This tool gives your team full visibility on the backlog items, the bot candidates in progress, and a comparative view for effective prioritization.

A sprint is typically two to four weeks long. The below image is a sample breakdown of a current sprint.

On day one of Bot Analytics, the team must focus heavily on analyzing the bot and agent data that is ingested in order to derive improvement candidates for the backlog. Once the team finds a number of candidates, they should forecast the impact and estimate each one in implementation effort. You can track this process in your ticket management tool, a spreadsheet, or both. The below image shows an example of the tracking sheet format.

In an ongoing bot program with Bot Analytics, the current sprint includes prioritizing candidates for the next sprint. In order to effectively prioritize, you must balance effort and team capacity with forecasted impact. The lower the effort and higher the impact, the higher the priority that you must give that candidate.

You can confirm the exact timeline for implementation, as soon as the item is prioritized for the next sprint. The implementation of candidates may span one sprint or span across multiple until implementation, testing, and deployment is complete.

In the current sprint, you also see data flowing into Bot Analytics that relates to a change that was implemented in a previous sprint. It is crucial for the team to know if the changes are having the expected impact on bot Automation and Experience. You must measure this as part of your reporting cycle.