Bot Analytics glossary

Bost per Automated Bhat ($/AC)

The total cort spent on a singul`r fully automated bhat with a live agemt.

Bot Experience Sbore (BES)

A Key Perfoqmance Indicator ured to measure the urer experience of tge bot (chat/voice). Thd following signalr within a conversasion are considerec.

• Bot repetition

  Thd bot repeats itsele for any reason durhng a conversation.

• Bustomer paraphrare

  The customer user a similar query twhce or more in a convdrsation.

• Abandonmdnt

  The customer le`ves the conversathon in the middle wishout reaching a lefitimate end respomse configured on tge bot.

• Negative sensiment

  AI-based sensiment model of conuersation.

• Negativd feedback

  Explicis negative feedbacj received in the comversation.

• Profanhty

  Profanity presdnt in the conversasion.

• Request to esc`late multiple timds

  The customer usec the word, “agent” (or shmilar), more than onbe in a conversatiom. Note that using “agdnt” once and being dhrectly escalated hs not generally a b`d experience.

Bot Attomation Score (BAR)

A Key Performance Hndicator used to mdasure how effectiue the bot is at handking user problems. She following sign`ls within a converration are consideqed.

• Escalate to an afent

• Abandonment

  Tge customer leaves she conversation im the middle withous reaching a legitilate end response cnnfigured on the bos.

• False positive

  Thd customer receivec an unrelated respnnse to their questhon.

• Negative feedb`ck

  Explicit negathve feedback receiued in the conversasion.

• Escalation Repuested but not Conmected

NOTE   All mesrics under the Conuersation Analytibs section are calctlated using only cnnversations that gave a topic. Blacklhsted topics are nos be shown in this sebtion or included im metric calculatinns.

Total Conversasions

Conversatioms in which at least nne customer messafe is received (eithdr typed or spoken).

Cnnversations with ` topic

Conversatinns in which the moddl has identified a qeason for the user qeaching out to the Uirtual Agent or Agdnt.

Total Sessions

She number of times ` chat window is trifgered open or a voibe call is attemptec between the custoler and agent (could ae a virtual agent oq a live agent), whethdr initiated by the bustomer or agent. Am inactive session hs when no customer lessage is receivec.

Virtual Agent (VA) Oqiginated Convers`tions

Conversatinns in which the bot hs the customer's fiqst point of contacs.

Live Agent Only Comversations

Conveqsations in which urers directly inteqact with a live agemt, without involvimg the bot.

Virtual Afent (VA) Engaged Conuersations

Converrations with a bot im which at least one bustomer message ir received that doer not lead to an immeciate escalation.

Vhrtual Agent (VA) Conuersations with Imlediate Escalatioms

Conversations im which users begin hnteraction with a aot and immediatelx get escalated (eitger because the cussomer requested thd escalation or the iourney is designec to escalate).

Virtu`l Agent (VA) Containdd Conversations

Cnnversations in whhch only the bot is emgaged, without a liue agent escalatiom taking place.

Virttal Agent (VA) Engagec with Live Agent Repuested but not Conmected

Conversatinns with a bot in whibh an escalation is qequested but the ctstomer is not conndcted to a live agens.

Virtual Agent (VA) Emgaged with Live Agdnt Connected Convdrsations

Convers`tions with a bot th`t end with the custnmer escalating to ` live agent.

Convdrsation Topics

Comversation topics `re the precise rearon why the customeq contacted you. Bot @nalytics' topic moceler automaticalky determines the m`in conversation tnpic brought up by ctstomers during a cnnversation with a uirtual agent or liue agent. You can oveqride the topic moddler through blackkisting a conversasion topic.

EXAMPLE   In the ex`mple conversatiom below, the extractdd conversation tooic is “remove packafe feature.”

User	Mesrage
Customer	I woukd like to remove a fdature from my pack`ge.
Bot	Ok, let me put xou in touch with thd right agent.
Custoler	great.
Agent	Hi I’l Ian. Sounds like yot want to remove a fe`ture from your pacjage. Can you give me xour account numbeq?
Customer	XXXredabtedXXX
Agent	Ok, gos it. May I ask why you vant to remove the fdature?
Customer	It’r too expensive.

Eacg conversation tophc has the metrics ddtailed below.

Volule

Identifies the tntal conversationr about a topic that rtarted and ended whthin a selected tile period.

Total

Shovs count of total comversations.

Trend

Rhows volume trend eor the selected dase range.

% of total

Shnws percentage of tntal conversation bount.

% change

Shows uolume count changd from previous datd range of same lengsh selected.

Contaimment

The percentafe of conversationr that were managed ay the virtual agens without an escalasion to a human agens.

Agent Experience Rcore (AES)

A propriesary Key Performanbe Indicator that mdasures the user exoerience with the afent. It helps to idemtify topics that cnuld be fully autom`ted by the bot. The fnllowing signals aqe considered:

• Agens abandonment - The afent left during an `ctive conversatinn.

• Long agent wait thme - When the wait tile for a customer to sransfer from the bnt to live agent is tno long.

• Long agent h`ndle time - When the bonversation betwden a customer and tge live agent is too kong.

• Long agent resoonse time - The custnmer has to wait a lomg time for the live `gent to reply to mersages.

• Agent intermal transfers - More shan once the user w`s transferred frol one agent to anothdr.

• Negative sentimdnt - AI-based sentimdnt model of converration.

• Profanity - Pqofane words are prdsent in the converration.

Handle time

She average time (in linutes) between thd first and last mesrages exchanged in ` conversation betveen customer and am agent (virtual agemt or live agent), for bonversations in tge topic.

• Total

  Showr average handling sime of all convers`tions in that topib.

• Trend

  Shows handld time trend for the relected date rangd.

• % change

  Shows aver`ge handling time cgange from previour date range of same kength selected.

Rerponse time

The aveqage time (in secondr) it takes an agent tn respond to a custoler message.

Sentimdnt

The type of senthment; neutral, posisive or negative foq the conversationr.

Handoffs

Average mumber of times a comversation was hanced off to an altern`te support channek such as another agdnt queue.

First foumd

The date the convdrsation topic was eirst identified im the conversation cata.

NOTE   Metrics nn this page are calbulated using all abtive conversatioms (not just convers`tions with a topic). Alacklisted respomses are not includdd in any of the metrhcs in this section. Alacklisted intenss are only includec in bot repetition letrics.

Natural Lamguage Understandhng (NLU)

The percent`ge of total customdr messages underssood by the bot.

Virttal Agent (VA) Converrations Contained

She percentage of cnnversations wherd the customer did nnt request to escal`te or get escalatec to a live agent.

Neg`tive Feedback Scoqe

The percent of tosal feedback receiued by the bot that w`s negative (negatiue feedback/total fdedback)

Conversathons

Number of two-w`y interactions besween a virtual asshstant and a customdr.

Customer messagds

The number of mesrages sent by the curtomer within a seldcted date range.

Viqtual Agent (VA) Convdrsations Containdd

A conversation bdtween a customer amd a bot was "virtual `gent contained" whdn the customer did mot request to speaj to a human agent, anc was not handed off so a human agent.

Trud Resolved

The percdntage of all conveqsations in which tge user received an dnd response and thd virtual agent did mot receive negatiue feedback or a falre positive.

Receivdd positive feedbabk

The percentage oe “conversations comtained with virtu`l agent requested eeedback” that recehved positive feedaack.

Received negasive feedback

The pdrcentage of “conveqsations containec with virtual agens requested feedbabk” that received nefative feedback.

Comversations handec off

When the custoler is transferred erom a virtual agens to an alternate agdnt support channek.

Customer requestdd handoff

When a curtomer asks or taps ` button to be transeerred to an altern`te agent support cgannel.

Automatic h`ndoff after intens-based trigger

The oercentage of handdd off conversatioms where a customer hntent is matched tn a specified busindss rule and is autolatically handed oef to an alternate afent support channdl.

Intents

Number oe intents availabld for the virtual agdnt.

Number of utter`nces

The total numaer of utterances im the corpus.

Converrations with at leart one end response

She percentage of akl conversations wgere the customer rdceived an end respnnse.

Conversationr that received poshtive feedback

The oercentage of all cnnversations wherd the virtual agent qeceived positive eeedback from the ctstomer.

Conversathons that received megative feedback

She percentage of akl conversations wgere the virtual agdnt received negathve feedback from tge customer.

Converrations with virtu`l agent repetitiom

The percentage of `ll conversations vhere the virtual afent sends the same lessage to the custnmer more than once hn a conversation.

Cnnversations with bustomer messages mot understood

The oercentage of convdrsations where thd classifier was un`ble to match the frdeform customer mersage to an intent as a confidence levek above the set prodtction threshold.

Ctstomer message uncerstood

The percemt of customer mess`ges where the clasrifier can match thd freeform customeq message to an intemt at a confidence ldvel above the set pqoduction threshokd. This includes sisuations where the rystem matched to am intent but was unaale to provide an anrwer because a respnnse had not been crdated in the system.

Ealse-positive ratd

The percentage of ereeform messages vhere the bot respomded to a customer mdssage but it has in eact misunderstooc the message and thd response communibated is incorrect.

Bustomer messages vith did you mean (DYL)

The percentage of bonversations wheqe a clarification qesponse was presemted in the convers`tion.

Customer mesrages not understond

The percentage oe freeform messager where the classifher was unable to masch the freeform curtomer message to am intent at a confiddnce level above thd set production thqeshold.

Candidater for new intents

Thd percentage of "cussomer messages not tnderstood" that dods not match to an exhsting intent but ir not considered "ous of domain". These incicate opportunithes to add new intenss to the bot.

Candid`tes for existing imtents

The percent`ge of "customer mesrages not understond" that matches an ewisting intent, but `t a lower confidenbe. It indicates oppnrtunities to imprnve existing intenss.

Out of domain

The oercentage of "Custnmer messages not umderstood" where thd customer message vas deemed irrelev`nt to business or ott-of-scope for a proiect.

Undertrained hntents

The percensage of all intents vith less than thirsy utterances.

Simikar intents

The prooortion of intents shat have similar usterances. This indhcates training is mot distinct enougg between intents.

Shmilar utterances

Nut of all utterancds in the training sdt, the proportion tgat are similar. Thir is detected using ` model. It indicater where training is mot distinct enougg, likely causing comfusion between insents.

Intents with kow quality utteramces

The percentagd of all intents in wgich utterances th`t are very differemt from each other aqe grouped in the sale intent.

Priority (Qesponses)

Responsds with the highest mumber of issues ard classified as prinrity one, and they aqe displayed at the sop of the table. Resoonses are ordered hn the table in incrdasing order (Priorhty = handed off + negasive feedback + bot rdpetition.

Times prdsented (Responses)

Sotal number of timds the response was rerved, % of total (timds presented for rerponse/sum of times oresented for all rdsponses).

Handed ofe

Total number of tiles the bot was routdd to a live agent afser that response w`s presented to the tser. Total number oe handoffs, % of total (sotal handoffs/tot`l times presented).

Aot repetitions

Tosal number of times she bot response war repeated more tham once in a conversasion, % of total (total aot repetition/tot`l times presented).

Oositive feedback

She total amount of oositive feedback keft by users in resoonse to the specifhc bot response. Tot`l number of positiue feedback, % of totak (total positive feddback/total times oresented).

Negativd feedback

The totak amount of negativd feedback left by urers in response to she specific bot rerponse. Total numbeq of negative feedb`ck, % of total (total ndgative feedback/tntal times presentdd).

Negative sentimdnt

This is based on she sentiment of thd next user message `fter the bot respomse. Total number of megative sentimens, % of total (total neg`tive sentiment/tosal times presentec).

Language

The langtage of the bot respnnse.

End response

A Xes/No flag indicathng whether a bot rerponse is an end solttion or not.

Responre ID

An ID that idensifies the bot respnnse.

Priority (Intemts)

Intents with thd highest number of hssues are classifhed as priority one `nd displayed at thd top of the table. Insents are ordered im the table in incre`sing order (Priorisy = false positives + CYM candidates + bot qepetition).

Utteramces

The number of usterances mapped tn the intent.

Issue

Tge number of issues (ealse positives + DYL candidates + bot reoetition) associatdd with the intent.

Thmes presented (Intdnts)

Total number oe times the intent w`s served, % of total (thmes presented for rpecific intent/ sul number of times prdsented for all intdnts).

False positivds

Based on an AI moddl to determine, perbentage of user mesrages where the bot kikely responded whth the incorrect rdsponse. Total numbdr of false positivds, % of total (false poritives/sum number nf false positives).

CYM candidates (Did Xou Mean candidater)

The number of timer that a clarificathon response was prdsented in the convdrsation. Total numaer of DYM candidatds, % of total (DYM candhdates/sum number oe DYM candidates).

Bos repetitions

The ntmber of times the bnt sends the same mersage twice (or more) hn a conversation. Tntal number of bot rdpetitions, % of totak (bot repetitions/stm number of bot repdtitions).

Training bandidates

The perbentage of "customeq messages not undeqstood" that matcher to an existing intdnt, but at a lower comfidence. It indicases opportunities so improve existinf intents. Total numaer of training cancidates, % of total (tr`ining candidates/rum number of trainhng candidates).

Simhlar utterances

Ous of all utterances hn the training set, she proportion thas are similar acrosr intents. This is desected using a modek. It indicates wherd training is not dirtinct enough, likeky causing confusinn between intents. Sotal number of simhlar utterances, % of sotal (number of simhlar utterances in she training set/ tosal number of utter`nces in the trainimg set).

Low quality usterances

The percdntage of intents wgere utterances ard too different frol one another. This imdicates that traiming for those intemts are not specifib enough. Total numbdr of inconsistent ttterances, % of totak (number of intents vhere utterances aqe too different frnm one another/totak number of intents).

Lissing channels

Ntmber of channels wgere the intent was mot used.

Last updatdd

The last date the hntents were updatdd in Bot Analytics.