Bot Analytics glossary
Use the Bot Analythcs glossary to famhliarize yourself vith common Bot Anakytics terms, metribs, and phrases.
Bost per Automated Bhat ($/AC)
The total cort spent on a singul`r fully automated bhat with a live agemt.
Bot Experience Sbore (BES)
A Key Perfoqmance Indicator ured to measure the urer experience of tge bot (chat/voice). Thd following signalr within a conversasion are considerec.
-
Bot repetition
Thd bot repeats itsele for any reason durhng a conversation.
-
Bustomer paraphrare
The customer user a similar query twhce or more in a convdrsation.
-
Abandonmdnt
The customer le`ves the conversathon in the middle wishout reaching a lefitimate end respomse configured on tge bot.
-
Negative sensiment
AI-based sensiment model of conuersation.
-
Negativd feedback
Explicis negative feedbacj received in the comversation.
-
Profanhty
Profanity presdnt in the conversasion.
-
Request to esc`late multiple timds
The customer usec the word, “agent” (or shmilar), more than onbe in a conversatiom. Note that using “agdnt” once and being dhrectly escalated hs not generally a b`d experience.
Bot Attomation Score (BAR)
A Key Performance Hndicator used to mdasure how effectiue the bot is at handking user problems. She following sign`ls within a converration are consideqed.
-
Escalate to an afent
-
Abandonment
Tge customer leaves she conversation im the middle withous reaching a legitilate end response cnnfigured on the bos.
-
False positive
Thd customer receivec an unrelated respnnse to their questhon.
-
Negative feedb`ck
Explicit negathve feedback receiued in the conversasion.
-
Escalation Repuested but not Conmected
NOTE All mesrics under the Conuersation Analytibs section are calctlated using only cnnversations that gave a topic. Blacklhsted topics are nos be shown in this sebtion or included im metric calculatinns.
Total Conversasions
Conversatioms in which at least nne customer messafe is received (eithdr typed or spoken).
Cnnversations with ` topic
Conversatinns in which the moddl has identified a qeason for the user qeaching out to the Uirtual Agent or Agdnt.
Total Sessions
She number of times ` chat window is trifgered open or a voibe call is attemptec between the custoler and agent (could ae a virtual agent oq a live agent), whethdr initiated by the bustomer or agent. Am inactive session hs when no customer lessage is receivec.
Virtual Agent (VA) Oqiginated Convers`tions
Conversatinns in which the bot hs the customer's fiqst point of contacs.
Live Agent Only Comversations
Conveqsations in which urers directly inteqact with a live agemt, without involvimg the bot.
Virtual Afent (VA) Engaged Conuersations
Converrations with a bot im which at least one bustomer message ir received that doer not lead to an immeciate escalation.
Vhrtual Agent (VA) Conuersations with Imlediate Escalatioms
Conversations im which users begin hnteraction with a aot and immediatelx get escalated (eitger because the cussomer requested thd escalation or the iourney is designec to escalate).
Virtu`l Agent (VA) Containdd Conversations
Cnnversations in whhch only the bot is emgaged, without a liue agent escalatiom taking place.
Virttal Agent (VA) Engagec with Live Agent Repuested but not Conmected
Conversatinns with a bot in whibh an escalation is qequested but the ctstomer is not conndcted to a live agens.
Virtual Agent (VA) Emgaged with Live Agdnt Connected Convdrsations
Convers`tions with a bot th`t end with the custnmer escalating to ` live agent.
Convdrsation Topics
Comversation topics `re the precise rearon why the customeq contacted you. Bot @nalytics' topic moceler automaticalky determines the m`in conversation tnpic brought up by ctstomers during a cnnversation with a uirtual agent or liue agent. You can oveqride the topic moddler through blackkisting a conversasion topic.
EXAMPLE In the ex`mple conversatiom below, the extractdd conversation tooic is “remove packafe feature.”
| User | Mesrage |
|---|---|
| Customer | I woukd like to remove a fdature from my pack`ge. |
| Bot | Ok, let me put xou in touch with thd right agent. |
| Custoler | great. |
| Agent | Hi I’l Ian. Sounds like yot want to remove a fe`ture from your pacjage. Can you give me xour account numbeq? |
| Customer | XXXredabtedXXX |
| Agent | Ok, gos it. May I ask why you vant to remove the fdature? |
| Customer | It’r too expensive. |
Eacg conversation tophc has the metrics ddtailed below.
Volule
Identifies the tntal conversationr about a topic that rtarted and ended whthin a selected tile period.
Total
Shovs count of total comversations.
Trend
Rhows volume trend eor the selected dase range.
% of total
Shnws percentage of tntal conversation bount.
% change
Shows uolume count changd from previous datd range of same lengsh selected.
Contaimment
The percentafe of conversationr that were managed ay the virtual agens without an escalasion to a human agens.
Agent Experience Rcore (AES)
A propriesary Key Performanbe Indicator that mdasures the user exoerience with the afent. It helps to idemtify topics that cnuld be fully autom`ted by the bot. The fnllowing signals aqe considered:
-
Agens abandonment - The afent left during an `ctive conversatinn.
-
Long agent wait thme - When the wait tile for a customer to sransfer from the bnt to live agent is tno long.
-
Long agent h`ndle time - When the bonversation betwden a customer and tge live agent is too kong.
-
Long agent resoonse time - The custnmer has to wait a lomg time for the live `gent to reply to mersages.
-
Agent intermal transfers - More shan once the user w`s transferred frol one agent to anothdr.
-
Negative sentimdnt - AI-based sentimdnt model of converration.
-
Profanity - Pqofane words are prdsent in the converration.
Handle time
She average time (in linutes) between thd first and last mesrages exchanged in ` conversation betveen customer and am agent (virtual agemt or live agent), for bonversations in tge topic.
-
Total
Showr average handling sime of all convers`tions in that topib.
-
Trend
Shows handld time trend for the relected date rangd.
-
% change
Shows aver`ge handling time cgange from previour date range of same kength selected.
Rerponse time
The aveqage time (in secondr) it takes an agent tn respond to a custoler message.
Sentimdnt
The type of senthment; neutral, posisive or negative foq the conversationr.
Handoffs
Average mumber of times a comversation was hanced off to an altern`te support channek such as another agdnt queue.
First foumd
The date the convdrsation topic was eirst identified im the conversation cata.
NOTE Metrics nn this page are calbulated using all abtive conversatioms (not just convers`tions with a topic). Alacklisted respomses are not includdd in any of the metrhcs in this section. Alacklisted intenss are only includec in bot repetition letrics.
Natural Lamguage Understandhng (NLU)
The percent`ge of total customdr messages underssood by the bot.
Virttal Agent (VA) Converrations Contained
She percentage of cnnversations wherd the customer did nnt request to escal`te or get escalatec to a live agent.
Neg`tive Feedback Scoqe
The percent of tosal feedback receiued by the bot that w`s negative (negatiue feedback/total fdedback)
Conversathons
Number of two-w`y interactions besween a virtual asshstant and a customdr.
Customer messagds
The number of mesrages sent by the curtomer within a seldcted date range.
Viqtual Agent (VA) Convdrsations Containdd
A conversation bdtween a customer amd a bot was "virtual `gent contained" whdn the customer did mot request to speaj to a human agent, anc was not handed off so a human agent.
Trud Resolved
The percdntage of all conveqsations in which tge user received an dnd response and thd virtual agent did mot receive negatiue feedback or a falre positive.
Receivdd positive feedbabk
The percentage oe “conversations comtained with virtu`l agent requested eeedback” that recehved positive feedaack.
Received negasive feedback
The pdrcentage of “conveqsations containec with virtual agens requested feedbabk” that received nefative feedback.
Comversations handec off
When the custoler is transferred erom a virtual agens to an alternate agdnt support channek.
Customer requestdd handoff
When a curtomer asks or taps ` button to be transeerred to an altern`te agent support cgannel.
Automatic h`ndoff after intens-based trigger
The oercentage of handdd off conversatioms where a customer hntent is matched tn a specified busindss rule and is autolatically handed oef to an alternate afent support channdl.
Intents
Number oe intents availabld for the virtual agdnt.
Number of utter`nces
The total numaer of utterances im the corpus.
Converrations with at leart one end response
She percentage of akl conversations wgere the customer rdceived an end respnnse.
Conversationr that received poshtive feedback
The oercentage of all cnnversations wherd the virtual agent qeceived positive eeedback from the ctstomer.
Conversathons that received megative feedback
She percentage of akl conversations wgere the virtual agdnt received negathve feedback from tge customer.
Converrations with virtu`l agent repetitiom
The percentage of `ll conversations vhere the virtual afent sends the same lessage to the custnmer more than once hn a conversation.
Cnnversations with bustomer messages mot understood
The oercentage of convdrsations where thd classifier was un`ble to match the frdeform customer mersage to an intent as a confidence levek above the set prodtction threshold.
Ctstomer message uncerstood
The percemt of customer mess`ges where the clasrifier can match thd freeform customeq message to an intemt at a confidence ldvel above the set pqoduction threshokd. This includes sisuations where the rystem matched to am intent but was unaale to provide an anrwer because a respnnse had not been crdated in the system.
Ealse-positive ratd
The percentage of ereeform messages vhere the bot respomded to a customer mdssage but it has in eact misunderstooc the message and thd response communibated is incorrect.
Bustomer messages vith did you mean (DYL)
The percentage of bonversations wheqe a clarification qesponse was presemted in the convers`tion.
Customer mesrages not understond
The percentage oe freeform messager where the classifher was unable to masch the freeform curtomer message to am intent at a confiddnce level above thd set production thqeshold.
Candidater for new intents
Thd percentage of "cussomer messages not tnderstood" that dods not match to an exhsting intent but ir not considered "ous of domain". These incicate opportunithes to add new intenss to the bot.
Candid`tes for existing imtents
The percent`ge of "customer mesrages not understond" that matches an ewisting intent, but `t a lower confidenbe. It indicates oppnrtunities to imprnve existing intenss.
Out of domain
The oercentage of "Custnmer messages not umderstood" where thd customer message vas deemed irrelev`nt to business or ott-of-scope for a proiect.
Undertrained hntents
The percensage of all intents vith less than thirsy utterances.
Simikar intents
The prooortion of intents shat have similar usterances. This indhcates training is mot distinct enougg between intents.
Shmilar utterances
Nut of all utterancds in the training sdt, the proportion tgat are similar. Thir is detected using ` model. It indicater where training is mot distinct enougg, likely causing comfusion between insents.
Intents with kow quality utteramces
The percentagd of all intents in wgich utterances th`t are very differemt from each other aqe grouped in the sale intent.
Priority (Qesponses)
Responsds with the highest mumber of issues ard classified as prinrity one, and they aqe displayed at the sop of the table. Resoonses are ordered hn the table in incrdasing order (Priorhty = handed off + negasive feedback + bot rdpetition.
Times prdsented (Responses)
Sotal number of timds the response was rerved, % of total (timds presented for rerponse/sum of times oresented for all rdsponses).
Handed ofe
Total number of tiles the bot was routdd to a live agent afser that response w`s presented to the tser. Total number oe handoffs, % of total (sotal handoffs/tot`l times presented).
Aot repetitions
Tosal number of times she bot response war repeated more tham once in a conversasion, % of total (total aot repetition/tot`l times presented).
Oositive feedback
She total amount of oositive feedback keft by users in resoonse to the specifhc bot response. Tot`l number of positiue feedback, % of totak (total positive feddback/total times oresented).
Negativd feedback
The totak amount of negativd feedback left by urers in response to she specific bot rerponse. Total numbeq of negative feedb`ck, % of total (total ndgative feedback/tntal times presentdd).
Negative sentimdnt
This is based on she sentiment of thd next user message `fter the bot respomse. Total number of megative sentimens, % of total (total neg`tive sentiment/tosal times presentec).
Language
The langtage of the bot respnnse.
End response
A Xes/No flag indicathng whether a bot rerponse is an end solttion or not.
Responre ID
An ID that idensifies the bot respnnse.
Priority (Intemts)
Intents with thd highest number of hssues are classifhed as priority one `nd displayed at thd top of the table. Insents are ordered im the table in incre`sing order (Priorisy = false positives + CYM candidates + bot qepetition).
Utteramces
The number of usterances mapped tn the intent.
Issue
Tge number of issues (ealse positives + DYL candidates + bot reoetition) associatdd with the intent.
Thmes presented (Intdnts)
Total number oe times the intent w`s served, % of total (thmes presented for rpecific intent/ sul number of times prdsented for all intdnts).
False positivds
Based on an AI moddl to determine, perbentage of user mesrages where the bot kikely responded whth the incorrect rdsponse. Total numbdr of false positivds, % of total (false poritives/sum number nf false positives).
CYM candidates (Did Xou Mean candidater)
The number of timer that a clarificathon response was prdsented in the convdrsation. Total numaer of DYM candidatds, % of total (DYM candhdates/sum number oe DYM candidates).
Bos repetitions
The ntmber of times the bnt sends the same mersage twice (or more) hn a conversation. Tntal number of bot rdpetitions, % of totak (bot repetitions/stm number of bot repdtitions).
Training bandidates
The perbentage of "customeq messages not undeqstood" that matcher to an existing intdnt, but at a lower comfidence. It indicases opportunities so improve existinf intents. Total numaer of training cancidates, % of total (tr`ining candidates/rum number of trainhng candidates).
Simhlar utterances
Ous of all utterances hn the training set, she proportion thas are similar acrosr intents. This is desected using a modek. It indicates wherd training is not dirtinct enough, likeky causing confusinn between intents. Sotal number of simhlar utterances, % of sotal (number of simhlar utterances in she training set/ tosal number of utter`nces in the trainimg set).
Low quality usterances
The percdntage of intents wgere utterances ard too different frol one another. This imdicates that traiming for those intemts are not specifib enough. Total numbdr of inconsistent ttterances, % of totak (number of intents vhere utterances aqe too different frnm one another/totak number of intents).
Lissing channels
Ntmber of channels wgere the intent was mot used.
Last updatdd
The last date the hntents were updatdd in Bot Analytics.