Manage forms for AI evaluation

You can use the Auto QM Evaluation Form Manager to create your own AI evaluation forms. Auto QM uses these forms to score contacts. You can create as many forms as you wish, but only the latest three forms are active for each interaction, even if more than three forms are assigned to the selected agent, groups, or teams. In the forms, you can configure the threshold for each metadata-based question.

If your organization has an Analytics Enterprise Plus license, you can also add your own custom questions to an evaluation form. This feature allows you to enhance the form in the following ways:

Create your own questions and group them by section.
Choose your own question names, descriptions, and answers.
Provide an example for each answer.

The customized questions are added to a question library. Then, when you create an evaluation form, you can choose custom questions from the library.

For advice on creating AI evaluation forms, go to Advice for writing AI evaluation forms.

Prerequisites

Your organization has the Analytics Enterprise license.
To manage custom questions or add additional form-level evaluation instructions for Auto QM, your organization must have the Analytics Enterprise Plus license.
You have the Edit Auto QM permission.
For audio contacts that are evaluated with a form that uses Calabrio’s standard questions, all of the following are true:
- The contact has multi-channel audio that separates the speakers.
- The conversation is in English.
- The conversation contains three or more messages.

Page location

Application Management > Auto QM Configuration > Auto QM Evaluation Form Manager

Procedures

Auto QM evaluation forms have two main components: sections and questions with answer options.

BEST PRACTICE Click Save Draft frequently. The form preview updates whenever you click Save Draft.

Step 1: Create a new form (required)

Click New form.
Enter the form name in the Form name field.
Enter a description in the Form description field. The character limit is 3,000.
(Optional) To establish a score that agents should meet, select Enable Target Score.
1. (Optional) To adjust the cutoff scores between the different scoring categories, click and drag the sliders.
2. (Optional) Edit the names for the scoring categories, if needed.
3. (Optional) To let the people who can view the completed evaluation see the names for the scoring categories, select Show performance range name on evaluation. To keep these names out of completed evaluations, leave this box cleared.
(Optional) To weight section and question percentages manually, select Manual Weighting. If you want Calabrio ONE to calculate weighting automatically and evenly, skip this step.

NOTE The total weighting of the sections and questions must be 100%.
(Optional, Analytics Enterprise Plus license only) In the Additional AI Evaluation Instructions field, enter additional descriptions specific to your contact center that could be helpful in the evaluation process. This information helps Auto QM to gain more knowledge and better understand your agent conversations. For example, you can add business jargon, abbreviations, or acronyms. The character limit is 3,000.

Step 2: Create sections (required)

Each form can have up to five sections, and each section can have up to five questions.

Click Add section. You can create up to five sections within each form.
Click the pen icon to edit the section name.
(Manual weighting only) If you have enabled manual weighting, enter a number in the Section weight % field.
Enter a number in the Section target % field. This is the target score that the company wants the agent to achieve at a section level.
Repeat steps 1–4 until you are finished adding sections.
(Optional, manual weighting only) To give all sections the same weight, click Balance Section Weight.

Step 3: Add questions (required)

Click Add Question. You can view the drop-down list of all AI-scored questions and select up to five questions for each section.
Select a category from the Question category drop-down list.
Select a question from the Question drop-down list.

NOTE For more information about evaluation criteria, go to About the Auto QM evaluation criteria. For information about how to create custom questions, go to Create a custom question. For information about how to use the Intelligent Phrase Matching category, go to .
(Manual weighting only) If you enabled manual weighting, enter a number in the Question weight % field.
(Optional) Enter a description in the Question description field. The character limit is 1,000. This description appears on the published evaluation form. Auto QM does not use this description.
(Optional) To make an answer a key performance indicator (KPI), select KPI failure and then select Form or Section. This allows you to configure KPI-based failures at the form or section level. If the agent does not achieve the appropriate percentage, they automatically fail the section or form and will be assigned a score of zero.
Where necessary, add time thresholds, hold events, and so on for specific questions.
Repeat steps 1–7 until you are finished adding questions to all sections.
(Optional, manual weighting only) To give all questions in the section the same weight, click Balance Question Weight.

Step 4: Publish an evaluation form (required)

IMPORTANT Publish a form only when you are completely finished creating it. You cannot edit a published form.

Click Publish. The form is published immediately and can now be assigned to contacts. For more information about assigning Auto QM forms to contacts, see Create Analytics tasks.

Create a custom question

You can create custom questions tailored to your business, including options, descriptions, and scoring criteria, which can be shared across evaluations. Custom questions are available if your organization has the Analytics Enterprise Plus license.

On the Auto QM Evaluation Form Manager page, click Manage custom questions.
Click New question.
Enter the question name in the Question name field. The character limit is 100 for the question name. For advice on writing questions, go to Advice for writing AI evaluation forms.
Enter a question description in the Question description field. The character limit is 1,000. For advice on writing question descriptions, go to Advice for writing AI evaluation forms.
In the Options section, enter at least two options for the question and then enter an Option description for each answer to help the AI evaluate responses more accurately. To change the order of the options, click the arrow icons. The character limit for the option name is 30, and the limit for the option description is 500. You can also include N/A as an option for each question. Click Add options to add more options. For advice on writing options and option descriptions, go to Advice for writing AI evaluation forms.
Click Publish. Alternatively, click Save draft, if you want to save a draft and publish it later.

IMPORTANT After you publish a custom question, you cannot edit it.

Create a question with intelligent phrase matching

To tell Auto QM to look for specific phrases in the contact, you can use intelligent phrase matching. With intelligent phrase matching, you provide examples of specific phrases that should affect the evaluation score if they are present in the contact.

As you are creating a new evaluation form, select Intelligent Phrase Matching from the Question category drop-down list.
Enter a question for Auto QM to answer in the Question field. This question should describe what you are trying to learn about the contact and why looking for specific phrases is necessary.
(Optional) Enter a description in the Question description field. This description appears on the published evaluation form. Auto QM does not use this description.
Select AI Match or Exact Match from the Match type drop-down list. If you select AI Match, Auto QM uses AI to identify similar phrases used in the contact, even if they do not exactly match the phrases you provide. If you select Exact Match, Auto QM identifies only exact matches for the phrases you provide.
Select Agent, Customer, or Any from the Sent by drop-down list.
In the Phrases field, enter phrases that you want Auto QM to identify in contacts. Each phrase can be up to 50 characters.

EXAMPLE Questions about compliance are a good fit for intelligent phrase matching because they do not require deeper context or tone analysis, and the compliance-related phrases that occur in a contact are predictable and standardized.
Question: Did the agent provide the mandatory credit disclosure?
Question description: Agents must inform the customer that any credit-related offer is subject to approval and additional terms. This disclosure is required by compliance policy during conversations involving credit cards, loans, or lines of credit. Auto QM will evaluate whether one of the approved phrases was present in the call.
Answer options: Yes, No
Evaluation logic (based on phrases): Look for one or more of the following approved phrases:
“This offer is subject to credit approval.”
“Terms and conditions apply.”
“Subject to approval and eligibility.”
“Approval is not guaranteed.”
“All loans are subject to credit approval.”
If none of these phrases are detected, the answer is No.

Move a section

Click the arrows next to the section title.

Move a question within a section

Click the arrows next to the question title.

Move a question to a different section

Click the three dots next to the question title.
Select Move to [section title].

Delete a section

Click the three dots icon and then click Delete Section.

Delete a question

Click the three dots icon and then click Delete Questions.

Manage existing evaluation forms

On the Auto QM Evaluation Form Manager page, you can complete the following actions:

To view a form, click a magnifying glass icon.
To edit a draft form, click the pencil icon, edit the information, and click Save draft or Publish.
To delete a form, click the trash can icon and then click Yes. Alternatively, click the pencil icon, click Delete, and then click Yes.
To clone a form, click the clone icon, enter a name for the form, and click Save. If you change your mind about cloning a form, click the trash can icon.
To publish a form, click the arrow icon.
To archive a form, click the cabinet icon and then click Yes.

NOTE If a form is included in a running Auto QM task, you cannot edit, delete, or archive the form.

Manage existing custom questions

On the Manage custom questions page, you can complete the following actions:

To view a custom question, click the magnifying glass icon.

To edit a draft custom question, click the pencil icon, edit the information, and click Save draft or Publish.
To delete a custom question, click the trash can icon and then click Yes. Alternatively, click the pencil icon, click Delete, and then click Yes.
To clone a custom question, click the clone icon, enter a name for the question, and click Save. If you change your mind about cloning the question, click the trash can icon.
To publish a custom question, click the arrow icon.
To archive a custom question, click the cabinet icon and then click Yes.