Advice for writing AI evaluation forms

Auto QM evaluation scores rely on well-written, well-organized evaluation forms. Follow the advice below to develop smart forms that generate meaningful evaluations.

Advice for writing multiple forms

Auto QM uses Analytics tasks to assign forms to contacts for evaluation. Auto QM can score each contact with up to three forms. If you are creating multiple AI evaluation forms, make sure Auto QM uses the best forms for each contact. If a contact is eligible to be evaluated by more than three Auto QM tasks, Auto QM will score the contact using only the three most recently created tasks. See Create Analytics tasks for more information about Auto QM tasks.

Advice for sections

Use sections to group related evaluation questions in a clear and logical way. For example, you might create separate sections for categories like professionalism, customer effort, or empathy to keep your questions organized and easy to navigate so that evaluators do not have to scroll up and down. For more examples of section names, go to About the Auto QM evaluation criteria.

Examples

Use a section with this name	To group these kinds of questions
Professionalism	Questions about the agent’s professional demeanor and communication style
Greeting	Questions evaluating how the agent opened the interaction and established rapport
Resolution	Questions measuring how effectively the agent solved customer issues
Empathy	Questions assessing the agent’s ability to understand and relate to customer feelings
Customer effort	Questions examining how easy the agent made the experience for the customer
Compliance	Questions verifying adherence to required policies and disclosures

Advice for questions

A question contains specific evaluation criteria that Auto QM assesses when evaluating a conversation. Write questions as clear, actionable statements that focus on observable agent behaviors, in terms of something the agent did or didn't do explicitly or implicitly. Each question should examine only one behavior or concept to ensure precise feedback. Finally, write “yes or no” questions: questions that call for an answer of either “yes” or “no.” “Yes or no” questions typically provide the most consistent results.

Examples

Begin each question with "Did the agent...," such as the following examples:

Did the agent properly authenticate the customer before discussing account details?
Did the agent recap the outcome of the conversation to ensure the customer's understanding?
Did the agent express empathy and provide reassurance when the customer mentioned their problem?
Did the agent offer additional assistance before ending the conversation?

Advice for question descriptions

You add question descriptions as part of creating evaluation forms for Auto QM. The question description provides detailed information that helps Auto QM understand the question and accurately evaluate the interaction between the customer and agent. Question descriptions should clarify the intent behind the question and explain what behaviors or actions to look for in the interaction. The description should address nuances, clarify terminology, and help Auto QM recognize when and how the behavior should appear in an interaction.

Example

In your evaluation form, you ask the following question:

Did the agent recap the outcome of the conversation to ensure the customer's understanding?

You add the following question description:

Did the agent recap the outcome of the conversation to ensure clarity? A recap occurs when the agent performs an action and then confirms what they did, specifically using the past tense. A recap is not simply answering a question or providing information about what will happen in the future. The recap must clearly confirm an action that was already taken.

Advice for option names

An option is an answer to a question. Options are the possible answers to an evaluation question that Auto QM selects from when answering the question.

Create option sets that match your custom question type and evaluation needs. For example, you could add some of the following options:

Examples

Option type	Option names
Binary options (Yes/No questions)	Yes, No, N/A (when some questions might not apply to all conversations) Met, Not Met
Satisfaction scales (3-point)	Unsatisfied, Neutral, Satisfied Negative, Neutral, Positive
Performance scales (3 to 5 points)	Below expectations, Meets expectations, Exceeds expectations Poor, Fair, Good, Excellent 1, 2, 3, 4, 5 (with clear descriptions for each number) BEST PRACTICE Avoid creating scales with more than five points because finer distinctions become increasingly subjective and lead to inconsistent evaluations. In general, binary or three-point scales (for example, negative, neutral, or positive) provide more consistent results.
Categorical options (mutually exclusive)	Product A, Product B, Product C, None of the above Email, Chat, Phone, In-person, Other Technical issue, Billing question, Account management, Product information, Complaint

For example, an evaluation form has the following question and options:

Question: Did the agent recap the outcome of the conversation to ensure the customer's understanding?
Options: Yes, no, or N/A.

The following interaction demonstrates how yes, no, or N/A might occur during an interaction:

Yes: "Great, so I've processed your refund of $49.99 which will appear in three to five business days. I've sent the prepaid shipping label to your email for returning the damaged product. Once we receive it, we'll ship the replacement model. Does that sound okay?"
No: "That's right. Is there anything else you need help with today?" (The agent answered the question without a recap)
N/A: "That would need to be handled by our retention department. Let me transfer you to the appropriate team who can help you with that request."
N/A: "We close at 8pm tonight. You're welcome. Have a good day!" (In response to customer who asked about office hours)

Advice for option descriptions

Writing good option descriptions is one of the most important part of building an Auto QM evaluation form. You add option descriptions as part of creating custom questions for Auto QM. The Option description field explains what each scoring option represents and gives Auto QM clear criteria for when it should select each option. This field helps to establish consistent scoring standards by defining the boundaries between different options.

Give each option its own description that precisely articulates what conditions must be present in the conversation for that option to be the correct choice.

Example 1: Conversation recap

You have an evaluation form with the following question and options:

Question: Did the agent recap the outcome of the conversation to ensure the customer's understanding?
Options: Yes, No, or N/A.

You add the following option descriptions:

Yes: Select this option when the agent explicitly summarizes actions taken during the call, conclusions reached, and any agreed steps the customer needs to take. The recap must occur before the call conclusion, and the agent must ensure a mutual understanding of what was accomplished.
No: Select this option when actions are taken or points are agreed upon during the call, but the agent fails to summarize these actions or points before ending the conversation. Alternatively, select this option when the recap is incomplete and misses key elements discussed.
N/A: Select this option when a recap would be inappropriate or unnecessary due to context, such as calls that get transferred to another department, conversations with only one self-evident point that both parties are clearly aligned on, interactions where no actions were taken or discussed, or inquiries that were outside the agent's scope.

Example 2: Confirm that all concerns are addressed

You have an evaluation form with the following question and options:

Question: Did the agent ask the customer if all their concerns were addressed or if further assistance was needed?
Options: Yes, No, or N/A.

You add the following option descriptions:

Yes: Select this option if the agent explicitly asks the customer if all their concerns were addressed today or if further assistance is needed, or if the agent sends out a closing statement confirming that the customer can follow up if they have concerns that were not addressed or need additional assistance. This option should be selected when the agent demonstrates proactive customer service by checking if the customer's needs have been met and offering additional assistance if necessary, or by leaving the door open for the customer to reach out again.
No: Select this option if the agent does not ask the customer if all their concerns were addressed today or if further assistance was needed, and does not send a closing statement confirming that the customer can follow up if they have concerns that were not addressed or need additional assistance. This option should be selected when the agent does not proactively check if the customer needs any additional help or if there are any unresolved issues.
N/A: Select this option if the conversation is dropped, incomplete, or there is not enough information to evaluate whether the agent asked the customer if all their concerns were addressed today or if further assistance was needed. This option should be selected when the conversation does not provide sufficient context or when the question is not applicable to the specific conversation being evaluated.

Example 3: Professional closing

You have an evaluation form with the following question and options:

Question: Did the agent use a professional closing?
Options: Yes, No, or N/A.

You add the following option descriptions:

Yes: Select this option if the agent used a professional closing in the customer interaction. This includes using appropriate language, tone, and formatting to conclude the conversation in a polite and professional manner. The closing should leave a positive impression on the customer and indicate that the agent is available for further assistance if needed.
No: Select this option if the agent did not use a professional closing in the customer interaction. This may include using informal language, an abrupt ending, or failing to provide a clear conclusion to the conversation. The closing may leave a negative impression on the customer or give the impression that the agent is not interested in further assistance.
N/A: Select this option if the question of whether the agent used a professional closing is not applicable to the conversation. This may occur in situations where the conversation did not reach a point where a closing is necessary or if the conversation was abruptly ended by the customer or agent before a proper closing could be provided.”

Advice for option examples

Option examples are conversation snippets that illustrate when Auto QM should select a particular option. You can add option examples to help Auto QM understand how to apply the criteria consistently. Option examples are not required, but they can help Auto QM understand specific contexts in which to select an option.

Example 1: Conversation recap

You have an evaluation form with the following custom question and options:

Question: Did the agent recap the outcome of the conversation to ensure the customer's understanding?
Options: Yes, No, or N/A.

You add the following option examples:

Yes: "Great, so I've processed your refund of $49.99 which will appear in 3–5 business days, and I've sent the prepaid shipping label to your email for returning the damaged product. Once we receive it, we'll ship the replacement model. Does that sound right?"
No: "That's right. Is there anything else you need help with today?" (The agent just answered the question without a recap.)
N/A: "That would need to be handled by our retention department. Let me transfer you to the appropriate team who can help you with that request."
N/A: "We close at 8 pm tonight. You're welcome. Have a good day!" (In response to a customer simply asking about office hours)

Example 2: Confirm that all concerns were addressed

You have an evaluation form with the following question and options:

Question: Did the agent ask the customer if all their concerns were addressed or if further assistance was needed?
Options: Yes, No, or N/A.

You add the following option examples:

Yes: "Thank you for contacting us today. Is there anything else I can help you with or any other concerns you would like me to address?"
Yes: "Before we end the conversation, I want to make sure I have addressed all your concerns. Are there any other issues you would like me to assist you with?"
Yes: "If you have any more questions or concerns, please feel free to reach out again. Have a great day!"
Yes: "Should you still need assistance, please feel free to reply to this text."
N/A: "I'm sorry, but I'm unable to assist with that. Please contact our customer support team for further assistance."

Example 3: Professional closing

You have an evaluation form with the following question and options:

Question: Did the agent use a professional closing?
Options: Yes, No, or N/A.

You add the following option examples:

Yes: "Thank you for contacting us. If you have any more questions, feel free to reach out. Have a great day!"
Yes: "I appreciate your time. If there's anything else I can help you with, please let me know. Have a wonderful day!"
No: "K, bye"
No: "That's all. Bye"
N/A: "I'm sorry, but we don't have any information on that item. Please contact our lost and found department directly."”

Advice for option points

Option points are numbers assigned to each answer option that determine how the option contributes to the overall evaluation score. Auto QM uses points to calculate question and section scores and ultimately the entire evaluation's final score. Setting appropriate point values ensures your scoring system properly emphasizes what matters most in your quality program.

BEST PRACTICE Use percentage-based scoring out of 100, where 100 represents the best possible performance and 0 represents the worst. For options that shouldn't affect scoring (like N/A scenarios), select Ignore. This option removes that question from the evaluation calculation and redistributes the points from that question to the other questions in the same section.

Examples

For this kind of answer option...	Use this kind of points system
Binary option (Yes/No)	Yes: 100—Awards full points when the criteria are met No: 0—Awards no points when the criteria are not met N/A: Ignore—Excludes the question from the score calculation when it is not applicable
Satisfaction scale (3-point)	Satisfied: 100—Full points for meeting all expectations Neutral: 50—Half points for acceptable but not exceptional performance Unsatisfied: 0—No points for poor performance N/A: Ignore
Performance scale (5-point)	Create equally spaced intervals like this: 5 - Excellent: 100 4 - Good: 75 3 - Average: 50 2 - Below Average: 25 1 - Poor: 0 N/A: Ignore
Categorical options	When one option is not automatically better than others (for example, when the question asks about the reason the customer called), you can pick one of these options: If all the options are neutral choices, assign equal points (for example, 100 points) to all options. If some options represent better outcomes than others, assign higher point values to the better options. If the option that Auto QM selects should not affect the evaluation score, select Ignore.

Example

The table below illustrates how the section name, question text, question description, option name, option points, option description, and option example work together in an AI evaluation form.

Section name	Winning the booking
Question text	Did the consultant attempt to win the booking?
Question description	This question evaluates whether the consultant made an effort to secure the booking during their interaction with the customer. The consultant should have used language that encourages the customer to commit to the booking, such as asking directly to proceed with the booking or highlighting an offer.
Option 1 (point value)	Yes (100)
Option 1 description	The consultant actively attempted to secure the booking. This could include using phrases like "Shall we go ahead and book that?", "Shall we go for that today to get that price?", or "Let’s get that booked up, shall we?" indicating a clear invitation to finalize the booking.
Option 2 (point value)	No (0)
Option 2 description	The consultant did not make an attempt to secure the booking. This may occur if the consultant did not directly ask the customer to commit or failed to suggest taking action toward completing the booking.