Bot performance dashboard for messaging
The new Bot Performance Dashboard enables brands to view and analyze their messaging bot performance over time, and measure the impact of agent annotations on the performance of bots designed by Conversation Builder.
Using the new dashboard, brands can track the bot funnel, including assigned conversations, handled conversations, intent match rate and conversations closed by the bot.
The dashboard contains reports which display data at two levels:
Brand level: Enables users to track bot quality trends and the impact of annotations over time, including:
- How many conversations are assigned to bots? How many are handled?
- What’s the intent match rate? Is it improving over time?
- What is the annotation rate over time? What is the ROI of the annotations?
Bot level: Enables users to analyze and optimize each bot, including:
- Does the intent match rate improve when we are investing the time to annotate?
- What is the trend of unmatched intents? Has it decreased over time?
- Which bot annotations are submitted? Do all bots improve over time when annotations are added by agents?
Primary datasets
- Messaging Intents activity
- Bot Intents annotation
Primary filters
- Date
- Bot Name
- Skill
Minimum time frame: Hourly
Brand Level Analysis
Bot Performance Dashboard, brand level analysis dashboard
Bot Level Analysis
Intent Annotation Metrics
The following table includes new Intent Annotation metrics that can be tracked in the Bot Performance dashboard:
| Metric name | Analysis Type | Formula | Description |
| NO. OF ANNOTATIONS CREATED | Agents Annotation | - | Number of annotations created by the bot |
| NO. OF EXPOSED ANNOTATIONS | Agents Annotation | - | Number of annotations that were exposed to at least one human agent, generated by Agent Workspace. This metric will count distinct annotations, meaning if an annotation was exposed twice, it will be counted only once. |
| ANNOTATIONS EXPOSURE RATE | Agents Annotation | NO. OF ANNOTATIONS EXPOSED / NO. OF ANNOTATIONS CREATED | Number of annotations that were exposed out of the Number of annotations that were created by the BOT when it did not find any matching intent. |
| NO. OF SUBMITTED ANNOTATIONS | Agents Annotation | - | Number of annotations that were submitted by a human agent. |
| ANNOTATIONS SUBMISSION RATE | Agents Annotation | NO. OF SUBMITTED ANNOTATIONS / NO. OF ANNOTATIONS CREATED | Percentage of submitted annotations out of the created annotations |
| SUBMITTED ANNOTATIONS WITH AN INTENT | Agents Annotation | - | Number of annotations that were completed and got submitted with any intent |
| SUBMITTED ANNOTATIONS WITH AN INTENT RATE | Agents Annotation | SUBMITTED ANNOTATIONS WITH AN INTENT / NO. OF SUBMITTED ANNOTATIONS | Number of annotations submitted with intent out of the submitted annotations |
| SUBMITTED ANNOTATIONS WITH NO INTENT | Agents Annotation | - | Number of annotations that were completed and got submitted as "I don't know" |
| SUBMITTED ANNOTATIONS WITH NO INTENT RATE | Agents Annotation | SUBMITTED ANNOTATIONS WITH NO INTENT/ NO. OF SUBMITTED ANNOTATIONS | Number of annotations submitted with no intent out of the submitted annotations |
Messaging Intents activity
The following table includes new Messaging Intent Activity metrics which can be tracked in the Bot Performance dashboard:
| Metric name | Analysis Type | Formula | Description |
| BOT HANDLED CONVERSATIONS | Bot Productivity | - | The distinct number of conversations where the bot has sent at least one message or transferred to any skill or agent |
| BOT CLOSED CONVERSATIONS | Bot Productivity | - | Conversations that were closed by the bot |
| BOT CLOSED CONVERSATIONS RATE | Bot Productivity | BOT CLOSED CONVERSATIONS / BOT HANDLED CONVERSATIONS | The percentage of conversations closed by the bot out of the conversations that were handled by the bot (where the bot has sent at least one message or transferred to any skill or agent) |
| NO. OF MATCHED INTENTS | Bot Productivity | - | Number of consumer matched intents that the bot managed to match for all intent types, including NLU and pattern types |
| NO. OF UNMATCHED INTENTS | Bot Productivity | - | Number of consumer intents that the bot did not manage to match (counted only for NLU intent type) |
| MATCHED INTENT RATE | Bot Productivity | NO. OF MATCHED INTENTS / NO OF ANALYZED INTENTS | The percentage matched intents out of the total analyzed intents |
| NO. OF MATCHED NLU INTENTS | Bot Productivity | - | Number of consumer intents that the bot managed to match from NLU type (as defined in Conversation Builder) |
| NO OF ANALYZED INTENTS | Bot Productivity | NO. OF MATCHED NLU INTENTS + NO. OF UNMATCHED INTENTS | Total number of analyzed intents (NO. OF MATCHED INTENTS +NO. OF UNMATCHED INTENTS) |
| NO OF ANALYZED NLU INTENTS | Bot Productivity | NO. OF MATCHED NLU INTENTS + NO. OF UNMATCHED INTENTS | Total number of analyzed NLU intents (NO. OF MATCHED NLU INTENTS +NO. OF UNMATCHED INTENTS) |
| MATCHED NLU INTENT RATE | Bot Productivity | NO. OF MATCHED NLU INTENTS / NO OF ANALYZED NLU INTENTS | The percentage of matched NLU intents out of the number of analyzed consumer messages for NLU intents |
Missing Something?
Check out our Developer Center for more in-depth documentation. Please share your documentation feedback with us using the feedback button. We'd be happy to hear from you.