The new Bot Performance Dashboard enables brands to view and analyze their messaging bot performance over time, and measure the impact of agent annotations on the performance of bots designed by Conversation Builder.

Using the new dashboard, brands can track the bot funnel, including assigned conversations, handled conversations, intent match rate and conversations closed by the bot.

The dashboard contains reports which display data at two levels:

Brand level: Enables users to track bot quality trends and the impact of annotations over time, including: How many conversations are assigned to bots? How many are handled? What’s the intent match rate? Is it improving over time? What is the annotation rate over time? What is the ROI of the annotations?

Bot level: Enables users to analyze and optimize each bot, including: Does the intent match rate improve when we are investing the time to annotate? What is the trend of unmatched intents? Has it decreased over time? Which bot annotations are submitted? Do all bots improve over time when annotations are added by agents?

Primary datasets

  • Messaging Intents activity
  • Bot Intents annotation

Primary filters

  • Date
  • Bot Name
  • Skill

Minimum time frame: Hourly

Brand Level Analysis

Bot Level Analysis

Intent Annotation Metrics

The following table includes new Intent Annotation metrics that can be tracked in the Bot Performance dashboard:

Metric name Analysis Type Formula Description
NO. OF ANNOTATIONS CREATED Agents Annotation - Number of annotations created by the bot
NO. OF EXPOSED ANNOTATIONS Agents Annotation - Number of annotations that were exposed to at least one human agent, generated by Agent Workspace. This metric will count distinct annotations, meaning if an annotation was exposed twice, it will be counted only once.
ANNOTATIONS EXPOSURE RATE Agents Annotation NO. OF ANNOTATIONS EXPOSED / NO. OF ANNOTATIONS CREATED Number of annotations that were exposed out of the Number of annotations that were created by the BOT when it did not find any matching intent.
NO. OF SUBMITTED ANNOTATIONS Agents Annotation - Number of annotations that were submitted by a human agent.
ANNOTATIONS SUBMISSION RATE Agents Annotation NO. OF SUBMITTED ANNOTATIONS / NO. OF ANNOTATIONS CREATED Percentage of submitted annotations out of the created annotations
SUBMITTED ANNOTATIONS WITH AN INTENT Agents Annotation - Number of annotations that were completed and got submitted with any intent
SUBMITTED ANNOTATIONS WITH AN INTENT RATE Agents Annotation SUBMITTED ANNOTATIONS WITH AN INTENT / NO. OF SUBMITTED ANNOTATIONS Number of annotations submitted with intent out of the submitted annotations
SUBMITTED ANNOTATIONS WITH NO INTENT Agents Annotation - Number of annotations that were completed and got submitted as "I don't know"
SUBMITTED ANNOTATIONS WITH NO INTENT RATE Agents Annotation SUBMITTED ANNOTATIONS WITH NO INTENT/ NO. OF SUBMITTED ANNOTATIONS Number of annotations submitted with no intent out of the submitted annotations

Messaging Intents activity

The following table includes new Messaging Intent Activity metrics which can be tracked in the Bot Performance dashboard:

Metric name Analysis Type Formula Description
BOT HANDLED CONVERSATIONS Bot Productivity - The distinct number of conversations where the bot has sent at least one message or transferred to any skill or agent
BOT CLOSED CONVERSATIONS Bot Productivity - Conversations that were closed by the bot
BOT CLOSED CONVERSATIONS RATE Bot Productivity BOT CLOSED CONVERSATIONS / BOT HANDLED CONVERSATIONS The percentage of conversations closed by the bot out of the conversations that were handled by the bot (where the bot has sent at least one message or transferred to any skill or agent)
NO. OF MATCHED INTENTS Bot Productivity - Number of consumer matched intents that the bot managed to match for all intent types, including NLU and pattern types
NO. OF UNMATCHED INTENTS Bot Productivity - Number of consumer intents that the bot did not manage to match (counted only for NLU intent type)
MATCHED INTENT RATE Bot Productivity NO. OF MATCHED INTENTS / NO OF ANALYZED INTENTS The percentage matched intents out of the total analyzed intents
NO. OF MATCHED NLU INTENTS Bot Productivity - Number of consumer intents that the bot managed to match from NLU type (as defined in Conversation Builder)
NO OF ANALYZED INTENTS Bot Productivity NO. OF MATCHED NLU INTENTS + NO. OF UNMATCHED INTENTS Total number of analyzed intents (NO. OF MATCHED INTENTS +NO. OF UNMATCHED INTENTS)
NO OF ANALYZED NLU INTENTS Bot Productivity NO. OF MATCHED NLU INTENTS + NO. OF UNMATCHED INTENTS Total number of analyzed NLU intents (NO. OF MATCHED NLU INTENTS +NO. OF UNMATCHED INTENTS)
MATCHED NLU INTENT RATE Bot Productivity NO. OF MATCHED NLU INTENTS / NO OF ANALYZED NLU INTENTS The percentage of matched NLU intents out of the number of analyzed consumer messages for NLU intents