What do you evaluate in your chatbots? Some ideas

You have a chatbot up and running, offering help to your customers. But how do you know whether the help you are providing is correct or not? Evaluating chatbots can be complex, especially because it is affected by many factors. 

We have gathered some ideas based on our experience in helping our clients improve their bots:

  • If you can get your hands on real-world evaluation data (external datasets pertaining to your domain, with test utterances and their corresponding intent), you have everything you need to carry out a proper performance evaluation. We usually compute a confusion matrix that allows us to easily measure accuracy, precision and recall (more about these terms here). Apart from that, another thing we usually measure is whether there are cases where the model’s prediction is “unclear” (i.e. the difference in confidence score between the first and second candidates is small), which is often an indication that there is potential overlap between two intents (or their training utterances). Some bot platforms include tools to help you perform these evaluations, and there are also some third party model evaluators around the web.
  • If real-world evaluation data is not available, we usually use our own data to build evaluation sets by taking utterances that have not been used for the training set. Rather than having a single evaluation set, we construct multiple ones using different modules (core, colloquial, polite…) and carry out multiple evaluation iterations, testing how the bot performs with different language registers.
  • In the evaluation work we’re doing for end clients, another concept we work with is “in scope” vs. “out of scope” queries – including “out of scope” utterances in the evaluation data is key to identify both true negatives and false positives. For this, what we often do is to take data from our datasets for other industries/verticals (e.g. testing a Banking bot using utterances from the Travel industry).

All these steps help us measure the usefulness of our chatbots or training datasets. You can use any of them to evaluate the Free Dataset we offer, created with our Multilingual Synthetic Data technology, centered on Customer Support: feel free to download it here and give us your feedback!



For more information, visit our website and follow Bitext on Twitter or LinkedIn.


Subscribe Here!