Bot accuracy can now be increased, easily. How? Increasing training data, not by hand, but using automatically-generated query variations. We have benchmarked Rasa and other platforms and accuracy goes up to 93% with Bitext artificial training data tech.
One of the main chatbots’ problems is their need for large amounts of training data, as commented in Improving Rasa’s results. Part I post. As we said, chatbots are only able to recognize a specific intent if you add it a big number of sentences that express it. Until now, that has been done in a manual time-consuming way that was clearly inefficient.
To solve this problem Bitext offers an artificial training data technology that allows to automatically generate many different variants from a query with the same meaning as the original one, automating the generation of bot training set. We wanted to prove that a well-known chatbot platform like Rasa can benefit from this approach, by comparing a bot trained with hand-tagged sentences with other one achieved by our technology (there called as NLG).
Our tests show that if you train your bot with automated-generated variants, it gets improvements with outstanding outcome - 93% accuracy. However, if you just add 1 or 2 sentences per intent, it gets unreliable results (3% accuracy). And even if you train with 10 sentences per intent, it only gets mediocre results (68% accuracy). To get real accuracy, you should be able of generating thousands of sentences variations for each intent (or do it automatically with our artificial training data technology).
Let’s have a closer look at the test
We have done two different tests (A and B). Both of them use the five following intents related to the management of lights in the house, with the same five types of slots (action, object, place, percentage and hour):
- Switch on the lights (switch on the lights in the living room)
- Switch off the lights (switch off the lights in the living room)
- Change the color of the lights (change the lights to blue)
- Dim the lights (dim the living room lights to 20%)
- Program lights for a specific hour (program the garden lights for 21:00)
First test: from few sentences per intent
In the first test (A) we have trained two different bots. The first model (A1) was trained with only 12 hand-tagged sentences, while the second one (A2) was trained with a set of 455 sentences. These sentences were the result of automatically generated variants of the sentences of A1 by using the Bitext artificial training data system. We used the same 114 complete independent sentences to evaluate both models, and got these results regarding both intent identification and slots filling:
Second test: to hundreds of sentences
In the second test (B) the only change was in the number of sentences used in training and evaluation sets. In this case, the first bot (B1) was trained with a hand-tagged training set of 50 sentences (10 per intent). The second one (B2) was trained with 906 variants generated by Bitext system. We used the same 226 independent sentences for evaluating both models. Now let´s have a look at the results:
To sum up, Bitext artificial training data system allows you create huge training sets effortlessly. If you only want to write one or two sentences per intent, our system is able to generate the rest of variants needed to go from inaccurate and unreliable results to great precision. And even if you want to write dozens of variants per intent, our system will also increase your accuracy in an impactful way, reaching excellent results.
While we completed these tests with Rasa only, our conclusions are relevant for ML-based bot platforms in general, like Microsoft LUIS, Amazon LEX, Wit.AI or Google Dialogflow. You can check the experiment with more detail here.
So, let’s get down to business and check how your platform can benefit from this brand-new technology. Try our test now and see for yourself how your training corpora can improve for the better with the help of Bitext. Register on our API and goes straight to Variant Generation service, in Ecommerce tab.