Your data holds secrets. Uncovering them is absolutely essential to business success. But mining volumes of text-based data for insights poses a big resource challenge for companies, which is why text analytics tools have become so business critical. There are two different approaches in the market machine learning and deep linguistic analysis, as we mention in different articles. In this post, we will dig more in depth in both of them.Text analytics tools tend to follow a particular text analysis approach. The most common is machine learning, which is based on statistical and mathematical models. Machine learning tools examine large amounts of data in search of patterns and then generate code that lets users recognize those patterns across in their incoming data.
Less commonly used – and understood – is the linguistic approach to text analysis, which is based on knowledge of language and its structure (grammar, ontology, vocabulary). Linguistic tools are able to make sense of the structure of language at all levels (morphology, syntax, and semantics).
The idea that companies need to choose between these two approaches – using either machine learning or linguistic analysis to achieve their text analytics objectives – is a common misconception that needs clearing up for the sake of the progress of the Big Data industry.
Machine learning and linguistic analysis are complementary and cooperative approaches that, when properly combined, provide the most effective way of extracting high-quality insights from big data. It can be said that deep linguistic analysis enriches and potentiates machine learning and allows for even more accurate insight extraction.
Deep linguistic analysis uses knowledge of language to extract structure from the text. It understands and makes sense of the complexities and nuances of human expressions, such as negation (“I never liked it”) and conditionality (“I'd like it if it were cheaper”). Machine learning typically handles text more topically, as a flat set of strings, where sentences like "dog bites man" and "man bites dog" look the same.
This poses a limitation on the amount of insight that machine learning is able to extract. By taking into account the structure of language, linguistic analysis clarifies differences which are vital to understanding customer opinion. Consider these two sentences, which have similar wording but entirely different meanings: “I don’t plan to buy this product” and “If I don’t buy this product today, I will buy it tomorrow.” If they are categorized as the same, a potential hot customer may be discarded as uninterested.
When we consider the richness and complexity of language, it becomes clear that an approach that is able to handle linguistic subtlety, contradiction and sentiment is vastly powerful. But machine learning is every bit as crucial as linguistic analysis.
Linguistic analysis tools do not actually extract insights from texts, they simply enrich the possible insights that can be extracted via machine learning. So both tools are essential to any business user who wants to quickly and easily gain the most insight from volumes of unstructured text-based data.
Download our paper to see how linguistics accelerates the training of Machine Learning engines.