The Future of Natural Language Processing

Language has always been a key component of communication and a vital part of human evolution. It has clarified the ways we interpret our environment and how others perceive it. Since the beginning of evolution, humans have transformed from using hieroglyphs to letters and numbers, and have now given birth to countless programming languages in order to communicate with machines.

The concept of enabling computers to manipulate language as humans do has been around for several decades. For example, in 1950 Alan Turing proposed a novel method now called the Turing test to evaluate a machine's ability to exhibit intelligent behaviour equivalent to a human. In this test the computer has to answer questions imitating itself like a human to trick a subject into believing that they are having a conversation with a human. The main assumption of this test was that a machine's ability to manipulate and understand language can act as a marker to its intelligence.

The term "natural language processing" (NLP) refers to the application of artificial intelligence (AI) to spoken or written languages. Today, NLP is so prevalent in most of the day-to-day applications we use that we rarely realize it. When you use Alexa, you are speaking with a device that uses natural language processing technology to take the sound waves you produce as input and convert them into text to execute your command; The same technology is applied when you are engaging in a conversation with a chatbot, where the chatbot takes the text you send as input and tries to classify the intent and extract keywords from your message so that it can execute a specific action, like booking a flight on a specific date.

You may already be familiar with many NLP applications such as autocorrection, translation and conversational AI. However, NLP is the cornerstone of numerous applications we use every day without even noticing. This article is a primer on how NLP became one of the most prominent use cases of AI and how it can help innovate and optimize various industries.

The Tech Behind Successful NLP

NLP has been a buzzword in the tech space for decades. The history of NLP cannot be considered complete without mentioning ELIZA, a chatbot program developed at MIT's Artificial Intelligence Laboratory from 1964 to 1966 designed by Joseph Weizenbaum. It was a simple chatbot which used predefined linguistic rules to mimic the tone and responses of a psychologist. Although its rule-based conversation engine worked well for very simple questions like, "how are you?", when engaged in complex conversations it became obvious that the algorithm was simply repeating words mentioned by the questioner to build up an answer, but did not actually understand neither the context nor the meaning of the question.

Since the "statistical revolution" of Natural Language Processing of the late 1980s and mid-1990s, much natural language processing research has relied heavily on linguistic rules and statistics, before diverging towards the more context-centered knowledge base approach in early 2000s.

A knowledge-based system was a type of AI system, that seeked to capture human expertise in structured formats and store them in a repository called the "knowledge base". The major drawback with this approach was that in the context of natural language it is extremely complex and time-consuming to include domain rules and patters for every open ended natural language input. Expert systems, so-called because they rely on human expertise, are examples of knowledge-based systems. After extensive research, NLP technologies evolved more towards now-famous Deep Learning based algoritms which incorporates more context representation and understanding. More recently, after the advent of GPU powered computing and neural networks, most of NLP applications today are built using Deep Learning concepts.

Deep Learning is a type of machine learning technique which uses artificial neural networks to learn patterns from large volumes of unstructured data. Deep learning is best useful when the data in hand is unstructured and extremely big in volume. This made deep learning based algorithms the perfect match for NLP due to inherent unstructured nature of language and the massive availability of text corpuses to train on.

Deep learning (DL) took over in the 2010s, and deep neural networks became common in natural language processing, owing to results demonstrating that such techniques can achieve state-of-the-art results in many natural language tasks, including language generation, text classification, information extraction and many others.

Previously, chatbots were used only for customer interaction with limited conversation capabilities because they were generally rule-based, but with the advent of deep learning based Natural Language Processing, chatbots can now handle more contextual conversations and perceive messages and respond similar to a human. This is not the only application of NLP that has emerged as a game changer. The following are some use cases that demonstrate the potential of NLP and how it can empower various industries.

NLP Use Cases

NLP itself serves as a catch-all for a number of related technologies. There are many applications that depend heavily on NLP.

Machine Translation

Even in such a connected world, language is sometimes one of the biggest barriers for collaboration, and just a couple of decades back there was a high level friction when organizations from different parts of the world wanted to exchange information with each other, primarily due to language barriers.

Machine Translation is the task of converting text or speech from language to another, using algorithms which preserve the meaning of the input text, and generate affluent text in the output language. This technology can potentially help decrease or even eliminate the language barrier in communication within individuals and organizations.

For businesses, machine translation is extremely powerful in situations where having the content translated by a human translator would be unscalable and inefficient. For example, modern machine translation systems can translate message, contracts, agreements and other critical documents in realtime enabling organizations to do cross border collaborations effectively.

On the other hand autonomous translations systems are also immensely powerful in connecting individuals across borders. Integrated machine translation systems are commonly used in social media platforms like LinkedIn and Twitter where the posts of users can be translated into the reader’s own language with a single click increasing accessibility and engagement.

Similar to other NLP domains, machine translation algorithms were initially built on top of rule-based linguistics to translate between languages. This approach was not only product inaccurate translations but it was also not a scalable pathway to build a translation engine that is embedded with the linguistic rules and structures of every language.

The most recent application of Machine Translation is Neural Machine Translation,

More recently neural network based approaches are being heavily used in translation systems, where contextual neural network algorithms like transformers are used to learn the underlying linguistic patterns in multiple languages before translating similar to how we humans perceive and understand language. Although this approach is more compute intensive than any other previous approach it is also more accurate and scalable as the number of languages to translate between increase. Deep learning based neural machine translation is rapidly becoming the standard technology for developing any sort of language translation applications.

Document Intelligence

Massive volumes of unstructured documents has always been a problem for business executives to go through, making it inefficient to extract and store crucial fragments of information from these documents efficiently.

Document Intelligence is the task of extracting critical information from unstructured documents in a structured format. A good example would be extracting the key points from a ~100 page contract such as the costs, expiry data and terms. Extracting this information not only makes it easy to store and these key fragments of information from large volumes of documents but also accelerates decision making by providing business stakeholders with information that they must focus on.

Modern document intelligence systems use a combination of NLP and computer vision techniques to identify and extract structured information. These algorithms try to understand both the visual layout and linguistic structure of documents to recognize and extract important fragments of information.

In the banking and finance sectors one of the main uses of document intelligence is in contract analysis. The contract drafting and negotiation stage of the contract lifecycle relies heavily on the negotiators’ sophisticated understanding of the full risks and opportunities that exist within a given contract, and their ability to craft a document and come to an agreement that optimizes the chance of a successful relationship. Document intelligence tools can help business leaders to swiftly identify crucial information in large contract documents without investing much time in manually reading through a large number of documents to identify the same information.This in-turn enables leaders to make faster decisions and identify key areas of contracts which are crucial.

In the insurance industry, document intelligence is used in claims processing and claims validation. For example, insurance companies use document intelligence to classify insurance coverage by autonomously extracting and analyzing claim proposals provided as documents.

HR and recruitment is another area where document intelligence can be immensely impactful. An average corporate recruiter receives atleast 300 resumes for a job opening. And by norm the recruiter has to go through each and every resume manually to filter out only a very subset of candidate resumes which match a fixed criteria. Going through 300 resumes is not only inefficient but also increases the possibility of missing out skilled candidates. This creates an inefficient cycle where on one hand, recruiters are overloaded with resumes and on the other candidates spend a long time in the pipeline, leading overall to an unproductive recruitment pipeline. Our team at Rootcode AI solved this problem through Aphelia. Aphelia is a intelligent document information extraction platform which can extract unstructured data from resumes and present them in a more structured format. The potential of using AI in the recruiting process is best for large companies and could radically alter and accelerate the hiring process. The extraction components used in Aphelia are powered by deep learning algorithms, unlike ruled based extraction systems, so that language models can consider both the visual layout of the document and the context of the text when extracting information from resumes.

Sentiment Analysis

Sentiment analysis is a natural language processing technique used to determine the collective feeling expressed in a text. Unstructured data, such as the text of an email, a Facebook comment, or a tweet, can be subjected to sentiment analysis and can be classified as positive, negative or neutral. Sentiment analysis tools can enable consumer businesses to understand the collective sentiment of consumers around their brand and products.

Monitoring both positive and negative sentiment from surveys responses is a pathway into what customers enjoy about your product or service. If you see trends within those responses, you can act on them to make real changes based on sentiment analysis. This can turn a potentially negative experience into a positive customer experience for anyone who interacts with your brand.

Sentiment analysis in business empowers companies to spot negative or positive sentiments about their product or service with precision, and take necessary steps to address those areas. Sentiment analysis can prevent complacency by showing how customer sentiment tracks over time. For example, you could see an uptick in consumer sentiment and trace that data back to its most likely cause.

Retailers across the globe are using sentiment analysis to benchmark products, develop new marketing strategies, and drive positive business outcomes. It also enables them to implement a short-term marketing campaign to better address the needs of the customers and stay updated with the latest retail industry trends. Sentiment analysis solutions also allow consumer businesses to leverage social listening and receive real-time updates on negative discussions to respond and address issues immediately.

Conversational AI

Conversational AI is a popular use case of NLP which focuses on building conversational AI agents that can mimic human conversational abilities to engage in conversations or resolve inquiries of users. Conversational AI agents can be mainly categorized into two, Open domain conversational agents, such as Google LaMDA, which focus on engaging in completely free form general conversations with humans by replicating human contextual memory and language understanding. These agents need to be able to handle a large number of possible conversations across several domains, making them extremely complex to design and build.

Closed domain conversational agents on the other hand are conversational agents which can only engage and respond to conversations surrounding a specific domain, like a sales chatbot. Although closed-domain conversational agents cannot engage in free form open conversations like open-domain conversational agents, closed domain agents are generally more practical and can drive higher engagements and conversions when used as a specialized sales tool.

One of the most practical industry applications of conversational AI is customer support.

Every consumer business with a growing customer base has the problem of human customer support agents being flooded with an overwhelming volume of inquiries from users. This results in a bottleneck scenario where on one hand support agents are overwhelmed with inquiries and on the other, customers’ are being put on hold without a response. We call this the “message-overload bottleneck” and this is a problem for all consumer sectors including retail, healthcare and ecommerce. And although scaling up the human support team is a possible solution it can cause businesses to incur exponential increases in operational costs as the business scales.

Conversational AI is a straightforward solution to this problem. Chatbots powered by machine learning and NLP enable businesses to give real-time responses to customer inquiries. This enables human customer support agents to productively spend their time by engaging in conversations that actually require human empathy and intelligence. Conversational AI tools can also engage in conversations across multiple channels like WhatsApp, Messenger, web and Instagram simultaneously without putting customers on hold. One of the biggest benefits of conversational AI is that, the ROI realization of a conversational AI agent is faster than the ROI period of any other AI solution, since the conversational AI agent can start handling conversations from the moment it’s deployed.

When we worked with major Ecommerce brands like Spa Ceylon to build their conversational AI agents, our team at Rootcode AI realized that the “message overload bottleneck” was a general problem across all Ecommerce retailers with a high volume of customers. And we wanted to solve this problem so that every Ecommerce retailer can utilize the capabilities of conversational AI to optimize their customer support experience to the maximum. And that led to the development of ConverseUp.

ConverseUp is a conversational AI platform through which ecommerce businesses can integrate their product databases from various ecommerce platforms like Shopify, Woocommerce, etc., and train and deploy their conversational AI agents seamlessly while providing a rich interface for human support agents to take control of conversations whenever they want. You can read more about our work on ConverseUp and how we are building an entire conversational AI platform for Ecommerce businesses over here.

Conclusion

A few decades ago understanding human language was one of the most difficult tasks for a computer. Fast-forward to today, due to progressive changes in NLP we now have language models and systems which perceive and generate natural language indistinguishable from humans.

Natural language processing has been around for a while, although it has only recently been making huge strides in terms of advancement. Through Machine Translation, NLP has given users an advanced way of eliminating language barriers without any discrepancies. Through Document Intelligence, users have been able to extract relevant information which greatly reduces time and manpower. Through Sentiment Analysis, many chatbots have been able to classify queries as positive, negative or neutral. Meanwhile, through Conversational AI, many queries of customers can be mitigated in a conversational skill similar to humans.

According to the research firm, MarketsandMarkets, the NLP market would grow at a rate of 20.3% (from 11.6 billion in 2020 to USD 35.1 billion by 2026). Research firm Statista is even more optimistic. According to their October 2021 article, NLP would catapult 14-fold between the years 2017 and 2025.

We can argue that recent advances in NLP make it appealing for investment by practitioners and tech enthusiasts. With increased adoption in healthcare, finance, and insurance, the NLP market is rapidly expanding. NLP is a collection of technologies, and practitioners must determine which of the underlying systems will provide the greatest business benefit and when. The future of NLP is bright, as more advancements will result in a better user experience, opening up new markets.