What is Natural Language Processing and how does it works ?

In 1950 Alan Mathison Turing (often called as the father of modern computing) published his famous paper titled “Computing Machinery and Intelligence”. That was a seminal paper on the topic of artificial intelligence. In here Alan introduced a game called "The Imitation Game” and he proposed a variation of this game that involves the computer.

Then Alan Turing expressed an opinion that if a machine could have a conversation with a human and trick the human into thinking the machine was a person itself, then it was artificially intelligent. That was named as Turing test and became the first step to the birth of artificial intelligence.

Natural Language Processing

Communicate between the computer and a human is the most essential part of the Turing test. Natural Language Processing(NLP) is an area of artificial intelligence and computer science which concern the interactions between the computer and human (natural) languages. It is a fascinating technology and used in many products over the past years. The ultimate goal of NLP is to make interactions between computers and humans similar to interactions between humans and humans.

Computers are operated on computer programming languages such as Java, Python, C, etc. These languages are used to give instructions to machines. Humans communicate with each other by using natural language such as English, Russian, French, etc. Natural language is a language that is native to people. The computer is unable to understand natural language. But natural language processing computer is can understand natural language and communicate with humans. 

How natural language processing works?

Followings are the main component of NPL

• Natural Language Understanding(NLU)
• Natural Language Generation (NLG)

Let’s find out what is NLG and NLU.

Natural Language Understanding

NLU provides a meaning to the input made in the form of sentences in text or speech format. This is what speech-to-text and speech recognition does. 

* Understanding a speech

If the input is in speech format, speech recognition system is going to use Hidden Markov Models (HMMs). These are statistical models that turn speech format input to text by making mathematical calculations. 

Here, Hidden Markov Models listen to the speech and brake it into small units which length is between 10 and 20 milliseconds. A phoneme is one of the units of sound that distinguish one word from another in a particular language. Then those units are compared with the pre-recorded speech to determine the phoneme in each unit of speech. After that, it looks at the series of phonemes and statistically determines the most likely words and sentences in the speech. And convert the information of the speech into the form of text.

Now computer needs to get actual meaning of the text. It tries to identify each word in the text. This is called Part-of-Speech tagging. A Part-Of-Speech Tagger (POS Tagger) is a piece of software that reads the text in particular language and assigns parts of speech to each word such as noun, verb, adjective, etc. NLP systems have a set of grammar rules and a lexicon. (Lexicon specifies which words in a language can be recognized). Finally, The computer applies grammar rules and lexicon into the text in the natural language and determines the meaning of the speech.

Natural Language Generation(NLG)

NLG translates a computer’s artificial language into text or speech. First, NLP system determines what information needs to translate into text. Then, it organizes the information representing structure. The computer can form complete sentences using a lexicon and a set of grammar rules in NLG system. After that, the text-to-speech engine analyzes the text using a prosody model, which determines breaks, duration, and pitch. Finally, the text-to-speech engine uses a speech database and puts together all the recorded phonemes to form one coherent string of speech.

Usage of Natural Language Processing

NLP is a fascinating technology and used in many products over the past years. Let’s find out some of the common applications for NLP.

• Question Answering       
•   Search engines use NLP to give best search result.
• Summarization        
• NLP is used to generate shorter summaries from long documents or articles. Then humans can understand it quicker.
• Filtering spams
• Email service providers use NLP to identify spams. Here, spam filters scan the whole text in all the emails and attempt to understand the meaning of that text to determine if it’s spam or not. 
• Chatbots
• Chatbots help customers to get the right point without delay, answering customer questions and directing them to relevant resources and products at any time.

NLP will become an important technology in bridging the gap between human communication and digital data.