Understanding Natural Language Processing To Build Better Bots

Science fiction, for all those who love them, has set our standards for technology pretty high. Stanley Kubrick's box-office smash hit “2001: A Space Odyssey” was made way back in 1968 but can be safely called the prototype of modern day artificial intelligence. He was certainly way ahead of his time and even today our expectations from AI and the reality has a big gap. Let us find out how a computer is made to 'understand' human language, which in turn becomes the base of all AI platforms.

Now, you may think why is it so important for chatbot development? Once we cover the basics of NLP, it becomes easier to design bots more efficiently by defining the limits of its learning.

NLP - Where It All Started

Alan Turing, in his famous experiment from the 1950s marked the beginning of this journey by defining standard parameters to call any machine, 'smart'. Through the 1970-80s, coding came into picture and linguistics experts collaborated with coders to produce software like SHRDLU, CHAT-80 etc. 

However, the biggest challenge was to grammatically interpret a sentence.

Machine learning, came next in the 1990s, which was the result of an increase in statistical research in NLP. Machine learning is the most popular model today and is based on probability. This essentially means that the more data one feeds, the better the model becomes.

So, where has so many years of advancement lead us to? The results are amazing today and we can accurately process sentences with up to 98% accuracy.

Where We Stand Today? 

Although, it is developing at a staggering speed, NLP and AI face two main problems today. Let us throw some light on what these are and how are they affecting chatbot services. 

1. Multiple ways of expressing

“What is the temperature today?”, “How cold is it outside?”, “Is it warm enough to go out today?” and the list is endless. There could a million ways of expressing the same thing, in this case a weather query. From the point of view of NLP, it becomes tedious and needs creating associations between the different words.

2. Words or sentences are contextual

Isn't this the basis of language? The same word may mean different things in different contexts or even different in the same context for two different individuals. We, as humans sometimes fail to grasp the contextual meaning of words and sentences and here we are, trying to teach a machine the same. 

In essence, AI cannot be improved solely through a probabilistic model and needs more contribution from the users themselves. A better understanding of NLP can help chatbot developers to produce bots that are truly like the AI from sci-fi movies we love so much.