What is natural language processing?
Natural language processing (NLP) is the branch of artificial intelligence (AI) that deals with communication: How can a computer be programmed to understand, process, and generate language just like a person?
While the term originally referred to a system’s ability to read, it’s since become a colloquialism for all computational linguistics. Subcategories include natural language generation (NLG) — a computer’s ability to create communication of its own — and natural language understanding (NLU) — the ability to understand slang, mispronunciations, misspellings, and other variants in language.
How natural language processing works
Natural language processing works through machine learning (ML). Machine learning systems store words and the ways they come together just like any other form of data. Phrases, sentences, and sometimes entire books are fed into ML engines where they’re processed based on grammatical rules, people’s real-life linguistic habits, or both. The computer then uses this data to find patterns and extrapolate what comes next. Take translation software, for example: In French, “I’m going to the park” is “Je vais au parc,” so machine learning predicts that “I’m going to the store” will also begin with “Je vais au.” All the computer needs after that is the word for “store.”
Common uses of natural language processing
Machine translation is one of the better NLP applications, but it’s not the most commonly used. Search is. Every time you look something up in Google or Bing, you feeding data into the system. When you click on a search result, the system sees this as confirmation that the results it’s found are right and uses this information to better search in the future.
Chatbots work the same way: They integrate with Slack, Microsoft Messenger, and other chat programs where they read the language you use, then turn on when you type in a trigger phrase. Voice assistants like Siri and Alexa also kick into gear when they hear phrases like “Hey, Alexa.” That’s why critics say these programs are always listening: If they weren’t, they’d never know when you need them. Unless you turn an app on manually, natural language processing programs must operate in the background, waiting for that phrase.
Even if they are always there, NLP isn’t Big Brother. Natural language processing does more good for the world than bad. Just imagine your life without Google search. Or spellcheck, which uses NLP to compare the words you type to ones in the dictionary. Comparing the two data sets allows spellcheckers to identify what’s wrong and to offer suggestions.
Business benefits of natural language processing
Search and spellcheck are so commonplace, we take often them for granted, especially at work where NLP offers radical productivity gains. Want to know how many vacation days you have left? Don’t call HR. Save time and ask Talla, a chatbot that searches company policies for an answer. On the phone and need last quarter’s numbers? Mention them during your conversation and audio search startup SecondMind will show the answer on your screen. The company boasts its integrated search tool makes accounting and customer resource calls up to ten times shorter.
Natural language processing also helps job recruiters sort through resumes, attract diverse candidates, and hire more qualified workers. Spam detection uses NLP to keep unwanted email out of your inbox; programs like Outlook and Gmail use it to sort messages from certain people into folders you create.
Tools like sentiment analysis help companies quickly discern whether Tweets about them are good or bad so they can triage customer concerns. Sentiment analysis doesn’t just process words on social media, it breaks down the context in which they appear. Only 30 percent of English words are positive, says Skye Morét, data visualizer at analysis firm Periscopic — the rest are neutral or negative. So NLP helps businesses more fully understand a post: What’s the consumer emotion behind those neutral words?
Traditionally, corporations used natural language processing to classify feedback as positive or negative. But Ryan Smith, senior vice president of social and innovation at FleishmanHillard, says today’s tools identify more precise emotions, like sadness, anger, and fear.
Natural language processing for social good
In addition to helping companies process data, sentiment analysis also helps us understand society. Periscopic, for example, has paired NLP with visual recognition to create the Trump-Emoticoaster, a data engine that processes language and facial expressions in order to monitor President Donald Trump’s emotional state.
Similar tech could also prevent school shootings: At Columbia University, researchers have processed 2 million Tweets posted by 9,000 at-risk youth, looking for the answer to one question: How does language change as a teen comes closer and closer to getting violent?
“Problematic content can evolve over time,” says program director Dr. Desmond Patton. As at-risk youth grow closer to the brink, they reach out for help, using language. Natural language processing then flags problematic emotional states so that social workers can intervene.
Like Periscopic, Columbia pairs sentiment analysis with image recognition to improve accuracy. Patton says computer vision breaks down pictures attached to the Tweets, then machine learning processes them together with the language to tell “the actual emotionality of an image. Is this image about grief? Is this image about threats?...What else is happening in an image that helps us understand more complexly?” In addition to school shootings, the Columbia program hopes to also prevent gang violence.
Natural language processing for personal improvement
Natural language processing can also help you monitor your own emotional state. Woebot is an electronic therapist that connects with users via a Facebook Messenger chatbot or through a stand-alone app. There’s no high-level sentiment analysis here yet, though. Woebot essentially tracks only depression and anxiety, looking for words that may indicate users face an emergency situation.
The future of natural language processing
Woebot uses NLP to search for keyphrases, but the communication is so clunky, no one would ever confuse the app for a human being. The longer NLP is on the market, though, the better it gets, with some programs communicating so sophisticatedly we need tools like Botometer and BotOrNot to tell us if we’re talking to a real person.
As bot-driven accounts pop up on Twitter and Facebook, the next wave of tech may very well be NLP that detects NLP. Both Botometer and BotOrNot work by analyzing language for computer communication characteristics. Fortunately, we still live in an age where this can be accurately predicted. While advanced, today’s natural language processing is nowhere near perfect: Despite the fact that Woebot fully relies on it to function, CEO Alison Darcy said natural language understanding is the app’s biggest technical problem. “We're still at the very beginning in terms of this tech,” she told Inside AI’s Rob May.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.