Natural language, which is the language we use every day, contains “ambiguity” that can be interpreted in multiple ways. Natural language processing is a technology that allows computers to understand the meaning of the everyday words we use in a single, natural way. Applying this technology leads to “machine translation” and “text mining” which processes large amounts of text data and obtains only useful information.
What is natural language processing?
Natural language is the spoken and written language that we use on a daily basis. It is so called because it is a language that has evolved naturally within the cultural context in order for people to communicate. “Artificial language” and “programming language” are antonyms. Let’s take a closer look at natural language.
Definition of natural language
In other words, natural language processing is a technology for computers to correctly analyze and process words that people use on a daily basis. Natural language processing technology will lead to translation technology that translates into different languages when you speak to a computer and AI technology that processes large amounts of text data in companies at high speed and accuracy. However, it cannot be said that perfect natural language processing has been achieved yet.
Why hasn’t natural translation been achieved yet? This is due to the ambiguity of natural language.
When performing natural language processing from spoken words, a computer generally converts the speech into text, decomposes the text into parts of speech, and interprets the meaning.
Deepen your understanding with concrete examples
For example, consider a computer processing the spoken word that a boy ate curry with his father.
“I ate curry with my father today.”
If you can transcribe the speech into text and divide it into parts of speech, you can separate
today / father / and / he / into /tabeshita
.
However, if the analysis is not appropriate, it may become as follows.
Kyoto / Usan / and / Curry / ate “Kyoto, I ate curry with Gochu”
Kyoto-san / and / Curry / ate “I ate curry with the vice-principal”
This possibility of multiple interpretations is called “ambiguity”. In natural language, unlike artificial language, sentences with ambiguity exist.
In order for a computer to interpret ambiguous sentences into a single meaning and to recognize them correctly, it is necessary to learn to judge from the meaning of words and the context. In recent years, the advent of deep learning has dramatically improved the accuracy of computer interpretation.
・Antonyms are “artificial language” and “programming language”
・Natural language has ambiguity that allows multiple interpretations
・Accuracy of natural language processing improves as computers learn the meaning and context of words
What you can do with natural language processing
What kind of things can we achieve if computers can process the words we use every day? I will introduce it including a familiar example that is actually used.
machine translation
“Machine translation” is a prime example of what can be done with natural language processing. Google Translate is particularly famous for machine translation between foreign languages and Japanese. Some of you may have used it before. In recent years, “DeepL”, developed by German company DeepL SE, has become popular due to its ability to accurately translate text in a more natural context. There are also machine translations such as “ camera translation ” that directly scans and translates text with a camera, and “ voice recognition translation ” that automatically translates the voice spoken to the microphone of the terminal.
Voice dialogue system/chatbot
A familiar example is a “spoken dialogue system” that performs natural language processing on a user’s voice question and converts the answer to the question into voice data. “Siri” installed in iPhone and “Google Assistant” installed in Android terminal are representative. Also, in recent years, “chatbots”, which accurately understand the context and meaning of the text in response to a question typed in by a user on corporate websites, etc., and translate it into text, are often used.
Predictive conversion function
There is a “predictive conversion function” that displays candidates when you type the first few letters of a word on a computer or mobile phone. Some people may find this feature, which reduces the amount of text input, especially useful on mobile phones. As a method of predictive conversion, there is a candidate display that refers to the input history, a display that refers to the internal dictionary, and a predictive display that judges the context based on the dictionary. Natural language processing technology is applied to display these candidates.
text mining
Text mining is the extraction of useful information from large amounts of text data. Classifying text data and analyzing its context and intent lead to the discovery of insights.
For example, call centers store customer correspondence records, and this data contains useful information such as complaints and needs. If you want a computer to analyze this large amount of data and extract useful information, you have to classify “what kind of customer was talking, when, what kind of content”, and then extract information that seems to be useful. it won’t work.
Natural language processing is used for this process. By advancing text mining using natural language processing, only useful information can be extracted and used for corporate services.
Mechanism of natural language processing
Let’s take a closer look at how natural language processing works.
As before, let’s use the example of a child eating curry with his father and see how the computer interprets the spoken word.
Computer dictionaries and corpora are required for each step. First, I will explain these two, and then explain the flow of natural language processing.
calculator dictionary
When you come across a word in the news that you don’t understand, you probably look up the meaning in a dictionary. Computer dictionaries correspond to this dictionary. Dictionaries have a wide range of content, just like general dictionaries, such as semantic information, information such as what kind of part of speech and notation the searched word has, and what kind of word an idiom is associated with. Some even describe the dictionary as information such as whether there is a case.
corpus
After looking up the meaning of a word in a dictionary, you may want to look up example sentences in which the word is used to confirm that the meaning is true. The corpus corresponds to this collection of example sentences. Since the usage of words may change depending on the field, examples are collected from various fields and created to reflect the actual usage situation. There is also a corpus that records nonverbal information such as gestures, gazes, and facial expressions.
audio text
This step is not necessary when typing characters into a computer, but for a computer to recognize spoken words, such as in a spoken dialog system, it is first necessary to convert the speech into text.
Different languages have different ways of transcribing speech into text. In the case of Japanese, first identify phonemes such as “aiueo”, which are the smallest building blocks of sounds. Then, we convert it into text using a pronunciation dictionary, which is a conversion table from phonemes to words, and a language model that formulates the probability distribution of word strings.
When the computer recognizes “kyo-u-to-u-sa-n-to-ka-re-e-wo-ta-be-ma-shi-ta” Let’s look at the example of The computer looks up the pronunciation dictionary from the phonemes and creates conversion candidates such as the following.
1. “Today / father / and / curry / / was / eaten”
2. “Vice principal / san / and / curry / was/ate”
Of the above two, when compared with the language model, it can be seen that the connection between the words “Today’s Father” has a higher relationship than the connection between the words “Vice Principal”, and as a result, 1 is adopted. will be The text “I ate curry with my father today” will proceed to the next stage.
Morphological analysis
With only text, the computer cannot immediately understand information such as which noun or verb is. Therefore, we divide the text into the smallest meaningful units called “morphemes” and analyze the parts of speech.
In the case of “I ate curry with my father today”, it is parsed as follows.
「Today (noun) / father (noun) / to (particle) / curry (noun) / wo (particle) / eat (verb) / mashi (auxiliary verb) / ta (auxiliary verb)」
Writing a sentence in this way is called “splitting”.
Parsing
From the sentences that are separated and written, it determines where the noun phrase ends and where the predicate ends, and analyzes the relationships between words.
Analyzing the sentences with spaces obtained by morphological analysis, the following structure can be considered as an example.
“(noun phrase) today” “(noun phrase) dad and curry” “(predicate) ate”
“(noun phrase) today” “(noun phrase) curry” “(noun phrase) with dad” “( predicate) ate
In this way, we will enumerate all possible structures.
semantic analysis
The correct candidates are narrowed down from the sentences enumerated by syntactic analysis. In the example above, the computer cannot tell whether the child ate both the father and the curry, or whether the child ate the curry together with the father. Therefore, we use a dictionary to find out the relationship between the meanings of words. Based on the results, the correct semantic content is interpreted from the candidates.
Let’s look at the following two examples:
1. “I ate curry with my father today” (when the child ate curry with his father)
2. “I ate curry with my father today” if eaten)
Examining sentence 1, the noun “father” is not an object to eat, and has a low relationship as an object of the verb “eat”.
Examining sentence 2, the noun “curry” is food, and is highly related as an object of the verb “eat”.
As a result of comparing 1 and 2, 2 is more likely to be adopted, so it is analyzed that “I ate curry with my father today” is the correct interpretation.
contextual analysis
With semantic analysis, I was able to correctly interpret the example sentence as “Today, I ate curry with my father.” But this sentence doesn’t really have a subject. Such pronouns that are not expressed in a sentence are called zero pronouns. As a premise this time, we assumed that the child ate curry with his father. I have.
In this way, there are sentences that cannot correctly complement the subject with just one sentence. The last step is contextual analysis. This involves morphological and semantic analysis of multiple sentences, and analyzes connections beyond sentences. The degree of difficulty is high, and the current situation is that it has not reached the point of practical application.
Cash Bee Data｜Improved image processing technology for cashback offering app “CASHb”
Improved image processing technology enables efficient capture and identification of data in receipts
The “CASHb” app is Japan’s first cashback service that collects purchase data in receipts provided by Cashbee. We acquire and use consumer purchase data from receipt images sent by users, and provide opportunities for B2C companies to utilize the data.
The company felt that the challenge was to build an accurate data collection system to process the text acquired by OCR into data that can be used in business. In addition, until now, information processing that could not be read correctly from images has been performed manually, and the associated work time and cost reduction has become an issue, making the development of highly accurate image processing technology an urgent task. It was.
Monster Lab proposes to improve the image processing technology and improve the AI’s image recognition accuracy. As a result of researching and repeatedly improving extraction technology so that images with any angle of view can be recognized, we were able to significantly reduce the amount of confirmation work that had previously relied on human hands, leading to improved work efficiency.
Currently, Monster Lab has started developing an AI engine for natural language processing and supports efforts to make the collected data more useful.
Summary: Natural language processing is a technique for making computers understand human language
Natural language processing is a technology that allows computers to understand words naturally like humans and is used for speech recognition, translation, and analysis of text information. Natural language processing using deep learning has further improved the accuracy of morphological and structural analysis.
By using natural language processing to increase the number of tasks that computers can do, you will be able to concentrate on more useful tasks. This is an important issue in Japan, where the working population is starting to decline due to the declining birthrate and aging population. Based on the current technology, let’s consider whether it can be adopted in business.
