Some of you reading this article may be wondering what you can do with machine learning.
Machine learning can be done in six major ways, and is currently being used in a wide range of fields.
However, unless you have studied machine learning to some extent, you will not know what machine learning can do and what it is used for.
Therefore, this time, I will explain in detail about “what machine learning can do” and “application examples of machine learning corresponding to what it can do”.
In the second half, I will introduce what machine learning cannot do and how to use machine learning successfully in business, so please read it to the end.
What is machine learning in the first place?
What is machine learning
Machine learning is “learning by machines (computers)”. In machine learning, computers learn a large amount of data, and while repeating a large amount of trial and error, they autonomously extract rules and patterns to build algorithms and models.
In general, image recognition algorithms trained on a large amount of image data, natural language processing algorithms trained on a large amount of text, etc. are in circulation, and are being used in various aspects of society. .
Reinforcement learning, which not only learns from a large amount of data but also returns better results through repeated trial and error, is also attracting attention. I’m here.
Relationship with AI/deep learning
In the latter half of the 2010s, the attention of machine learning increased, and the third AI boom occurred. The reason for this is the growing interest in a technology called deep learning, which is one of the machine learning techniques.
With deep learning, by studying a large amount of data such as images and text, it is possible to autonomously acquire the laws of the data and build highly accurate models without being taught by humans.
It’s easy to understand if you think of deep learning as the core of the machine learning that drives the current AI boom. Deep learning is part of machine learning, which is part of AI ( artificial intelligence ).
Machine learning can be divided into 4 types
Machine learning can be categorized into four types:
- Supervised learning
- Unsupervised learning
- Semi-supervised learning
- Reinforcement learning
I will explain below.
Supervised learning
Supervised learning is a learning method that derives patterns and rules from a large amount of pre-prepared data (supervised data) that is learned by a computer.
This learning method is used for various tasks, mainly image recognition and numerical prediction, and the learning process of supervised learning is divided into “learning” and “inference.”
“Learning” derives rules and patterns while learning a lot of data and builds a model ( algorithm ), while “inference” returns results for new data based on the built model.
Unsupervised learning
Unlike supervised learning, unsupervised learning does not learn a large amount of data, but instead analyzes the structure and features of the data, groups and simplifies the data.
” Clustering ” and “dimensionality reduction” are typical methods of this unsupervised learning .
Semi-supervised learning
Semi-supervised learning is a combination of supervised and unsupervised learning.
Semi-supervised learning is used when there is not enough data to predict an accurate model and neither the power nor the resources to acquire more data.
Using this method, you can increase the size of your training data.
But there is no way to be sure that the algorithm produced 100% correct labels. As a result, it is less reliable than traditional supervised learning.
Reinforcement learning
Reinforcement learning is a method that improves the strategy that the machine takes while learning.
In reinforcement learning, instead of giving correct data like supervised learning, a reward is set for each result of taking a series of actions, and the machine learns by trial and error to maximize the reward. This will improve accuracy.
The learning process of reinforcement learning can be easily understood by imagining the process of learning to ride a bicycle.
Example) Process to become able to ride a bicycle
- Take a ride
- Fall down
- Change how you ride (equivalent to “strategy” in reinforcement learning)
- Can ride a little (equivalent to “reward” in reinforcement learning)
- You will be able to ride gradually by changing the way you ride
- By repeating this trial and error, you will eventually be able to ride smoothly
List of things machine learning can do
Here are six things you can do with machine learning:
- Regression (supervised learning)
- Classification (supervised learning)
- Dimensionality reduction (unsupervised learning)
- Clustering (unsupervised learning)
- Recommendation
- Anomaly detection
I will explain each.
Regression
Regression is one of ” supervised learning ” in machine learning, and it is a learning method when dealing with quantities.
An example of using regression is “analyze past data and predict how many times new customers will visit from customer data”.
The concept of regression is to predict unknown numbers from known numbers. There are two types of regression: “linear regression,” which regresses with a straight line, and “nonlinear regression,” which regresses with a smooth line other than a straight line.
If you want to learn more about machine learning regression, check out the following articles:
Classification
Whereas regression was numerical prediction, classification is an algorithm that predicts label membership.
For example, classification is used as a learning method when classifying and predicting label information such as “dog” and “cat” or “buy” and “non-purchase” category information. As a familiar example of using classification, it is easy to understand “discrimination of spam mail”.
Also, the classification
- Random forest
- Logistic regression
- Support Vector Machine (SVM)
There are 3 categories above.
Random forest is a method using decision trees. A decision tree is a method of setting thresholds and classifying, and is used when making decisions, such as risk management, using a tree-shaped graph. By using this method, even if there is a decision tree with poor accuracy, the system can maintain accuracy as a whole.
Logistic regression is an adaptation of linear regression, and although regression is in the name, it is actually a technique used for classification problems.
Basically, we deal with problems where the target variable can be classified as either 1 or 0.
For example, logistics regression is used to find out which channel is the most effective for ad delivery, or to predict the risk of developing a certain disease based on multiple factors such as lifestyle habits.
Support Vector Machines (SVMs) are supervised learning that can be used for both classification and regression.
It is a popular algorithm because it is easy to implement for non-linear identification, has high identification ability, and produces highly accurate analysis results.
Dimensionality reduction
Dimensionality reduction is a technique to reduce the dimensionality (number of features) in data.
Even in machine learning, if there are too many dimensions unnecessarily, a phenomenon called “curse of dimensionality” occurs, in which the algorithm of the problem grows exponentially in proportion to it, and a decrease in accuracy may be seen.
Therefore, by compressing and visualizing data using dimensionality reduction, we take measures against the “curse of dimensionality”.
A technique often used for such dimensionality reduction is “principal component analysis”.
Principal component analysis is a technique that compresses and expresses multidimensional data with variations into a smaller number of dimensions while trying to preserve the original information as much as possible. Converting height and weight into BMI, which indicates the degree of obesity, is a typical example of principal component analysis.
Clustering
Clustering is a method that is an extension of classification, and is a typical unsupervised learning method that “collects similar data groups by category”.
There are two types of clustering algorithms: hierarchical cluster analysis and non-hierarchical cluster analysis.
Hierarchical cluster analysis is a method of classification in a hierarchical structure, in which the most similar combinations of data are grouped together. This method of analysis is not suitable for mass data classification and is useful to perform on small data of interest.
Next, I will explain the “Ward method” and “group mean” of hierarchical cluster analysis.
Ward Law
Ward’s method is a method of clustering by combining the two closest groups among the existing groups. This method is relatively commonly used for hierarchical cluster analysis.
Group mean
The county average is a method of comparing the distances between data within two groups to reduce the distance between groups. This method can prevent the chain effect, which is a phenomenon in which extra groups are created.
Non-hierarchical cluster analysis
Non-hierarchical clustering is a non-hierarchical classification method in which the number of clusters to be handled is determined in advance and the data is classified into that number of groups.
Compared to hierarchical cluster analysis, it is suitable for handling large amounts of data such as big data.
Next, I will explain the k-means method of non-hierarchical cluster analysis.
K-means method
The k-means method is a method used to divide the given data into groups (clusters) that are similar. The name of the k-means method comes from classifying into a predetermined number of clusters “k” using “means of clusters”.
The k-means method starts with using k vectors and classifies the data while adjusting the vectors. By the way, it is also used in part of the analysis of “Tebleau”, a major software company for data analysis.
Recommendation
Recommendation is a method of presenting products and information that are considered valuable to the target person more individually.
This is not limited to the Internet world. For example, at McDonald’s, suggesting “Would you like some fries with your meal?” can be considered a recommendation.
Also, recommendations aren’t necessarily personalized. In addition, there are types of recommendations, and there is the possibility of actively asking the user to take action, but there are also those that are neither completely passive nor active, and are recommended by “instruction with the user”.
Anomaly detection
Anomaly detection is a method used to detect and infer machine failures and outliers in data analysis. This is the identification of observational results and expected patterns that do not match other data in a dataset, such as by data mining.
It plays an active role in various situations such as detecting unauthorized use of credit cards and detecting system failures. Here are three representative methods.
k-nearest neighbor method
The k-neighborhood method is a method of classifying and anomaly detection using the distance between data.
This method classifies the data to be newly discriminated using the range created by the determined k pieces of data and the principle of majority.
A major feature of this method is that the value of k greatly affects the results.
Principal component analysis
As mentioned earlier, principal component analysis is the reduction of features in a dataset by extracting features.
During dimensionality reduction, by using principal component analysis, the outliers are represented by other axes to the extent that the data values do not completely disappear, and by approximating other data, the outliers are virtually eliminated. get rid of it.
One-class SVM
One-class SVM can be said to be a method for outlier detection that sets a discrimination boundary by literally applying the SVM method and distinguishes normal values and outliers.
What machine learning can do from six use cases
So far, we have discussed the use cases of machine learning, but I will explain how these are useful in real life.
- Sales and demand forecast
- Spam/fraud detection
- Data visualization
- Text mining tools
- Product recommendation
- Anomaly/failure detection
I will explain each below.
(1) Sales and demand forecast
“Multiple regression analysis” is used for sales and demand forecasts. Multiple regression analysis combines multiple factors to create a formula that predicts results such as sales and customer numbers.
Building a sales forecast model using multiple regression forecasting is easy once you have prepared sales data for each store, store attribute items, and trade area data.
A multiple regression model can be easily created in Excel, and if you prepare data for each store here, you can create a more accurate prediction model.
(2) Spam/fraud detection
Anomaly detection is used for spam and fraud detection. Typical examples include spam detection and credit card fraud.
Spam emails are sometimes automatically sorted into the spam folder, which learns and classifies words that are often used in spam emails.
(3) Data visualization
When given data, it is important to visualize the data and understand its characteristics. Since there are no set rules for the axis of visualization, it is important to think about what you want to know and visualize it.
④Text mining tools
Text mining is data mining for character strings.
Data mining is a technology that extracts knowledge by applying statistics and artificial intelligence to a large amount of data, and it has become an indispensable technology as opportunities to handle big data have increased.
⑤ Product recommendation
Product recommendation is a method of recommending products and services that attract the user’s interest.
The main method of product recommendation is to propose products and services that match the customer’s preferences based on the browsing history and purchase history of the websites and EC sites visited. The “recommendations for you” that you see on sites etc. are due to this function.
(6) Abnormality/failure detection
Anomaly detection is a method of machine learning a large number of measured values to detect whether or not an unknown pattern is abnormal.
It is used in a variety of industries, such as detecting failures in factories, fraudulent transactions, and equipment monitoring.
Examples of what machine learning can’t do
Even seemingly all-purpose machine learning has its weak points.
- Learning complex concepts with less data
- Running on new types of data
These two.
A specific example of learning complex concepts with a small amount of data'' isinputting pictures and outputting intentions from human gestures towards cars”.
Gestures to cars include hitchhikers putting their hands down, construction workers raising their hands to stop the car, and cyclists raising their hands to signal a turn. , there are many intentions and types.
And it is not easy to collect such a large amount of gestures that are perceived differently by each person as data.
For this reason, it is difficult for machine learning to output the complex concept of the intention of a gesture toward a car from a small amount of data such as photographs.
A specific example of running on new types of data'' isX-ray chest imaging using datasets of images from different medical centers”.
For example, suppose you build a supervised learning system to learn to diagnose pneumonia.
When applying the system, if the image data was collected with correct practices at medical center A, but the image data at medical center B was skewed or showed wounds or other objects, the system performance will be significantly degraded.
If humans learn from images from medical center A, they are likely to be able to adapt to slightly different types of images such as those from medical center B. Because it turns out that the patient is only lying obliquely.
However, AI systems are not very good at generalizing and understanding these new types of data, resulting in poor diagnostic performance.
What is Python used for machine learning
Python is a versatile, simple and popular language. The biggest feature of Python is that “code input is short”.
Short code has three main advantages.
- Easy to study
- Less code entry time
- Easy-to-understand error locations
Python was not originally created for AI, but in recent years it has been attracting attention in AI and machine learning. Since it can be handled by individuals, even programming beginners can easily start learning.
How to use machine learning successfully in business
Now that machine learning has become popular in the world, some people may be thinking, “This and that should be possible with machine learning.”
However, there are fields that machine learning is not good at, and it is not possible to cover everything with machine learning.
When using machine learning in business, it is necessary to understand that machine learning is not a panacea that can do anything, and to identify its characteristics and use it.
Also, when starting a business that utilizes machine learning, clarify as much as possible what you want to solve by using machine learning, and then investigate whether the problem can really be solved using machine learning. I have to.
If they are not clear, there is a possibility that the output and results expected in the hypothesis stage will not be achieved.
Also, when launching a new business using machine learning, you may find some business hints by referring to the latest AI use cases and the business details of growing AI companies.
What we should do now that machine learning has advanced
As machine learning advances and jobs are taken over by AI, the skills required of us will be different than ever before.
Specifically, it will be required to “show direction by making AI and robots subordinates”.
Therefore, in the future, we will need “thinking ability to tackle complex data and issues,” “communication ability,” and “basic knowledge of AI.”
Repetition of simple tasks is AI’s specialty, and it represents the type of occupation that will replace humans. From now on, we should play the role of leading AI and guiding it in the right direction.
It is also important to acquire basic knowledge of AI and to keep up with the latest AI trends in order to be able to properly indicate the direction of AI.
First, let’s start with something easy to do, such as studying AI through lectures and books, or subscribing to AINOW’s e-mail newsletter to get the latest AI news.
Summary
In this article, we will discuss in detail what machine learning can do. There are six things that machine learning can do, and these technologies are being used in various situations around the world.
However, you should have understood that machine learning is not all-purpose, and that there are multiple things that “cannot be done” and “we are inferior to humans”.
If you want to “develop machine learning” or “use machine learning in business” in the future, it would be a good idea to understand what machine learning can and cannot do.
