Real-world applications of anomaly detection

Anomalies can happen in any system. Technically, you can always find a never-seen-before event that could not be found in the system's historical data. The implications of detecting those observations in some contexts can have a great impact (positive and negative).

In the field of law enforcement, anomaly detection could be used to reveal criminal activities (supposing you are in an area where the average person is honest enough to identify criminals standing out of the distribution).

In a network system, anomaly detection can help at finding external intrusions or suspicious activities of users, for instance, an employee who is accidentally or intentionally leaking large amounts of data outside the company intranet. Or maybe a hacker opening connections on non-common ports and/or protocols. In the specific case of Internet security, anomaly detection could be used for stopping new malware from spreading out by simply looking at spikes of visitors on non-trusted domains. And even if cyber security is not your core business, you should protect your network with data-driven solutions that can monitor and alert you in case of unrecognized activities.

Another similar example is authentication systems for many major social networks. Dedicated security teams have developed solutions that can measure each single activity, or sequence of them, and how distant those are from the median behavior of other users. Every time the algorithm marks an activity as suspicious, the system will prompt you with additional verifications. Those techniques can dramatically reduce identity theft and offer greater privacy protection. Likewise, the same concept can be applied to financial fraud, as we have seen in the previous example.

Anomalies generated by human behavior are among the most popular applications, but also the toughest. It is like a chess game. On one side, you have subject matter experts, data scientists, and engineers developing advanced detection systems. On the other side, you have hackers, aware of the game, studying their opponent's moves. That's why those kinds of systems require a lot of domain knowledge and should be designed to be reactive and dynamic.

Not all of the anomalies are originated from the "bad guys". In marketing, anomalies can represent isolated, but highly profitable customers who can be targeted with tailored offers. Their different and particular interests and/or profitable profile can be used to detect the outlying customers. For example, during an economy recession period, finding a few potential customers who are increasing their profit despite the mass trend could be an idea for adapting your product and redesigning your business strategy.

Other applications are medical diagnosis, hardware fault detection, predictive maintenance, and many more. Those applications also require agility.

Business opportunities, just like new malware, can rise every day and their life cycle could be very short, from hours to a few weeks. If your system is slow to react, you could be too late and will never catch up to your competitors.

Human detection systems are not able to scale and generally suffer from generalization. Deviations from normal behavior are not always obvious and it could be hard for an analyst to remember the whole history to compare to, which is the core requirement for anomaly detection. The situation complicates if the anomaly pattern is hidden inside abstract and non-linear relationships of entities in your data. The need for intelligent and fully automated systems that can learn complex interactions and provide real-time and accurate monitoring is the next frontier of innovation in the field.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset