Anomaly detection is a very common problem that can be found in many applications.

At the start of this chapter, we described a few possible use cases and highlighted the major types and differences according to the context and application requirements.

We briefly covered some of the popular techniques for solving anomaly detection using shallow machine learning algorithms. The major differences can be found in the way features are generated. In shallow machine learning, this is generally a manual task, also called feature engineering. The advantage of using deep learning is that it can automatically learn smart data representations in an unsupervised fashion. Good data representations can substantially help the detection model to spot anomalies.

We have provided an overview of H2O and summarized its functionalities for deep learning, in particular the auto-encoders.

We have implemented a couple of proof-of-concept examples in order to learn how to apply auto-encoders for solving anomaly detection problems.

For the digit recognition, we ranked each image according to an anomaly score given by the model reconstruction error.

A similar approach could also be further extended to applications such as signature verification, author handwriting recognition of manuscripts, or fault detection via image pictures.

The digit recognition example was a type of individual point outlier detection. It used a shallow architecture made of only one single hidden layer.

For the ECG example, we used a deeper architecture and showed an additional detection technique based on the compressed feature representation instead of the fully reconstructed one. We used the encoder part of the network to compress the non-linear relationships of the raw data into a smaller dimensionality space. The newer representation can then be used as a pre-process step in order to apply classic anomaly detection algorithms such as Gaussian Multivariate Distribution. By reducing to a two-dimensional space, we could even visualize the data points and identify anomalies at the frontier of the main elliptical distribution.

Nevertheless, auto-encoders are not the only way of doing anomaly detection using deep learning. You can also follow a supervised approach where you take out part of the information from your data and try to estimate based on the remaining information. The predicted value will represent your normal expected behavior and deviations from this value would represent your anomalies. For example, in case of time series, you could use recurrent neural networks (RNNs), or their evolution in long short-term memory (LSTM), as a regression model to predict what is going to be the next numerical value of a time sequence and then use the error between the predicted and observed value as an anomaly score.

We preferred to focus on this semi-supervised approach because it can be applied to many applications and also because it is nicely implemented in H2O.

Another important detail is that the majority of the code snippets were written for data analysis, manipulation, and visualization. By using H2O, we used the built-in classes to implement deep neural networks in just a couple of lines of code. This is quite impressive compared to the overhead of other frameworks. Moreover, the H2O estimators and models offer a wide range of customizable parameters and different configurations. On the other hand, we found H2O to be quite limited in extending its usage for scopes that are not currently supported. Overall, it is a very promising technology and there is much room for further improvement.

Be aware that the techniques covered in this chapter served only as proof-of-concept for how deep learning could be applied to anomaly detection. There are many gotchas and pitfalls to consider, both technical and practical, when dealing with production data. We will cover a few of them in Chapter 10, Building a Production-Ready Intrusion Detection System.