Overview of Face-Detection Algorithms

Creating reliable face detection has long been a problem, and solving it is a crucial step toward other facial analysis algorithms such as face recognition, tracking, or facial feature detection (e.g., eyes or nose). It is a challenging problem as different face poses, camera orientations, lighting conditions, and imaging quality could make the detection algorithms fail. With that said, much progress has been made in the last decade, and numerous approaches have been proposed and developed.

Yang et al. [3] extensively surveyed these algorithms and divided them into four categories:

Many statistical models and machine-learning methods have been proposed for face detection, such as support vector machine, neural network, or Bayes classifier. However, the most widely used method is the one based on boosting, which we will touch on in the next section.

Viola–Jones Face-Detection Algorithms

The most popular and widely used face detection technique is the one developed by Viola and Jones [4]. It was developed as a generic object-detection framework and outperformed previously proposed approaches in terms of both performance and running time. The main challenge of face detection for these model-based or template-based approaches is the need to scan through every possible location and scale of an image in order to find the locations of the faces and reduce the false positives.

However, since an image usually contains only a few (0–5) faces, schemes like this end up spending lots of computing cycles on aggregating the information and evaluating candidates that are mostly nonface regions. The main contributions of the Viola–Jones face detector include a new image representation for fast feature computation, a boosting-base feature selection method, and a “cascade” classifier structure to quickly discard a large number of nonface regions. We will detail each of them next.

The Viola–Jones face detector starts with computing the features over a detection area on an intensity image. The features computed are rectangle features of different sizes. Three different kinds of rectangle features are used: two-rectangle features with two regions of the same size adjacent to each other horizontally and vertically. A three-rectangle feature consists of three neighboring rectangular regions, and a four-rectangle feature consists of two diagonal pairs of rectangular regions. For all cases, the feature value is computed as the difference between the sums of the pixel values within the black regions and the white regions. Figure 2.1 shows four examples of the rectangle features. You could think of these features as possible signatures for the face regions. The process of face detection is to look for the signatures, trim out nonface regions where the signatures do not exist, and identify the face regions with salient signatures.

In order to compute theses feature quickly, a new image representation, integral image, is introduced. Let the original image be I and let S be the integral image representation of I. The integral image S computes at each image location (x, y) a value S(x, y), which is the sum of the pixel values above and to the left of (x, y), inclusively, on the original image. The computation of all the values in S can be done in a pass-through all the pixels on I following the recursion:

$S(x,y)=S(x-1,y)+S(x,y-1)-S(x-1,y-1)+I(x,y)$ (2.1)

(2.1)

The recursion can be illustrated in the left image of Figure 2.2, which shows that the area of the rectangle ACIG can be computed by subtracting the area of rectangle ABED from the sum of those of rectangle ACFD and ABHG. Once the integral image representation S of the original image I is computed, the sum of original pixel values within any rectangle can be computed by table lookup. For example, as shown in the right image of Figure 2.2, the sum of the rectangle ABCD in I can be computed by S(C) − S(B) − S(D) + S(A), where S(·) indicates the value of a particular location on image S. Note that, with the integral image, we could compute the sum of any rectangle region on image I using a constant number of operations, e.g., four lookup operations and three numerical operations. This allows us to compute the rectangle features in an efficient manner.

With the computed features, one can use this favorite classifier and train a face model for detection using these rectangle features. However, with just a 24-by-24 detection window, there are about 160,000 possible rectangle features. It will be impractical and computationally infeasible to compute all of the features every time when we would like to know if a certain detection region contains a face. The question, then, is how to choose a small subset of features and build a classifier in an efficient manner.

Feature selection is a problem that has long been studied in the field of machine learning, and many approaches have been proposed. The Viola–Jones face detector uses a variant of boosting [5], as it could allow the classifier to efficiently prune out most of the irrelevant features during the training stage. Conventionally, boosting is a classification scheme used to boost the classification result of simpler learning algorithms. In the literature, the simpler learning algorithm is termed as a weak learner, meaning that its classification performance could be just slightly better than random guessing. For instance, for a two-class classification problem, the classification performance might be slightly over 50%. The boosting scheme adopts a round-based procedure and iteratively evaluates the weak learner on the training set. In each round, a set of weights is kept for the training set and the weight for the wrongly classified example is increased so that the weak learner could focus on learning the hard examples. At the end of each round, the weak learner is assigned a weight based on how well it does on the weighted training set. The better the learner performs, the larger the weight it is assigned. The output of the boosting procedure is then a weighted combination of these learners.

The boosting procedure can be seen as a process that produces a weighted sum of a large set of classification functions where strong-performing functions receive larger weights. It can also be used to incorporate the feature selection procedure if the weak learner is a function of a single feature. The Viola–Jones detector proposes the following function as the weak learner:

$h(x,f,p,\theta )=1\mathrm{if}pf(x)<p\theta ,0\mathrm{otherwise}$ (2.2)

(2.2)

Here, h is the classification function that takes as parameters x, the image region within the detection window, f, the rectangle feature being considered, p, the polarity that determines the direction of the sign and, θ, the threshold for the feature and produces a decision regarding whether the detection window contains a face. Since the classification function h is a function of only one single feature, the boosting procedure amounts to selecting features that are more relevant. A key reason the Viola–Jones detector adopts boosting as a feature selection process is its efficiency as compared to other feature selection methods, which typically require an additional loop on the number of examples.

Figure 2.3 shows the first two features selected by the boosting procedure. These two features imply that a signature for a frontal face is a darker region (the eyes) or a sandwiched dark–light–dark pattern in the middle.

To further reduce the false positive rate and increase the detection accuracy, the Viola–Jones face-detection algorithm constructs a cascade of boosted classifiers as shown in Figure 2.4. The reasoning behind the cascaded classifier is that most of the false positives could be pruned out and rejected at the early stages. The classifiers at the earlier stages are tuned so that they could reject most of the false faces while at the same time maintaining a decent detection rate. Subsequent classifiers are trained using the positive examples that pass every previous stage and as a result, classifiers at later stages actually deal with harder examples. Interested readers could refer to Ref. [4] for more details.

At the time of publication, the method was running 15 frames per second on a Pentium III processor. However, even with the computing power of the modern day cell phones, it is still difficult for the original algorithm to run in real time on mobile platforms. We will introduce some modified face detection methods after we introduce face recognition algorithms.

Overview of Face-Recognition Algorithms

Given an input image and a detected face region, face recognition refers to the problem of identifying the detected face as one of those in a face database. The problem has been studied for over 30 years, and research has resulted in many successful commercial applications. However, two main challenges still remain: face pose variation and illumination changes.

Numerous approaches have been proposed due to the potential extensive applications. For example, in the security application domain, face recognition technology can allow law enforcement officers to search for a particular person in large video databases. In this section, we focus on image-based face recognition technologies and give an overview of different face recognition approaches.

The approaches to face recognition can be categorized into three classes [6]:

Over the past few decades, much progress has been made toward the detection of faces in still images and the extraction of the face features. At the same time, significant advances have also been made in the area of face recognition. Researchers have shown that holistic approaches are effective on large datasets. Feature-based approaches also enjoy high recognition rates, and compared with their holistic counterparts, they are shown to be more resistant to pose and lighting changes. However, their performance highly depends on the reliability of underlying feature detection and localization methods. For instance, the eye detectors might not work if they are based on the geometry model of open eyes. In the next few sections, we will introduce some representative methods from each genre, especially Eigenfaces [7] and Fisherfaces [8] due to their success and popularity.

Holistic Face-Recognition Algorithms—Eigenfaces and Fisherfaces

Given a query image, the main idea behind Eigenfaces is trying to find its nearest neighbor from the face dataset and then assigning the identity of the nearest neighbor to it. The question, then, is how to represent the face region. Assuming we could crop and scale the face region to an N-by-N square, we could denote the entire face region as x, where x is a feature vector of dimension N². Given a face dataset consisting of M faces, denoted as y₁, y₂, … y_n, the Eigenfaces approach tries to find y_k, such that y_k minimizes the distance function $\left|\right|\mathbf{x}-{\mathbf{y}}_{\mathbf{i}}\left|\right|$.

Note that x sits in a very high dimensional space. For example, if the face region were a 256-by-256 image patch, x would be a vector of 65,536 dimensions. Although the N-by-N dimensional space represents all possible images of size N-by-N, the actual distribution of face images might actually live in a subspace that has a much smaller dimension due to the similarity of different human faces. Figure 2.5 illustrates that the image dataset might sit in a lower-dimensional space, indicated by the solid ellipsoid.

To find such a lower-dimensional space, the Eigenfaces approach performs principal component analysis (PCA) on the image dataset. PCA is a statistical method used to find the principal components or eigenvectors of sample data. These principal components characterize how the sample data are distributed. For instance, the principal components would be the variance along the x- and y-axis for data that are Gaussian distributed in a 2D space. In face recognition, these eigenvectors are called Eigenfaces and can be thought of as canonical faces that characterize the face image distribution. Figure 2.6 shows the process of using Eigenfaces for face recognition.

The Eigenfaces approach has been shown robust to noise such as partial occlusions, blurring, or background changes, and has yielded good performance on standard datasets. However, the drawback of Eigenfaces is its lack of discriminant power. The reason for this is that Eigenfaces do not take class information (i.e., face identities) into account. Instead, all the training images are fed into the PCA module at once with no class-specific treatment. Fisherfaces [8] was proposed to remedy such a drawback.

The motivation behind Fisherfaces is that since the training set is labeled with face identities, it will make more sense to use a class-specific linear method for both dimension reduction and recognition. Instead of PCA, Fisherfaces uses linear discriminant analysis (LDA) for dimension reduction. Given a set of face images with C classes, the Fisherfaces approach computes the between-class scatter matrix:

$S_{\mathrm{B}}=\sum _{i=1}^C{N}_i({\mu }_i-\mu ){({\mu }_i-\mu )}^{\mathrm{T}}$ (2.3)

(2.3)

and the within-class scatter matrix:

$S_{\mathrm{W}}=\sum _{i=1}^C\sum _{x_k\in X_i}({\mu }_i-x_k){({\mu }_i-x_k)}^{\mathrm{T}}$ (2.4)

(2.4)

where N_i denotes the number of images within class X_i, µ the mean feature vector of all the images, and µ_i the mean feature vector of the images of class X_i. The Fisherfaces approach computes the eigenvectors W such that

$W=\mathrm{argmax}\frac{\left|W^{\mathrm{T}}{S}_{\mathrm{B}}W\right|}{\left|W^{\mathrm{T}}{S}_{\mathrm{W}}W\right|}$ (2.5)

(2.5)

The difference from PCA is that LDA maximizes the ratio of between-class scatter and the within-class scatter. The idea behind this is to project the feature vectors to a new space so that feature vectors from the same class are closer while those from different classes are further away. This can be seen in Figure 2.7. The solid circles and triangles represent two different classes in the 2D space. The dotted line is the subspace formed after PCA is performed, while the dashed line is that after LDA. Note that on the dotted line, the projected locations of the triangles and circles are mixed together, making it difficult to differentiate the two classes. On the other hand, on the dashed line, the projected circles are well separated from the triangles. This shows that LDA is more discriminant than PCA.

Other features have also been proposed, such as edges or intensities. Interested readers could refer to Ref. [6] for more details.

Feature-Based Face-Recognition Algorithms

Earlier face-recognition methods are mostly feature-based. For instance, they recognize faces using information such as the width of the head, distances between the eyes and from the eyes to the mouth, or the distances and angles between eye corners, mouth extrema, nostrils, chin, and so on. The most successful feature-based face-recognition algorithm is the elastic bunch graph matching system (EBGM) [9]. The approach represents the face regions with Gabor wavelets. That is, each image is convolved with the Gabor filter to generate wavelet coefficients:

$g(x,y)=\exp \left(-\left[\frac{x^2}{2{\sigma }_x^2}+\frac{y^2}{2{\sigma }_y^2}\right]+2\pi i[u_{0}x+v_{0}y]\right)$ (2.6)

(2.6)

where σ_x and σ_y denote the width of the spatial Gaussian and u₀ and v₀ denote the frequency of the complex sinusoid.

For each face in the training set, a set of N fiducial points, e.g., the pupils, corners of the mouth, and nose tip are labeled. A graph G is then constructed to represent the face, where each vertex of G corresponds to each fiducial. A feature representation, jet, is defined to describe the local image patch surrounding a particular image pixel. It is essentially a set of wavelet coefficients at different image orientations and scales. For each graph constructed, the jet for each fiducial point is also extracted. Given an input face image, the EBGM face recognition system extracts the fiducial points by matching it against an offline built face model and represents it with a graph. Face recognition is done by finding the nearest neighbor of its graph representation. Figure 2.8 illustrates the process of graph building in the EBGM face recognition system. The elastic bunch graph matching approach has also been successfully applied to other domains, such as face detection, pose estimation, gender classification, and general object recognition.

Hybrid Face Recognition Algorithms

Finally, the hybrid approach combines the features from the previously mentioned genre. For instance, in Ref. [10], instead of computing the Eigenfaces for the entire dataset, the authors compute the Eigenfaces for each view point and show that it performs better. Taking the idea even further, they introduce the concept of Eigenfeatures, such as Eigeneyes and Eigenmouths and show that with a small number of eigenvectors, the Eigenfeatures approach outperforms Eigenfaces.

Face Detection on Mobile Platforms

A face detector based on the cascaded classifiers runs efficiently on a PC but would need further optimization to run on mobile platforms. Although there are dedicated hardware implementations—e.g., smart digital cameras for face detection—real-time software implementation of face detection is still a challenge. Several strategies have been proposed to address this issue, categorized as follows:

Face-Detection Applications on Mobile Platforms

Face detection is usually the first step towards many face-related technologies, such as face recognition or verification. However, face detection itself can have very useful applications. The most successful applications of face detection would probably be photo taking. When you take a photo of your friends, the face detection algorithm built into your digital camera detects where the faces are and adjusts the focus accordingly.

Head pose estimation is another application that heavily relies on face detection. Estimating the head pose is useful in the settings of automated guided cars, where an in-car device runs the head pose estimation algorithm to detect the drowsiness of the driver. In Ref. [15], the authors show a real-time head pose estimation system that can identify five poses on a mobile platform.

Another good usage of face detection/tracking is the support of mobile device interaction. In Ref. [16], the authors propose an interaction method with the mobile device based on tracked face position. With mixed interaction spaces (MIXIS), their system allows the mobile device to track the location of the device in relation to a tracked feature in the real world. The main idea of MIXIS is to use the space and information around the mobile device as input instead of limiting interaction with mobile devices to the device itself (Figure 2.10). To this end, they track the location of the user’s face to estimate the location of the device relative to the tracked face, which forms a 4D input vector, i.e., x, y, z, and tilt. The difference with other face-tracking applications is that in their system, it is not the face that moves but the mobile device, which minimizes exhaustion related to frequent head movements. With the face location tracked, the system can support applications such as a Pong game or map panning and zooming. Figure 2.10 shows the concept of MIXIS, where the user is able to interact with the mobile device by moving it in different directions. This is achieved through the face-tracking module on the device. The image on the right-hand side shows a Pong game application.

Face Recognition and Verification on Mobile Platforms

Compared to face detection, the algorithms for face recognition typically run much faster. Therefore, with a fast face-detection implementation, face recognition can typically run on mobile platforms without many modifications. With that said, face recognition enables many novel applications on mobile platforms. In Ref. [17], the authors present a system for face annotation. The user takes a photo using a mobile phone and the system detects the faces in the photo, recognizes and identifies the faces, and finally posts the photo with annotated faces to the user’s social network websites. Figure 2.11 shows a high-level illustration of this system.

The system was implemented on the Android platform on the Nvidia Tegra SoC board. To start with, it detects the face region using the Android API. It then builds a custom Ada-boost eye detector to identify the landmarks on the face. Given the landmark locations, the system scales and registers the input photo so the image is better aligned with those in the database. Later, the system extracts Gabor wavelet features [18] on the face region and dimensionally reduces using PCA and LDA. Finally, the system uses a K-nearest neighbor classifier for face recognition.

Although the Gabor wavelet features are reported to be more invariant to illumination and pose changes, they are more computationally expensive. On an ARM CPU, the authors report a 5.1 s per-frame performance. Using an embedded GPU, the authors are able to reduce the computational cost to one second. The entire system achieves over 90% on a traditional face dataset.

Another major application is to use face authentication to unlock the mobile phone [19]. That is, instead of asking the users to type or gesture, the mobile phone can unlock itself by recognizing the face of its user. Similar ideas have been implemented, such as the face unlock feature on Android, or other mobile platforms [19,20]. The pipeline for face authentication is similar to that in Figure 2.11, except that the number of stored faces is much smaller, as frequent users of a mobile phone are typically limited to one or two. The feature extraction part could differ to accommodate for different applications. Next, we briefly discuss some examples and considerations for feature design.

In Ref. [19], the authors propose to use local binary pattern (LBP) [21] for face recognition on the mobile platform for its computational efficiency and discriminant power. An LBP is a representation of a pixel on an image. For a pixel, its LBP can be formed by comparing the pixel to each of its eight neighbors. When its pixel value is greater than the neighbor’s value, mark “1,” otherwise, mark “0.” This gives an 8-digit binary number for a pixel and is called the LBP for the pixel. Figure 2.12 shows an example of LBP for the center pixel.

To use LBP as the feature for facial authentication, the authors first divide the detected face region into several grids. The LBP is computed for each pixel in each grid. The binary number can be turned into a decimal value. For instance, the binary number 00001000 is eight in decimal base. With these values, the authors compute a histogram for each cell and use the concatenated histogram to represent the detected face. The concatenated histogram is compared to that of the stored face to determine if there is a match. Figure 2.13 shows the process of using LBP for face authentication. The authors later augment the authentication system by combining face and speech recognition. Interested readers could refer to Ref. [22].

Another face verification system is presented in Ref. [20]. The authors use template matching in the frequency domain to determine if the query image is authentic. During the training stage, the face regions are detected and extracted from the training images. These detected regions are scaled to ensure all faces from the training set have the same dimension. Finally, a template is constructed from the training set. In addition to a global face template, to improve the accuracy, the authors extract subregions around facial features, such as eyes and nose, and build several part templates. Given a query image, the system matches it with the global template while at the same time extracts the regions around the facial features to produce several fragments, each of which is matched with the part templates to produce a local matching score. The global and local matching results are combined to determine if the query image is authentic. In all cases, fast Fourier transform is applied to all the images and the correlation is done in the frequency domain to improve computational efficiency. Figure 2.14 illustrates the process of the system.

Facial Feature Tracking on Mobile Platforms

The Active Appearance Model

So far, we have been discussing the use of facial features, such as eyes and nose tip, for facial recognition or verification. However, being able to track the facial features reliably over time is itself useful. For instance, by recognizing and tracking the facial features, you could imagine a videoconferencing application where an avatar speaks on your behalf and its facial geometry morphs as you speak or move your head. Figure 2.15 shows a videoconferencing system where the software on one end tracks the facial features and sends their location information over the Internet. The motivation behind this is that in a mobile environment, the network bandwidth is limited, and hence it is not possible to use conventional videoconferencing software that sends all the image data over the network. With a facial feature tracker, the application on one end could just send the landmark locations over and the application on the other end could render an avatar and control the avatar with the received facial information.

As another example, imagine applications that use the facial feature information to infer where the users are looking and then render the user interface accordingly. The availability of the detailed feature information could also enable finer-grained interface control than using the head pose, e.g., allowing disabled users to control the interface by eye blinking. It can also be applied to authentication, as we have discussed previously.

In this subsection, we introduce the state-of-the-art method for facial feature tracking, and how it can be optimized on mobile platforms in the next subsection. We conclude the section with its applications, particularly in the area of authentication.

The conventional approach toward this end is the active appearance model (AAM) [23]. The authors propose this as a means to interpret an image. Their approach interprets an unseen image by matching it against a synthesized or rendered image from an offline model. The approach falls in the category of what is called analysis-by-synthesis in the literature. For example, in order to analyze face images, the authors generate a statistical model for face shape variation. The model is generated offline with a set of face images, and each image contains n hand-labeled landmark points. Figure 2.16 shows an example of a hand-labeled face image. The red dots represent the landmark points.

For a labeled face image with n landmark points, we could represent its shape with a 2n dimensional feature vector $X=(x_1,y_1,x_2,y_2,\dots x_n,y_n)$, where x_i, y_i is the image coordinate of landmark point i. Given a set of N aligned labeled images, the AAM approach computes the mean vector $\overline{X}$ as the mean shape and models the shape of each face X as $\overline{X}+\delta$, where δ is the shape variation. AAM assumes δ is Gaussian distributed and applies PCA to find the canonical variations. Finally, the shape of each face can be written as $\overline{X}+Pb$, where P is the matrix whose columns are the top-k eigenvectors from PCA and b is the k dimensional shape parameter.

The above procedure only models the shape variations of the face. To model the appearance, the AAM approach first warps each example image so that it aligns with the mean shape. For each example face image, let g be the vector representing the intensities of the pixels covered by the mean shape and $\overline{g}$ be the mean appearance vector of all the images. Similarly, the appearance of the face could also be modeled as $g=\overline{g}+{\delta }_{\mathrm{g}}$, where δ_g represents the appearance variations and, in turn, can be written as δ_g=P_gb_g, where P_g is the matrix whose columns are the top-k eigenvectors from PCA done on the appearance feature vectors and b_g is the k dimensional appearance parameter. Figure 2.17 shows examples of synthesized images using the AAM model.

Given a statistical face model and an unseen image, how can we then synthesis a face image that matches the unseen image well in order to analyze it? Using a face detector, the AAM approach translates and scales the face region to approximately align it with the face model. Let c=(b, b_g) be the model parameters from which a model image I_m is generated from and I_i be the input image. The matching problem can be formulated as finding the optimal model c, such that the $|\delta I{|}^2=|I_{\mathrm{m}-I_i}{|}^2$ difference between the model image and the input image is minimized, which is a nonlinear high-dimensional optimization problem. To find the best model parameters c, the authors propose to learn the mapping between δI and δc. The assumption is that the spatial pattern in δI provides information regarding how to adjust the model parameters. Toward this end, they model the mapping linearly, i.e., δc=AδI and find A using multivariate regression with given sample of known δc and δI. The training set of δc and δI are generated in an offline process, where a known model parameter c is randomly perturbed to generate variations of I_m. The deltas of the model images and the model parameters are collected to form the training set to find A.

Given a new image and the mapping A, the matching process begins with the initial δI. It then updates the model parameter c with adjustment δc as predicted as AδI, followed by the generation of a new model image, from which δI is recomputed. The procedure repeats until δI is small enough that no improvement is possible for updating the model parameters.

Applications

To demonstrate the applicability of the mobile AAM, the authors propose a mobile biometrics system [25]. The system provides a verification layer where the users could use their face and voice for remote login without typing username and password.

The system first determines the face appearance by detecting the face region. Instead of using the rectangle features, the LBP representation is used to summarize the image statistics for a candidate region. A cascade of classifiers is built based on the LPB features in order to reject false positives. Later, the AAM is used to localize individual facial features, such as eyes or nose. These facial features are used to remove irrelevant background. They are also used to fit the unseen face image into a predefined face shape model so that the scale, shape, and orientation of input image is normalized to that of those in the face database. The processed face image is then sent for face verification [19] after being normalized for lighting. Figure 2.18 illustrates this process.

Note that the authors could have tried to recognize the face using the detected face region without using AAM. However, the variations in face orientations, face expressions, lighting conditions, and background clutter could degrade the face recognition system. The accuracy of face recognition could be further improved when these factors are removed or reduced.

The system also uses voice recognition to improve the accuracy of authentication. As seen in Figure 2.19, the image data and voice data are fed separately into the system and processed independently. The face module takes and processes the image to give a score on how likely it is that a face belongs to the authentic users. The same goes for the voice module. In the data fusion step, another classifier takes the scores from both modules and determines whether to grant access.

Other Applications on Mobile Platforms

The facial features are very useful in another application domain that needs higher-level face analysis, such as face expression recognition. For example, in Ref. [26], the authors present a system that uses sensor data from a tablet to infer the reactions of the user when he/she is watching movies. The inferred reactions are transformed to a movie rating automatically. One module inside their system is the face analysis system, which tracks the face, eyes, and lips. The locations and sizes of eyes and lips are used as features for user reaction recognition.

To cite another example, Visage system [27] tracks the face, detects the facial features, and uses the facial features to recognize user emotion, e.g., sad or happy. Two proof-of-concept applications are built on top of Visage to demonstrate its applicability. For instance, the Street View+ shows different viewpoints of Google Street View based on detected head poses, while the MoodProfile detects and records users’ emotion when they use a particular application. Figure 2.20 illustrates the flow chart of the Visage system.

The Visage system starts with phone pose detection based on the sensor on the phone. The pose information is used to compensate for the tilting of the face images. Face detection based on the Viola–Jones detector is done to locate the face region. Later on, the AAM approach is used to detect the facial features on the face. Finally, these facial features are used to classify the user emotion into one of the seven major classes.