2.4.1.1 AND Rule

In AND rule [4, 51, 1], the final decision is a conjunction of the individual matcher decisions. AND rule is explained in Eq. (2.1). The matchers can be imagined to form a serial combination. It results in a match if and only if all the inputs result into a match. Systems using AND rule for fusion exhibit a high FRR. Because of it, this type of fusion gives the highest level of security. These systems, however, have a very low, even as low as 0% FAR.

(2.1)

2.4.1.2 OR Rule

In OR rule [51, 1], the final decision is a disjunction of the individual matcher decisions. OR rule is explained in Eq. (2.2). The matchers can be imagined to form a parallel combination. It results in a match if any one of the inputs results in a match. Systems using OR rule for fusion exhibit a higher FAR. Because of it, this type of fusion gives the lowest level of security. These systems, however, have very low, even as low as 0% FRR.

(2.2)

2.4.1.3 Majority Voting

In this rule, the final decision is the one that is in the majority among all the matchers as shown in Eq. (2.3). If majority of the matchers are reliable and accurate, then the final decision can be trusted otherwise, an accurate matcher may fail to affect the final output decision in the presence of majority of less accurate matchers.

(2.3)

2.4.1.4 Maximum Rule

The maximum rule [41] gives the final output score as the maximum of all the matcher scores. A mathematical formula is given in Eq. (2.4). The accuracy of the matcher with the maximum match-score will determine the accuracy of the final decision.

(2.4)

2.4.1.5 Minimum Rule

In minimum rule [41], the final output score is the minimum of all the matcher scores shown in Eq. (2.5). The accuracy of the matcher with the minimum match-score will determine the accuracy of the final decision.

(2.5)

2.4.1.6 Sum Rule

Sum rule [34, 51] is similar to the OR rule, but for match scores. The final score is the sum of the individual match scores (Eq. (2.6)). This rule has been shown to perform better than product rule and Linear Discriminant Analysis (LDA) [40, 12]. The best combination results are obtained by applying simple sum rule [34]. Sum rule is not significantly affected by the probability estimation errors and this explains its superiority [23]. The ROC performances of sum rule are always better than that of the product rule [49].

(2.6)

2.4.1.7 Product Rule

Product rule [51] is similar to the AND rule, but for match scores. The final score is the product of all the input scores. This rule performs as if the matchers are arranged in a series combination (Eq. (2.7)). This rule assumes independence between individual modalities. If the scores are normalized in the range [0, 1] then the final match score will always be less than the smallest of the match scores.

(2.7)

2.4.1.8 Arithmetic Mean Rule

In arithmetic mean rule [51], final fused score is the arithmetic mean of the match scores of all the matchers (Eq. (2.8)). The match scores should be in the same numerical range i.e. normalized so that they can be averaged out properly.

(2.8)

2.4.2.1 Weighted Sum Rule

Weighted sum rule [31, 34] is a modification of the simple sum rule. In this rule, a weighted factor is multiplied by the score and then sum is calculated as given in Eq. (2.9). Each matcher carries a weight, W_i. Matcher weights are calculated heuristically and empirically. Generally, the matcher weights are directly proportional to their accuracy. The more accurate a matcher is, the more weight it carries.

(2.9)

where

2.4.2.2 Weighted Product Rule

This rule is a modification of the simple product rule. In this rule, a weighted factor is multiplied by the score and then the product is calculated as given in Eq. (2.10). Each matcher carries some weight, W_i. Matcher weights are calculated using some heuristic methods. Matcher weights are directly proportional to their accuracy. The more accurate a matcher is, the more weight it carries.

(2.10)

where

2.4.2.3 User Weighting

In user weighting [19, 41], each user is assigned a specific weight into each modality. The final score is calculated by multiplying the assigned weights with each modality’s score as given in Eq. (2.11). User weights are calculated heuristically and through empirical measurements. This method provides an improvement over the matcher weighing method. Performance improvement is seen with reduced FRR as the user convenience and user’s habituation with the system increases. Automatic learning and update of system parameters help in reducing the errors associated with an individual, thus improving system performance [19]. Not all users have all of their traits up to an acceptable level/standard and so assigning equal weights to such traits will not exploit their full potential and may dampen their uniqueness. This will also cause their false rejection. User specific weighing and user-specific thresholds will accommodate more people into the system, thus reducing the system’s FRR.

(2.11)

2.4.2.4 Fisher Linear Discriminant (FLD)

Fisher’s linear discriminant [2, 58] is a classification method that projects high-dimensional data onto a line and performs classification in this one-dimensional space. The projection maximizes the distance between the means of the two classes while minimizing the variance within each class. The Fisher Linear Discriminant (FLD) gives a projection matrix W that reshapes the scatter of a data set to maximize class separability, defined as the ratio of the between-class scatter matrix to the within-class scatter matrix. This projection defines features that are optimally discriminating. For that purpose FLD finds such a W to maximize the criterion function defined in Eq. (2.12).

(2.12)

where, S_w and S_b are the within-class scatter matrix and the between-class scatter matrix, respectively. These two matrices are defined in Eq. (2.13) and Eq. (2.14).

(2.13)

(2.14)

In Eq. (2.13) and (2.14), is the i^th sample of class j, µ_j is the mean of class j, c is the number of classes, N_j is the number of samples in class j, and µ is the mean of all classes.

2.4.2.5 Support Vector Machine (SVM)

SVM [25, 34, 3, 2, 14] is a binary classifier that maps the input vector x_i ∈ R^d to output labels y_i = −1, +1 for i = 1, 2, 3, . . . , l, where l is the number of patterns to be classified. A binary classifier is used to map the n-dimensional data points into the label space of two classes (genuine and imposter). The main objective of SVM is to find an optimal separating hyperplane that separates the dataset into two groups and maximizes the separation between them. The separating Hyperplane could be linear or nonlinear [14]. A general form of SVM can be represented as,

(2.15)

Where, α_i is Lagrange multipliers, Ω is indexed to support vectors for which α_i ≠ 0, y_i is output labels, b is bias term, K (x, x_i) is kernel function and x is the input vector to be classified.

The kernel function decides the nature of the separating surface. This function can be polynomial (see Eq. (2.16)) or a Radial Bias Function (RBF) (see Eq. (2.17)).

(2.16)

(2.17)

The choice of kernel function depends upon the satisfaction of Mercer’s conditions. It has been shown that the polynomial classifier performs better than the RBF classifier kernel function [2]. The final decision is made depending on whether f (x) is above or below a threshold.

The experimental comparison of many fusion techniques along with SVM is made in [2] and it shows an FAR of [1.07, 2.09]% and an FRR of [0.0, 1.5]%. This is the best classifier among the classifiers studied in [2]. These results are well within acceptable limits and as it can be seen an FRR of 0% has been achieved.

2.4.2.6 Multi Layer Perceptron (MLP)

An MLP [55, 2] is a supervised learning neural network consisting of many layers of nodes or neurons as shown in Fig. 2.2. The input layer is denoted by i and accepts inputs like a particular pattern or a feature vector from the biometric system. The next layer is called as a hidden layer since it exists only between the input layer and the output layer. There can be more than one hidden layer and they are denoted by j. The final layer is the output layer, denoted by k, and provides the outputs to the outside world.

Figure 2.2: Multilayer perceptron network.

All the layers are fully connected; meaning a node in one layer is connected to every node in the next layer. Each of these connections or branches carries a specific weight. The weight matrices are w_ij for weights between the input and the hidden layer and W_jk for the weights between the hidden layer and the output layer. These weights are obtained during the training phase.

Input to each node in the input layer undergoes identity transformation, I_i = O_i where I_i is the input and O_i is the output of the i^th node. Input I_j to the hidden layer is given by the following transformation.

(2.18)

where w_ij is the weight of branch between i to j and O_i is the output of the i^th node from the input layer.

The output of the hidden layer is given by,

(2.19)

where I_j and O_j are the input and output of the j^th node, respectively. f () is an activation function of the node.

I_k is the input to the output layer is given by the following transformation.

(2.20)

where, W_jk is the weight of branch between j node to k node and O_j is the output of the j^th node from the hidden layer.

Once these inputs are available at the output layer, they undergo similar transformation as given by Eq. (2.19).

MLP is a supervised learning network. That means the desired output is available and the MLP is trained to deliver an output close to the desired output (e.g. for learning the AND rule, when the input is two 0s, the desired output is a 0). During the training phase, many input patterns are presented to the MLP along with the desired output and the weights w_ij and W_jk are adjusted in such a way that actual output is as close to the desired output as possible. The learning phase is completed when the MLP can deliver actual output very close to the desired output. For continuous adjustment of the weights w_ij and W_jk, MLP uses a back-propagation algorithm [15, 44, 28]. This algorithm calculates the error which is the difference between actual and desired output. This error is propagated backwards from the output layer until all the weights up to the input layer are adjusted. The weights are adjusted as follows,

(2.21)

where, w_kj (t + 1) and w_kj (t) are the weights connecting nodes k and j at iteration (t + 1) and t, respectively and η is the learning rate.

The difference between the actual output (δ_k) and the desired output (d_k) is calculated as follows.

(2.22)

This error is then back propagated through the entire network and the weights are adjusted as follows.

(2.23)

2.4.2.7 Mixture-of-Experts (MOE)

Mixture-of-Expert rules [25] consists of two layers as shown in the Fig. 2.3.

Layer of local experts: This layer consists of a number of local experts. Each local expert contains some classifiers. These classifiers work on a single modality. Each classifier uses its own matching algorithms and comes up with local score. These local scores are then passed along to the second layer.

Layer of gating network: This layer consists of a gating network [25]. This network combines the local scores into a final fused score using hard switching fusion network, sum rule and product rule in rule-based fusion, support vector machines, multilayer perceptrons, and binary decision trees in learning-based fusion. This core is then used to classify the user as genuine or an imposter.

The gating network may be one of the following.

1) Adaptive Hard-Switching Networks: Final fused score is given by Eq. (2.24).

(2.24)

In this mode of operation, W_i = 1 for all i and W_j = 0 for all j and j ≠ i. Effectively, the final score is the score of any one of the matcher.

Figure 2.3: Block diagram of Mixture-of-Experts.

2) Adaptive Weighted Combination Networks: Final fused score is given by Eq. (2.25).

(2.25)

In this mode of operation, the classifier weights are nonzero. They M fall in the range [0, 1] such that .

It is a weighted sum of all the classifier rules. This is termed as a soft fusion method. Matcher learning techniques are used to determine an optimum value for constants W_i .

3) Adaptive Nonlinear Fusion Networks: The previous two type of gating networks produce a linear separation between the imposter and genuine distributions. For more information on experimental results of the same please refer [25]. These types of gating networks make use of non-linear classifiers such as SVM or decision-based neural networks. It gives more flexible decision boundaries. The experimental results in [25] show this type of fusion to be the best in terms of performance.

α_j are Lagrange multipliers and S contains indexes to the support vectors s_j . B is a bias multiplier and K (s, s_j) is a kernel function. The experimental results in [25] show this type of fusion to be the best in terms of performance.

2.4.2.8 Bimodal Fusion (BMF)

Bimodal fusion [10] fuses features from two modalities, hence the name. Both the modalities undergo with the different feature extraction phases. The features obtained after feature extraction phases are then fused to form a combined feature vector. In this approach, multimodal fusion is based on late fusion and its variants. Detailed discussion of this method can be found in [22, 35]. Late fusion or fusion at the score level involves combining the scores of the different classifiers, each of which has made an independent decision. The distinguishing factor here is the separation of the different modalities in the feature extraction phase.

2.4.2.9 Cross-Modal Fusion

In cross-modal fusion [10], the modalities go through a feature extraction phase and the obtained features are then passed through another feature extraction technique in order to obtain cross-modal features. The cross-modal features are based on optimizing cross-correlations in a rotated cross-modal subspace. The features are fused in a cross-modal subspace. The individual combination methods which are used to extract features for the cross-modal fusion, try to combine the synchrony between the features fused. Therefore, features like facial movements and voice that have close synchronization between them have generally benefited from such fusion techniques. Chetty et al. [10] explain a cross modal technique using Latent Semantic Analysis (LSA) and Canonical Correlation Analysis (CCA). The LSA technique tries to extract underlying semantic relationship between the face and the audio vectors while the CCA technique tries to find a linear mapping that maximizes the cross-correlation between the feature sets combined.

2.4.2.10 3-D Multimodal Fusion

3-D multimodal fusion [10] technique requires 3D features. The 2D samples that are captured by the sensor are used to make a 3D model of that modality. Modalities such as face, fingerprint are prominently used in their 2D as well as 3D forms. This requires appropriate supplementary information to construct 3D features from many 2D features. These features include frontal and side views for 3D face generation, sweep sensors outputs in the form of strips from the front and side views of a fingerprint. Various internal anatomical relationships between the modalities make it easy to find external and visible relationship between them from the sensor outputs. These include head movement during pronunciation of a particular letter or word. This fusion technique can also help the system to perform liveliness checks by capturing and checking for such movements and hence provide protection against spoof attacks, either manual or computer generated.

2.4.2.11 Canonical Correlation Analysis (CCA), and Kernel Canonical Correlation Analysis (KCCA)

In these methods, association between two feature vectors are measured. CCA is a classical multivariate method dealt with linear dependencies of the feature vectors and KCCA is a method that generalizes the classical linear CCA to nonlinear setting. These methods are described in the following.

Canonical Correlation Analysis (CCA): In this approach [57, 56], only two feature vectors are considered for fusion. A correlation function is required to be defined between these two feature vectors. CCA finds a pair of linear transformations one for each of the vectors to maximize the correlation coefficient between extracted features. Then CCA finds additional pairs of variables that are maximally correlated subject to the constraint that they are uncorrelated with previous variables. This process is repeated until all correlation features of two vectors are extracted. Based on this idea, the canonical correlation features of two vectors are extracted. These features are used as discriminant information between the groups of feature vectors. CCA finds a pair of a linear transformation between the two vectors such that it maximizes the correlation coefficient between them. This simplicity of having linear transformation leads to the disadvantage of this method not being able to identify all or most of the useful descriptors possible. Also, the CCA suffers from small sample size problem.

Kernel Canonical Correlation Analysis (KCCA): Kernels are functions or methods of mapping data into higher dimensional feature spaces. The KCCA [57, 56] projects the features from lower dimensions into higher dimensional Hilbert spaces [43]. This higher dimensional feature maximized the correlation coefficient between two vectors. It is used to extract the nonlinear canonical correlation features associated with the modalities.

2.4.2.12 Simple and Weighted Average

Simple averaging [52] of output is also called as Basic Ensemble Method (BEM). It is used when the individual classifiers produce a continuous output. It is assumed that classifiers are mutually independent and that there is no bias toward any of the classifiers means the accuracy and the efficiency of each classifier are same. Therefore, they all have the same weights. The final score is calculated in Eq. (2.26).

(2.26)

Weighted averaging [51] of output is called as Generalized Ensemble Method (GEM). The assumption in BEM about the mutual independence of classifiers and their unbiasness is not a practical one. It is very natural to have classifiers having the different accuracy and thus should carry the different weights. The final score is given by Eq. (2.27)

(2.27)

where

2.4.2.13 Optimal Weighting Method (OWM)

OWM [52, 51] is an effective way of combining the classifiers linearly. But, there might exist nonlinear effects of classifier combination and interactions among the classifiers. Ignoring these interactions may result in inaccurate results.

2.4.2.14 Likelihood Ratio-Based Biometric Score Fusion

This method of fusion [33, 32] uses the likelihood ratio test [33, 41]. It is a form of density-based score level fusion. It requires the explicit estimation of genuine and imposter score densities [41]. Accuracy of the density-based fusion technique depends upon the accuracy with which the densities are calculated. The likelihood ratio test is as shown in the following.

According to the Neyman-Pearson theorem [33]

(2.28)

(2.29)

where,

X = [X₁, X₂, . . . , X_M] | : | match sores of M different matchers.

According to the Neyman-Pearson theorem, theorem, the optimal test to determine whether the user is a genuine user or an imposter is the likelihood ratio test given by Eq. (2.29), at a given FAR, α. The threshold η is decided such that the GAR is maximized. When the underlying densities are known, the rule guarantees that there is no other rule with a higher GAR than this. The densities ƒ_gen(x) and ƒ_imp(x) are calculated from the training set.

2.4.2.15 Borda Count Method

Borda count method [30, 41] is used in rank level fusion [30]. In rank level fusion, each user is assigned a rank by all the matchers based on their relative similarity with the input query sample. This is useful in the identification scenario rather than in verification scenario. Ranks give a relative ordering among the users, starting from the user with the highest rank to the one with the lowest rank. Unlike match scores, ranks do not give any details about the measure of similarity or dissimilarity between the input sample and the one stored in the database. But the ranking gives more details than just a match or non-match decision. In the borda count method, the ranks assigned to the users by biometric matchers are summed up to form a statistic s_k for user k (see Eq. (2.30)).

(2.30)

where, r_j,k is the rank assigned to the k^th user by the j^th matcher for j = 1, . . . , R and k = 1, . . . , M .

The ranks are revised based on this statistic s_k for each user. The user with the least value for s_k gets the highest rank. Borda count assumes that all the matchers have equal accuracies, this cannot always be true. Also a simple summation of ranks will put users with average ranks on the same levels with those that have scored the best rank for one matcher but worst for the other (e.g. a user with a rank of 3 from two matchers will have a final statistic, s_k as 6). A user with rank of 1 for the first matcher and 5 for the other matcher will also have the same statistic of 6. Such a high discrepancy in a user’s ranks provided by the different matchers clearly indicates the difference in accuracies of those matchers, something that is not considered by the Borda count method.

2.4.2.16 Logistic Regression Method

Logistic Regression [41] is a generalization of the Borda count method [30]. A weighted sum of ranks assigned to a user is performed for each user to form a statistic s_k (see Eq. (2.31)).

(2.31)

where r_j,k is the rank assigned to the k^th user by the the j^th matcher and w_j is the weight assigned to j^th matcher for j = 1, . . . , R and k = 1, . . . , M .

The weights are determined in a training phase by logistic regression method [48]. A matcher with higher accuracy is assigned a higher weight. This overcomes the drawback of the Borda count method related to the assumption of all matchers being of equal accuracy. However, Logistic regression method does require a training phase to calculate the weights accurately.

2.4.2.17 Kernel Fischer Discriminant Analysis (KFDA)

KFDA [17] is used to solve the problem of nonlinear decision boundary between genuine and imposter distributions. Conventional methods degrade the fusion accuracy when decision boundary is linear. It can control the security level (FAR, FRR) on a threshold as it does not make a unique decision boundary like SVM and other methods.

The KFDA works as follows. The input feature vectors are first projected into a very large, possibly infinite functional space via nonlinear mapping. Then Linear Discriminant Analysis (LDA) is applied there. The LDA projects 2- dimensional data onto a line. This makes it easier to discriminate it linearly and maximize the ratio of inter-class scatter to intra-class scatter.

2.4.2.18 Minimum Cost Bayesian Classifier

Minimum cost Bayesian classifier [2, 54] is used in decision level fusion. This works on the principle of minimization of the expected cost function called Bayes Risk. The mathematical formulation of the problem is shown in Eq. (2.32).

(2.32)

where, i, j ∈ {0, 1}, C_ij is the cost of deciding a = i when w = j is present and w = 10 denotes either the presence or absence of the claimed identity. Considering these four conditions of a and w, Eq. (2.32) becomes Eq. (2.33).

(2.33)

The posteriori authentication probability is given in Eq. (2.34) using Bayes’ theorem.

(2.34)

In Eq. (2.34), f(x|w) is a likelihood function. The joint density of local authentication probabilities is given in Eq. (2.35).

(2.35)

If P (w = 1) is replaced by g then Eq. (2.35) becomes Eq. (2.36).

(2.36)

Therefore,

(2.37)

For n sensors the total likelihood function is given in Eq. (2.38).

(2.38)

Where f_i(x_i|w) is a likelihood function for sensor i. The sensors are assumed to be independent and thus the total likelihood function is a product of likelihood functions of all the sensors. Value of ’a’ is decided with the help of standard 0-1 cost function.

(2.39)

The most important part in all these calculations is the accurate modeling of the likelihood function. The choice of a proper function is pivotal in deciding the quality of the probability fusion.

The experimental comparison of many fusion techniques along with Minimum risk Bayesian classifier is made in [2] and it shows an FAR of [0.63, 1.5]% and an FRR of [0.0, 1.75]%. This is amongst the best classifier among the classifiers studied in [2]. These results are well within acceptable limits and as it can be seen an FRR of 0% has been achieved.

2.4.2.19 Decision Tree

A Decision tree [2, 27] consists of nodes and branches. The nodes are tests that are performed on a particular attribute of the data and the leaf corresponds to a particular label. The path from the root node to the leaf node then consists of a series of tests that classify the input data into a particular class. Alternatively, a decision tree gives a graphical representation of the classification process. They recursively partition the data points in an axis-parallel manner. At each node, the test splits the input on some discriminating feature and thus provides a natural way of feature selection.