where X F,ij, x1,ij and x2,ij denote the low-pass subband coefficients of the fused image and two source images respectively.
Step 5: High-pass subband coefficients are fused using the maximum variance rule, i.e., select the coefficient of the subband that has the maximum variance or absolute value maximum selection (AVMS) fusion rule or Laplacian mixture model (LMM) – based fusion rule has been reported by Unni V. S. [19].
Step 6: Fused images are reconstructed using the selected fused coefficients via inverse wavelet transform.
6.2.5Compressive sensing
The Nyquist-Shannon sampling theorem states that the reconstructed signal rate is higher than twice the maximum frequency. The sampling rate of some signals becomes very high due to their high bandwidth, for example, due to the acquisition of images. Therefore, the need for CS comes into the picture, where signals are compressed and represented in a small number of coefficients. Islam et al. [20] discussed the acquisition of information using CS without sampling input signals. If a signal is a “sparse sample”, thenthe acquired reconstructed signal can be accurate for all calculations. The sparsity of the signal is defined on a similar basis, such as DCT or wavelet transform. Incoherence between signals and measurements plays an important role where the information developed is widely spread out. CS provides an accurate measurement for reconstructed compressive sensed output in a non-adaptivemanner [19]. In CS first we generate a sparse representation of the signal. Then we generate some measurements of dimension M, using a measurement matrix of dimension M × N. Finally, the reconstruction is carried out from these measurement.
Suppose we have f = {f1, f2 . . . fN} of N real-value samples of a signal, which can represented by the transform coefficients x. That is:
where ψ = {ψ1, ψ2 . . . ψN} is an N × N transform basis matrix, which determines the domain where the signal is sparse, and x = {x1, x2 . . . xN} is an N dimensional coefficient vector with xi = ⟨x, ψi⟩. We assume that x is K-sparse, meaning that there are only significant elements in x with K ≪ N.
Suppose a general linear measurement process computes inner products M < N between f and a collection of vectors ϕj giving yj = f , ϕj > i = 1. . . N; j = 1. . . M.
If ϕ denotes an M × N matrix with ϕj as row vectors, then the measurements y = (y1, y2. . .yn) are given by
where y is an M-dimensional observation vector and ϕ is an M × N random measurement matrix and A = ϕψ is called the measurement matrix. We can see that CS reduces the signal of N dimension to an observation signal of M dimension, so a dimension reduction takes place for the accurate reconstruction of the signal, and the sensing matrix ϕ should satisfy the restricted isometric property (RIP) explained by R. G. Baraniuk et al. [21, 22]. In other words, the K-RIP ensures that all submatrices of ϕ of size M × K are close to isometry, and therefore the distance (and information) preserving. Random matrices are an example of measurement matrix whose entries are i.i.d (identically distributed) Bernoulli and Gaussian matrices. The signal reconstruction problem uses y to reconstruct the N-length signal, x that is K-sparse, given ϕ and ψ since M < N, it is an ill condition problem and there are infinite x' satisfying Ax' = y. To solve ill-conditioned problems of this kind, a conventional approach is used to minimize the lp norm. Define the lp norm vector x as for 1 ≤ p < ∞and lxp = max |xi|. For example, consider the l2 norm to solve this ill condition. The optimization problem is given by
Many algorithms have been generated to solve these CS problems. They include linear programming (LP) techniques, greedy algorithms, gradient-based algorithms and iterated shrinkage algorithms. Although l1 and l2 norm has strong possibilities for exact recovery, it has disadvantages in terms of computational cost and implementation complexity. Orthogonal matching pursuit (OMP) algorithms are developed to address these problems.
6.2.5.1Block compression sensing
In block CS, random matrices are derived from the sampling of an image block by block, suggested by M. Y. Baig et al. [18]. Consider an Ic × Ir image with N = IcIr pixels in total. The image is split into n small blocks with dimension B × B each and sampled with the same matrix in block CS. Let xi represent the vectorized signal of the i-th block through raster scanning and i = 1. . . n, n = N/B2. Construct a B2 × B2 i.i.d. (identically distributed) Gaussian matrix, and by obtaining orthogonal matrix Θ by orthonormalizing the Gaussian matrix, M B(≪ B2) rows are picked randomly from Θ to produce M B × B2 measurement matrix ϕB. The corresponding yi is expressed in Equation (6.31):
To get M (n × M B), compress the sensed output y of an entire data. The output of the previous stage is given as input block by block in cognition through the matrix ϕ, given in Equation (6.32):
Block CS is memory efficient, since we just need to store an M B × B2 Gaussian matrix, rather than a full M × N one, and block CS also has several merits. The first merit is that the encoder does not consume more time for computation since linear projection sends each block for all pixels from the image. Second, the calculation is feasible due to the smaller value of ϕB by an initial approximation method.
6.2.5.2SPL reconstruction algorithm
The reconstruction process is done using smooth projected Landweber (SPL) iteration. The wiener filtering was constructed using a Landweber approach for the purpose of removing blocking artefacts and noise. The approximation to the image at iteration K + 1, x(k + 1), is produced from x(k) as
for each block i,
we terminate the iteration when |Dk+1 − Dk| < ϵ, where
6.2.5.3Thresholding
Projected Landweber (PL) algorithm uses hard thresholding for denoising. For the purpose of setting a proper Γ for hard thresholding, the universal threshold method is used:
The parameter σ(k) is estimated using a robust median estimator,
6.3An integrated approach to image fusion
The main objective of fusion techniques is to identify the most essential elements of an image and to retain them in the original image. This transformed into smaller subbands without any loss of information. The demands of an effective image fusion technique lies in the quality of the image, computational complexity and computational time. Based on different applications, there are varying demands on computation time, complexity, environmental constraints and information quality. The main requirement of a fusion method is that it must be able to identify essentially the largest elements in input images and transmit them without missing any elements in the fusion image. The disadvantage of the spatial domain approach is the production of spatial distortion in the image fusion. The required results obtained with spectral distortion are worst, while predicting the image from the database. The problem of spatial distortion can be overcome by transforming domain approaches to image fusion. The image details and directional coefficients are taken into account when multiscale transform-based fusion is performed. The relative coefficients of an image are fused by different sets of rules for relevant coefficients. The conventional approach overcomes the cost values and the problem of higher computational complexity. A multiscale transform-based fusion approach takes image details and direction coefficients into account. It can accomplish high fused image quality. According to the properties of different coefficients, different effective fusion rules are required to fuse the relevant coefficients. For the purpose of avoiding low computational complexity and storage or cost reduction problems, a conventional CS-based fusion approach is required. Based on the aforementioned facts an integrated image fusion method is proposed that separately deals with the low-pass- and high-pass-frequency subbands and reduces the storage and computational problem via a CS framework.
6.3.1Image fusion using PCNN and PCA
In the proposed method, images are initially decomposed into low-pass subbands and high-pass subbands using wavelet transform. DWTs are used for image decomposition. The high-frequency subbands contain the coefficients of an image, which have both real and imaginary terms corresponding to sharp depth changes and salient features in the input data. The energy of the high-frequency coefficients of a clear image is much larger than that of a blurred one. Therefore, the variance of the focused image is greater than that of a blurred image. We will be using this variance property for fusing high-frequency subbands. We will be using a PCA approach for the fusion process. For low-pass-frequency subband fusionwe will use a PCNN model-based fusion approach as it incorporates the human visual perception of images and utilizes neighbourhood information. This PCNN-based image fusion is done based on activity-level measurement.
6.3.2Algorithm
| Step 1: | Consider the input images as IA and IB. |
| Step 2: | Decompose the input images into high-pass and low-pass subbands as ILA, ILB and IHA, IHB respectively using wavelet transform. For example, use DWT. |
| Step 3: | Fuse the low-pass subband using a PCNN model-based fusion rule as mentioned earlier and obtain a fused low-pass subband image as ILF. |
| Step 4: | Fuse the high-pass subbands using a PCA-based fused algorithm as mentioned earlier, and obtain a fused high-pass subband as IHF. |
| Step 5: | Use inverse wavelet transform to get back the fused image as IF' . |
6.3.3Image fusion using PCNN and PCA in compressive sensing framework
In this proposed method, high-pass subband coefficients are fused using a PCA approach and low-pass subband coefficients are fused using PCNN. A CS framework is incorporated to reduce the processing rate and the quality of the fused image.
6.3.4Algorithm
| Step 1: | Decompose source images into high-pass and low-pass subbands using wavelet transform. |
| Step 2: | Generate measurements by projecting the high-pass and low-pass subbands into a projection matrix. Use a Gaussian measurement matrix for better efficiency. |
| Step 3: | Recover the wavelet coefficients via a projected Landweber iteration algorithm. |
| Step 4: | Fuse the low-pass subband using PCNN. |
| Step 5: | Fuse the high-pass subband using a PCA-based fusion algorithm. |
| Step 6: | Use inverse wavelet transform to get back the fused image. |
6.4Comparative results analysis
True information and natural colour preservation are essential for any kind of image fusion method. Fusion performance is mainly evaluated using subjective and objective performance matrices. However, the results from the target data can be accurate and expensive in computational complexity and time. Hence, here the performance of different multisensor image fusion methods is evaluated as a function of their generated objective performance matrices. The following list are objective fusion measures:
- Extract important feature points from an input image.
- Measure the ability of the fusion process to transmit the information into an output image as accurately as possible.
From the resultant values the different performance metrics are validated from the fused images. Here all objective matrices are bounded between 0 and 1. For example, consider an objective matrix 0 ≤ QAB/F (m, n) ≤ 1 where a value of 0 equals the complete loss of information, at location (m, n), as transferred from A to F or B to F, and a value of 1 corresponds to an “ideal fusion” with no loss of information. Hence for all objective performance matrices a value of 0 denotes a complete loss of information and a value of 1 means perfect natural colour fusion.
6.4.1Objective evaluation methods
Preservation of information plays an important role in the natural appearance of fused images, and measures of fusion success must take this into account either within their total fusion efficiency or in explicit natural measures. A well-known gradient-based objective fusion evaluation framework was adopted that evaluates the preservation of input gradients in fused images to focus the investigation into metric configurations and representation spaces for comparing information between input and fused images.
In this work, the initial focus is on matrices that evaluate the performance of the fusion method that combines two input multifocus images and their resultant fused image. One of the earliest fusion-specific objective metrics was based on the idea of estimating gradient preservation. Another important body of work focuses on using mutual information and information theory measures of statistical dependence between two signals that are well suited to comparing multimodal data in general, and image quality metrics such as the structural similarity index have also been successfully applied to the evaluation of fusion, where they measure structural similarities between localized sections of each input and the fused image.
6.4.1.1Sobel operator
The Sobel operator, typically known as a Sobel filter, is utilized in image processing and computer vision because the edges of an image are detected by its kernel function. This approximates the calculation of the gradient of an image intensity function with a discrete differentiation operator. At each point in the image, the result of the Sobel operator is based on convolving the image using a separable, small and integer valued filter in horizontal and vertical direction and is therefore relatively inexpensive in terms of computation. On the other hand, the gradient approximation produced is relatively crude, in particular for high-frequency variations in the image. The operator can use two 3 × 3 kernels, which are convolved with the custom image to calculate approximations of the derivatives – one for vertical changes and the other one for horizontal. Consider A as the input image, and Gx and Gy are two other images that at each factor incorporate the horizontal and vertical derivative approximations. The computations are as follows:
The gradient strength is given by
The gradient orientation is given by
6.4.1.2Sigmoid function
A sigmoid function is a continuous non-linear activation function. The word sigmoid derives from the fact that the function is S-shaped. Statisticians use it in reference to a logistic function, utilizing x as an input, f(x) as output with respect to t as a distinct aspect term. The sigmoid function can be given as
The characteristic of the sigmoid function has a smooth continuous function whose output value ranges between −1 to 1.
6.4.1.3Gradient preservation fusion evaluation framework
A gradient information preservation–based fusion evaluation framework, also called a QAB/F metric, is associated with gradient information that is transferred from source input images to fused images. The efficiency of such a fusion algorithm depends upon the amount of gradient information transferred. In this work we have used the well-known Sobel edge operator for edge detection. Specifically, assuming two input images A and B and the resulting fused image F, a Sobel edge operator is applied to produce the strength g and orientation α (α ∈ |0π|) information for each input and fused image pixel (m, n).
Using the foregoing parameters, relative strength and orientation change factors Δg and Δα between each fused image and input are derived, for example:
These factors are the basis of the gradient information preservation measure QAF obtained by orientation change elements and sigmoidal mapping of quality. This quantity represents the relative loss of information in the resultant fused image and its coefficients kg, σg and kα, σα determining the exact shape of the sigmoidal mapping:
where kg = −15, σg = 0.5, Γg = 1.0006 and kα = −22, σα = 0.8, Γα = 1.0123.
F represents the input A at (m, n). The gradient function is expressed as
The total fusion performance QAB/F is calculated by preserving input images QAF and QBF sum of gradient element weights where the weight elements wA and wB represent perceptual significance of every input image pixel. The range is 0 ≤ QAB/F ≤ 1, where QAB/F = 0 implies an entire loss of input data and QAB/F = 1 shows a perfect combination with no loss of input data. In the simplest form, the perceptual weights wA and wB take the values of the corresponding gradient strength parameters gA and gB:
Effectively QAB/F measures the preservation of the most important input information at each location across the scene in the fused image.
6.4.1.4Entropy
Entropy can be described as a measure of the amount of disorder in a system. It is conveyed as the spread of states that a system can adopt. A low-entropy framework possesses a small number of such states, while a high-entropy framework involves an expansive number of states.
On account of an image, these states compare to the grey-level value, which corresponds to individual pixels. For instance, in an 8-bit pixel of an image there are 256 possible states. If all states are similarly involved, as they are on account of an image that has been perfectly histogram equalized, the spread of states is a maximum, just like the entropy of an image. Then again, if the image has been thresholded, so that two exclusive states are involved, the entropy is low. In the event that the majority of pixels have the same value, the entropy of the image is zero.
With zero entropy,
where N is the number of grey levels in the image and p(i) is the normalized probability of occurrence of each grey level.
6.4.1.5Standard deviation
In probability and statistics analysis, the standard deviation (SD) is represented by the symbol σ and is a measure that is used for evaluating the amount of variation or scattering of an arrangement of information qualities. An SD near 0 shows that the data points tend to be very close to the mean of the set, while a high SD indicates that the data points are spread out over a wider range of values.