11.1.1.1 Early fusion – fusion at the observation level

Fusion can be achieved at the observation level, i.e., through the direct joint analysis of the pixel values (with or without calibration procedures). For that purpose, pan-sharpening is a well-known technique that integrates the geometric details of a high resolution panchromatic image and the color information of a low spatial resolution multispectral (or hyperspectral) image to produce a high spatial resolution multispectral (or hyperspectral) image. Pan-sharpening methods usually use the panchromatic image to replace the high frequency part in the low resolution image [11]. Other fusion algorithms have been proposed to merge multispectral (or hyperspectral) and panchromatic (but also multispectral and hyperspectral) images to combine complementary characteristics in terms of spatial and spectral resolutions [12–14]). A review of such methods can be found in [14]. Eventually, super-resolution is another approach relying on early fusion of several sensors [15].

11.1.1.2 Intermediate fusion – fusion at the attribute/feature level

Data sources can be merged at the feature level. Features (spectral indices, texture-based, etc.) are computed for each source separately or for both of them and fed into the same classifier through a unique feature set [16]. Examples of remote-sensing pipelines involving fusion at the attribute level can be found in [17–22]. For instance, [18] proposed a conditional random field (CRF) model for building detection using InSAR and orthoimage features. Reference [20] merged Lidar and optical aerial image features for forest stand extraction: the proposed approach involves several steps (segmentation, classification, and regularization) and fusion is performed at each of them. Improvement is noticed for each step. More recently, deep convolutional neural networks were used to perform data fusion at the attribute level [23]. Reference [22] applied for instance deep forests to Lidar and hyperspectral imagery features. A detailed review can be found in [24]. Several datasets and challenges have been released in the last decades, under the aegis of the IEEE GRSS society [25–27].

11.1.1.3 Late fusion – fusion at the decision level

Late decision fusion happens after the classification process: the outputs of multiple independent classifiers are combined in order to provide a more reliable decision. Such classification results can be either label maps or class membership probability maps. Various late decision fusion methods have been proposed. Most of them can be divided into different categories: consensus rules (majority voting), probabilistic approaches (Bayesian fusion), credibilist or evidential ones, and possibilist ones.

The probabilistic, evidential, and credibilist decision fusion approaches are generic and can be applied to different fusion problems. They only require class “membership” measures (probabilities or belief masses depending on the approach) for each source and for each class, or at least a confidence measure for each source.

Possibilist methods use fuzzy logic-based fusion rules [28–30]. They require one to define weights [31] in order to better deal with the uncertainty of the different sources. Such generic approaches have been applied to remote-sensing data [32].

Evidential approaches are a generalization of probabilistic ones. They include the well-known Dempster–Shafer fusion rule [33]. In remote sensing, this rule has often been used to merge classifications or alarm detection results. For instance, [34–36] applied the Dempster–Shafer rule to combine several different supervised building detectors based on different remote-sensing modalities (optical, Lidar, radar). This rule was also used by [37] for the fusion of different road obstacle detectors in the context of intelligent vehicle development. Reference [38] used the Dempster–Shafer rule for the fusion of urban footprints detected at different dates out of satellite archival images. References [39,40] applied this rule to detecting changes in an unsupervised classification context.

Another evidential fusion rule is the Yager rule. It was applied by [41] to combine different road obstacle detectors for vehicle navigation. Other evidential rules have been proposed more recently: the Dezert–Smarandache rule may be mentioned [42]. Other new rules have also been proposed by [43,44]: they extend Dempster–Shafer rules in order to achieve a better management of the conflict between sources. However efficient in some cases, Dempster–Shafer remains a theoretical complex framework that does not easily apply when dealing with heterogeneous and multiple data.

Another important issue consists in defining the input of the different fusion rules. Different situations can be taken into account depending on whether a global confidence measure is affected for each source, or, conversely, whether class posterior probabilities for a given source are directly available for each pixel (directly provided by the initial classifier). In the former situation, global confidence measures affected to each source are calculated from the confusion matrices (indeed, a confusion matrix provides the probability of an object labeled class A by one source to belong in fact to class B). Validation data are thus necessary to calculate these weights. Furthermore, for evidential methods and especially Dempster–Shafer methods, uncertainty classes (i.e. unions of original classes) must necessarily be defined. Hence, belief masses associated to these classes must be computed. If a global confidence is associated to a source, some solutions have been proposed. For instance, we have Appriou's method [45], or more recently the method presented in [46,47]. If a class membership measure is affected to each pixel for each source and for each class, it is also possible to derive belief masses for these uncertainty (union) classes. Finally, [38] propose an alternative way to integrate these two kinds of information.

Last but not least, the last category of late fusion approaches consists in supervised learning. They automatically learn from training examples the best way to merge input sources. Per-source posterior class probabilities are concatenated and considered as a feature vector. This feature vector is then provided as an input to a classifier which is trained to perform the best possible fusion. Thus, they can be considered as being at the interplay between late and intermediate fusion (classifiers previously applied to each source can then be considered as a kind of feature generators, then referred to as auto-context classification [48]). Such approaches have been used with different classifiers: random forest [49,50], Adaboost [50], support vector machines [51,52]. These supervised learning-based methods enable good results but require a sufficient amount of training data to model the classes and avoid over-fitting (especially for deep learning).

In addition, it must here be said that most fusion methods mentioned in this section generate measures to assess the conflict between two sources.

Late approaches have often been applied to remote-sensing fusion problems [53–56,32,51,57–61]. Fusion methods operating at the decision level can be applied in two situations, considering if they try to merge multiple classifiers applied to either the same data source, or to multiple sources. For instance, [53] combined neuronal and statistical maximum likelihood classifiers using several consensus theory rules (i.e., majority voting, complete agreement) to classify multispectral and hyperspectral images. References [57,58] merged posterior probabilities from maximum likelihood classifications of optical images with prior information about classes derived out of, respectively, digital terrain models or digital surface models, as well as information from existing land cover databases. Reference [55] investigated the fusion of multitemporal thematic mapper images, using decision fusion-based methods (i.e., joint likelihood and weighted majority fusion). A characterization of the spatial organization of SAR image elements is investigated by [54] merging the responses of multiple low-level detectors applied to the same image within a Dempster–Shafer scheme. Reference [32] investigated the use of fuzzy decision rules to combine the classification results of a conjugate gradient neural network and a fuzzy classifier over an IKONOS image. Reference [61] combined convolutional neural networks and random forest classifiers using a multiplication Bayesian scheme.

11.2.1.1 Fuzzy rules

The first tested fusion approach is based on fuzzy rules [30]. Fuzzy rule theory states that a fuzzy set A in a reference set of classes $\mathcal{L}$ is a set of ordered pairs:

$A=[(c,P_A^{(c)}(x)|c\in \mathcal{L})],$

where the membership probability of A in $\mathcal{P}$ is given by $P_A^{(c)}:\mathcal{L}\to [0,1]$. The measure of conflict ($1-K$) between two sources is given as [68]

$K=\underset{c\in \mathcal{L}}{\sup }\min (P_A^{(c)}(x),P_B^{(c)}(x)).$

In order to account for the fact that fuzzy sets with a strong fuzziness possibly hold unreliable information, each fuzzy set i is weighted according to a pointwise confidence measure $w_i$ [32]:

$w_i=\frac{{\sum }_{k=0,k\ne i}^n{H}_{\alpha QE}(P_k)}{(n-1){\sum }_{k=0}^n{H}_{\alpha QE}(P_k)}.$

n is the number of sources and $H_{\alpha QE}$ is a fuzziness measure called α-quadratic entropy (QE) [31]. Each fuzzy set i is weighted by the fuzziness degree of all other fuzzy sets (i.e., classifications). If the fuzziness degree of the other sets is high, the weight of a given source i will be high too. For all the following fusion rules, the fuzzy sets have been weighted as ${\tilde{P}}_A^{(c)}(x)=w_A\cdot P_A^{(c)}(x)$, ${\tilde{P}}_B^{(c)}(x)=w_B\cdot P_B^{(c)}(x)$, $P_A^{(c)}(x)$ and $P_B^{(c)}(x)$ being the original membership probabilities and $w_A$, $w_B$ their corresponding pointwise measures.

In further experiments, all fuzzy rules have been tested as input using the probabilities weighted by the pointwise measure.

The following fusion rules based on fuzzy logic were considered:

1. 1.  Minimum rule (Min) as the intersection of two fuzzy sets $P_A$ and $P_B$, given by the minimum of their membership probabilities (conjunctive behavior):

$\forall c\in \mathcal{L}{(P_A\cap P_B)}^{(c)}(x)=P_{fusion}^{(c)}(x)=\min ({P_A}^{(c)}(x),P_B^{(c)}(x)).$

(11.4)

2. 2.  Maximum rule (Max) as the union between the two fuzzy sets $P_A$ and $P_B$, given by the maximum of their membership probabilities (disjunctive behavior):

$\forall c\in \mathcal{L}{(P_A\cup P_B)}^{(c)}(x)=P_{fusion}^{(c)}(x)=\max ({P_A}^{(c)}(x),P_B^{(c)}(x)).$

(11.5)

2. 3.  Compromise operator:

$P_{fusion}^{(c)}(x)=\{\begin{array}{c}\begin{array}{cc}\hfill & \max (T_1,\min (T_2,(1-K)\left)\right)\hfill \\ \hfill & \text{if}(1-K)\ne 1,\hfill \end{array}\hfill \\ \begin{array}{cc}\hfill & \max (P_A^{(c)}(x),P_B^{(c)}(x))\hfill \\ \hfill & \text{if }(1-K)=1.\hfill \end{array}\hfill \end{array}$

(11.6)

1. $T_1=\frac{\min (P_A^{(c)}(x),P_B^{(c)}(x))}{K}$, $T_2=\max (P_A^{(c)}(x),P_B^{(c)}(x))$.
2. It can be noticed that the operator behavior is conjunctive when the conflict between A and B is low ($1-K\approx 0$), and disjunctive when the conflict is high ($1-K\approx 1$). When the conflict is partial, the operator behaves in a compromise way [68].
3. 4.  Compromise modified: Since a pure compromise fusion rule would favor $T_1$, [69] proposed to measure the intra-class conflict as $f_c=\mathrm{abs}(C_{\text{best1}}-C_{\text{best2}})$ with $C_{\text{best1}}={\mathrm{argmax}}_{c\in \mathcal{L}}{P}_S^{(c)}(x)$ and $C_{\text{best2}}={\mathrm{argmax}}_{c\in \mathcal{L}\setminus C_{\text{best1}}}{P}_S^{(c)}(x)$. They set a conflict threshold $t_c$ (e.g., $t_c=0.25$ for experiments in Sect. 11.3) to be used as follows:

2. 5.  Prioritized operators (referred to as Prior 1 for Eq. (11.7) and Prior 2 for Eq. (11.8)):

$P_{fusion}^{(c)}(x)=\max (P_A^{(c)}(x),\min (P_B^{(c)}(x),K)),$

(11.7)

$P_{fusion}^{(c)}(x)=\min (P_A^{(c)}(x),\max (P_B^{(c)}(x),(1-K)\left)\right).$

(11.8)

1. If the conflict between A and B is high (i.e., $K\approx 0$), only $P_A^{(c)}(x)$ is taken into account (prioritized) and $P_B^{(c)}(x)$ is considered as a specific piece of information. Thus, the fusion result depends on the order of the sources A and B.
2. 6.  An accuracy dependent (AD) operator [32], integrating both local and global confidence measurements:

$P_{fusion}^{(c)}(x)=\max (\min (w_i.P_i^{(c)}(x),f_i^{(c)}(x)),i\in [1,n]),$

(11.9)

1. where $f_i^{(c)}$ is the global confidence of source i regarding class c, $P_i$ is a class membership of source i, and $w_i$ is a normalization factor (see Eq. (11.3)). This operator ensures that only reliable sources are taken into consideration for each class, via the predefined coefficients $f_i^{(c)}$. The idea seems interesting. Nevertheless, the final result depends on the reliability of the classifier and also on the availability of ground truth data, which is mandatory to generate the $f_i^{(c)}$ term.

11.2.1.2 Bayesian combination and majority vote

A straightforward approach is to sum or multiply the input class membership probabilities as a Bayesian sum (∼ majority vote) or product [70]:

$P_{fusion\mathrm{\_}sum}^{(c)}(x)=P_A^{(c)}(x)+P_B^{(c)}(x),$

$P_{fusion\mathrm{\_}product}^{(c)}(x)=P_A^{(c)}(x)\times P_B^{(c)}(x).$

Those rules will be referred to as Bayesian sum and Bayesian product, respectively.

11.2.1.3 Margin-based rules

The aim of these rules is to take into account the confidence, measured by the classification margin, of each source. The classification margin is defined for each pixel x and each source s as the difference between the two highest class probabilities:

$margi{n}^{(s)}(x)=P_s^{(C_{\text{best1}})}(x)-P_s^{(C_{\text{best2}})}(x),$

with $C_{\text{best1}}^{(s)}(x)={\mathrm{argmax}}_{c\in \mathcal{L}}{P}_s^{(c)}(x)$ and $C_{\text{best2}}^{(s)}(x)={\mathrm{argmax}}_{c\in \mathcal{L}\setminus \{C_{\text{best1}}^{(s)}(x)\}}{P_s}^{(c)}(x)$.

Fusion can then be carried out preferring for each pixel the most confident source, i.e. the one with the highest margin. This fusion rule (referred to as margin-Max) selects, for each pixel, the source for which the margin between the two highest probabilities is the highest, with sources $\mathcal{S}=A,B$ and the classes $\mathcal{L}={\{c_i\}}_{1⩽i⩽n}$:

$\forall x,\forall c\in \mathcal{L}{P}_{fusion}^{(c)}(x)=P_{S_{best}}^{(c)}(x),$

where $S_{best}={\mathrm{argmax}}_{\mathcal{S}\in \mathcal{C}}margi{n}^{(s)}(x)$.

The classifier confidence information provided by the margin can also be used to weight the class probabilities of each source in the Bayesian sum and product (respectively, margin Bayesian sum weighted, margin Bayesian product weighted):

$P_{fusion\mathrm{\_}sum}^{(c)}(x)=\frac{P_A^{(c)}(x)\cdot margi{n}^{(A)}(x)+P^{B(c)}(x)\cdot margi{n}_B^{(c)}(x)}{margi{n}^{(A)}(x)+margi{n}^{(B)}(x)},$

$P_{fusion\mathrm{\_}product}^{(c)}(x)={(P_A^{(c)}(x))}^{\frac{margi{n}^{(A)}(x)}{margi{n}^{(A)}(x)+margi{n}^{(B)}(x)}}\times {(P_B^{(c)}(x))}^{\frac{margi{n}^{(B)}(x)}{margi{n}^{(A)}(x)+margi{n}^{(B)}(x)}}.$

11.2.1.4 Dempster–Shafer evidence theory

According to the Dempster–Shafer (DS) formalism, an information from a source s for a class c can be given as a mass function $m_c|m_c\in [0,1]$ [33]. Dempster–Shafer's evidence theory rule assumes simple classes $c\in \mathcal{L}$ as well as composed classes, which were hence limited to two simple classes at most [69].

Masses associated to each simple class are directly the class membership probabilities:

$m_s^{(c)}(x)=P_s^{(x)}.$

For mixed classes, $\forall c_1,c_2\in \mathcal{L},\forall$ pixel x and $\forall s\in \mathcal{S}$, two versions were tested, denoted $DS{V}_1$ and $DS{V}_2$, respectively:

$\begin{array}{cc}\hfill m_s^{(c_1\cup c_2)}(x)& =(P_s^{(c_1)}(x)+P_s^{(c_2)}(x))\hfill \\ \hfill & \times (1-\max (P_s^{(c_1)}(x),P_s^{(c_2)}(x)\left)\right)\hfill \\ \hfill & +\min (P_s^{(c_1)}(x),P_s^{(c_2)}),\hfill \end{array}$

$\begin{array}{cc}\hfill m_s^{(c_1\cup c_2)}(x)& =\frac{1}{2}(P_s^{(c_1)}(x)+P_s^{(c_2)}(x))\hfill \\ \hfill & \times (1-\max (P_s^{(c_1)}(x),P_s^{(c_2)}(x)\left)\right)\hfill \\ \hfill & +\min (P_s^{(c_1)}(x),P_s^{(c_2)}).\hfill \end{array}$

This leads to a mass $m_s^{(c_1\cup c_2)}(x)\in [0,1]$, being 1 if $P_s^{(c_1)}=0$, $P_s^{(c_2)}=1$ or $P_s^{(c_2)}=0$, $P_s^{(c_1)}=1$. In both versions, all masses are normalized such that ${\sum }_{c\in \mathcal{L}}^{(s)}{m}_c^{(s)}(x)=1$.

The fusion rule is based on the following conflict measure between two sources A and B:

$K(x)=\sum _{\begin{array}{c}\hfill c,d\in {\mathcal{L}}^{\prime }\hfill \\ \hfill c\cap d\ne \varnothing \hfill \end{array}}^{(s)}=m_A^{(c)}(x)m_B^{(c)}(x),$

$c,d\in \mathcal{L}$ being mixed classes with $c\cap d=\varnothing$.

The fusion is performed by

$m_{fusion}^{(c)}(x)=\frac{1}{1-K(x)}\sum _{\begin{array}{c}\hfill c_1,c_2\in {\mathcal{L}}^{\prime }\hfill \\ \hfill c_1\cap c_2=c\hfill \end{array}}^{(s)}{m}_A^{(c)}(x)m_B^{(c)}(x).$

11.2.1.5 Supervised fusion rules: learning based approaches

In addition to the previous standard fusion rules, learning-based supervised methods were also tested [51,52,23]. Such methods consist in learning how to best merge both sources (based on a ground truth). A classifier is trained to label feature vectors corresponding to the concatenation of class membership measures from both sources. Thus, such a strategy can be considered at the interplay between late and intermediate fusion (classifiers applied independently to each source can then be considered as a kind of feature generators). It is similar to auto-context approaches. A drawback stems from the fact that they require a significant amount of reference data.

In next experiments, two classifiers were considered for supervised fusion: random forests (RFs) [71] and support vector machines (SVMs) [72] with a linear or a radial basis function (rbf) kernel.

11.2.2.1 Model formulation(s)

The problem is formulated in terms of an energy E that has to be minimized over the whole image I in order to retrieve a labeling C of the entire image which corresponds to a minimum of E. As commonly adopted in the literature, E consists of two terms, one related to the data fidelity, and one to prior spatial knowledge, setting constraints on class transitions between the different pairs of neighboring pixels N. Several options were considered for the different energy terms. We have

$E(P_{fusion},C_{fusion},C)=\sum _{x\in I}{E}_{data}(C(x))+\lambda \sum _{\begin{array}{c}\hfill x,y\in N\hfill \\ \hfill x\ne y\hfill \end{array}}{E}_{reg}(C(x),C(y)),$

where

$E_{data}(C(x))=f\left(P_{fusion}\right(C(x)\left)\right),E_{reg}(C(x)=C(y))=g\left(P_{fusion}\right(C(x)),P_{fusion}(C(y)),C_{fusion}(x),C_{fusion}(y),I(x),I(y)),E_{reg}(C(x)\ne C(y))=h\left(P_{fusion}\right(C(x)),P_{fusion}(C(y)),C_{fusion}(x),C_{fusion}(y),I(x),I(y)).$

The final label map corresponds to the configuration C which minimizes E over I.

The data term is a fit-to-data attachment term. It relies on the probability distribution $P_{fusion}$, defined by a function $f(.)$. The function f ensures that if the probability for a pixel x to belong to class $C(x)$ is close to 1, $E_{data}$ will be small. It will not impact the total energy E. Conversely, if the probability for a pixel x to belong to class $C(x)$ is low, $E_{data}$ will be near its maximum and will penalize such a configuration. The following options were tested:

$\mathit{\text{Option 1}}\text{: }f(t)=-\log (t),$

$\mathit{\text{Option 2}}\text{: }f(t)=1-t.$

Earlier experiments [20] verified that Option 2 tends to smooth the classification map more than Option 2. Thus, the data term will be selected among these options depending on the targeted application and on the input data. Option 1 will be selected to keep small regions as long as they are relevant according to class probabilities, while Option 2 will be used to obtain smoother maps with wider flat areas [20].

The regularization term $E_{reg}$ defines the interactions between a pixel x and its eight neighbors, setting a constraint to smooth the initial classification map. Several options were also considered. They were all based on an enhanced Potts model [64] in order to guarantee smoother label changes.

The term $V(x,y,\epsilon )$ for image contrast is the same for all regularization term formulations. It is based on [66,74] and defined by

$V(x,y)=\frac{1}{dim}\sum _{i\in [0,dim]}{V}_i{(x,y)}^{\epsilon },$

$V_i(x,y)=\exp \left(\frac{-{(I_i(x)-I_i(y))}^2}{2\cdot MeanGrad(I)}\right)$ and $MeanGrad(I)=\frac{1}{Card(N)}{\sum }_{x,y\in N;x\ne y}{(I_i(x)-I_i(y))}^2$. dim is the dimension of the image I, $I_i(x)$ is the intensity for x in band i in the multispectral image I; $\epsilon \in [0,\infty [$.

11.2.2.2 Optimization

Once the energy E has been defined, it has to be minimized in order to get a labeling configuration C (i.e., a classification map) of the entire image, which corresponds to a minimum of the energy E. This model can be expressed as a graphical model and solved as a min-cut problem [73,75].

The graph-cut algorithm employed here is the quadratic pseudo-Boolean optimization (QPBO)¹ [75,76]. QPBO is a classical graph-cut method that builds a probabilistic graph where each pixel is a node. The minimization is computed by finding the minimal cut. Contrary to several standard graph-cut methods for which the pairwise term $E_{regul}$ can only consider the two configurations $(C(x)=C(y))$ and $(C(x)\ne C(y))$, QPBO enables one to integrate more constraints, defining a pairwise term $E_{regul}$ differently according to these four configurations $(C(x)=0,C(y)=0)$, $(C(x)=0,C(y)=1)$, $(C(x)=1,C(y)=0)$ and $(C(x)=1,C(y)=1)$.

QPBO performs binary classification. Extension to the multiclass problem is performed using an α-expansion routine [73]. Each label α is visited in turn and a binary labeling is solved between that label and all others, thus flipping the labels of some pixels to α. These expansion steps are iterated until convergence and at the end the algorithm returns a labeling C of the entire image which corresponds to a minimum of the energy E.

11.2.2.3 Parameter tuning

The regularization term E (Eq. (11.21)) is controlled by up to four parameters, depending on the retained formulation for $E_{reg}$: λ, γ, β and ε. Each of them is attached to a particular sub-term of E.

A simple Potts model can be obtained using the following parameterizations:

A greedy way for parameter optimization was presented in [66]. The value ${\lambda }_{\text{opt}}$ maximizing the classification result for a Potts model is assumed to be the same value as the one maximizing the fusion model classification result. Hence, ${\lambda }_{\text{opt}}$ is first computed using a simple Potts model. Then, with ${\lambda }_{\text{opt}}$ and $\gamma =0$, ${\beta }_{\text{opt}}$ is found. Similarly, using ${\lambda }_{\text{opt}}$ and $\gamma =1$, the value ${ϵ}_{\text{opt}}$ is computed. Lastly, the trade-off parameter ${\gamma }_{\text{opt}}$ maximizing results of the model is chosen in the $[0,1]$ interval. The process is iterated, optimizing parameters in the same order at each iteration according to the current parameter set.

Such a strategy can be performed by quantitative cross-validation when sufficient reference data is available. Otherwise, it can be empirically performed by qualitative (visual) evaluation of the results. The set of parameters yielding the nicest and smoothest possible result while following the real object contours can thus be identified. This solution is relevant when regularization also targets improving the visual quality and the interpretability of classification results in operational contexts.

In practice, a set of parameters defined for a classification problem and a decision fusion rule is stable enough to be used in other, similar situations.

11.3.4.1 Source comparison

The MS image is characterized by a high spatial resolution and few bands, while the HS one has a low spatial resolution and a hundred(s) of bands. As expected, the SVM classifier applied over these images led to:

• •  sharp object delineation in the MS image due to its good spatial resolution, but also a lot of artifacts (see Figs. 11.3, 11.4, and 11.5);

• •  a good discrimination of the different classes in the HS image. However, blurry object delineation is also noticed, due to its low spatial resolution (see Figs. 11.3, 11.4, and 11.5).

The corresponding classification accuracies are listed in Table 11.2: better results are retrieved using the HS image.

11.3.4.2 Decision fusion classification

10 different decision fusion rules were first tested and compared over the three datasets. The quantitative results provided in Table 11.1 lead us to consider the compromise, Bayesian product, margin-Max and Dempster–Shafer rules to be the most efficient rules. The comparison must also take into consideration the visual inspection of the results, as ground truth data remains very limited on these datasets. For Pavia University, four of the best accuracies were reached for Min, compromise, Bayesian product, and Dempster–Shafer rules. In practice, the Min/Compromise rules give the most satisfactory rendering, especially regarding the Self-Blocking Bricks class which is a conflicting class (see Fig. 11.3, magenta color class). The two other rules seem to overestimate this class and to a greater extent consider the HS classification map in the fusion process. This explains their higher accuracy (Table 11.1). The Min rule acts in a cautious way when taking the best of the lowest memberships, while the compromise rule acts depending on the degree of conflict between sources. The Bayesian product rule is a good and simple trade-off if the initial classification maps are not highly conflicting. Otherwise, the result will be degraded by wrong information.

Concerning Pavia Center, all the rules seem accurate (Fig. 11.4, e.g., with the Dempster–Shafer rule), with an overall accuracy higher than 98% (Table 11.1). When visually inspecting the results, all rules gave similar good results excepting Prior 1, showing a result guided by the HS classification map rather than the MS one.

The Toulouse dataset is the largest one, with up to 15 classes. This explains the lower accuracies reached for this dataset. The best results were given by the Max, Prior 2, Bayesian sum and Dempster–Shafer rules. In practice, the Max, prior and sum rules seem to overestimate certain classes; especially tile roofing and vegetation. The best qualitative results are given by the Min, compromise and Dempster–Shafer rules. Despite a satisfactory accuracy, the AD rule exhibits many misclassifications regarding tile roofing (i.e., underestimation), metal roofing 1 (i.e., overestimation), and an erroneous detection of the gravel roofing. This is mainly due to the global accuracy measure which is included in the rule and calculated thanks to a limited ground truth data.

However, due to the very limited amount of reference data, the quantitative accuracies do not necessarily transcribe the real potential of the fusion rules. The best ones from a quantitative and practical qualitative point of view are the compromise, the Bayesian product and the Dempster–Shafer rules.

In this study, VHR-MS images as well as HS ones at lower resolution were generated from VHR HS original images. Thus, working on such synthetic datasets leads to quite optimistic results, but this is sufficient to assess the different fusion rules. Besides, the fusion method is flexible enough for instance to integrate a specific process to deal with shadows in a diachronic acquisition context.

11.3.4.3 Regularization

Global regularization was applied to enhance the classification results and eliminate the artifacts. Table 11.2 presents the optimization results for the best fusion rules per dataset. Indeed, the optimization procedure permits one to enhance further the classification. Quantitatively, it slightly enhances the decision fusion classification (by 1–2%) but offers a better visual rendering with an elimination of the artifacts, a better decimation of the classes borders, and a regularization of the scattered pixels (Figs. 11.3, 11.4 and 11.5). These optimized maps seem better in modeling the real scene. The optimization effect is more visible over Pavia University and Toulouse Center. Concerning Pavia Center, the decision fusion gives already good results and thus, the optimized maps are only slightly improved (Table 11.2). Results obtained over the Pavia datasets are comparable to other studies (e.g., [17]). For Pavia University; the painted metal sheets are better recovered and no mismatches with the surrounding road are noticeable. The proposed method permits one to extract some bitumen buildings that were difficult to differentiate from roads (i.e., upper right and lower right, Fig. 11.5), even if the gravel buildings could still be better refined. For Pavia center, the global rendering is enhanced with a minimization of the classification artifacts.

11.4.2.1 Initial classifications

Both sources are classified individually. The Sentinel-2 time series is labeled using a random forest (RF) classifier trained from 50,000 samples per class. RF is used to have a framework similar to the one from [91], of which the LC maps are intended to be available at a national scale. The SPOT 6/7 image is classified using a deep Convolutional Neural Network (CNN) [98], because of its high ability to efficiently exploit context and texture information from VHR image. The training process of the CNN used 10,000 samples per class (from which 10% were kept for cross-validation). Both classifiers produce membership probabilities for the five classes.

11.4.2.2 First regularization

Both fusions (i.e., for the 5-class nomenclature and the urban/non-urban one) are performed according to Sect. 11.2, involving a per-pixel decision fusion followed by a spatial regularization.

11.4.2.3 Binary classification and fusion

The “urban/non-urban” fusion requires one to derive binary class probabilities from results of the previous steps. Buildings from the 5-class fusion result are considered as seeds of urban areas and used to define a prior to be in an urban area (see Fig. 11.7): a linearly decreasing function assigning a probability is applied surrounding all buildings. This probability to be in an urban area starts from 1 and reaches a value of 0 after a distance of 100 m to a building.

Class posterior probabilities from the Sentinel-2 image RF 5-class classification are converted to a binary classification with an urban area class (u), and a non-urban area class (¬u): $P(u)=P(b)+P(r)$ and $P(\neg u)=1-P(u)$, with $P(b)$ and $P(r)$ the class probabilities for buildings and roads, respectively.

Then, the prior probability map to be in an urban area according to previous building detection is merged together with binary class probabilities from the Sentinel-2 RF classifier.

11.4.4.1 Five-class classifications

For the sake of visibility, the results are shown over a restricted area of 0.64 km² (Figs. 11.10, 11.9 and 11.11). There, the original classifications exhibit several errors: the impact of data fusion can be clearly demonstrated. Quantitative evaluation over image is for five tiles of 3000 × 3000 m size, totaling an area of 45 km², distributed over the entire 648 km² study zone. Each classification is compared to the class labels of the ground truth (Fig. 11.8). Evaluation measures for individual classifications, fusion and regularization, all using five classes, are shown in Table 11.3.

Initial classifications. Original classifications confirm the initial observation that the SPOT 6/7 CNN result tends to preserve small objects. However, some confusions between (bare soil) crops and built-up areas occur. The Sentinel-2 RF classification overall behaves better, but it mixes buildings and roads due to its coarse spatial resolution. The results from the individual classifications on the SPOT 6/7 and Sentinel-2 images are shown on Fig. 11.9. This area was selected to illustrate problems of both classifications and improvements obtained with fusion and regularization (better results are observed in all other areas). Confusion between water and built areas can be noticed. This phenomenon is caused by the fact that the water area database used to generate training samples included ponds at the bottom of careers appearing white and very similar to built-up areas.

Per pixel fusion. This first fusion is performed at the SPOT 6/7 resolution (1.5 m) as it aims at achieving the most fine detection of building objects.

Among the classic fusion rules proposed by [69], the Min fuzzy rule produces the best results, following the objects' borders most precisely while producing the least class confusions (Fig. 11.10 and Table 11.3). Considering Fig. 11.10, all rules managed to eliminate the wrongly classified building patch (top-left), preferring the Sentinel-2 classification over the SPOT 6/7 one. The Min fusion rule follows the field contours a bit less smoothly than the other ones. The industrial area (at the center of the displayed area) is still confused with water in both results (such a confusion can be explained by the presence of very white water area training samples).

The RF/SVM supervised fusions initially tend to produce patches of buildings rather than separate buildings due to missing training data (and thus constraints) between buildings and roads (Fig. 11.8). Indeed the ground truth contains gaps between buildings and so the classifiers tend to aggregate individual buildings as there are no training data available around building to prevent the classifier from doing it. Adding an additional sixth buffer class “around buildings” helps to refine the contours and to obtain a higher level of detail than just using the Min or Bayes rules. It preserves more details of individual buildings, but can cause some confusion between buildings and this “buffer” class in the center of very wide buildings in industrial areas (see Fig. 11.10). It can also erase small patches of buildings. However, this remains quite an exception and is not a real problem for the final goal of detection of urban areas. The same observations are made on all areas, even those on which the supervised fusion model is not trained. Thus, the result of the supervised fusion using a rbf SVM classifier with buffer class is used in subsequent steps.

Regularization. The parameters are as follows: $\lambda =10$, $\gamma =0.7$, $\epsilon =50$. They were selected by cross-validation, optimizing visual qualities of the regularization result. The contrast term is calculated from the SPOT 6/7 image: it especially aims at obtaining the finest detection of building objects. The result is shown in Fig. 11.11. The regularization performs well, smoothing out small noisy patches and yielding a visually more appealing result. However, accuracy measures of the regularization remain similar to the ones of the fusion (cf. Table 11.3).

11.4.4.2 Urban footprint extraction

Raw urban area maps directly derived using the binary class probabilities are shown in Fig. 11.12. Again, they are first merged using a per pixel fusion rule. The supervised learning based fusion approaches cannot be used here since no training data for urban/non-urban areas was available. Several rules yield visually similar results; the Min fuzzy fusion rule is eventually chosen.

Global regularization is then performed with the same $(\lambda ,\gamma ,\epsilon )$ parameters as for the 5-class regularization, but the image contrast term is now calculated from the Sentinel-2 image. The result obtained is shown in Fig. 11.12. Compared to Sentinel-2 detection, roads and small misclassifications have been removed. Compared to the dilated building mask from previous step, the objects' borders fit better to the true urban area borders.

As no true reference data for urban areas is available, strict evaluation is not possible. However, the detected artificialized area can be compared to binary ground truth maps derived from the following other related databases:

Strict quantitative evaluation is not possible, but such data can provide some hints: accuracies are provided in Table 11.4. Besides, a visual comparison with the different sources is shown in Fig. 11.13.

The following aspects can be underlined:

The dilated BD Topo® ground truth generally yields the highest agreement with classifications in terms of accuracy measures. The fusion and regularization steps improve the accuracy measures over the individual input classifications with the exception of the binary regularization input, which can be explained by the fact that both have been produced by dilatation.