3.2.1 Image Classification and the VGG Network

Network architectures developed for data fusion heavily rely on the results achieved in image classification using CNNs. Image classification is a very active research field, and recent developments can be tracked from the results reported at the annual ILSVCR competition [1]. Most notable networks from this competition are LeNet-5 [16], AlexNet [17], VGG [18], GoogLeNet [19], DenseNet [20] and ResNet [21].

Most commonly used image classifier that is also adopted in many data fusion architectures is the VGG developed by the Visual Geometry Group at University of Oxford. VGG is composed of blocks of 2 to 3 convolution-ReLU (Rectified Linear Units) pairs and a max-pooling layer at the end of each block; see Fig. 3.2A. The depth or feature channel size of the blocks increases with the depth of the network. Finally, three fully connected layers and a soft-max function map the features into label probabilities at the tail of the VGG network. Several variations of the VGG network exist, which differ in the number of blocks. For example, VGG-16 network indicates that the network consists of 13 layers of convolutional-ReLU pairs as well as three fully connected layers resulting in 16 blocks; see Fig. 3.2A. The latest implementations add batch normalization and drop-out layers to the originally reported VGG network in order to improve training and the generalization of the network [22].

In VGG and other networks, the convolution is implemented as shared convolution, not as a locally connected layer [15, p. 341] to handle fewer parameters, speed up the training, and use less memory. Shared weights within one channel assumes that the same features appear at various locations of the input images. This assumption is generally valid for remote sensing and indoor images. It is worth to mention that, since the ceiling or floor regions are located at the top and bottom of the images, using locally connected layer might provide an interesting insight and research direction for segmentation of indoor images.

The channels in the convolutional layers can be interpreted as filters, highlighting patterns in an image that the network has learned from the training data. In the first convolution layer, these patterns, referred to as features, are associated with basic/primitive image content; for instance, corners, edges, etc. Deeper in the network, the features describe complex relationships of these basic image content; for more details see [15, p. 360].

It is noteworthy that VGG networks are not considered as the best network architecture for image classification problems due to their accuracy, the number of parameters learned, and the inference time. The reader can find comparisons of various CNNs in [23]. The latest deep models that outperform VGG contain significantly more convolutional layers. To train these deeper networks is harder due to the propagation of potential numerical errors stemming from very low residuals in the training step. For this reason, ResNet utilizes residual connections that allow for training a 1000-layer deep network; see [21]. While this field is rapidly evolving, current data fusion networks typically adopt the VGG network due to its simple architecture.

3.2.2 Architectures for Pixel-level Labeling

As opposed to image classification, pixel-level labeling requires annotating the pixels of the entire image. Consequently, the output is an array similar to the size of the input. This can be achieved by utilizing a single image classifier network and by discarding the classifier tail of VGG, i.e. removing the fully connected layer and soft-max, and instead, utilizing a series of unpooling operations along with additional convolutions. An unpooling operation allows for increasing the width and height of the convolutional layer and decreases the number of channels. This results in generating an output at the end of the network that has the original image size; see Fig. 3.2B. This concept is referred to as encoder–decoder network, such as SegNet [6]. SegNet adopts a VGG network as encoder, and mirrors the encoder for the decoder, except the pooling layers are replaced with unpooling layers; see Fig. 3.2B. An advantage of utilizing an image classifier is that the weights trained on image classification datasets can be used for the encoder. This can be considered a benefit as the image classification datasets are typically larger, such that the weights learned using these datasets are likely to be more accurate. Adopting these weights as initial weights in the encoder part of the network is referred to as transfer learning. Here, we mention another notable network, called U-Net, which has a similar encoder–decoder structure but the encoder and decoder features are connected forming a U-shaped network topography [24].

The main disadvantage of encoder–decoder networks is the pooling-unpooling strategy which introduces errors at segment boundaries [6]. Extracting accurate boundaries is generally important for remote sensing applications, such as delineating small patches corresponding to buildings, trees or cars. SegNet addresses this issue by tracking the indices of max-pooling, and uses these indices during unpooling to maintain boundaries. DeepLab, a recent pixel-level labeling network, tackles the boundary problem by using atrous spatial pyramid pooling and a conditional random field [25]. Additionally, the latest DeepLab version integrates ResNet into its architecture, and thus benefits from a more advanced network structure.

3.2.3 Architectures for RGB and Depth Fusion

It is reasonable to expect that using additional information about the scene, such as depth, would improve the segmentation and labeling tasks. In this chapter, our focus is on end-to-end deep learning models that fuse RGB and depth data using CNNs; hence we investigate three different approaches:

Furthermore, instead of depth representation, one can also derive 3D normal vector (needle) images from the depth images and use them as input. Note that surface normal vector representation of the 3D space might be more general in terms of describing objects, since it does not depend on camera pose. It is clear that the depth values change when the camera pose changes, but the normals are fixed to the local plane of the object; hence, they are view independent. This property is expected to allow for more general learning of various objects. 3D normal vectors can be represented as a three-channel image that contains the X, Y, Z components of the vector at each pixel. We also derive a one-channel input by labeling the normal vectors based on their quantized directions. This one-channel normal label image allows for simpler representation, and thus requires fewer network parameters. These two normal representations are investigated for all fusion approaches presented above.

3.2.4 Datasets and Benchmarks

Successful training of deep learning methods requires large set of labeled segments. Generating accurate labels are labor intensive, and therefore, open datasets and benchmarks are important for developing and testing new network architectures. Table 3.1 lists the most notable benchmark datasets that contain both RGB and depth data. Currently, the Stanford 2D-3D-Semantics Dataset [27] is the largest database of images captured from real scenes along with depth data as well as ground truth labels. In the remote sensing domain, the ISPRS 2D Semantic Labeling benchmark [28] is the most recent overhead imagery dataset that contains high resolution orthophotos, images with labels of six land cover categories, digital surface models (DSM) and point clouds captured in urban areas.

3.3.1 Datasets and Data Splitting

The Stanford 2D-3D-Semantics Dataset (2D-3D-S) [27] contains RGB and depth images of 6 different indoor areas. The dataset were mainly captured in office rooms and hallways, and small part of it in a lobby and auditorium. The dataset is acquired by a Matterport Camera system, which combines cameras and three structured-light sensors to capture 360-degree RGB and depth images. The sensor provides reconstructed 3D textured meshes and raw RGB-D images. The dataset contains annotated pixel-level labels for each image. The objects are categorized into 13 classes: ceiling, floor, wall, column, beam, window, door, table, chair, bookcase, sofa, board, and clutter. The sofa class is underrepresented in the dataset, and thus categorized as clutter in our tests.

Due to the large size of the dataset, we use Area 1 of the Stanford dataset for training, and Area 3 for testing. This strategy allows for estimating the generalization error of the networks, because the training and testing are performed in two independent scenes. The training dataset is divided into 90% training and 10% validation sets, respectively, resulting in 9294 and 1033 images. The validation set is used for evaluating the network performance at each epoch of the training. The test data, i.e. Area 3, contains 3704 images. An overview of the training, validation and test datasets can be found in Table 3.2.

The ISPRS 2D Semantic Labeling Dataset is an airborne image collection, consisting of high resolution true orthophotos and corresponding digital surface models (DSMs) derived with dense image matching. The images are annotated manually into six land use and object categories, i.e. impervious surfaces, building, low vegetation, tree, car and clutter/background. The dataset acquired in a city (Potsdam) and a village (Vaihingen). Here, the Vaihingen dataset is investigated, which contains 33 high resolution orthophoto mosaics. The images have a ground sampling distance (GSD) of 9 cm, and are false-colored using the near-infra-red, red and green spectral bands (NRG). Table 3.2 lists the data used for training, validation, and testing.

3.3.2 Preprocessing of the Stanford Dataset

This subsection presents the methods used for image resizing, depth filtering and normalization. Sample images can be seen in the first two columns of Fig. 3.3.

Resizing. Original image size of the dataset is $1080\times 1080$ pixels. Deep learning models use significantly smaller image size due to memory and computation considerations. We should note that larger image size does not necessarily lead to more accurate result. We adopt the strategies developed by other authors [26,6], and resize the images to $224\times 224$ pixels. Sample RGB images can be seen in the first row of Fig. 3.3.

Depth filtering. Depth images in the dataset contain missing pixels. Since missing pixels can significantly influence the training and testing process, depth interpolation is performed by applying image inpainting strategies [31,32]. This approach, first, calculates initial guesses of the missing depth value using local statistical analysis, and then one iteratively estimates the missing depth values using discrete cosine transform. Sample depth images can be seen in the second row of Fig. 3.3.

Data normalization. RGB values are normalized according to the input layer of the neural networks. Range of depth values depends on the scene and the maximum operating range of the sensor. For indoor scenes, depth values are typically up to few meters. Therefore, we use 0 and 10 m range to normalize the data. Values larger than 10 m are truncated to this maximum value.

Calculation of normal vectors. Surface normal vectors are provided with the dataset. Examples of 3D normal images can be seen in the third row of Fig. 3.3.

3.3.3 Preprocessing of the ISPRS Dataset

This subsection presents the preprocessing steps for the ISPRS dataset. These steps include resizing and cropping of the orthophotos and the calculation of the normal vectors. Sample images can be seen in the last two columns of Fig. 3.3.

Resizing and cropping. The orthophoto mosaics from the ISPRS dataset are roughly $2000\times 2500$ pixels. In order to reduce the images to the desired $224\times 224$ pixel size, first, the images are cropped into $448\times 448$ tiles, providing an about 60% overlap. Then the cropped areas are resized to $224\times 224$ using nearest neighborhood interpolation. The final dataset consists of 12,664 images. Sample images can be seen in the first row of Fig. 3.3.

Depth filtering. The data are already filtered for outliers, and therefore, no data preprocessing is needed. Sample depth images can be seen in the second row of Fig. 3.3.

Calculation of normal vectors. Surface normal data is directly not available from the ISPRS dataset. Therefore, surface normals are derived from the DSM using least squares plane fitting. Given a point, the computation of a normal at that point is performed as follows. First, the eight neighbor grid points (${\mathbf{\text{z}}}_i$, $i=1,...,8$) are obtained around the point (${\mathbf{\text{z}}}_c$), and then their differences from the point are calculated, i.e. ${\overline{\mathbf{\text{z}}}}_i={\mathbf{\text{z}}}_i-{\mathbf{\text{z}}}_c$. Then the points are filtered based on the vertical differences and a predefined threshold. Given ${\overline{\mathbf{\text{z}}}}_i$, $i=1,...,k$, $k⩽8$, we formulate a constrained optimization problem:

$\widehat{\mathbf{\text{n}}}=\underset{||\mathbf{\text{n}}||=1}{\mathrm{argmin}}{\mathbf{\text{n}}}^{\top }(\sum _{i=1}^k{\overline{\mathbf{\text{z}}}}_i{\overline{\mathbf{\text{z}}}}_i^{\top })\mathbf{\text{n}},$

where n is the surface normal, $\widehat{\mathbf{\text{n}}}$ is its estimate. This problem can be solved with singular value decomposition; the $\widehat{\mathbf{\text{n}}}$ solution is the eigenvector of ${\sum }_{i=1}^k{\bar{\mathbf{z}}}_i{{\bar{\mathbf{z}}}_i}^{\top }$ corresponding to the smallest eigenvalue. Sample 3D normal images can be seen in the third row of Fig. 3.3.

3.3.4 One-channel Normal Label Representation

The one-channel surface normals are derived from the 3D normals by mapping the three-channel normal vectors into a unit sphere. This mapping can be simply done by transforming the normal unit vectors into spherical coordinates, i.e. $\theta ={\tan }^{-1}(\frac{n_y}{n_x})$ and $\varphi ={\cos }^{-1}(n_z)$, since $r=||\mathbf{\text{n}}||=\sqrt{n_x^2+n_y^2+n_z^2}=1$. At this point, the angle values have to be associated with integer numbers (labels) in order to represent the normal values in 8-bit image format, i.e. $(\theta ,\varphi )\to [0,...,255]$. Here, we opted to divide the sphere at every 18 degrees, which results in 200 values. These values are saved as gray image. This one-channel representation was calculated for both the Stanford and ISPRS datasets. Sample one-channel normal images can be seen in the fourth row of Fig. 3.3.

3.3.5 Color Spaces for RGB and Depth Fusion

We investigate two possibilities to fuse RGB and depth information through color space transformations. The goal is to keep the three-channel structure of RGB images, but substitute one of the channels with either the depth or normal labels.

Normalized RGB color space. Let r, g and b be the normalized RGB values, such that

$r=\frac{R}{R+G+B};g=\frac{G}{R+G+B};b=\frac{B}{R+G+B}.$

In this color space representation, one of the channels is redundant due to the fact that it can be recovered from the other two channels using the condition $r+g+b=1$. Therefore, for example, we can replace the blue channel with depth D or normals (one-channel representation) N, normalized to $[0,1]$. This results in fused images rgD and rgN. Sample rgD and rgN images can be seen in the fifth and sixth rows of Fig. 3.3. Note that in terms of energy, blue represents about 10% of the visible spectrum in average.

HSI color space. The second color space fusion is based on hue, saturation and intensity (HSI) representation. In this representation, intensity I is replaced with depth D or normals N, which results in HSD and HSN representation, respectively, where

$H=\{\begin{array}{c}{\cos }^{-1}\frac{0.5[(r-g)+(r-b)]}{\sqrt{{(r-g)}^2+(r-g)(g-b)}},\hfill \\ 2\pi -{\cos }^{-1}\frac{0.5[(r-g)+(r-b)]}{\sqrt{{(r-g)}^2+(r-g)(g-b)}},\hfill \end{array}$

$S=1-3\min (r,g,b).$

Sample HSI and HSN images can be seen in the seventh and eighth rows of Fig. 3.3.

Finally, we investigate a representation which is a transformation of the HSD or HSN representation back to the RGB space. Therefore, we calculate

$(R_d,G_d,B_d)=\{\begin{array}{c}(b,c,a),h=H,\text{ for }H<\frac{2}{3}\pi ,\hfill \\ (a,b,c),h=H-\frac{2}{3}\pi ,\text{for }\frac{2}{3}\pi ⩽H<\frac{4}{3}\pi ,\hfill \\ (c,a,b),h=H-\frac{4}{3}\pi ,\text{ for }\frac{4}{3}\pi ⩽H<2\pi ,\hfill \end{array}$

where

$a=D(1-S),$

$b=D(1+\frac{S\cos (h)}{\cos (\frac{\pi }{3}-h)}),$

$c=3D-(a+b),$

for color space modification using depth D or $(R_n,G_n,B_n)$ calculated using (3.5), where

$a=N(1-S),$

$b=N(1+\frac{S\cos (h)}{\cos (\frac{\pi }{3}-h)}),$

$c=3N-(a+b),$

for color space modification using normals N. Sample HSI and HSN images can be seen in the last two rows row of Fig. 3.3.

3.3.6 Hyper-parameters and Training

Due to the similarity in the network structures of investigated networks, the same hyper-parameters are used in the experiments for all three networks. We use pretrained weights for the VGG-16 part of the networks, which are trained on the large-scale ImageNet dataset [1]. When the network has an additional one-channel branch, such as the depth branch of FuseNet, or it has an additional input channel, i.e. four-channel network, the three-channel pretrained weights of the input convolution layer of VGG-16 are averaged and used for initializing the extra channel. A stochastic gradient descent (SGD) solver and cross-entropy loss are used for training. The entire training dataset is passed through the network at each epoch. Within an epoch, the mini batch size is set to 4 and the batch was randomly shuffled at each iteration. The momentum and weight decay was set to 0.9 and 0.0005, respectively. The learning rate was 0.005 and decreased by 10% at every 30 epochs. The best model is chosen according to the performance on the validation set measured after each epoch. We run 300 epochs for all tests, but the best loss is typically achieved between 100–150 epochs.

3.4.1 Results and Discussion on the Stanford Dataset

Results for the various network and input structures are presented in Table 3.3. Fig. 3.4 shows the accuracies of each class for different networks. The baseline solution, the SegNet with RGB inputs, achieves 76.4% global accuracy in our tests. If only depth images are used in SegNet, then the accuracy drops by 5%. Fusing RGB and depth through color space transformations results in noticeable decrease ($\sim 1\text{\%}$–3%) in the network performance.

Table 3.3

Results on Stanford indoor dataset; Training: 90% of Area 1, 9294 images; Validation: 10% of Area 1, 1033 images; Testing: 100% of Area 3, 3704 images. *Network abbreviations: the name of the networks consists of two parts: the first is the base network, i.e. SegNet, FuseNet and VNet, see Fig. 3.2, following the input type. Note that, in some cases, the networks are slightly changed to be able to accept the different input size. For instance, RGBD and RGBN represent four-channel inputs. RGB+D and RGB+N indicates that the network has separate branches for depth or normal, respectively. X_nY_nZ_n is three-channel input of 3D surface normals.

Network*	GlobalAcc	MeanAcc	Mean IoU
Segnet RGB	76.4%	56.0%	70.8%
Segnet Depth Only	71.0%	50.8%	67.5%
Segnet HSD	73.4%	53.6%	74.2%
Segnet HSN	77.4%	57.1%	71.2%
Segnet RdGdB${}_{d(HSI)}$	73.5%	49.7%	64.1%
Segnet RnGnB${}_{n(HSI)}$	74.6%	51.4%	67.2%
Segnet rgD	76.1%	55.6%	71.3%
Segnet rgN	76.3%	53.4%	69.2%
Segnet RGBD	75.4%	53.6%	68.5%
Segnet RGBN	76.7%	57.4%	74.4%
Segnet RGBDN	68.7%	45.2%	58.9%
FuseNet RGB+D	76.0%	54.9%	73.7%
FuseNet RGB+N	81.5%	62.4%	76.1%
FuseNet RGB+XnYnZn	81.4%	62.4%	78.1%
VNet RGB+D	78.3%	54.2%	68.5%
VNet RGB+N	80.0%	56.1%	70.4%

The four-channel variants of the SegNet, i.e. SegNet RGBD and SegNet RGBN, produce similar results to the SegNet RGB. In these cases, the information from the depth data clearly makes no contribution to the segmentation process; the network uses only the pretrained weights of SegNet RGB, and the weights associated with the depth are neglected.

The FuseNet architecture with depth input, i.e. FuseNet RGB+D, produces slightly worse results than the baseline solution in our tests; it particularly achieves 76.0% global accuracy, which is 0.4% less than the accuracy produced by SegNet, which can be considered insignificant. The mean IoU in this case is better by $\sim 3\text{\%}$ than the baseline. Note that this 76% accuracy is similar to the result reported in [26] on the SUN RGB-D benchmark dataset. In our tests, we found that FuseNet with normals, i.e. FuseNet RGB+N, provides the best results in terms of global and mean accuracy. The $\sim 5\text{\%}$ margin between either the SegNet RGB or FuseNet RGB+D and the RGB+N is significant. The 3D normal representation provides the same global accuracy as the FuseNet RGB+N, while it outperforms by 2% margin in the mean IoU.

Discussion. Results suggest that depth does not improve the overall accuracy for the adopted dataset. Using normals instead of depth, however, improves almost all types of RGB and depth fusion approaches with a significant margin. Since the test dataset is a different area (Area 3) than the training (Area 1), these results indicate that the normals improve the generalization ability of the network. In other words, normals are better general representation of indoor scenes than depth.

In order to support this hypothesis, we also evaluated the networks using data from the Area 1 only. No samples were taken from Area 3 neither for the training nor for the testing. These results are presented in Table 3.4. In this training scenario, the images from Area 1 are divided into 10% (1033 images) training and 90% (9294 images) validation sets. We used this extreme split in order to rely on the pretrained SegNet weights; consequently, the achieved accuracies presented in Table 3.4 are significantly smaller than in the first test scenario. Table 3.4 shows no or a slight performance gap between depth and normal data for all approaches. FuseNet shows the best results by effectively combining RGB and depth; FuseNet outperforms SegNet with 6% margin as opposed to the results presented in Table 3.3; note that similar improvements were reported by previous studies [26]. It is important to note that FuseNet is able to combine RGB and depth with very low number of training samples, which prompts the question whether the network actually learns the depth features, or just the addition operator applied between the network branches is an efficient, but deterministic way to share depth information with the VGG-16 network.

The per-class accuracies presented in Fig. 3.4 show that each network has a different accuracy per each class. FuseNet RGB+D provides the best overall accuracy metrics, but other networks, such as SegNet HSD, performs better in majority of classes, such as column, beam, chair. This indicates that an ensemble classifier using these networks might outperform any individual network.

3.4.2 Results and Discussion on the ISPRS Dataset

Results for the ISPRS Vaihingen dataset are listed in Table 3.5, the baseline solution is the three-channel SegNet NRG network, which achieves 85.1% global accuracy. Note that SegNet NRG is the same network as SegNet RGB which was discussed in the previous section; we use NRG term to emphasize that the ISPRS images are false color infra-red, red, green images. As expected, the accuracy drops down by about 10% using only depth data. All network solutions produce similar results close to the baseline solution. SegNet NRGD, NRG+N and VNet NRG+N slightly outperform the SegNet NRG in all three metrics; though, the difference is not significant. The best results are provided by the VNet NRG+N achieving 85.6% global, 50.8% mean accuracy and 60.4% mean IoU. Per-class accuracies presented in Fig. 3.5. Here, we see that all networks have a very similar performance in all classes. A slight difference can be seen in the clutter class, where SegNet NRGD outperforms the other networks.

Discussion. Overall, almost all accuracies presented in Table 3.5 are within the error margin and are around 85%. Using normals have insignificant improvement due to the fact that almost all surface normals point in the same direction (upward).

Table 3.5 shows very slight improvement of VNet over FuseNet, while FuseNet NRG+D does not achieve the accuracy of the baseline SegNet architecture. It is noteworthy that the reported global accuracies on the ISPRS Vaihingen dataset using either SegNet or FuseNet are around 91% [11]. The main reason of this relatively large performance gap between Table 3.5 and [11] is potentially due both to the difference in evaluation technique and the random selection of images in the training process.

Although SegNet NRG has similar results to VNet or FuseNet, there are examples, where data fusion using depth outperforms SegNet; see an example in Fig. 3.6. The red rectangle shows a section, where SegNet NRG does not detect a building due to the color of the building roof is very similar to the road; see Figs. 3.6D and 3.6C for ground truth. Unlike SegNet, FuseNet NRG+D partially segment that building, see Fig. 3.6E, due to the depth difference between the roof and road surface in the depth image presented in Fig. 3.6B. Also, note that the depth image is not sharp around the building contours, and the prediction image of FuseNet NRG+D follows this irregularity; see Fig. 3.6B and 3.6E. This indicates that the quality of the depth image is important to obtain reliable segment contours.

Using normals instead of depth as input also does not improve the global accuracy significantly. Most object categories are horizontal, such as grass or a road, and therefore, normals point upwards containing no distinctive information of the object class itself. In fact, after analyzing the confusion matrices, we found that FuseNet NRG+N slightly decreased (0.7%) the false positives of the building class and improved the true positives for the impervious surface class with the same ratio. This indicates that the network better differentiate between buildings and roads. The reason for this that some roofs in the dataset are not flat, such as gable or hipped roof. In these cases, normals have a distinguishable direction that guides the network to correctly identify some buildings. Two examples can be seen in the first two rows of Fig. 3.7. The figures demonstrate that the FuseNet NRG+N has sharper object edges than a single SegNet NRG classifier. Finally, the last row in Fig. 3.7 presents a case when the NRG based network does not classify correctly an object based on the ground truth; see the object boxed by the red rectangle. The ground truth and FuseNet NRG+N mark that object as car, and SegNet as building. This example demonstrates that normal or depth data has semantical impact on interpreting a patch.