5.2.1 Visible Pedestrian Detection

A large variety of methods have been proposed to perform pedestrian detection using visible information in the last 20 years. Papageorgiou et al. [43] firstly abandoned motion cues to train a visible pedestrian detection system, utilizing Haar wavelets transform and support-vector machines (SVMs) [8]. The authors also collected a new visible pedestrian database (MIT) to evaluate their method. Viola and Jones [52] designed the integral images with a cascade structure for fast feature computation and efficient visible pedestrian detection, applying the AdaBoost classifier for automatic and accurate feature selection. Wu et al. [55] firstly proposed sliding window detectors for pedestrian detection, applying multiscale Haar wavelets and support-vector machines (SVMs) [8]. Dalal et al. [9] proposed the histograms of oriented gradient (HOG) descriptors along with a cascaded linear support-vector network for more accurate pedestrian detection. Dollar et al. [11] improved the HOG descriptors by designing the integrate channel features (ICFs) descriptors with an architecture of multichannel feature pyramids followed by the AdaBoost classifier. The feature representations of ICF have been further improved through multiple techniques including ACF [12], LDCF [39], SCF [1], and Checkerboards [62].

In recent years, DNN based approaches for object detection [19,18,45,25] have been widely adopted to improve the performance of visible pedestrian detection. Sermanet et al. [47] applied convolutional sparse coding to pretrain convolutional neural networks (CNNs) for feature extraction and fully-connected (FC) layers for classification. Tian et al. [50] developed extensive part detectors using weakly annotated humans to handle occlusion problems occurred in pedestrian detection. Li et al. [34] presented a scale-aware DNN which adaptively combines the outputs from a large-size sub-network and a small-size one to generate robust detection results at different scales. Cai et al. [5] designed the unified multiscale DNN to combine complementary scale-specific ones, applying various receptive fields to identify pedestrians in different scales. Zhang et al. [60] proposed a coarse-to-fine classification scheme for visible pedestrian detection by applying region proposal networks [45] to generate high-resolution convolutional feature maps, which are followed by the AdaBoost classifier for final classification. Mao et al. [38] designed a powerful deep convolutional neural networks architecture by utilizing the information of aggregating extra features to improve pedestrian detection performance without additional inputs in inference. Brazil et al. [4] developed a novel multitask learning scheme to improve visible pedestrian detection performance with the joint supervision on weakly box-based semantic segmentation and pedestrian detection, indicating that the box-based segmentation masks can provide sufficient supervision information to achieve additional performance gains.

We summarize the mentioned visible pedestrian detection methods in Table 5.1. However, pedestrian detectors trained on visible images are sensitive to changes of illumination, weather, and occlusions. These detectors are very likely to be stuck with images captured during nighttime.

5.2.2 Infrared Pedestrian Detection

Infrared imaging sensors, which provide excellent visible cues during nighttime, have found their importance for around-the-clock robotic applications, such as autonomous vehicle and surveillance system.

Nanda et al. [40] presented a real-time pedestrian detection system that works on low quality thermal videos. Probabilistic templates are utilized to capture the variations in human targets, working well especially for the case when object contrast is low and body parts are missing. Davis et al. [10] utilized a generalized template and an AdaBoosted ensemble classifier to detect people in widely varying thermal imagery. The authors also collected a challenging thermal pedestrian dataset (OSU-T) to test their method. Suard et al. [49] developed image descriptors, based on histograms of oriented gradients (HOG) with a support-vector machine (SVM) classifier, for pedestrian detection applied to stereo thermal images. This approach achieved good results for window classification in a video sequence. Zhang et al. [59] investigated the approaches derived from visible spectrum analysis for the task of thermal pedestrian detection. The author extended two feature classes (edgelets and HOG features) and two classification models (AdaBoost and SVM cascade) to the thermal images. Lee et al. [32] presented a nighttime part-based pedestrian detection method which divides a pedestrian into parts for a moving vehicle with one camera and one near-infrared lighting projector. The confidence of detected parts can be enhanced by analyzing the spatial relationship between every pair of parts, and the overall pedestrian detection result is refined by a block-based segmentation method. Zhao et al. [64] proposed a robust approach utilizing the shape distribution histogram (SDH) feature and the modified sparse representation classification (MSRC) for thermal pedestrian detection. Biswas et al. [3] proposed the multidimensional templates based on local steering kernel (LSK) descriptors to improve the pedestrian detection accuracies in low resolution and noisy thermal images. Kim et al. [29] presented a new thermal infrared radiometry aggregated channel feature (TIR-ACF) to detect pedestrians in the far thermal images at night.

We summarize the above-mentioned infrared pedestrian detection methods in Table 5.2. However, false detections are frequently caused due to strong solar radiation and background clutters in the daytime thermal images. With the development of multimodal sensing technology, it is possible to generate more stable detection results by simultaneously capturing multimodal information (e.g., visible, thermal and depth) of the same scene, which provide complementary information about objects of interest.

5.2.3 Multimodal Pedestrian Detection

It is experimentally demonstrated that multimodal images provide complementary information about objects of interest. As a result, pedestrian detectors trained using multimodal images can generate more robust detection results than using the visible or thermal images alone.

Grassi et al. [21] proposed a novel information fusion approach to detecting pedestrians, by determining the regions of interest in the video data through a lidar sensor and a thermal camera. Yuan et al. [58] proposed a multispectral based pedestrian detection approach which employs latent variable support-vector machines (L-SVMs) to train the multispectral (visible and near-thermal) pedestrian detection model. A large-size multispectral pedestrian dataset (KAIST) is presented by Hwang et al. [26]. The KAIST dataset contains well-aligned visible and thermal image pairs with dense pedestrian annotations. The authors further developed a new multimodal aggregated feature (ACF+T+THOG) followed by the AdaBoost classifier for target classification. The ACF+T+THOG concatenate the visible features generated by ACF [12] and the thermal one generated by T+THOG, which contains the thermal image intensity (T) and the thermal HOG [9] features (THOG).

Recently, DNN based approaches for visible pedestrian detection have been widely adopted to design the multimodal pedestrian detectors. Wagner et al. [53] proposed the first application of DNN for multimodal pedestrian detection. The detections in [26] are considered as proposals, which are classified with a two-stream R-CNN [19] applying concatenation fusion in the late stage. The authors further evaluated the performance of architectures with different fusion stages, and the optimal architecture is in the late fusion stage. Liu et al. [28] investigated how to adopt the Faster R-CNN [45] model for the task of multimodal pedestrian detection and designed four different fusion architectures in which two-stream networks are integrated at different stages. Experimental results showed that the optimal architecture is the Halfway Fusion model which merges two-stream networks at a high-level convolutional stage. Xu et al. [57] designed a method to learn and transfer cross-modal deep representations in a supervised manner for robust pedestrian detection against bad illumination conditions. However, this method is based on information of visible channel only (during the testing stage) and its performance is not comparable with ones based multispectral data (e.g., Halfway Fusion model [28]) König et al. [30] modified the visible pedestrian detector RPN+BDT [60] to build Fusion RPN+BDT architecture for multimodal pedestrian detection. The Fusion RPN concatenates the two-stream RPN on the high-level convolutional stage and achieves the state-of-the-art performance on KAIST multimodal dataset.

We summarize the mentioned multimodal pedestrian detection methods in Table 5.3. It is worth it to mention in this chapter that our approach is definitely different from the above methods in two aspects. Firstly, a number of feature fusion strategies for multimodal pedestrian detection is firstly evaluated comparing with other basic fusion schemes. Secondly, we make use of semantic segmentation information as supervision for multimodal feature learning.

5.3.1 Multimodal Feature Learning/Fusion

The architecture of two-stream deep convolutional neural networks (TDCNNs) for multimodal feature learning and fusion is illustrated in Fig. 5.4. TDCNN learn semantic feature descriptions in the visible and thermal channels individually. The visible stream of TDCNN contains five convolutional layers (from Conv1-V to Conv5-V) and pooling ones (from Pool1-V to Pool4-V), which is the same as the thermal stream of TDCNN (from Conv1-T to Conv5-T and from Pool1-V to Pool4-V). Each stream of TDCNN takes the first five convolutional layers (from Conv1 to Conv5) and pooling ones (from Pool1 to Pool4) in VGG-16 as the backbone.

Previous researches revealed that feature fusion at a late stage can generate semantic feature maps which are appropriate for complex high-level detection tasks [53,28,30]. On the basis of this conclusion, the multimodal feature fusion layer is deployed after the Conv5-V and Conv5-T layers in TDCNN to combine the feature maps generated by the visible and thermal streams. The multimodal feature maps are generated as

$\mathbf{m}=\mathcal{F}(\mathbf{v},\mathbf{t}),$

where $\mathcal{F}$ is the feature fusion function, $\mathbf{v}\in {\mathbb{R}}^{B_v\times C_v\times H_v\times W_v}$ and $\mathbf{t}\in {\mathbb{R}}^{B_t\times C_t\times H_t\times W_t}$ are the feature maps generated by the visible and thermal streams, respectively, and $\mathbf{m}\in {\mathbb{R}}^{B_m\times C_m\times H_m\times W_m}$ is the multimodal feature maps. It should be noted that B, C, H and W denote the number of batch size, channels, heights and widths of the respective feature maps. Considering that the feature maps generated by the visible and thermal streams are the same size, we set $B_t=B_c$, $C_v=C_t$, $W_v=W_t$, $H_v=H_t$. Three different feature fusion functions are considered: concatenation (${\mathcal{F}}^{\text{concat}}$), maximization (${\mathcal{F}}^{\text{max}}$) and summation (${\mathcal{F}}^{\text{sum}}$).

Concatenation fusion. Concatenation function ${\mathcal{F}}^{\text{concat}}$ is the most widely used operation to integrate feature maps generated in visible and thermal channels [53,28,30]. ${\mathbf{m}}^{\text{concat}}={\mathcal{F}}^{\text{concat}}(\mathbf{v},\mathbf{t})$ stacks v and t across the individual channels c as

$\{\begin{array}{c}{\mathbf{m}}_{B,2c-1,H,W}^{\text{concat}}={\mathbf{v}}_{B,c,H,W},\hfill \\ {\mathbf{m}}_{B,2c,H,W}^{\text{concat}}={\mathbf{t}}_{B,c,H,W},\hfill \end{array}$

where $c\in (1,2,...,C)$ and $\mathbf{m}\in {\mathbb{R}}^{B\times 2C\times H\times W}$. Considering that the feature maps generated in visible and thermal streams are directly stacked across feature channels using concatenation function, the correlation of features generated in visible and thermal streams will be further learned in subsequent convolutional layers.

Maximization fusion. ${\mathbf{m}}^{\text{max}}={\mathcal{F}}^{\text{max}}(\mathbf{v},\mathbf{t})$ takes the maximum of v and t at the same spatial locations h, w as

${\mathbf{m}}_{B,C,h,w}^{\text{max}}=\text{max}\{{\mathbf{v}}_{B,C,h,w},{\mathbf{t}}_{B,C,h,w}\},$

where $h\in (1,2,...,H)$, $w\in (1,2,...,W)$ and $\mathbf{m}\in {\mathbb{R}}^{B\times C\times H\times W}$. The maximization fusion function is utilized to select the features which are most distinct in either the visible or thermal streams.

Summation fusion. ${\mathbf{m}}^{\text{sum}}={\mathcal{F}}^{\text{sum}}(\mathbf{p},\mathbf{q})$ computes the summation of v and t at the same spatial locations h, w as

${\mathbf{m}}_{B,C,h,w}^{\text{max}}={\mathbf{v}}_{B,C,h,w}+{\mathbf{t}}_{B,C,h,w},$

where $h\in (1,2,...,H)$, $w\in (1,2,...,W)$ and $\mathbf{m}\in {\mathbb{R}}^{B\times C\times H\times W}$. The summation fusion function is utilized to integrate the feature maps generated in visible and thermal streams using equal weighting scheme. Thus, multimodal feature maps will be stronger by combining the weak features generated in the visible and thermal streams. The performances of multimodal pedestrian detectors utilizing these three fusion functions (${\mathcal{F}}^{\text{concat}}$, ${\mathcal{F}}^{\text{max}}$, ${\mathcal{F}}^{\text{sum}}$) are comparatively evaluated in Sect. 5.4.3.

5.3.2 Multimodal Pedestrian Detection

In this subsection, we design two different DNN based models with and without a scene-aware mechanism for the task of multimodal pedestrian detection. Their overall architectures are shown in Fig. 5.5.

5.3.2.2 Scene-aware DNN model

Pedestrian samples have significantly different multimodal characteristics in daytime and nighttime scenes as shown in Fig. 5.6. Therefore, it is reasonable to deploy multiple scene-based sub-networks to handle intra-class object variances. Based on this observation, we further present a scene-aware DNN model to improve detection performance in daytime and nighttime scenes by considering the scene information encoded in multimodal images.

For this purpose, we firstly design a simple scene prediction model to estimate the probability of being daytime scene or nighttime one. As shown in Fig. 5.7, the scene prediction network (SPN) contains one pooling layer (SP-Pool), three continuous fully-connected layers (SP-FC1, SP-FC2, SP-FC3), and the classification layer (Soft-max). Each pair of multimodal images are entered into the first five convolutional layers and four pooling ones of TRPN to extract feature maps in individual visible and thermal channels. The two-stream feature maps are integrated utilizing the concatenate fusion layer (Concat). Inspired by the spatial pyramid pooling layer which can resize the feature maps to the same spatial resolution [24], the pooling layer SP-Pool resizes the multimodal feature maps to a fixed spatial size of 7×7 using symmetric bilinear interpolation from the nearest neighbors. Attached after the SP-FC3 is Soft-max, which is the classification layer of the scene prediction model. The outputs of Soft-max are ${\omega }_d$ and ${\omega }_n=(1-{\omega }_d)$, which compute the probability of being day scene or night one. We define the error term of scene prediction ${\mathcal{L}}_{SP}$ as

${\mathcal{L}}_{SP}=-{\hat{\omega }}_d\cdot \text{log}({\omega }_d)-{\hat{\omega }}_n\cdot \text{log}({\omega }_n),$

(5.8)

where ${\omega }_d$ and ${\omega }_n=(1-{\omega }_d)$ are the predicted scene weights for daytime and nighttime, ${\hat{\omega }}_d$ and ${\hat{\omega }}_n=(1-{\hat{\omega }}_d)$ are the scene labels. The ${\hat{\omega }}_d$ is set to 1 when the scene labels are annotated as daytime. Otherwise, we set ${\hat{\omega }}_d=0$.

We incorporate scene information into the baseline DNN model in order to generate more accurate and robust results for various scene conditions. Specifically, the scene-aware two-stream region proposal networks (STRPNs) consist of four sub-networks (D-Conv-CS, N-Conv-CS, D-Conv-BB, and N-Conv-BB) to generate scene-aware detection results (CS and BB) as shown in Fig. 5.5B. D-Conv-CS and N-Conv-CS generate feature maps for classification under daytime and nighttime scenes, respectively. The outputs of D-Conv-CS and N-Conv-CS are combined utilizing the weights computed in the SPN model to produce the scene-ware classification scores (CSs). D-Conv-BB and N-Conv-BB generate feature maps for box regression in day and night scene conditions, respectively. The outputs of D-Conv-BB and N-Conv-BB are integrated using the weights computed in the SPN model to calculate the scene-ware bounding boxes (BBs). The loss term for detection $L_{det}$ is also defined based on Eq. (5.5), while $c_i^s$ is computed as the weighted sum of classification score in daytime scene $c_i^d$ and one in nighttime scene $c_i^n$ as

$c_i^s={\omega }_d\cdot c_i^d+{\omega }_n\cdot {c_i}^n,$

(5.9)

and $b_i^s$ is the scene-aware weighted combination of $b_i^d$ and $b_i^n$, which are calculated by D-Conv-BB and N-Conv-BB sub-networks, respectively, as

$b_i^s={\omega }_d\cdot b_i^d+{\omega }_n\cdot {b_i}^n.$

(5.10)

5.3.3 Multimodal Segmentation Supervision

According to some recent research work [25,4], the performance of anchor-based object detectors can be improved using the information of semantic segmentation as a strong cue. The underlying reason is that box-based segmentation masks are able to supply additional effective supervision to make the feature maps in the shared layers more distinctive for the downstream object detectors. We integrate the segmentation supervision scheme with TRPN and STRPN to obtain more accurate multimodal pedestrian detection results.

Given feature maps generated from two-stream visible and thermal channels, different segmentation supervision results will be achieved by integrating the feature maps in different levels (feature-level and decision-level). In order to explore the optimal fusion architecture for the multimodal segmentation supervision task, we develop two different multimodal segmentation supervision architectures, denoted as feature-level multimodal segmentation supervision (MS-F) and decision-level multimodal segmentation supervision (MS). As illustrated in Fig. 5.8A, the MS-F contains a concatenate fusion layer (Concat), a 3×3 convolutional layer (Conv-Seg) and the output layer (Seg). Each pair of multimodal images are entered into the first five convolutional layers and four pooling ones of TRPN to extract feature maps in individual visible and thermal channels. The two-stream feature maps are integrated utilizing the Concat layer. The MS-F generate segmentation prediction (Seg) as a supervision to make the two-stream feature maps more distinctive for multimodal pedestrian detector. In comparison, as illustrated in Fig. 5.8B, the MS utilizes two 3×3 convolutional layers (Conv-Seg-V and Conv-Seg-T) to generate visible and thermal segmentation prediction as the supervision to make the visible and thermal feature maps more distinctive for multimodal pedestrian detector. The segmentation outputs of the MS is a combination of Conv-Seg-V and Conv-Seg-T.

It is reasonable to explore whether the performance of segmentation supervision able to be improved by integrating the scene information. Two different scene-aware multimodal segmentation supervision architectures (SMS-F and SMS) are developed based on the architectures of multimodal segmentation supervision models (MS-F and MS). As shown in Fig. 5.8C–D, multimodal segmentation outputs are generated using daytime and nighttime segmentation sub-networks (D-Seg and N-Seg) applying the scene-aware weighting mechanism. It should be noted that SMS-F consists of two sub-networks, while SMS consists of four sub-networks. The performance of these four different multimodal segmentation supervision architectures (SM-F, SM, SMS-F, and SMS) are evaluated in Sect. 5.4.

The loss term of segmentation supervision is defined as

$L_{Seg}=\sum _{i\in C}\sum _{j\in B}[-{\hat{s}}_j\cdot \text{log}(s_{ij}^s)-(1-{\hat{s}}_j)\cdot \text{log}(1-{s_{ij}}^s)],$

where $s_{ij}^s$ is the segmentation prediction, C is the segmentation streams (MS-F and SMS-F consist of one multimodal stream while MS and SMS consist of one visible stream and one thermal stream), B are training samples for box-based segmentation supervision. We set ${\hat{s}}_j=1$ if the sample is located in the region within any bounding box of ground truth. Otherwise, we set ${\hat{s}}_j=0$. As for the scene-aware multimodal segmentation supervision architectures SMS-F and SMS, the scene-aware segmentation prediction $s_{ij}^s$ is computed by the scene-aware weighted combination of daytime and nighttime segmentation prediction $s_{ij}^d$ and $s_{ij}^n$, respectively, as

$s_{ij}^s={\omega }_d\cdot s_{ij}^d+{\omega }_n\cdot {s_{ij}}^n.$

The loss terms defined in Eqs. (5.7), (5.2), and (5.11) are combined to conduct multitask learning of scene-aware pedestrian detection and segmentation supervision. The final multitask loss function is defined as

$\mathcal{L}={\mathcal{L}}_{Det}+{\lambda }_{sp}\cdot {\mathcal{L}}_{SP}+{\lambda }_{seg}\cdot {\mathcal{L}}_{Seg}$

where ${\lambda }_{sp}$ and ${\lambda }_{seg}$ are the trade-off coefficient of loss term ${\mathcal{L}}_{SP}$ and ${\mathcal{L}}_{Seg}$, respectively.

5.4.1 Dataset and Evaluation Metric

The public KAIST multimodal pedestrian benchmark dataset [26] are utilized to evaluate our proposed methods. The KAIST dataset consists of 50,172 well-aligned visible-thermal image pairs with 13,853 pedestrian annotations in training set and 2252 image pairs with 1356 pedestrian annotations in the testing set. All the images in KAIST dataset are captured under daytime and nighttime lighting conditions. According to the related research work [28,30], we sample the training images every two frames. The original annotations under the “reasonable” evaluation setting (pedestrian instances are larger than 55 pixels height and over 50% visibility) are used for quantitative evaluation. Considering that many problematic annotations (e.g., missed pedestrian targets and inaccurate bounding boxes) are existed in the original KAIST testing dataset, we also utilize the improved annotations manually labeled by Liu et al. [37] for quantitative evaluation.

The log-average miss rate (MR) proposed by Dollar et al. [12] is used as the standard evaluation metric to evaluate the quantitative performance of multimodal pedestrian detectors. According to the related research work [26,28,30], a detection is considered as a true positive if the IoU ratio between the bounding boxes of the detection and any ground-truth label is greater than 50% [26,28,30]). Unmatched detections and unmatched ground-truth labels are considered as false positives and false negatives, respectively. The MR is computed by averaging miss rate at nine false positives per image (FPPI) rates which are evenly spaced in log-space from the range ${10}^{-2}$ to 10⁰ [26,28,30].

5.4.2 Implementation Details

According to the related research work [60,28,30], the image-centric training framework is applied to generate mini-batches and each mini-batch contains one pair of multimodal images and 120 randomly selected anchor boxes. In order to make the right balance between foreground and background training samples, we set the ratio of positive and negative anchor boxes to 1:5 in each mini-batch. The first five convolutional layers in each stream of TDCNN (from Conv1-V to Conv5-V in the visible stream and from Conv1-T to Conv5-T in the thermal one) are initialized using the weights and biases of VGG-16 [48] DNN pretrained on the ImageNet dataset [46] in parallel. Following the RPN designed by Zhang et al. [60], other convolutional layers are initialized with a standard Gaussian distribution. The number of channels in SP-FC1, SP-FC2, SP-FC3 are empirically set to 512, 64, 2, respectively. We set ${\lambda }_r=5$ in Eq. (5.5) following [60] and ${\lambda }_s=1$ in Eq. (5.13) according to the visible segmentation supervision method proposed by Brazil et al. [4]. All of multimodal pedestrian detectors are trained using the Caffe deep learning framework [27] with stochastic gradient descent [65] for the first two epochs in the learning rate (LR) of 0.001 and one more epoch in the LR of 0.0001. We clip gradients when the L2 norm of the back-propagation gradient is larger than 10 to avoid exploding gradient problems [44].

5.4.3 Evaluation of Multimodal Feature Fusion

In order to explore the optimal feature fusion scheme for multimodal pedestrian detection, we compare the TRPN with different feature fusion layers. As exposed in Sect. 5.3.1, three different feature fusion functions (concatenation, maximization, and summation) are utilized to build the three different TRPN models (TRPN-Concat, TRPN-Max, and TRPN-Sum). The TRPN models are trained using the loss term of detection ${\mathcal{L}}_{Det}$. The detection performances of TRPN-Concat, TRPN-Max, and TRPN-Sum are quantitatively compared in Table 5.4 and Table 5.5 using the log-average miss rate (MR) proposed by Dollar et al. [12]. In addition, qualitative comparisons of the detection performances of the three different TRPN models are conducted by showing some detection results in Fig. 5.9.

We can observe that the multimodal pedestrian detection performance is affected by the functions of feature fusion. Our experimental comparisons show that the performance of TRPN-Sum is better than the TRPN-Concat and TRPN-Max, resulting in lower MR on both original and improved KAIST testing datasets. Surprisingly, the widely used fusion function (concatenation) to integrate feature maps generated in visible and thermal channels [53,28,30] performs worst among the three different feature fusion functions. As described in Sect. 5.3.1, the feature maps generated in visible and thermal streams are directly stacked across feature channels using concatenation function. Thus, the correlation of features generated in visible and thermal streams will be further learned using the Conv-Det layer in the TRPN-Concat model. As shown in Fig. 5.6, such correlation is different for daytime and nighttime multimodal pedestrian characteristics. It is difficult to build up the correlation between visible and thermal feature maps using a simple convolutional encoder (Conv-Det). On comparison, TRPN-Max and TRPN-Sum models achieve better detection results by using either maximum or summation function to define the correlation between visible and thermal feature maps. It should be noticed that the TRPN-Sum can successfully detect the pedestrian targets whose human-related characteristics are not distinct in both visible and thermal images. Different from the maximum function that selects the most distinct features in the visible and thermal streams, the summation function is able to integrate the weak features generated in the visible and thermal streams to generate a stronger multimodal one to facilitate more accurate pedestrian detection results.

5.4.4 Evaluation of Multimodal Pedestrian Detection Networks

The scene prediction network (SPN) is essential and fundamental in our proposed scene-aware multimodal pedestrian detection networks. We first evaluate whether the scene prediction networks (SPNs) can accurately compute the scene weights ${\omega }_d$ and ${\omega }_n=(1-{\omega }_d)$, which supply vital information to integrate the scene-aware sub-networks. The KAIST testing dataset is utilized to evaluate the performance of SPN. Please note that the KAIST testing dataset consists of 1455 image pairs captured in daytime scene conditions and 797 in nighttime. Given a pair of multimodal images, the SPN computes a daytime scene weight ${\omega }_d$. The scene condition will be predicted correctly if ${\omega }_d>0.5$ during daytime or ${\omega }_d<0.5$ during nighttime. In order to investigate whether visible images or thermal ones can supply the most reliable information to predict the scene conditions, the performances of scene prediction utilizing the feature maps generated only in the visible stream (SPN-V) or thermal one (SPN-T) are evaluated individually. We illustrate the architectures of SPN-V, SPN-T, and SPN in Fig. 5.10. The prediction results of these three scene prediction networks are compared in Table 5.6.

We can observe that the SPN-V are able to generate accurate prediction of scene conditions for daytime scenes (97.94%) and nighttime ones (97.11%). The underlying reason is that the visible images display different brightness in daytime and nighttime scene conditions. The scene prediction performance using SPN-T are not comparable with SPN-T (daytime 93.13% vs. 97.94% and nighttime 94.48% vs. 97.11%). This is a reasonable result as the relative temperature between the pedestrian, and surrounding environment is not very different in daytime and nighttime scenes. Although thermal images cannot be utilized for scene prediction individually, it supplies supplementary information for the visible images to boost the performance of scene prediction. By integrating the complementary information of visible and thermal images, the SPN can generate more accurate prediction of scene conditions compared with SPN-V (daytime 98.35% vs. 97.94% and nighttime 99.75% vs. 97.11%). In addition, we show some fail cases using SPN in Fig. 5.11. The SPN may generate false scene predictions when the brightness is very low in daytime or the street lights provide high illumination in nighttime. Overall, the scene conditions can be accurately predicted utilizing the SPN by integrating visible and thermal features.

We further evaluate whether the performance of multimodal pedestrian detector can be boosted by applying our proposed scene-aware weighting mechanism by comparing the performance of the baseline TRPN and scene-aware TRPN (STRPN) models. The loss term for scene prediction defined in Eq. (5.8) and loss term for detection defined in Eq. (5.2) are combined to jointly train SPN and STRPN. Although the TRPN model is an effective framework to integrate visible and thermal streams for robust pedestrian detection, it cannot differentiate human-related features under daytime and nighttime scenes when generating detection results. With comparison, STRPN utilize the scene-aware weighting mechanism to adaptively integrate multiple scene-aware sub-networks (D-Conv-CS, N-Conv-CS, and D-Conv-BB, N-Conv-BB) to generate detection results (CS and BB).

As shown in Tables 5.7 and 5.8, the quantitative comparative results of TRPN and STRPN are conducted using the log-average miss rate (MR) as the evaluation protocols. Applying the scene-aware weighting mechanism, the detection performances of the STRPN are significantly improved comparing with TRPN for all scenes on both original and improved KAIST testing datasets. We can observe that integrating the Concat fusion scheme with a global scene-aware mechanism, instead of a single Conv-Det layer, facilitates better learning of both human-related features and correlation between visible and thermal feature maps. It also worth mentioning that such performance gain (TRPN-Concat 32.60% MR v.s. STRPN-Concat 29.62% MR on the original KAIST testing dataset [26] and TRPN-Concat 21.12% MR v.s. STRPN-Concat 17.58% MR on the improved KAIST testing dataset [37]) is achieved at the cost of small computational overhead. Measured on a single Titan X GPU, the STRPN model takes 0.24 s to process a pair of multimodal images in KAIST dataset while TRPN needs 0.22 s. These experimental results show that the global scene information can be infused into scene-aware sub-networks to boost the performance of the multimodal pedestrian detector.

5.4.5 Evaluation of Multimodal Segmentation Supervision Networks

In this section, we investigate whether the performance of multimodal pedestrian detection can be improved by incorporating the segmentation supervision scheme with STRPN. As described in Sect. 5.3.3, four different multimodal segmentation supervision (MS) models including MS-F (feature-level MS), MS (decision-level MS), SMS-F (scene-aware feature-level MS) and SMS (scene-aware decision-level MS) are combined with STRPN to build STRPN, STRPN+MS-F, STRPN+MS and STRPN+SMS-F and STRPN+SMS respectively. Multimodal segmentation supervision models generate the box-based segmentation prediction and supply the supervision to make the multimodal feature maps more distinctive. The detection results (MR) of the STRPN, STRPN+MS-F, STRPN+MS and STRPN+SMS-F and STRPN+SMS are compared in Table 5.9 and Table 5.10.

Through the joint training of segmentation supervision and pedestrian detection, all the multimodal segmentation supervision models, except for STRPN+MS-F in nighttime scene, achieve performance gains. The reason is that semantic segmentation masks provide additional supervision to facilitate the training of more sophisticated features to enable more robust detection [4]. Meanwhile, we observe that the choice of fusion architectures (feature-level or decision-level) can significantly affect the performance of multimodal pedestrian detection results. As illustrated in Tables 5.9 and 5.10, the detection results of decision-level multimodal segmentation supervision models (MS and SMS) are much better than the feature-level models (MS-F and SMS-F). The reason is that decision-level models can generate more effective supervision information by infusing visible and thermal segmentation information directly for learning human-related features for multimodal pedestrian detection. It will be our future research work to explore the optimal segmentation fusion scheme for the effective supervision of multimodal pedestrian detection. More importantly, we can observe that the performance of segmentation supervision can be significantly improved by applying the scene-aware weighting mechanism. More accurate segmentation prediction can be generated by adaptively integrating the scene-aware segmentation sub-networks. Some comparative segmentation predictions utilizing MS-F, MS, SMS-F, and SMS are shown in Fig. 5.12. The STRPN+SMS can generate more accurate segmentation predictions which supply better supervision information to facilitate the training of human-related features for multimodal pedestrian detection task.

In order to show the improvements gains achieved by different scene-aware modules, we visualize the feature maps of TRPN, STRPN, and STRPN+SMS in Fig. 5.13. It can be observed that STRPN can generate more distinctive human-related features by incorporating scene-aware weighting mechanism into TRPN for better learning of multimodal human-related features. More importantly, further improvements can be achieved by STRPN+SMS through the scene-aware segmentation modules to supervise the learning of multimodal human-related features.

5.4.6 Comparison with State-of-the-Art Multimodal Pedestrian Detection Methods

We compare the designed STRPN and STRPN+SMS models with the current state-of-the-art multimodal pedestrian detection algorithms: ACF+T+THOG [26], Halfway Fusion [28] and Fusion RPN+BDT [30]. The log–log figure of MR vs. FPPI is plotted for performance comparison in Fig. 5.14. It can be observed that our proposed STRPN+SMS achieves the best detection accuracy (26.37% MR on the original KAIST testing dataset [26] and 14.95% MR on the improved KAIST testing dataset [37]) in all-day scenes (see Fig. 5.15). The performance gain on the improved KAIST testing dataset achieves a relative improvement rate of 18% compared with the best of current state-of-the-art algorithm Fusion RPN+BDT (18.23% MR on the improved KAIST testing dataset [37]). In addition, the detection performance of our proposed STRPN is comparable with the current state-of-art models. Furthermore, some detection results of the Fusion RPN+BDT and STRPN+SMS are visualized for qualitative comparison as shown in Fig. 5.16. We can observe that our proposed STRPN+SMS model can generate more accurate detection results in both daytime and nighttime scene conditions.

The computation efficiency of STRPN, STRPN+SMS and the current state-of-the-art methods [28,30] are illustrated in Table 5.11. Every method is executed 1000 times to compute the averaged runtime. We can observe that our proposed STRPN+SMS results in least runtime comparing with the current state-of-the-art methods [28,30] (STRPN+SMS 0.25 s vs. Halfway Fusion 0.40 s vs. Fusion RPN+BDT 0.80 s). The reason is that both the extra Fast R-CNN module [18] utilized in the Halfway Fusion model and the boosted decision trees [14] algorithm utilized in Fusion RPN+BDT model significantly decrease the computation efficiency. It should be noticed that our proposed scene-aware architectures can significantly boost the multimodal pedestrian detection results while only cause a small computational overhead (TRPN 0.24 s vs. STRPN 0.24 s vs. STRPN+SMS 0.25 s).