12.2.1 Generalized Distillation

Within the generalized distillation framework, our model is both related to the privileged information theory [9], considering that the extra modality (depth, in this case) is only used at training time, and, mostly, to the distillation framework. Indeed the core mechanism that our model uses to learn the hallucination network is derived from a distillation loss. More specifically, the supervision information provided by the teacher network (in this case, the network processing the depth data stream) is distilled into the hallucination network leveraging teacher's soft predictions and hard ground-truth labels in the loss function.

In this context, the closest works to our proposal are [21] and [10]. Luo et al. [21] addressed a similar problem to ours, where the model is first trained on several modalities (RGB, depth, joints and infrared), but tested only in one. The authors propose a graph-based distillation method that is able to distill information from all modalities at training time, while also passing through a validation phase on a subset of modalities. This showed to reach state-of-the-art results in action recognition and action detection tasks. Our work substantially differs from [21] since we benefit from a hallucination mechanism, consisting in an auxiliary network trained using the guidance distilled by the teacher network (that processes the depth data stream in our case). This mechanism allows the model to learn to emulate the presence of the missing modality at test time.

The work of Hoffman et al. [10] introduced a model to hallucinate depth features from RGB input for object detection task. While the idea of using a hallucination stream is similar to the one thereby presented, the mechanism used to learn it is different. In [10], the authors use a Euclidean loss between depth and hallucinated feature maps that is part of the total loss along with more than ten classification and localization losses, which makes its effectiveness very dependent on hyperparameter tuning to balance the different values, as the model is trained jointly in one step by optimizing the aforementioned composite loss. Differently, we propose a loss inspired to the distillation framework that not only uses the Euclidean distance between feature maps, and the one-hot labels, but also leverages soft predictions from the depth network. Moreover, we encourage the hallucination learning by design, by using cross-stream connections (see Sect. 12.3). This showed to largely improve the performance of our model with respect to the one-step learning process proposed in [10].

12.2.2 Multimodal Video Action Recognition

Video action recognition has a long and rich field of literature, spanning from classification methods using handcrafted features [22] [23] [24] [25] to modern deep learning approaches [26] [27] [28] [16], using either RGB-only or various multimodal data. Here, we focus on some of the more relevant works in multimodal video action recognition, including state-of-the-art methods considering the NTU RGB+D dataset, as well as architectures related to our proposed model.

The two-stream model introduced by Simonyan and Zisserman [15] is a landmark on video analysis, and since then has inspired a series of variants that achieved state-of-the-art performance on diverse datasets. This architecture is composed of an RGB and an optical flow stream, which are trained separately, and then fused at the prediction layer. The current state of the art in video action recognition [16] is inspired by such model, featuring 3D convolutions to deal with the temporal dimension, instead of the original 2D ones. In [17], a further variation of the two-stream approach is proposed, which models spatiotemporal features by injecting the motion stream's signal into the residual unit of the appearance stream. The idea of combining the two streams have also been explored previously by the same authors in [29].

Instead, in [5], the authors explore the complementary properties of RGB and depth data, taking the NTU RGB+D dataset as testbed. This work designed a deep autoencoder architecture and a structured sparsity learning machine, and showed to achieve state-of-the-art results for action recognition. Liu et al. [6] also use RGB and depth complementary information to devise a method for viewpoint invariant action recognition. Here, dense trajectories from RGB data are first extracted, which are then encoded in viewpoint invariant deep features. The RGB and depth features are then used as a dictionary to predict the test label.

All these previous methods exploited the rich information conveyed by the multimodal data to improve recognition. Our work, instead, proposes a fully convolutional model that exploits RGB and depth data at training time only, and uses exclusively RGB data as input at test time, reaching performance comparable to those utilizing the complete set of modalities in both stages.

12.3.1 Cross-stream Multiplier Networks

Typically in two-stream architectures, the two streams are trained separately and the predictions are fused with a late fusion mechanism [15] [17]. Such models use as input appearance (RGB) and motion (optical flow) data, which are fed separately into each stream, both in training and testing. Instead, in this work we use RGB and depth frames as inputs for training, but only RGB at test time, as already discussed (Fig. 12.2).

We use the ResNet-50-based [30] [31] model proposed in [17] as baseline architecture for each stream block of our model. In that paper, Feichtenhofer et al. proposed to connect the appearance and motion streams with multiplicative connections at several layers, as opposed to previous models which would only interact at the prediction layer. Such connections are depicted in Fig. 12.2 with the blue arrows. Fig. 12.3 illustrates this mechanism at a given layer of the multiple stream architecture, while it is actually implemented at the four convolutional layers of the Resnet-50 model. The underlying intuition is that these connections enable the model to learn better spatiotemporal representations, and help to distinguish between identical actions that require the combination of appearance and motion features. Originally, the cross-stream connections consisted of the injection of the motion stream signal into the other stream's residual unit, without affecting the skip path. ResNet's residual units are formally expressed as:

${\mathbf{x}}_{l+1}=f(h({\mathbf{x}}_l)+F({\mathbf{x}}_l,{\mathcal{W}}_l)),$

where ${\mathbf{x}}_l$ and ${\mathbf{x}}_{l+1}$ are lth layer's input and output, respectively, F represents the residual convolutional layers defined by weights ${\mathcal{W}}_l$, $h({\mathbf{x}}_l)$ is an identity mapping and f is a ReLU non-linearity. The cross-streams connections are then defined as

${\mathbf{x}}_{l+1}^a=f({\mathbf{x}}_l^a)+F({\mathbf{x}}_l^a\odot f({\mathbf{x}}_l^m),{\mathcal{W}}_l),$

where ${\mathbf{x}}^a$ and ${\mathbf{x}}^m$ are the appearance and motion streams, respectively, and ⊙ is the element-wise multiplication operation. Such mechanism implies a spatial alignment between both feature maps, and therefore between both modalities. This alignment comes for free when using RGB and optical flow, since the latter is computed from the former in a way that spatial arrangement is preserved. However, this is an assumption we cannot generally make. For instance, depth and RGB are often captured from different sensors, likely resulting in spatially misaligned frames. The alignment procedure, described in Sect. 12.4.2, is part of the pre-processing phase and it refers uniquely to the NTU RGB+D dataset.

Temporal convolutions. In order to augment the model temporal support, we implement 1D temporal convolutions in the second residual unit of each ResNet layer (as in [17]), as illustrated in Fig. 12.3. The weights $W_l\in {\mathbb{R}}^{1\times 1\times 3\times C_l\times C_l}$ are convolutional filters initialized as identity mappings at feature level, and centered in time, and $C_l$ is the number of channels in layer l.

Very recently in [32], the authors explored various network configurations using temporal convolutions, comparing several different combinations for the task of video classification. This work suggests that decoupling 3D convolutions into 2D (spatial) and 1D (temporal) filters is the best setup in action recognition tasks, producing best accuracies. The intuition for the latter setup is that factorizing spatial and temporal convolutions in two consecutive convolutional layers eases training of the spatial and temporal tasks (also in line with [33]).

12.3.2 Hallucination Stream

We also introduce and learn a hallucination network [10], using a new learning paradigm, loss function and interaction mechanism. The hallucination stream network has the same architecture as the appearance and depth stream models.

This network receives RGB as input, and is trained to “imitate” the depth stream at different levels, i.e. at feature and prediction layers. In this work, we explore several ways to implement such learning paradigm, including both the training procedure and the loss, and how they affect the overall performance of the model.

In [10] it is proposed a regression loss between the hallucination and depth feature maps, defined as:

$L_{\mathit{hall}}(l)={\lambda }_l{\Vert \sigma (A_l^d)-\sigma (A_l^h)\Vert }_2^2,$

where σ is the sigmoid function, and $A_l^d$ and $A_l^h$ are the l-th layer activations of depth and hallucination network. This Euclidean loss forces both activation maps to be similar. In [10], this loss is weighted along with another ten classification and localization loss terms, making it hard to balance the total loss. One of the main motivations behind the proposed new staged learning paradigm, described in Sect. 12.3.3, is to avoid the inefficient, heuristic-based tweaking of so many loss weights, a.k.a. hyperparameter tuning.

Instead, we adopt an approach inspired by the generalized distillation framework [14], in which a student model $f_s\in {\mathcal{F}}_s$ distills the representation $f_t\in {\mathcal{F}}_t$ learned by the teacher model. This is formalized as

$f_s=\underset{f\in {\mathcal{F}}_s}{\mathrm{arg\hspace{0.17em}min}}\frac{1}{n}\sum _{i=1}^n{L}_{GD}(i),n=1,...,N$

where N is the number of examples in the dataset. The generalized distillation loss is so defined as:

$L_{GD}(i)=(1-\lambda )\ell (y_i,\varsigma (f(x_i)))+\lambda \ell (s_i,\varsigma (f(x_i))),\lambda \in [0,1]f_s\in {\mathcal{F}}_s,$

ς is the softmax operator and $s_i$ is the soft prediction from the teacher network:

$s_i=\varsigma (f_t(x_i)/T),T>0.$

The parameter λ in Eq. (12.3) allows one to tune the loss by giving more importance either to imitating ground-truth hard or soft teacher targets, $y_i$ and $s_i$, respectively. This mechanism indeed allows the transfer of information from the depth (teacher) to the hallucination (student) network. The temperature parameter T in Eq. (12.4) allows one to smooth the probability vector predicted by the teacher network. The intuition is that such smoothing may expose relations between classes that would not easily be revealed in raw predictions, further facilitating the distillation by the student network $f_s$.

We suggest that both Euclidean and generalized distillation losses are indeed useful in the learning process. In fact, by encouraging the network to decrease the distance between hallucinated and true depth feature maps, it can help to distill depth information encoded in the generalized distillation loss. Thus, we formalize our final loss function as follows:

$L=(1-\alpha )L_{GD}+\alpha L_{\mathit{hall}},\alpha \in [0,1],$

where α is a parameter balancing the contributions of the two loss terms during training. The parameters λ, α and T are estimated by utilizing a validation set and discussed in Sect. 12.4.2.

In summary, the generalized distillation framework proposes to use the student-teacher framework introduced in the distillation theory to extract knowledge from the privileged information source. We explore this idea by proposing a new learning paradigm to train a hallucination network using privileged information, which we will describe in the next section. In addition to the loss functions introduced above, we also allow the teacher network to share information with the student network by design, through the cross-stream multiplicative connections. We test how all these possibilities affect the model's performance in the experimental section through an extensive ablation study.

12.3.3 Training Paradigm

In general, the proposed training paradigm, illustrated in Fig. 12.2, is divided in two core parts: the first part (steps 1 and 2 in the figure) focuses on learning the teacher network $f_t$, leveraging RGB and depth data (the privileged information in this case); the second part (steps 3 and 4 in the figure) focuses on learning the hallucination network, referred to as student network $f_s$ in the distillation framework, using the general hallucination loss defined in Eq. (12.5).

The first training step consists in training both streams separately, which is a common practice in two-stream architectures. Both depth and appearance streams are trained minimizing cross-entropy, after being initialized with a pre-trained ImageNet model for all experiments. Temporal kernels are initialized as $[0,1,0]$, i.e. only information on the central frame is used at the beginning—this eventually changes as the training continues. As in [34], depth frames are encoded into color images using a jet colormap.

The second training step is still focused on further training the teacher model. Since the model trained in this step has the architecture and capacity of the final one, and has access to both modalities, its performance represents an upper bound for the task we are addressing. This is one of the major differences between our approach and the one used in [10]: by decoupling the teacher learning phase with the hallucination learning, we are able to both learn a better teacher and a better student, as we will show in the experimental section.

In the third training step, we focus on learning the hallucination network from the teacher model, i.e., the depth stream network just trained. Here, the weights of the depth network are frozen, while receiving in input depth data. Instead, the hallucination network, receiving in input RGB data, is trained with the loss defined in Eq. (12.5), while also receiving feedback from the cross-stream connections from the depth network. We found that this helps the learning process.

In the fourth and last step, we carry out fine tuning of the whole model, composed of the RGB and the hallucination streams. This step uses RGB-only as input, and it also precisely resembles the setup used at test time. The cross-stream connections inject the hallucinated signal into the appearance RGB stream network, resulting in the multiplication of the hallucinated feature maps and the RGB feature maps. The intuition is that the hallucination network has learned to inform the RGB model where the action is taking place, similarly to what the depth model would do with real depth data.

12.4.1 Datasets

We evaluate our method on three datasets, while the ablation study is performed only on the NTU RGB+D dataset. Our model is initialized with ImageNet pre-trained weights and trained and evaluated on the NTU RGB+D dataset. We later fine-tune this model on each of the two smaller datasets for the corresponding evaluation experiments.

NTU RGB+D [18]. This is the largest public dataset for multimodal video action recognition. It is composed of 56,880 videos, available in four modalities: RGB videos, depth sequences, infrared frames, and 3D skeleton data of 25 joints. It was acquired with a Kinect v2 sensor in 80 different viewpoints, and includes 40 subjects performing 60 distinct actions. We follow the two evaluation protocols originally proposed in [18], which are cross-subject and cross-view. As in the original paper, we use about 5% of the training data as validation set for both protocols, in order to select the parameters λ, α and T. In this work, we use only RGB and depth data. The masked depth maps are converted to a three channel map via a jet mapping, as in [34].

UWA3DII [19]. This dataset consists on 1075 samples of RGB, depth and skeleton sequences. It features 10 subjects performing 30 actions captured in 5 different views.

Northwestern-UCLA [20]. Similarly to the other datasets, it provides RGB, depth and skeleton sequences for 1475 samples. It features 10 subjects performing 10 actions captured in 3 different views.

12.4.2 Pre-processing and Alignment of RGB and Depth Frames

The multiplicative cross-stream connections require both RGB and depth frames to be spatially aligned, since they are element-wise operations over the feature maps. The NTU RGB+D dataset is acquired using a Kinect sensor, which result in different dimensions and aspect ratios for RGB and depth frames. Fortunately, this dataset provides the joints' spatial coordinates in every RGB and depth frames, ${\mathit{rgb}}_{x,y}$ and ${\mathit{depth}}_{x,y}$, respectively, which we use to align both modalities. Because the other two smaller datasets do not provide such information, this alignment procedure is only applicable to the NTU RGB+D dataset. Consequently, experiments using the other datasets refer to the model with no cross-stream connections.

For every frame of a given video, we first compute the ratio ${\mathit{ratio}}_x^{A,B}=({\mathit{rgb}}_x^A-{\mathit{rgb}}_x^B)/({{\mathit{depth}}_x}^A-{\mathit{depth}}_x^B)$ $\forall A,B\in S$, using all depth and RGB x coordinates from the frame's well-tracked joints set S, and similarly for the y dimension. The video aspect ratio is then calculated as the mean between the median aspect ratio for x and the median aspect ratio for y dimensions. The RGB frames of a given video are scaled according to this ratio. Finally, both RGB and depth frames are overlaid by aligning both skeletons, and the intersection is cropped on both modalities. The cropped sections are then re-scaled according to the network's input dimension, in this case $224\times 224$.

Similarly to what was done in [17], we sample 5 frames evenly spaced in time for each video, both for training and testing. Sampling random frames for training results in similar accuracy values. For training, we also flip horizontally the video frames with probability $P=0.5$.

12.4.3 Hyperparameters and Validation Set

After validation, we have selected the following set of hyperparameters: $\alpha =0.5$, $\lambda =0.5$, $T=10$; slightly different values do not show a significant change in performance. The networks are optimized using ADAM [35] with default learning rate, except for the fine-tuning steps where the learning rate was decreased by a factor of 10.

Regarding the NTU RGB+D dataset, the validation set is not defined in the original paper where the dataset is presented [18]. For the sake of experiments reproducibility, we explain here how we defined the validation set. For the cross-subject protocol, we choose the subject #1 (from the training set), which corresponds to around 5% of the training set. For the cross-view protocol, we do the following: 1) create a dictionary of sorted videos for each key=action (from the training set); 2) set numpy random seed equal to 0; 3) sample 31 videos using numpy.random.choice for each action, which in the end will correspond to around 5% of the training set.

12.4.4 Ablation Study

In this section we discuss the results of the experiments carried out to understand the contribution of each part of the model and of the training procedure. Table 12.1 reports performances at the several training steps, different losses and model configurations.

Rows #1 and #2 refer to the first training step, where depth and RGB streams are trained separately. We note that the depth stream network provides better performance with respect to the RGB one, as expected.

The second part of the table (Rows #3–5) shows the results using Hoffman et al.'s method [10] – i.e., adopting a model initialized with the pre-trained networks from the first training step, and the hallucination network initialized using the depth network. Row #3 refers to the original paper [10] (i.e., using the loss $L_{\mathit{hall}}$, Eq. (12.1)), and rows #4 and #5 refer to the training using the proposed losses $L_{GD}$ and L, in Eqs. (12.3) and (12.5), respectively. It can be noticed that the accuracies achieved using the proposed loss functions overcome that obtained in [10] by a significant margin (about 6% in the case of the total loss L).

The third part of the table (rows #6–9) reports performances after the training step 2. Rows #6 and #7 refer to the accuracy provided by depth and RGB stream networks belonging to the model of row #8, taken individually. The final model constitutes the upper bound for our hallucination model, since it uses RGB and depth for training and testing. Performances obtained by the model in row #8 and #9, with and without cross-stream connections, respectively, are the highest in absolute since using both modalities (around 78–79% for the cross-subject and 81–82% for the cross-view protocols, respectively), largely outperforming the accuracies obtained using only one modality (in rows #6 and #7).

The fourth part of the table (rows #10–14) shows results for our hallucination network after the several variations of learning processes, different losses and with and without cross-stream connections.

Finally, the last row, #15, reports results after the last fine-tuning step which further narrows the gap with the upper bound.

12.4.5 Inference with Noisy Depth

Suppose that in a real test scenario we can only access unreliable sensors which produce noisy depth data. The question we now address is: to which extent can we trust such noisy data? In other words, at which level of noise does it become favorable to hallucinate the depth modality with respect to using the full teacher model (step 2) with noisy depth data?

The depth sensor used in the NTU dataset (Kinect), is an IR emitter coupled with an IR camera, and has very complex noise characterization comprising at least six different sources [36]. It is beyond the scope of this work to investigate noise models affecting the depth channel, so, for our analysis we choose the most common one, i.e., the multiplicative speckle noise. Hence, we inject Gaussian noise in the depth images I in order to simulate speckle noise: $I=I⁎n$, $n\sim \mathcal{N}(1,\sigma )$. Table 12.2 shows how performances of the network degrade when depth is corrupted with such Gaussian noise with increasing variance (NTU cross-view protocol only). Results show that accuracy significantly decreases w.r.t. to the one guaranteed by our hallucination model (77.21% – row #15 in Table 12.1), even with low noise variance. This means, in conclusion, that training a hallucination network is an effective way not only to obviate to the problem of a missing modality, but also to deal with noise affecting the input data channel.

Table 12.2

Accuracy of the model tested with clean RGB and noisy depth data. Accuracy of the proposed hallucination model, i.e. with no depth at test time, is 77.21%.

σ2	no noise	10−3	10−2	10−1	100	101	void
Accuracy	81.43%	81.34%	81.12%	76.85%	62.47%	51.43%	14.24%

12.4.6 Comparison with Other Methods

Table 12.3 compares performances of different methods on the various datasets. The standard performance measure used for this task and datasets is classification accuracy, estimated according to the protocols (training and testing splits) reported in the respective works we are comparing with.

The first part of the table (indicated by × symbol) refers to unsupervised methods, which achieve surprisingly high results even without relying on labels in learning representations.

The second part refers to supervised methods (indicated by △), divided according to the modalities used for training and testing. Here, we list the performance of the separate RGB and depth streams trained in step 1, as a reference. We expect our final model to perform better than the one trained on RGB-only, whose accuracy constitutes a lower bound for our student network. The values reported for our step 1 models for UWA3DII and NW-UCLA datasets refer to the fine-tuning of our NTU model. We have experimented training using pre-trained ImageNet weights, which led from 20% to 30% less accuracy. We also propose our baseline, consisting in the teacher model trained in step 2. Its accuracy represents an upper bound for the final model, which will not rely on depth data at test time.

The last part of the table (indicated by □) reports our model's performances at two different stages together with the other privileged information methods [10] [11]. For all datasets and protocols, we can see that our privileged information approach outperforms [10], which is the only fair direct comparison we can make (same training & test data). Besides, as expected, our final model performs better than “Ours – RGB model, step 1” since it exploits more data at training time, and worse than “ Ours – step 2”, since it exploits less data at test time. Other RGB+D methods perform better (which is comprehensible since they rely on RGB+D in both training and test) but not by a large margin. The method by Garcia et al. [11] is similar to these two in the sense that is also uses a hallucination network to cope with the missing depth modality. However, it takes a different look to it, by learning the hallucination stream through an adversarial strategy.

12.4.7 Inverting Modalities – RGB Distillation

The results presented in Table 12.4 address the opposite case of what is studied in the rest of the chapter, i.e., the case when RGB data is missing. In this case, the hallucination stream distills knowledge from the RGB stream in step 3 (Fig. 12.2).

We observe that the performance of the final model degrades by almost 1%, 76.41% vs. 77.21% (cf. line 15 of Table 12.2). A more consistent setting would be to modify the model, inverting the cross-stream connections in steps 3 and 4, thus having information flowing again from depth to RGB.