2.3.1 Auto-Encoder

An auto-encoder is a neural network that consists of an encoder network and a decoder network (Fig. 2.2). Letting $\mathbf{x}\in {\mathbb{R}}^d$ be an input variable, which is a concatenation of pixel values when managing images, an encoder maps it to a latent variable $\mathbf{z}\in {\mathbb{R}}^r$, where r is usually much smaller than d. A decoder maps z to the output ${\mathbf{x}}^{\prime }\in {\mathbb{R}}^d$, which is the reconstruction of the input x. The encoder and decoder are trained so as to minimize the reconstruction errors such as the following squared errors:

${\mathcal{L}}_{\mathrm{AE}}={\mathbb{E}}_{\mathbf{x}}[{\Vert \mathbf{x}-{\mathbf{x}}^{\prime }\Vert }^2].$

The purpose of an auto-encoder is typically dimensionality reduction, or in other words, unsupervised feature / representation learning. Recently, as well as the encoding process, more attention has been given to the decoding process which has the ability of data generation from latent variables.

2.3.2 Variational Auto-Encoder (VAE)

The variational auto-encoder (VAE) [34,35] regards an auto-encoder as a generative model where the data is generated from some conditional distribution $p(\mathbf{x}|\mathbf{z})$. Letting ϕ and θ denote the parameters of the encoder and the decoder, respectively, the encoder is described as a recognition model $q_{\varphi }(\mathbf{z}|\mathbf{x})$, whereas the decoder is described as an approximation to the true posterior $p_{\theta }(\mathbf{x}|\mathbf{z})$. The marginal likelihood of an individual data point ${\mathbf{x}}_i$ is written as follows:

$\log p_{\theta }({\mathbf{x}}_i)=D_{\mathrm{KL}}(q_{\varphi }(\mathbf{z}|{\mathbf{x}}_i)||p_{\theta }(\mathbf{z}|{\mathbf{x}}_i))+\mathcal{L}(\theta ,\varphi ;{\mathbf{x}}_i).$

Here, $D_{\mathrm{KL}}$ stands for the Kullback–Leibler divergence. The second term in this equation is called the (variational) lower bound on the marginal likelihood of data point i, which can be written as

$\mathcal{L}(\theta ,\varphi ;{\mathbf{x}}_i)=-D_{\mathrm{KL}}(q_{\varphi }(\mathbf{z}|{\mathbf{x}}_i)||p_{\theta }(\mathbf{z}))+{\mathbb{E}}_{\mathbf{z}\sim q_{\varphi }(\mathbf{z}|{\mathbf{x}}_i)}[\log p_{\theta }({\mathbf{x}}_i|\mathbf{z})].$

In the training process, the parameters ϕ and θ are optimized so as to minimize the total loss $-{\sum }_i\mathcal{L}(\theta ,\varphi ;{\mathbf{x}}_i)$, which can be written as

${\mathcal{L}}_{\mathrm{VAE}}=D_{\mathrm{KL}}(q_{\varphi }(\mathbf{z}|\mathbf{x})||p_{\theta }(\mathbf{z}))-{\mathbb{E}}_{\mathbf{z}\sim q_{\varphi }(\mathbf{z}|\mathbf{x})}[\log p_{\theta }(\mathbf{x}|\mathbf{z})].$

Unlike the original auto-encoder which does not consider the distribution of latent variables, VAE assumes the prior over the latent variables such as the centered isotropic multivariate Gaussian $p_{\theta }(\mathbf{z})=\mathcal{N}(\mathbf{z};\mathbf{0},\mathbf{I})$. In this case, we can let the variational approximate posterior be a multivariate Gaussian with a diagonal covariance structure:

$\log q_{\varphi }(\mathbf{z}|{\mathbf{x}}_i)=\log \mathcal{N}(\mathbf{z};{\mathit{\mu }}_i,{\mathit{\sigma }}_i^2\mathbf{I}),$

where the mean ${\mathit{\mu }}_i$ and the standard deviation ${\mathit{\sigma }}_i$ of the approximate posterior are the outputs of the encoder (Fig. 2.3). In practice, the latent variable ${\mathbf{z}}_i$ for data point i is calculated as follows:

${\mathbf{z}}_i={\mathit{\mu }}_i+{\mathit{\sigma }}_i\cdot \mathit{ϵ},\text{where}\mathit{ϵ}\sim \mathcal{N}(\mathbf{0},\mathbf{I}).$

2.3.3 Generative Adversarial Network (GAN)

The generative adversarial network (GAN) [36] is one of the most successful framework for data generation. It consists of two networks: a generator G and a discriminator D (shown in Fig. 2.4), which are jointly optimized in a competitive manner. Intuitively, the aim of the generator is to generate a sample ${\mathbf{x}}^{\prime }$ from a random noise vector z that can fool the discriminator, i.e., make it believe ${\mathbf{x}}^{\prime }$ is real. The aim of the discriminator is to distinguish a fake sample ${\mathbf{x}}^{\prime }$ from a real sample x. They are simultaneously optimized via the following two-player minimax game:

$\underset{G}{\min }\underset{D}{\max }{\mathcal{L}}_{\mathrm{GAN}},\text{where}{\mathcal{L}}_{\mathrm{GAN}}={\mathbb{E}}_{\mathbf{x}\sim p_{\mathrm{data}}}[\log D(\mathbf{x})]+{\mathbb{E}}_{\mathbf{z}\sim p(\mathbf{z})}[\log (1-D(G(\mathbf{z})))].$

From the perspective of a discriminator D, the objective function is a simple cross-entropy loss function for the binary categorization problem. A generator G is trained so as to minimize $\log (1-D(G(\mathbf{z})))$, where the gradients of the parameters in G can be back-propagated through the outputs of (fixed) D. In spite of its simplicity, GAN is able to train a reasonable generator that can output realistic data samples.

Deep convolutional GAN (DCGAN) [42] was proposed to utilize GAN for generating natural images. The DCGAN generator consists of a series of four fractionally-strided convolutions that convert a 100-dimensional random vector (in uniform distribution) into a $64\times 64$ pixel image. The main characteristics of the proposed CNN architecture are threefold. First, they used the all convolutional net [43] which replaces deterministic spatial pooling functions (such as maxpooling) with strided convolutions. Second, fully connected layers on top of convolutional features were eliminated. Finally, batch normalization [44], which normalize the input to each unit to have zero mean and unit variance, was used to stabilize learning.

DCGAN still has the limitation in resolution of generated images. StackGAN [45] was proposed to generate high-resolution (e.g., $256\times 256$) images by using two-stage GANs. Stage-I GAN generator in StackGAN generates low-resolution ($64\times 64$) images, which are input into Stage-II GAN generator that outputs high-resolution ($256\times 256$) images. Discriminators in Stage-I and Stage-II distinguish output images and real images of corresponding resolutions, respectively. More stable StackGAN++ [46] was later proposed, which consists of multiple generators and multiple discriminators arranged in a tree-like structure.

2.3.4 VAE-GAN

VAE-GAN [37] is a combination of VAE [34,35] (Sect. 2.3.2) and GAN [36] (Sect. 2.3.3) as shown in Fig. 2.5. The VAE-GAN model is trained with the following criterion:

${\mathcal{L}}_{\mathrm{VAE}-\mathrm{GAN}}={\mathcal{L}}_{\mathrm{VA}{\mathrm{E}}^{⁎}}+{\mathcal{L}}_{\mathrm{GAN}},$

${\mathcal{L}}_{\mathrm{VA}{\mathrm{E}}^{⁎}}=D_{\mathrm{KL}}(q_{\varphi }(\mathbf{z}|\mathbf{x})||p_{\theta }(\mathbf{z}))-{\mathbb{E}}_{\mathbf{z}\sim q_{\varphi }(\mathbf{z}|\mathbf{x})}[\log p_{\theta }(D(\mathbf{x})|\mathbf{z})],$

where ${\mathcal{L}}_{\mathrm{GAN}}$ is the same as that in Eq. (2.7). Note that VAE-GAN replaces the element-wise error measures in VAE (i.e., the second term in Eq. (2.4)) by the similarity measures trained through a GAN discriminator (i.e., the second term in Eq. (2.9)). Here, a Gaussian observation model for $D(\mathbf{x})$ with mean $D({\mathbf{x}}^{\prime })$ and identity covariance is introduced:

$p_{\theta }(D(\mathbf{x})|\mathbf{z})=\mathcal{N}(D(\mathbf{x})|D({\mathbf{x}}^{\prime }),\mathbf{I}).$

In this way, VAE-GAN improves the sharpness of output images, in comparison to VAE.

It has been said that GANs could suffer from mode collapse; the generator in GAN learns to produce samples with extremely low variety. VAE-GAN mitigates this problem by introducing the prior distribution for z. The learned latent space is therefore continuous and is able to produce meaningful latent vectors, e.g., visual attribute vectors corresponding to eyeglasses and smiles in face images.

2.3.5 Adversarial Auto-Encoder (AAE)

The adversarial auto-encoder (AAE) [38] was proposed to introduce the GAN framework to perform variational inference using auto-encoders. Similarly to VAE [34,35] (Sect. 2.3.2), AAE is a probabilistic autoencoder that assumes the prior over the latent variables. Let $p(\mathbf{z})$ be the prior distribution we want to impose, $q(\mathbf{z}|\mathbf{x})$ be an encoding distribution, $p(\mathbf{x}|\mathbf{z})$ be the decoding distribution, and $p_{\mathrm{data}}(\mathbf{x})$ be the data distribution. An aggregated posterior distribution of $q(\mathbf{z})$ is defined as follows:

$q(\mathbf{z})={\int }_{\mathbf{x}}q(\mathbf{z}|\mathbf{x})p_{\mathrm{data}}(\mathbf{x})d\mathbf{x}.$

The regularization of AAE is to match the aggregated posterior $q(\mathbf{z})$ to an arbitrary prior $p(\mathbf{z})$. While the auto-encoder attempts to minimize the reconstruction error, an adversarial network guides $q(\mathbf{z})$ to match $p(\mathbf{z})$ (Fig. 2.6). The cost function of AAE can be written as

${\mathcal{L}}_{\mathrm{AAE}}={\mathcal{L}}_{\mathrm{AE}}+{\mathbb{E}}_{q_{\varphi }(\mathbf{z}|\mathbf{x})}[\log D(\mathbf{z})]+{\mathbb{E}}_{\mathbf{z}\sim p(\mathbf{z})}[\log (1-D(\mathbf{z}))],$

where ${\mathcal{L}}_{\mathrm{AE}}$ represents the reconstruction error defined in Eq. (2.1).

In contrast that VAE uses a KL divergence penalty to impose a prior distribution on latent variables, AAE uses an adversarial training procedure to encourage $q(\mathbf{z})$ to match to $p(\mathbf{z})$. An important difference between VAEs and AAEs lies in the way of calculating gradients. VAE approximates the gradients of the variational lower bound through the KL divergence by Monte Carlo sampling, which needs the access to the exact functional form of the prior distribution. On the other hand, AAEs only need to be able to sample from the prior distribution for inducing $q(\mathbf{z})$ to match $p(\mathbf{z})$. AAEs therefore can impose arbitrarily complicated distributions (e.g., swiss roll distribution) as well as black-box distributions.

2.3.6 Adversarial Variational Bayes (AVB)

Adversarial variational Bayes (AVB) [39] also uses adversarial training for VAE, which enables the usage of arbitrarily complex inference models. As shown in Fig. 2.7, the inference model (i.e. encoder) in AVB takes the noise ϵ as additional input. For its derivation, the optimization problem of VAE in Eq. (2.4) is rewritten as follows:

$\underset{\theta }{\max }\underset{\varphi }{\max }{\mathbb{E}}_{p_{\mathrm{data}}(\mathbf{x})}[-D_{\mathrm{KL}}(q_{\varphi }(\mathbf{z}|\mathbf{x})||p(\mathbf{z}))+{\mathbb{E}}_{q_{\varphi }(\mathbf{z}|\mathbf{x})}\log p_{\theta }(\mathbf{x}|\mathbf{z})]=\underset{\theta }{\max }\underset{\varphi }{\max }{\mathbb{E}}_{p_{\mathrm{data}}(\mathbf{x})}{\mathbb{E}}_{q_{\varphi }(\mathbf{z}|\mathbf{x})}(\log p(\mathbf{z})-\log q_{\varphi }(\mathbf{z}|\mathbf{x})+\log p_{\theta }(\mathbf{x}|\mathbf{z})).$

This can be optimized by stochastic gradient descent (SGD) using the reparameterization trick [34,35], when considering an explicit $q_{\varphi }(\mathbf{z}|\mathbf{x})$ such as the centered isotropic multivariate Gaussian. For a black-box representation of $q_{\varphi }(\mathbf{z}|\mathbf{x})$, the following objective function using the discriminator $D(\mathbf{x},\mathbf{z})$ is considered:

$\underset{D}{\max }{\mathbb{E}}_{p_{\mathrm{data}}(\mathbf{x})}{\mathbb{E}}_{q_{\varphi }(\mathbf{z}|\mathbf{x})}\log \sigma (D(\mathbf{x},\mathbf{z}))+{\mathbb{E}}_{p_{\mathrm{data}}(\mathbf{x})}{\mathbb{E}}_{p(\mathbf{z})}\log (1-\sigma (D(\mathbf{x},\mathbf{z}))),$

where $\sigma (t)\equiv {(1+e^{-t})}^{-1}$ denotes the sigmoid function. For $p_{\theta }(\mathbf{x}|\mathbf{z})$ and $q_{\varphi }(\mathbf{z}|\mathbf{x})$ fixed, the optimal discriminator $D^{⁎}(\mathbf{x},\mathbf{z})$ according to Eq. (2.14) is given by

$D^{⁎}(\mathbf{x},\mathbf{z})=\log q_{\varphi }(\mathbf{z}|\mathbf{x})-\log p(\mathbf{z}).$

The optimization objective in Eq. (2.13) is rewritten as

$\underset{\theta }{\max }\underset{\varphi }{\max }{\mathbb{E}}_{p_{\mathrm{data}}(\mathbf{x})}{\mathbb{E}}_{q_{\varphi }(\mathbf{z}|\mathbf{x})}(-D^{⁎}(\mathbf{x},\mathbf{z})+\log p_{\theta }(\mathbf{x}|\mathbf{z})).$

Using the reparameterization trick [34,35], this can be rewritten with a suitable function ${\mathbf{z}}_{\varphi }(\mathbf{x},\mathit{ϵ})$ in the form

$\underset{\theta }{\max }\underset{\varphi }{\max }{\mathbb{E}}_{p_{\mathrm{data}}(\mathbf{x})}{\mathbb{E}}_{\mathit{ϵ}}(-D^{⁎}(\mathbf{x},{\mathbf{z}}_{\varphi }(\mathbf{x},\mathit{ϵ}))+\log p_{\theta }(\mathbf{x}|{\mathbf{z}}_{\varphi }(\mathbf{x},\mathit{ϵ}))).$

The gradients of Eq. (2.17) w.r.t. ϕ and θ as well as the gradient of Eq. (2.14) w.r.t. the parameters for D are computed to apply SGD updates. Note that AAE [38] (Sect. 2.3.5) can be regarded as an approximation to AVB, where $D(\mathbf{x},\mathbf{z})$ is restricted to the class of functions that do not depend on x (i.e., $D(\mathbf{x},\mathbf{z})\equiv D(\mathbf{z})$).

2.3.7 ALI and BiGAN

The adversarially learned inference (ALI) [40] and the Bidirectional GAN (BiGAN) [41], which were proposed at almost the same time, have the same model structure shown in Fig. 2.8. The model consists of two generators $G_{\mathbf{z}}(\mathbf{x})$ and $G_{\mathbf{x}}(\mathbf{z})$, which correspond to an encoder $q(\mathbf{z})$ and a decoder $p(\mathbf{x})$, respectively, and a discriminator $D(\mathbf{x},\mathbf{z})$. In contrast that the original GAN [36] lacks the ability to infer z from x, ALI and BiGAN can induce latent variable mapping, similarly to auto-encoder. Instead of the explicit reconstruction loop of auto-encoder, the GAN-like adversarial training framework is used for training the networks. Owing to this, trivial factors of variation in the input are omitted and the learned features become insensitive to these trivial factors of variation. The objective of ALI/BiGAN is to match the two joint distributions $q(\mathbf{x},\mathbf{z})=q(\mathbf{x})q(\mathbf{z}|\mathbf{x})$ and $p(\mathbf{x},\mathbf{z})=p(\mathbf{z})p(\mathbf{x}|\mathbf{z})$ by optimizing the following function:

$\underset{G_{\mathbf{x}},G_{\mathbf{z}}}{\min }\underset{D}{\max }{\mathcal{L}}_{\mathrm{BiGAN}},\text{where}{\mathcal{L}}_{\mathrm{BiGAN}}={\mathbb{E}}_{q(\mathbf{x})}[\log D(\mathbf{x},G_{\mathbf{z}}(\mathbf{x}))]+{\mathbb{E}}_{p(\mathbf{z})}[\log (1-D(G_{\mathbf{x}}(\mathbf{z}),\mathbf{z}))].$

Here, a discriminator network learns to discriminate between a sample pair $(\mathbf{x},\mathbf{z})$ drawn from $q(\mathbf{x},\mathbf{z})$ and one from $p(\mathbf{x},\mathbf{z})$. The concept of using bidirectional mapping is also important for image-to-image translation methods, which are described in the next section.

2.4.1 Pix2pix and Pix2pixHD

Pix2pix [7] is one of the earliest works on image-to-image translation using a conditional GAN (cGAN) framework. The purpose of this work is to learn a generator that takes an image x (e.g., an edge map) as input and outputs an image y in a different modality (e.g., a color photo). Whereas the generator in a GAN is only dependent on a random vector z, the one in a cGAN also depends on another observed variable, which is the input image x in this case. In the proposed setting in [7], the discriminator also depends on x. The architecture of pix2pix is shown in Fig. 2.9. The objective of the cGAN is written as follows:

${\mathcal{L}}_{\mathrm{cGAN}}(G,D)={\mathbb{E}}_{\mathbf{x},\mathbf{y}}[\log D(\mathbf{x},\mathbf{y})]+{\mathbb{E}}_{\mathbf{x},\mathbf{z}}[\log (1-D(\mathbf{x},G(\mathbf{x},\mathbf{z})))].$

The above objective is then mixed with ${\ell }_1$ distance between the ground truth and the generated image:

${\mathcal{L}}_{{\ell }_1}(G)={\mathbb{E}}_{\mathbf{x},\mathbf{y},\mathbf{z}}[{\Vert \mathbf{y}-G(\mathbf{x},\mathbf{z})\Vert }_1].$

Note that images generated using ${\ell }_1$ distance tend to be less blurring than ${\ell }_2$ distance. The final objective of pix2pix is

$\underset{G}{\min }\underset{D}{\max }{\mathcal{L}}_{\mathrm{pix2pix}}(G,D),\text{where}{\mathcal{L}}_{\mathrm{pix2pix}}(G,D)={\mathcal{L}}_{\mathrm{cGAN}}(G,D)+\lambda {\mathcal{L}}_{{\ell }_1}(G),$

where λ controls the importance of the two terms.

The generator of pix2pix takes the U-Net architecture [6], which is an encoder–decoder network with skip connections between intermediate layers (see Fig. 2.10). Because the U-Net transfers low-level information to high-level layers, it improves the quality (e.g., the edge sharpness) of output images.

More recent work named pix2pixHD [32] generates $2048\times 1024$ high-resolution images from semantic label maps using the pix2pix framework. They improved the pix2pix framework by using a coarse-to-fine generator, a multiscale discriminator architecture, and a robust adversarial learning objective function. The generator G consists of two sub-networks $\{G_1,G_2\}$, where $G_1$ generates $1024\times 512$ images and $G_2$ generates $2048\times 1024$ images using the outputs of $G_1$. Multiscale discriminators $\{D_1,D_2,D_3\}$, who have an identical network structure but operate at different image scales, are used in a multitask learning setting. The objective of pix2pixHD is

$\underset{G}{\min }((\underset{D_1,D_2,D_3}{\max }\sum _{k=1,2,3}{\mathcal{L}}_{\mathrm{cGAN}}(G,D_k))+\lambda \sum _{k=1,2,3}{\mathcal{L}}_{\mathrm{FM}}(G,D_k)),$

where ${\mathcal{L}}_{\mathrm{FM}}(G,D_k)$ represents the newly proposed feature matching loss, which is the summation of ${\ell }_1$ distance between the discriminator's ith-layer outputs of the real and the synthesized images.

2.4.2 CycleGAN, DiscoGAN, and DualGAN

CycleGAN [26] is a newly designed image-to-image translation framework. Suppose that x and y denote samples in a source domain X and a target domain Y, respectively. Previous approaches such as pix2pix [7] (described in Sect. 2.4.1) only lean a mapping $G_Y:X\to Y$, whereas CycleGAN also learns a mapping $G_X:Y\to X$. Here, two generators $\{G_{\mathbf{y}},G_{\mathbf{x}}\}$ and two discriminators $\{D_{\mathbf{y}},D_{\mathbf{x}}\}$ are jointly optimized (Fig. 2.11A). $G_{\mathbf{y}}$ translates x to ${\mathbf{y}}^{\prime }$, which is then fed into $G_{\mathbf{x}}$ to generate ${\mathbf{x}}^{\prime }$ (Fig. 2.11B). In the same manner, $G_{\mathbf{x}}$ translates y to ${\mathbf{x}}^{\prime }$, which is then fed into $G_{\mathbf{y}}$ to generate ${\mathbf{y}}^{\prime }$ (Fig. 2.11C). The ${\ell }_1$ distances between the input and the output are summed up to calculate the following cycle-consistency loss:

${\mathcal{L}}_{\mathrm{cyc}}(G_{\mathbf{x}},G_{\mathbf{y}})={\mathbb{E}}_{\mathbf{x}\sim p_{\mathrm{data}}(\mathbf{x})}[{\Vert \mathbf{x}-G_{\mathbf{x}}(G_{\mathbf{y}}(\mathbf{x}))\Vert }_1]+{\mathbb{E}}_{\mathbf{y}\sim p_{\mathrm{data}}(\mathbf{y})}[{\Vert \mathbf{y}-G_{\mathbf{y}}(G_{\mathbf{x}}(\mathbf{y}))\Vert }_1].$

The most important factor of the cycle-consistency loss is that paired training data is unnecessary. In contrast that Eq. (2.20) in pix2pix loss requires x and y corresponding to the same sample, Eq. (2.23) only needs to compare the input and the output in the respective domains. The full objective of CycleGAN is

${\mathcal{L}}_{\mathrm{cycleGAN}}(G_{\mathbf{x}},G_{\mathbf{y}},D_{\mathbf{x}},D_{\mathbf{y}})={\mathbb{E}}_{\mathbf{y}\sim p_{\mathrm{data}}(\mathbf{y})}[\log D_{\mathbf{y}}(\mathbf{y})]+{\mathbb{E}}_{\mathbf{x}\sim p_{\mathrm{data}}(\mathbf{x})}[\log (1-D_{\mathbf{y}}(G_{\mathbf{y}}(\mathbf{x})))]+{\mathbb{E}}_{\mathbf{x}\sim p_{\mathrm{data}}(\mathbf{x})}[\log D_{\mathbf{x}}(\mathbf{x})]+{\mathbb{E}}_{\mathbf{y}\sim p_{\mathrm{data}}(\mathbf{y})}[\log (1-D_{\mathbf{x}}(G_{\mathbf{x}}(\mathbf{y})))]+\lambda {\mathcal{L}}_{\mathrm{cyc}}(G_{\mathbf{x}},G_{\mathbf{y}}).$

DiscoGAN [27] and DualGAN [28] were proposed in the same period that CycleGAN was proposed. The architectures of those methods are identical to that of CycleGAN (Fig. 2.11). DiscoGAN uses ${\ell }_2$ loss instead of ${\ell }_1$ loss for the cycle-consistency loss. DualGAN replaced the sigmoid cross-entropy loss used in the original GAN by Wasserstein GAN (WGAN) [47].

2.4.3 CoGAN

A coupled generative adversarial network (CoGAN) [29] was proposed to learn a joint distribution of multidomain images. CoGAN consists of a pair of GANs, each of which synthesizes an image in one domain. These two GANs share a subset of parameters as shown in Fig. 2.12. The proposed model is based on the idea that a pair of corresponding images in two domains share the same high-level concepts. The objective of CoGAN is a simple combination of the two GANs:

$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{CoGAN}}(G_1,G_2,D_1,D_2)={\mathbb{E}}_{{\mathbf{x}}_2\sim p_{\mathrm{data}}({\mathbf{x}}_2)}[\log D_2({\mathbf{x}}_2)]+{\mathbb{E}}_{\mathbf{z}\sim p(\mathbf{z})}[\log (1-D_2(G_2(\mathbf{z})))]+\\ \hfill {\mathbb{E}}_{{\mathbf{x}}_1\sim p_{\mathrm{data}}({\mathbf{x}}_1)}[\log D_1({\mathbf{x}}_1)]+{\mathbb{E}}_{\mathbf{z}\sim p(\mathbf{z})}[\log (1-D_1(G_1(\mathbf{z})))].\end{array}$

Similarly to the GAN frameworks introduced in Sect. 2.4.2, CoGAN does not require pairs of corresponding images as supervision.

2.4.4 UNIT

Unsupervised Image-to-Image Translation (UNIT) Networks [30] combined the VAE-GAN [37] model (Sect. 2.3.4) and the CoGAN [29] model (Sect. 2.4.3) as shown in Fig. 2.13. UNIT also does not require pairs of corresponding images as supervision. The sets of networks $\{E_1,G_1,D_1\}$ and $\{E_2,G_2,D_2\}$ correspond to VAE-GANs, whereas the set of networks $\{G_1,G_2,D_1,D_2\}$ correspond to CoGAN. Because the two VAEs share the weights of the last few layers of encoders as well as the first few layers of decoders (i.e., generators), an image of one domain ${\mathbf{x}}_1$ is able to be translated to an image in the other domain ${\mathbf{x}}_2$ through $\{E_1,G_2\}$, and vice versa. Note that the weight-sharing constraint alone does not guarantee that corresponding images in two domains will have the same latent code z. It was, however, shown that, by $E_1$ and $E_2$, a pair of corresponding images in the two domains can be mapped to a common latent code, which is then mapped to a pair of corresponding images in the two domains by $G_1$ and $G_2$. This benefits from the cycle-consistency constraint described below.

The learning problems of the VAEs and GANs for the image reconstruction streams, the image translation streams, and the cycle-reconstruction streams are jointly solved:

$\underset{E_1,E_2,G_1,G_2}{\min }\underset{D_1,D_2}{\max }{\mathcal{L}}_{\mathrm{UNIT}},$

where

${\mathcal{L}}_{\mathrm{UNIT}}={\lambda }_1{\mathcal{L}}_{\mathrm{VAE}}(E_1,G_1)+{\lambda }_2{\mathcal{L}}_{\mathrm{GAN}}(E_2,G_1,D_1)+{\lambda }_3{\mathcal{L}}_{\mathrm{CC}}(E_1,G_1,E_2,G_2)+{\lambda }_1{\mathcal{L}}_{\mathrm{VAE}}(E_2,G_2)+{\lambda }_2{\mathcal{L}}_{\mathrm{GAN}}(E_1,G_2,D_2)+{\lambda }_3{\mathcal{L}}_{\mathrm{CC}}(E_2,G_2,E_1,G_1).$

Here, ${\mathcal{L}}_{\mathrm{VAE}}(\cdot )$ and ${\mathcal{L}}_{\mathrm{GAN}}(\cdot )$ are the same as those in Eq. (2.4) and Eq. (2.7), respectively. ${\mathcal{L}}_{\mathrm{CC}}(\cdot )$ represents the cycle-consistency constraint given by the following VAE-like objective function:

${\mathcal{L}}_{\mathrm{CC}}(E_a,G_a,E_b,G_b)=D_{\mathrm{KL}}(q_a({\mathbf{z}}_a|{\mathbf{x}}_a)||p_{\theta }(\mathbf{z}))+D_{\mathrm{KL}}(q_b({\mathbf{z}}_b|{\mathbf{x}}_a^{a\to b})||p_{\theta }(\mathbf{z}))-{\mathbb{E}}_{{\mathbf{z}}_b\sim q_b({\mathbf{z}}_b|{\mathbf{x}}_a^{a\to b})}[\log p_{G_a}({\mathbf{x}}_a|{\mathbf{z}}_b)].$

The third term in the above function ensures a twice translated resembles the input one, whereas the KL terms penalize the latent variable z deviating from the prior distribution $p_{\theta }(\mathbf{z})\equiv \mathcal{N}(\mathbf{z}|\mathbf{0},\mathbf{I})$ in the cycle-reconstruction stream.

2.4.5 Triangle GAN

A triangle GAN (Δ-GAN) [31] is developed for semi-supervised cross-domain distribution matching. This framework requires only a few paired samples in two different domains as supervision. Δ-GAN consists of two generators $\{G_{\mathbf{x}},G_{\mathbf{y}}\}$ and two discriminators $\{D_1,D_2\}$, as shown in Fig. 2.14. Suppose that ${\mathbf{x}}^{\prime }$ and ${\mathbf{y}}^{\prime }$ represent the translated image from y and x using $G_{\mathbf{x}}$ and $G_{\mathbf{y}}$, respectively. The fake data pair $({\mathbf{x}}^{\prime },\mathbf{y})$ is sampled from the joint distribution $p_{\mathbf{x}}(\mathbf{x},\mathbf{y})=p_{\mathbf{x}}(\mathbf{x}|\mathbf{y})p(\mathbf{y})$, and vice versa. The objective of Δ-GAN is to match the three joint distribution $p(\mathbf{x},\mathbf{y})$, $p_{\mathbf{x}}(\mathbf{x},\mathbf{y})$, and $p_{\mathbf{y}}(\mathbf{x},\mathbf{y})$. If this is achieved, the learned bidirectional mapping $p_{\mathbf{x}}(\mathbf{x}|\mathbf{y})$ and $p_{\mathbf{y}}(\mathbf{y}|\mathbf{x})$ are guaranteed to generate fake data pairs $({\mathbf{x}}^{\prime },\mathbf{y})$ and $(\mathbf{x},{\mathbf{y}}^{\prime })$ that are indistinguishable from the true data pair $(\mathbf{x},\mathbf{y})$.

The objective function of Δ-GAN is given by

$\underset{G_{\mathbf{x}},G_{\mathbf{y}}}{\min }\underset{D_1,D_2}{\max }{\mathbb{E}}_{(\mathbf{x},\mathbf{y})\sim p(\mathbf{x},\mathbf{y})}[\log D_1(\mathbf{x},\mathbf{y})]+{\mathbb{E}}_{\mathbf{y}\sim p(\mathbf{y}),{\mathbf{x}}^{\prime }\sim p_{\mathbf{x}}(\mathbf{x}|\mathbf{y})}[\log ((1-D_1({\mathbf{x}}^{\prime },\mathbf{y}))\cdot D_2({\mathbf{x}}^{\prime },\mathbf{y}))]+{\mathbb{E}}_{\mathbf{x}\sim p(\mathbf{x}),{\mathbf{y}}^{\prime }\sim p_{\mathbf{y}}(\mathbf{y}|\mathbf{x})}[\log ((1-D_1(\mathbf{x},{\mathbf{y}}^{\prime }))\cdot (1-D_2(\mathbf{x},{\mathbf{y}}^{\prime })))].$

The discriminator $D_1$ distinguishes whether a sample pair is from the true data distribution $p(\mathbf{x},\mathbf{y})$ or not. If this is not from $p(\mathbf{x},\mathbf{y})$, $D_2$ is used to distinguish whether a sample pair is from $p_{\mathbf{x}}(\mathbf{x},\mathbf{y})$ or $p_{\mathbf{y}}(\mathbf{x},\mathbf{y})$.

Δ-GAN can be regarded as a combination of cGAN and BiGAN [41]/ALI [40] (Sect. 2.3.7). Eq. (2.29) can be rewritten as

$\underset{G_{\mathbf{x}},G_{\mathbf{y}}}{\min }\underset{D_1,D_2}{\max }{\mathcal{L}}_{\mathrm{\Delta }\text{-GAN}},$

where

${\mathcal{L}}_{\mathrm{\Delta }\text{-GAN}}={\mathcal{L}}_{\mathrm{cGAN}}+{\mathcal{L}}_{\mathrm{BiGAN}},{\mathcal{L}}_{\mathrm{cGAN}}={\mathbb{E}}_{p(\mathbf{x},\mathbf{y})}[\log D_1(\mathbf{x},\mathbf{y})]+{\mathbb{E}}_{p_{\mathbf{x}}({\mathbf{x}}^{\prime },\mathbf{y})}[\log (1-D_1({\mathbf{x}}^{\prime },\mathbf{y}))]+{\mathbb{E}}_{p_{\mathbf{y}}(\mathbf{x},{\mathbf{y}}^{\prime })}[\log (1-D_1(\mathbf{x},{\mathbf{y}}^{\prime }))],{\mathcal{L}}_{\mathrm{BiGAN}}={\mathbb{E}}_{p_{\mathbf{x}}({\mathbf{x}}^{\prime },\mathbf{y})}[\log D_2({\mathbf{x}}^{\prime },\mathbf{y})]+{\mathbb{E}}_{p_{\mathbf{y}}(\mathbf{x},{\mathbf{y}}^{\prime })}[\log (1-D_2(\mathbf{x},{\mathbf{y}}^{\prime }))].$

2.5.1 Model Architecture

As illustrated in Fig. 2.15, each convolution layer (Conv) in the encoder is followed by a batch-normalization layer (Norm) and activation function (ReLU). Two max-pooling operations are placed in the middle of seven Conv+Norm+ReLU components, which makes the dimension of the latent representation be 1/16 of the input. Similarly, the decoder network consists of seven Conv+Norm+ReLU components except for the final layer, while max-pooling operations are replaced by un-pooling operations for expanding a feature map. The max-pooling operation pools a feature map by taking its maximum values, and the un-pooling operation restores the pooled feature into an un-pooled feature map using switches where the locations of the maxima are recorded. The final output of the decoder is then rescaled to the original input size. In case for a classification task, the rescaled output may be further fed into a softmax layer to yield a class probability distribution.

As discussed earlier, all encoder / decoder pairs are connected with the shared latent representation. Let ${\mathbf{x}}_r$, ${\mathbf{x}}_s$, and ${\mathbf{x}}_d$ be the input modalities for each encoder; RGB image, semantic label, and depth map, and ${\mathbf{E}}_r$, ${\mathbf{E}}_s$, and ${\mathbf{E}}_d$ be the corresponding encoder functions, then the outputs from each encoder r, which means latent representation, is defined by $\mathbf{r}\in \{{\mathbf{E}}_r({\mathbf{x}}_r),{\mathbf{E}}_s({\mathbf{x}}_s),{\mathbf{E}}_d({\mathbf{x}}_d)\}$. Here, the dimension of r is $C\times H\times W$, where C, H, and W are the number of channels, height, and width, respectively. Because the intermediate output r from all the encoders form the same shape $C\times H\times W$ by the convolution and pooling operations at all the encoder functions $\mathbf{E}\in \{{\mathbf{E}}_r,{\mathbf{E}}_s,{\mathbf{E}}_s\}$, we can obtain the output $\mathbf{y}\in \{{\mathbf{D}}_r(\mathbf{r}),{\mathbf{D}}_s(\mathbf{r}),{\mathbf{D}}_d(\mathbf{r})\}$ from any r, where ${\mathbf{D}}_r$, ${\mathbf{D}}_s$, and ${\mathbf{D}}_d$ are decoder functions. The latent representations between encoders and decoders are not distinguished among different modalities, i.e., the latent representation encoded by an encoder is fed into all decoders, and at the same time, each decoder has to be able to decode latent representation from any of the encoders. In other words, latent representation is shared for by all the combinations of encoder / decoder pairs.

For semantic segmentation, the U-net architecture with skip paths are also employed to propagate intermediate low-level features from encoders to decoders. Low-level feature maps in the encoder are concatenated with feature maps generated from latent representations and then convolved in order to mix the features. Since we use $3\times 3$ convolution kernels with $2\times 2$ max-pooling operators for the encoder and $3\times 3$ convolution kernels with $2\times 2$ un-pooling operators for the decoder, the encoder and decoder networks are symmetric (U-shape). The introduced model has skip paths among all combinations of the encoders and decoders, and also shares the low-level features in a similar manner to the latent representation.

2.5.2 Multitask Training

In the training phase, a batch of training data is passed through all forwarding paths for calculating losses. For example, given a batch of paired RGB image, depth, and semantic label images, three decoding losses from the RGB image / depth / label decoders are first computed for the RGB image encoder. The same procedure is then repeated for depth and label encoders, and the global loss is defined as the sum of all decoding losses from nine encoder–decoder pairs, i.e., three input modalities for three tasks. The gradients for the whole network are computed based on the global loss by back-propagation. If the training batch contains unpaired data, only within-modal self-encoding losses are computed. In the following, we describe details of the cost functions defined for decoders of RGB images, semantic labels, and depth maps.

RGB images. For RGB image decoding, we define the loss ${\mathcal{L}}_{\mathbf{r}}$ as the ${\ell }_1$ distance of RGB values as

${\mathcal{L}}_I=\frac{1}{N}\sum _{x\in P}{\Vert \mathbf{r}(x)-{\mathbf{f}}_r(x)\Vert }_1,$

where $\mathbf{r}(x)\in {\mathbb{R}}_{+}^3$ and ${\mathbf{f}}_r(x)\in {\mathbb{R}}^3$ are the ground truth and predicted RGB values, respectively. If the goal of the network is realistic RGB image generation from depth and label images, the RGB image decoding loss may be further extended to DCGAN-based architectures [42].

Semantic labels. In this chapter, we define a label image as a map in which each pixel has a one-hot-vector that represents the class that the pixel belongs to. The number of the input and output channels is thus equivalent to the number of classes. We define the loss function of semantic label decoding by the pixel-level cross-entropy. Let K be the number of classes, the softmax function is then written as

$p^{(k)}(x)=\frac{\exp (f_s^{(k)}(x))}{{\sum }_{i=1}^K\exp (f_s^{(i)}(x))},$

where $f_s^{(k)}(x)\in \mathbb{R}$ indicates the value at the location x in the kth channel ($k\in \{1\dots ,K\}$) of the tensor given by the final layer output. Letting P be the whole set of pixels in the output and N be the number of the pixels, the loss function ${\mathcal{L}}_s$ is defined as

${\mathcal{L}}_s=\frac{1}{N}\sum _{x\in P}\sum _{k=1}^K{t}_k(x)\log p^{(k)}(x),$

where $t_k(x)\in \{0,1\}$ is the kth channel ground-truth label, which is one if the pixel belongs to the kth class, and zero, otherwise.

Depth maps. For the depth decoder, we use the distance between the ground truth and predicted depth maps. The loss function ${\mathcal{L}}_d$ is defined as

${\mathcal{L}}_d=\frac{1}{N}\sum _{x\in P}|d(x)-f_d(x)|,$

where $d(x)\in {\mathbb{R}}_{+}$ and $f_d(x)\in \mathbb{R}$ are the ground truth and predicted depth values, respectively. We normalize the depth values so that they range in $[0,255]$ by linear interpolation. In the evaluation step, we revert the normalized depth map into the original scale.

2.5.3 Implementation Details

In the multimodal encoder–decoder networks, learnable parameters are initialized by a normal distribution. We set the learning rate to 0.001 and the momentum to 0.9 for all layers with weight decay 0.0005. The input image size is fixed to $96\times 96$. We use both paired and unpaired data as training data, which are randomly mixed in every epoch. When a triplet of the RGB image, semantic label, and depth map of a scene is available, training begins with the RGB image as input and the corresponding semantic label, depth map, and the RGB image itself as output. In the next step, the semantic label is used as input, and the others as well as the semantic label are used as output for computing individual losses. These steps are repeated on all combinations of the modalities and input / output. A loss is calculated on each combination and used for updating parameters. In case the triplet of the training data is unavailable (for example, when the dataset contains extra unlabeled RGB images), the network is updated only with the reconstruction loss in a similar manner to auto-encoders. We train the network by the mini-batch training method with a batch including at least one labeled RGB image paired with either semantic labels or depth maps if available, because it is empirically found that such paired data contribute to convergence of training.

2.6.1 Results on NYUDv2 Dataset

NYUDv2 dataset has 1448 images annotated with semantic labels and measured depth values. This dataset contains 868 images for training and a set of 580 images for testing. We divided test data into 290 and 290 for validation and testing, respectively, and used the validation data for early stopping. The dataset also has extra 407,024 unpaired RGB images, and we randomly selected 10,459 images as unpaired training data, while other class was not considered in both of training and evaluation. For semantic segmentation, following prior work [51,52,12], we evaluate the performance on estimating 12 classes out of all available classes. We trained and evaluated the multimodal encoder–decoder networks on two different image resolutions $96\times 96$ and $320\times 240$ pixels.

Table 2.1 shows depth estimation and semantic segmentation results of all methods. The first six rows show results on $96\times 96$ input images, and each row corresponds to MAE [1], single-task encoder–decoder with and without skip connections, single-modal multitask encoder–decoder, the multimodal encoder–decoder networks without and with extra RGB training data. The first six columns show performance metrics for depth estimation, and the last column shows semantic segmentation performances. The performance of the multimodal encoder–decoder networks was better than the single-task network (enc-dec) and single-modal multitask encoder–decoder network (enc-decs) on all metrics even without the extra data, showing the effectiveness of the multimodal architecture. The performance was further improved with the extra data, and achieved the best performance in all evaluation metrics. It shows the benefit of using unpaired training data and multiple modalities to learn more effective representations.

In addition, next seven rows show results on $320\times 240$ input images in comparison with baseline depth estimation methods including Make3d [53], DepthTransfer [54], Discrete-continuous CRF [55], Eigen et al. [11], and Liu et al. [50]. The performance was improved when trained on higher resolution images, in terms of both depth estimation and semantic segmentation. The multimodal encoder–decoder networks achieved a better performance than baseline methods, and comparable to methods requiring CRF-based optimization [55] and a large amount of labeled training data [11] even without the extra data.

More detailed results on semantic segmentation on $96\times 96$ images are shown in Table 2.2. Each column shows class-specific IOU scores for all models. The multimodal encoder–decoder networks with extra training data outperforms the baseline models with 10 out of the 12 classes and achieved 0.063 points improvement in MIOU.

2.6.2 Results on Cityscape Dataset

The Cityscape dataset consists of 2975 images for training and 500 images for validation, which are provided together with semantic labels and disparity. We divide the validation data into 250 and 250 for validation and testing, respectively, and used the validation data for early stopping. This dataset has 19,998 additional RGB images without annotations, and we also used them as extra training data. There are semantic labels of 19 class objects and a single background (unannotated) class. We used the 19 classes (excluding the background class) for evaluation. For depth estimation, we used the disparity maps provided together with the dataset as the ground truth. Since there were missing disparity values in the raw data unlike NYUDv2, we adopted the image inpainting method [57] to interpolate disparity maps for both training and testing. We used image resolutions $96\times 96$ and $512\times 256$, while the multimodal encoder–decoder networks was trained on half-split $256\times 256$ images in the $512\times 256$ case.

The results are shown in Table 2.3, and the detailed comparison on semantic segmentation using $96\times 96$ images are summarized in Table 2.4. The first six rows in Table 2.3 show a comparison between different architectures using $96\times 96$ images. The multimodal encoder–decoder networks achieved improvement over both of the MAE [1] and the baseline networks in most of the target classes. While the multimodal encoder–decoder networks without extra data did not improve MIOU, it resulted in 0.043 points improvement with extra data. The multimodal encoder–decoder networks also achieved the best performance on the depth estimation task, and the performance gain from extra data illustrates the generalization capability of the described training strategy. The next three rows show results using $512\times 256$, and the multimodal encoder–decoder networks achieved a better performance than the baseline method [56] on semantic segmentation.

2.6.3 Auxiliary Tasks

Although the main goal of the described approach is semantic segmentation and depth estimation from RGB images, in Fig. 2.16 we show other cross-modal conversion pairs, i.e., semantic segmentation from depth images and depth estimation from semantic labels on cityscape dataset. From left to right, each column corresponds to ground truth (A) RGB image, (B) upper for semantic label image lower for depth map and (C) upper for image-to-label lower for image-to-depth, (D) upper for depth-to-label lower for label-to-depth. The ground-truth depth maps are ones after inpainting. As can be seen, the multimodal encoder–decoder networks could also reasonably perform these auxiliary tasks.

More detailed examples and evaluations on NYUDv2 dataset is shown in Fig. 2.17 and Table 2.5. The left side of Fig. 2.17 shows examples of output images corresponding to all of the above-mentioned tasks. From top to bottom on the left side, each row corresponds to the ground truth, (A) RGB image, (B) semantic label image, estimated semantic labels from (C) the baseline enc-dec model, (D) image-to-label, (E) depth-to-label conversion paths of the multimodal encoder–decoder networks, (F) depth map (normalized to $[0,255]$ for visualization) and estimated depth maps from (G) enc-dec, (H) image-to-depth, (I) label-to-depth. Interestingly, these auxiliary tasks achieved better performances than the RGB input cases. A clearer object boundary in the label and depth images is one of the potential reasons of the performance improvement. In addition, the right side of Fig. 2.17 shows image decoding tasks and each block corresponds to (A) the ground-truth RGB image, (B) semantic label, (C) depth map, (D) label-to-image, and (E) depth-to-image. Although the multimodal encoder–decoder networks could not correctly reconstruct the input color, object shapes can be seen even with the simple image reconstruction loss.