Invertible Autoencoder for Domain Adaptation

Abstract

The unsupervised image-to-image translation aims at finding a mapping between the source ($\mathcal{A}$) and target ($\mathcal{B}$) image domains, where in many applications aligned image pairs are not available at training. This is an ill-posed learning problem since it requires inferring the joint probability distribution from marginals. Joint learning of coupled mappings ${\mathcal{F}}_{\mathcal{A}\mathcal{B}}:\mathcal{A}\to \mathcal{B}$ and ${\mathcal{F}}_{\mathcal{B}\mathcal{A}}:\mathcal{B}\to \mathcal{A}$ is commonly used by the state-of-the-art methods, like CycleGAN to learn this translation by introducing cycle consistency requirement to the learning problem, i.e., ${\mathcal{F}}_{\mathcal{A}\mathcal{B}}\left({\mathcal{F}}_{\mathcal{B}\mathcal{A}}\left(\mathcal{B}\right)\right)\approx \mathcal{B}$ and ${\mathcal{F}}_{\mathcal{B}\mathcal{A}}\left({\mathcal{F}}_{\mathcal{A}\mathcal{B}}\left(\mathcal{A}\right)\right)\approx \mathcal{A}$. Cycle consistency enforces the preservation of the mutual information between input and translated images. However, it does not explicitly enforce ${\mathcal{F}}_{\mathcal{B}\mathcal{A}}$ to be an inverse operation to ${\mathcal{F}}_{\mathcal{A}\mathcal{B}}$. We propose a new deep architecture that we call invertible autoencoder (InvAuto) to explicitly enforce this relation. This is done by forcing an encoder to be an inverted version of the decoder, where corresponding layers perform opposite mappings and share parameters. The mappings are constrained to be orthonormal. The resulting architecture leads to the reduction of the number of trainable parameters (up to 2 times). We present image translation results on benchmark datasets and demonstrate state-of-the art performance of our approach. Finally, we test the proposed domain adaptation method on the task of road video conversion. We demonstrate that the videos converted with InvAuto have high quality and show that the NVIDIA neural-network-based end-to-end learning system for autonomous driving, known as PilotNet, trained on real road videos performs well when tested on the converted ones.

1. Introduction

Inter-domain translation problem of converting an instance, e.g., image or video, from one domain to another is applicable to a wide variety of learning tasks, including object detection and recognition, image categorization, sentiment analysis, action recognition, speech recognition, and more. High-quality domain translators ensure that an arbitrary learning model trained on the samples from the source domain, can perform well when tested on the translated samples (Similarly, an arbitrary learning model trained on the translated samples should perform well on the samples from the target domain. Training in this framework is however, much more computationally expensive). The translation problem can be posed in the supervised learning framework, e.g., [1,2], where the learner has access to corresponding pairs of instances from both domains, or unsupervised learning framework, e.g., [3,4], where no such paired instances are available. This paper focuses on the latter case, which is more difficult but at the same time more realistic as acquiring the dataset of paired images is often impossible in practice.

The unsupervised domain adaptation is typically solved using generative adversarial networks (GAN) framework [5]. GANs constitute a family of methods that learn generative models from complicated real-world data. In order to teach the generator to synthesize semantically meaningful data from standard signal distributions, GANs train a discriminator to distinguish real samples in the training dataset from fake samples synthesized by the generator. The generator aims to deceive the discriminator by producing increasingly more realistic samples. Thus, the generator and discriminator play an adversarial game, during which the generator learns to produce samples from the desired data distribution and the discriminator eventually cannot make a better decision than randomly guessing whether a particular sample is fake or real. GANs have recently been successfully applied to image generation [6,7,8,9], image editing [1,3,10,11], video prediction [12,13,14], and many other tasks [15,16,17]. In the domain adaptation setting, the generator performs domain translation and is trained to learn the mapping from the source to the target domain and the discriminator is trained to discriminate between original images from the target domain and those provided by the generator. In this setting, the generator usually has the structure of the autoencoder. The two most common state-of-the-art domain adaptation approaches, CycleGAN [3] and UNIT [4], are built on this basic approach. CycleGAN addresses the problem of adaptation from domain $\mathcal{A}$ to domain $\mathcal{B}$ by training two translation networks, where one realizes the mapping ${\mathcal{F}}_{\mathcal{A}\mathcal{B}}$ and the other realizes ${\mathcal{F}}_{\mathcal{B}\mathcal{A}}$. The cycle consistency loss ensures the correlation between input image and the corresponding translation. In particular, to achieve cycle consistency, CycleGAN trains two autoencoders, where each minimizes its own adversarial loss and they both jointly minimize

$$\parallel {\mathcal{F}}_{\mathcal{A}\mathcal{B}}\left({\mathcal{F}}_{\mathcal{B}\mathcal{A}}\left(\mathcal{B}\right)\right){-\mathcal{B}\parallel }_2^2\mathrm{and}{\parallel {\mathcal{F}}_{\mathcal{B}\mathcal{A}}\left({\mathcal{F}}_{\mathcal{A}\mathcal{B}}\left(\mathcal{A}\right)\right)-\mathcal{A}\parallel }_2^2.$$

(1)

Cycle consistency loss is also incorporated into the recent implementations of UNIT. It is implicitly assumed that the model will learn the mappings ${\mathcal{F}}_{\mathcal{A}\mathcal{B}}$ and ${\mathcal{F}}_{\mathcal{B}\mathcal{A}}$ in such a way that ${\mathcal{F}}_{\mathcal{A}\mathcal{B}}={\mathcal{F}}_{\mathcal{B}\mathcal{A}}^{-1}$, however, it is not explicitly imposed. Consider a simple example. Assume the first autonecoder is a two-layer linear multi-layer perceptron (MLP) where the weight matrix of the first layer (encoder) is denoted as $E_1$ and the weight matrix of the second layer (decoder) is denoted as $D_1$. Thus, for an input $x_{\mathcal{A}}\in \mathcal{A}$ it outputs $y_{\mathcal{B}}\left(x_{\mathcal{A}}\right)=D_1{E}_1{x}_{\mathcal{A}}$. The second autoencoder then is a two-layer MLP with encoder weight matrix $E_2$ and decoder weight matrix $D_2$ that for an input data point $x_{\mathcal{B}}$ should produce output $y_{\mathcal{A}}\left(x_{\mathcal{B}}\right)=D_2{E}_2{x}_{\mathcal{B}}$. To satisfy cycle consistency requirement, the following should hold: $y_{\mathcal{A}}\left(y_{\mathcal{B}}\left(x_{\mathcal{A}}\right)\right)=\left(x_{\mathcal{A}}\right)$ and $y_{\mathcal{B}}\left(y_{\mathcal{A}}\left(x_{\mathcal{B}}\right)\right)=\left(x_{\mathcal{B}}\right)$. These two conditions are equivalent to $D_2{E}_2{D}_1{E}_1=I$ and $D_1{E}_1{D}_2{E}_2=I$. This holds for example when $D_1=E_2^{-1}$ and $D_2=E_1^{-1}$.

In contrast to this approach, we implicitly require ${\mathcal{F}}_{\mathcal{A}\mathcal{B}}={\mathcal{F}}_{\mathcal{B}\mathcal{A}}^{-1}$. Thus, in the context of the given simple example, we correlate encoders and decoders to satisfy inversion conditions $D_1=E_2^{-1}$ and $D_2=E_1^{-1}$. We avoid performing prohibitive inversions of large matrices and instead guarantee these conditions to hold through two steps: (i) introducing shared parametrization of encoder $E_2$ and decoder $D_1$ such that $D_1=E_2^{\top }$ ($E_1$ and $D_2$ is treated similarly) and (ii) appropriate training to achieve orthonormality $E_2^{\top }=E_2^{-1}$ and $E_1^{\top }=E_1^{-1}$, i.e., we train autoencoder $(E_2,D_1)$ to satisfy $D_1{E}_2{x}_{\mathcal{B}}=x_{\mathcal{B}}$ for arbitrary input $x_{\mathcal{B}}$ and autoencoder $(E_1,D_2)$ to satisfy $D_2{E}_1{x}_{\mathcal{A}}=x_{\mathcal{A}}$ for arbitrary input $x_{\mathcal{A}}$. Since the encoder and decoder are coupled as given in (i), such training leads to satisfying inversion conditions. Practical networks contain linear and non-linear transformations. We therefore propose specific architectures, which are invertible.

Figure 1 (see also its extended version, Figure A4, in the Appendix A) and Figure 2 illustrate the basic idea behind InvAuto. The plots were obtained by training a single autoencoder $(E,D)$ to reconstruct its input. InvAuto has shared weights satisfying $D=E^{\top }$ and inverted non-linearities and clearly obtains matrix $DE$ that is the closest to identity compared to other methods, i.e., vanilla autoencoder (Auto), autoencoder with cycle consistency (Cycle), and variational autoencoder (VAE) [18]. Note also that, at the same time, InvAuto requires half of the number of trainable parameters. This is because the encoder and decoder use the same parameters.

Figure 1. Heatmap of the values of matrix $DE$ for InvAuto (a,e), Auto (b,f), Cycle (c,g), and VAE (d,h) on MLP and ResNet architectures and MNIST and CIFAR datasets. Matrices E and D are constructed by multiplying the weight matrices of consecutive layers of multi-layer encoder and decoder, respectively, e.g., $E=E_L\dots E_2{E}_1$ and $D=D_L\dots D_2{D}_1$ for a $2L$-layer autoencoder. In case of InvAuto, $DE$ is the closest to the identity matrix.

Figure 2. Comparison of the mean squared error (MSE) $\mathrm{MSE}(DE-I)$ for InvAuto, Auto, Cycle, and VAE on MLP, convolutional, and ResNet architectures and MNIST and CIFAR datasets. Matrices E and D are constructed by multiplying the weight matrices of consecutive layers of encoder and decoder, respectively.

This paper is organized as follows: Section 2 reviews the literature, Section 3 explains InvAuto in details, Section 4 explains how to apply InvAuto to domain adaptation, Section 5 demonstrates experimental verification of the proposed approach, and Section 6 provides conclusions.

2. Related Work

Unsupervised image-to-image translation models were developed to tackle domain adaptation problem with unpaired datasets. A plethora of existing approaches utilize autoencoders trained in the GAN framework, where autoencoder serves as a generator, for this learning problem. This includes approaches based on conditional GAN [2,19] and methods introducing additional components to the loss function forcing partial cycle consistency [20]. Another approach [21] introduces two coupled GANs, where each generator is an autoencoder and the coupling is obtained by sharing a subset of weights between autoencoders as well as between discriminators. This technique was later extended to utilize variational autoencoders as generators [4]. The resulting approach is commonly known as UNIT. CycleGAN presents yet another way of addressing the image-to-image translation by specific training scheme that preserves the mutual information between input and translated images [22]. Both UNIT and CycleGAN constitute the most popular choices for performing image-to-image translation.

There also exist other learning tasks that can be viewed as instances of image-to-image translation problem. Among them, notable approaches focus on style transfer [23,24,25,26]. They aim at preserving the content of the input image while altering its style to mimic the style of the images from the target domain. This goal is achieved by introducing content and style loss functions that are jointly optimized. Finally, inverse problems, such as super-resolution, also fall into the category of image-to-image translation problems [27].

3. Invertible Autoencoder

Here we explain the details of the architecture of InvAuto. The architecture needs to be symmetric to allow invertibility, e.g., the layers should be arranged as $\underset{\mathrm{encoder}E}{\underbrace{(T_1,T_2,\dots ,T_M}},\underset{\mathrm{decoder}D}{\underbrace{T_M^{-1},T_{M-1}^{-1},\dots ,T_1^{-1})}}$, where $T_1,T_2,\dots ,T_M$ denote subsequent transformations of the signal that is being propagated through the network (M is the total number of those) and $T_1^{-1},T_2^{-1},\dots ,T_M^{-1}$ denote their inversions. Thus, the architecture is inverted layer by layer, where any layer of the encoder has its mirror inverted counterpart in the decoder. The autoencoder is trained to reconstruct its input. Below we explain how to invert different types of layers of the deep model.

3.1. Fully Connected Layer

Consider transformation $T^E$ of an input signal performed by an arbitrary fully connected layer of an encoder E parametrized with weight matrix W. Let x denote layer’s input and y denote its output. Thus,

$$T^E:y=Wx.$$

(2)

An inverse operation is then defined as

$${\left(T^E\right)}^{-1}:x=W^{-1}y,$$

(3)

We parametrize the counterpart layer of the decoder with a transpose of W, Thus, the considered encoder and decoder layers will share parametrization. Therefore, we enforce the counterpart decoder’s layer to perform transformation:

$$T^D:x=W^{\top }y.$$

(4)

By training the autoencoder to reconstruct its input on its output we will enforce orthonormality $W^{-1}=W^{\top }$ and Thus, equivalence of transformations ${\left(T^E\right)}^{-1}$ and $T^D$, i.e., ${\left(T^E\right)}^{-1}\equiv T^D$.

3.2. Convolutional Layer

Consider transformation $T^E$ of an input image performed by an arbitrary convolutional layer of an encoder E. Let x denote this layer’s vectorized input image and y denote corresponding output. 2D convolution can be implemented using matrix multiplication involving a Toeplitz matrix [28]. Toeplitz matrix is obtained from the set of kernels of the 2D convolutional filters. Thus, transformation $T^E$ and its inverse ${\left(T^E\right)}^{-1}$ can be explained with the same equations as the ones used before, Equations (2) and (3), however, now W is a Toeplitz matrix. We will again parametrize the counterpart layer of the decoder with a transpose of a Toeplitz matrix W. The transpose of the Toeplitz matrix is in practice obtained by copying weights from the considered convolutional layer to the counterpart decoder’s layer that is implemented as a transposed convolutional layer (also known as a deconvolutional layer). Therefore, as before, we enforce the counterpart decoder’s layer to perform transformation $T^D:x=W^{\top }y$ and by appropriate training ensure ${\left(T^E\right)}^{-1}\equiv T^D$.

3.3. Activation Function

Invertible activation function should be a bijection. In this paper, we consider a modified LeakyReLU activation function $\sigma$ and use only this non-linearity in the model. Consider transformation $T^E$ of an input signal performed by this non-linearity applied in the encoder E. This non-linearity is defined as

$$T^E:y=\sigma \left(x\right)=\left\{\begin{array}{cc}\frac{1}{\alpha }x,\hfill & \mathit{if}x\ge 0\hfill \\ \alpha x,\hfill & \mathit{otherwise}.\hfill \end{array}\right.$$

(5)

An inverse operation is then defined as

$${\left(T^E\right)}^{-1}:x={\sigma }^{-1}\left(y\right)=\left\{\begin{array}{cc}\alpha y,\hfill & \mathit{if}x\ge 0\hfill \\ \frac{1}{\alpha }y,\hfill & \mathit{otherwise}.\hfill \end{array}\right.$$

(6)

The corresponding non-linearity in the decoder will therefore realize the operation of an inverted modified LeakyReLU given in Equation (6). In the experiments we set $\alpha =2$.

3.4. Residual Block

Consider transformation $T^E$ of an input signal performed by a residual block [29] of an encoder E. We modify the residual block to remove the internal non-linearity as given in Figure 3a. The residual block is parametrized with weight matrices $W_1$ and $W_2$. Those are Toeplitz matrices corresponding to the convolutional and transposed convolutional layers of the residual block. Let x denote this block’s vectorized input and y denote its corresponding output. Thus, transformation $T^E$ is defined as

$$T^E:y=\sigma ((W_2·W_1+I)·x)$$

(7)

Figure 3. (a) Residual block. (b) Inverted residual block.

An inverse operation is then defined as

$${\left(T^E\right)}^{-1}:x={(W_2·W_1+I)}^{-1}{\sigma }^{-1}\left(y\right).$$

(8)

We will parametrize the counterpart residual block of the decoder with a transpose of matrix $W_2·W_1+I$ as given in Figure 3b. Therefore we enforce the counterpart decoder’s residual block to perform transformation:

$$T^D:x=(W_1^{\top }{W}_2^{\top }+I)y.$$

(9)

As before, at training will enforce orthonormality ${(W_2·W_1+I)}^{-1}={(W_2·W_1+I)}^{\top }$ and Thus, ${\left(T^E\right)}^{-1}\equiv T^D$.

3.5. Bias

We consider bias as a separate layer in the network. Then, handling biases is straightforward. In particular, the layer in the encoder that performs bias addition has its counterpart layer in the decoder, where the same bias is subtracted.

3.6. Experimental Validation of Orthonormality

In this section, we validate the concept of InvAuto. The goal of this section is to show that proposed shared parametrization and training enforce orthonormality and that at the same time the orthonormality property is not organically achieved by standard architectures. We compare InvAuto with previously mentioned vanilla autoencoder, autoencoder with cycle consistency, and variational autoencoder. We experimented with various datasets (MNIST and CIFAR-10) and architectures (MLP, convolutional (Conv), and ResNet). All the networks were designed to have two down-sampling layers and two up-sampling layers. Encoder’s matrix E and decoder’s matrix D are constructed by multiplying the weight matrices of consecutive layers of encoder and decoder, respectively.

We test orthonormality by reporting the histograms of the cosine similarity of each pair of rows of matrix E for all methods (Figure 4) along with their mean and standard deviation (Table 1) as we expect the cosine similarity to be close to 0 for InvAuto. We then show the ${\ell }_2$-norm of the rows of E as we expect the rows of InvAuto to have close-to-unit norm (Table 2). InvAuto enforces the encoder, and consequently the decoder, to be orthonormal. Other methods do not explicitly demand that and Thus, the orthonormality of their encoders is weaker. This observation is further confirmed by Figure 1 and Figure 2 shown before in the Introduction. In the Appendix A, we provide three more figures that complement Figure 2 (recall that the latter reports the MSE of $DE-I$). They show the MSE of the diagonal (Figure A1) and off-diagonal of $DE-I$ (Figure A2) as well as the ratio of the MSE of the off-diagonal and diagonal of $DE$ (Figure A3) for various methods. The reconstruction loss obtained for all methods is also shown in Appendix A (Table A1).

Figure 4. The histograms of cosine similarity of the rows of E for InvAuto (a,e), Auto (b,f), Cycle (c,g), and VAE (d,h) on MLP and ResNet architectures and MNIST and CIFAR datasets.

Table 1. Mean and standard deviation of cosine similarity of rows of E. InvAuto achieves cosine similarity that is the closest to 0. Best performer is in bold.

Dataset and Model	InvAuto	Auto	Cycle	VAE
MNIST	$\mathbf{0.001}$	$0.008$	$0.007$	$0.001$
MLP	$\pm \mathbf{0.118}$	$\pm 0.210$	$\pm 0.207$	$\pm 0.219$
MNIST	$\mathbf{0.001}$	$0.001$	$0.001$	$-0.001$
Conv	$\pm \mathbf{0.148}$	$\pm 0.179$	$\pm 0.176$	$\pm 0.190$
CIFAR	$\mathbf{0.001}$	$0.002$	$0.004$	$0.003$
Conv	$\pm \mathbf{0.145}$	$\pm 0.176$	$\pm 0.195$	$\pm 0.268$
CIFAR	$\mathbf{0.000}$	$0.000$	$0.000$	$0.001$
ResNet	$\pm \mathbf{0.134}$	$\pm 0.203$	$\pm 0.232$	$\pm 0.298$

Table 2. Mean and standard deviation of the ${\ell }_2$-norm of the rows of E. InvAuto achieves the ${\ell }_2$-norm of the rows that is the closest to the unit norm. Best performer is in bold.

Dataset and Model	InvAuto	Auto	Cycle	VAE
MNIST	$\mathbf{0.976}$	$1.326$	$1.268$	$1.832$
MLP	$\pm \mathbf{0.190}$	$\pm 0.095$	$\pm 0.095$	$\pm 0.501$
MNIST	$\mathbf{0.905}$	$1.699$	$1.780$	$1.971$
Conv	$\pm \mathbf{0.321}$	$\pm 0.732$	$\pm 0.779$	$\pm 0.794$
CIFAR	$\mathbf{0.908}$	$3.027$	$2.463$	$1.176$
Conv	$\pm \mathbf{0.219}$	$\pm 0.816$	$\pm 0.688$	$\pm 0.356$
CIFAR	$\mathbf{0.868}$	$2.890$	$2.650$	$1.728$
ResNet	$\pm \mathbf{0.078}$	$\pm 0.895$	$\pm 0.937$	$\pm 0.311$

Next we describe how InvAuto is applied to the problem of domain adaptation.

4. Invertible Autoencoder for Domain Adaptation

For the purpose of performing domain adaptation, we construct the dedicated architecture that is similar to CycleGAN, but we use InvAuto at the feature level of the generators. This InvAuto contains encoder E and decoder D that themselves have the form of autoencoders. Each of these internal autoencoders is used to do the conversion between the features corresponding to two different domains, and thus, the encoder E performs the conversion from the features corresponding to domain $\mathcal{A}$ into the features corresponding to domain $\mathcal{B}$. The decoder D, on the other hand, performs the conversion from the features corresponding to domain $\mathcal{B}$ into the features corresponding to domain $\mathcal{A}$. Since E and D form InvAuto, E realizes an inversion of D (and vice versa) and shares parameters with D. This introduces strong correlations between two generators and reduces the number of trainable parameters, which distinguishes our approach from CycleGAN. The proposed architecture is illustrated in Figure 5. The details of the architecture and training are provided in Appendix A.

Figure 5. The architecture of the domain translator with InvAuto $(E,D)$. $x_{\mathcal{A}}\in \mathcal{A}$ and $x_{\mathcal{B}}\in \mathcal{B}$ are the inputs of the translator. $y_{\mathcal{B}}$ is a converted image $x_{\mathcal{A}}$ into the $\mathcal{B}$ domain and $y_{\mathcal{A}}$ is a converted image $x_{\mathcal{B}}$ into the $\mathcal{A}$ domain. Invertible autoencoder $(E,D)$ is built of encoder E and decoder D, where each of those itself is an autoencoder. ${\mathrm{Enc}}_{\mathcal{A}},{\mathrm{Enc}}_{\mathcal{B}}$ are feature extractors, and ${\mathrm{Dec}}_{\mathcal{A}},{\mathrm{Dec}}_{\mathcal{B}}$ are the final layers of the generators ${\mathrm{Gen}}_{\mathcal{B}}$, i.e., (${\mathrm{Enc}}_{\mathcal{A}},E,{\mathrm{Dec}}_{\mathcal{B}}$), and ${\mathrm{Gen}}_{\mathcal{A}}$, i.e., (${\mathrm{Enc}}_{\mathcal{B}},D,{\mathrm{Dec}}_{\mathcal{A}}$), respectively. Discriminators ${\mathrm{Dis}}_{\mathcal{A}}$ and ${\mathrm{Dis}}_{\mathcal{B}}$ discriminate whether their input comes from the generator (True) or original dataset (False).

Next we describe the cost function that we use to train our deep model. The first component of the cost function is the adversarial loss [5], i.e.,

$$\begin{array}{cc}\hfill L_{\mathrm{adv}}({\mathrm{Gen}}_{\mathcal{A}},{\mathrm{Dis}}_{\mathcal{A}})=& {\mathbb{E}}_{x_{\mathcal{A}}\sim p_d\left(\mathcal{A}\right)}[log{\mathrm{Dis}}_{\mathcal{A}}\left(x_{\mathcal{A}}\right)]+\hfill \\ & {\mathbb{E}}_{x_{\mathcal{B}}\sim p_d\left(\mathcal{B}\right)}[log(1-{\mathrm{Dis}}_{\mathcal{A}}\left({\mathrm{Gen}}_{\mathcal{A}}\left(x_{\mathcal{B}}\right)\right))]\hfill \\ \hfill L_{\mathrm{adv}}({\mathrm{Gen}}_{\mathcal{B}},{\mathrm{Dis}}_{\mathcal{B}})=& {\mathbb{E}}_{x_{\mathcal{B}}\sim p_d\left(\mathcal{B}\right)}[log{\mathrm{Dis}}_{\mathcal{B}}\left(x_{\mathcal{B}}\right)]+\hfill \\ & {\mathbb{E}}_{x_{\mathcal{A}}\sim p_d\left(\mathcal{A}\right)}[log(1-{\mathrm{Dis}}_{\mathcal{B}}\left({\mathrm{Gen}}_{\mathcal{B}}\left(x_{\mathcal{A}}\right)\right))],\hfill \end{array}$$

(10)

where $p_d\left(\mathcal{A}\right)$ and $p_d\left(\mathcal{B}\right)$ denote the distribution of data from $\mathcal{A}$ and $\mathcal{B}$, respectively.

The second component of the loss function is the cycle consistency loss defined as

$$\begin{array}{cc}\hfill L_{cc}({\mathrm{Gen}}_{\mathcal{A}},{\mathrm{Gen}}_{\mathcal{B}})& ={\mathbb{E}}_{x_{\mathcal{A}}\sim p_d\left(\mathcal{A}\right)}\left[{\parallel x_A-{\mathrm{Gen}}_{\mathcal{A}}\left({\mathrm{Gen}}_{\mathcal{B}}\left(x_A\right)\right)\parallel }_1\right]\hfill \\ & +{\mathbb{E}}_{x_{\mathcal{B}}\sim p_d\left(\mathcal{B}\right)}\left[{\parallel x_B-{\mathrm{Gen}}_{\mathcal{B}}\left({\mathrm{Gen}}_{\mathcal{A}}\left(x_B\right)\right)\parallel }_1\right].\hfill \end{array}$$

(11)

The objective function that we minimize therefore becomes

$$\begin{array}{cc}\hfill L({\mathrm{Gen}}_{\mathcal{A}},{\mathrm{Gen}}_{\mathcal{B}},{\mathrm{Dis}}_{\mathcal{A}},{\mathrm{Dis}}_{\mathcal{B}})& =\lambda L_{cc}({\mathrm{Gen}}_{\mathcal{A}},{\mathrm{Gen}}_{\mathcal{B}})\hfill \\ & +L_{\mathrm{adv}}({\mathrm{Gen}}_{\mathcal{A}},{\mathrm{Dis}}_{\mathcal{A}})\hfill \\ & +L_{\mathrm{adv}}({\mathrm{Gen}}_{\mathcal{B}},{\mathrm{Dis}}_{\mathcal{B}}),\hfill \end{array}$$

(12)

where $\lambda$ controls the balance between the adversarial loss and cycle consistency loss. The cycle consistency loss enforces the orthonormality property of InvAuto.

5. Experiments

We next demonstrate experiments on domain adaptation problems. We compare our model against UNIT [4] and CycleGAN [3]. We used publicly available implementations of both methods available from https://github.com/mingyuliutw/UNIT/ and https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/. The details of our architecture and the training process are summarized in Appendix A.

5.1. Experiments with Benchmark Datasets

We considered the following domain adaptation tasks:

(i)

Day-to-night and night-to-day image conversion: we used unpaired road pictures recorded during the day and at night obtained from KAIST dataset [30].

(ii)

Day-to-thermal and thermal-to-day image conversion: we used road pictures recorded during the day with a regular camera and a thermal camera obtained from KAIST dataset [30].

(iii)

Maps-to-satellite and satellite-to-maps: we used satellite images and maps obtained from Google Maps [1].

The datasets for the last two tasks, i.e., (ii) and (iii), are originally paired, however, we randomly permuted them and train the model in an unsupervised fashion. The training and testing images were furthermore resized to $128\times 128$ resolution.

The visual results of image conversion are presented in Figure 6, Figure 7 and Figure 8 (Appendix A contains the same figures in higher resolution). We see that InvAuto visually performs comparably to other state-of-the-art methods.

Figure 6. (Left) Day-to-night image conversion. Zoomed image is shown in Figure A5 in Appendix A. (Right) Night-to-day image conversion. Zoomed image is shown in Figure A6 in Appendix A.

Figure 7. (Left) Day-to-thermal image conversion. Zoomed image is shown in Figure A7 in Appendix A. (Right) Thermal-to-day image conversion. Zoomed image is shown in Figure A8 in Appendix A.

Figure 8. (Left) Maps-to-satellite image conversion. Zoomed image is shown in Figure A9 in Appendix A. (Right) Satellite-to-maps image conversion. Zoomed image is shown in Figure A10 in Appendix A.

To evaluate the performance of the methods numerically we use the following approach:

• For the tasks (ii) and (iii), we directly calculated the ${\ell }_1$ loss between the converted images and the ground truth.
• For the task (i), we trained two autoencoders ${\mathsf{\Omega }}_{\mathcal{A}}$ and ${\mathsf{\Omega }}_{\mathcal{B}}$ on both domains, i.e., we trained each of them to perform high-quality reconstruction of the images from its own domain and low-quality reconstruction of the images from the other domain. Then we use these two autoencoders to evaluate the quality of the converted images, where high ${\ell }_1$ reconstruction loss of the autoencoder for the images converted to resemble those from its corresponding domain implies low-quality image translation.

Table 3 contains the results of the numerical evaluation and shows that the performance of InvAuto is similar to the state-of-the-art techniques that we compare InvAuto with and is furthermore contained within the performance range established by the CycleGAN (best performer) and UNIT (consistently slightly worst from CycleGAN).

Table 3. Numerical evaluation of CycleGAN, UNIT, and InvAuto with ${\ell }_1$ reconstruction loss.

	Methods
Tasks	CycleGAN	UNIT	InvAuto
Night-to-day	$0.033$	$0.227$	$0.062$
Day-to-nigth	$0.041$	$0.114$	$0.067$
Thermal-to-day	$0.287$	$0.339$	$0.299$
Day-to-thermal	$0.179$	$0.194$	$0.205$
Maps-to-satellite	$0.261$	$0.331$	$0.272$
Satellite-to-maps	$0.069$	$0.104$	$0.080$

5.2. Experiments with Autonomous Driving System

To test the quality of the image-to-image translations obtained by InvAuto, we use the NVIDIA evaluation system for autonomous driving described in detail in [31]. The system evaluates the performance of an already trained NVIDIA neural-network-based end-to-end learning platform for autonomous driving (PilotNet) on a test video using a simulator for autonomous driving. The system uses the following performance metrics for evaluation: autonomy, position precision, and comfort. We do not describe these metrics as they are described well in the mentioned paper. We only emphasize that these metrics are expressed as a percentage, where $100\%$ corresponds to the best performance. We collected the high-resolution videos of the same road during the day and night from the camera inside the car. Each video had ∼45 K frames. The pictures were resized to $512\times 512$ resolution for the conversion and then resized back to the original size of $1920\times 1208$. We used our domain translator as well as CycleGAN to convert the collected day video to a night video and also the collected night video to a day video (Figure 9). To evaluate our model, we used the aforementioned NVIDIA evaluation system, where the converted videos where used as testing sets for this system. We report results in Table 4.

Figure 9. (Left) Experimental results with autonomous driving system: day-to-night conversion. Zoomed image is shown in Figure A11 in Appendix A. (Right) Experimental results with autonomous driving system: night-to-day conversion. Zoomed image is shown in Figure A12 in Appendix A.

Table 4. Experimental results with autonomous driving system: autonomy, position precision, and comfort.

Video Type	Autonomy	Position Precision	Comfort
Original day	$99.6\%$	$73.3\%$	$89.7\%$
Original night	$98.6\%$	$63.1\%$	$86.3\%$
Day-to-night	$99.0\%$	$69.6\%$	$83.2\%$
InvAuto
Night-to-day	$99.3\%$	$68.0\%$	$84.7\%$
InvAuto
Day-to-night	$99.0\%$	$68.4\%$	$84.7\%$
CycleGAN
Night-to-day	$98.8\%$	$64.0\%$	$87.3\%$

The PilotNet model used for testing was trained mostly on day videos. Thus, it is expected to perform worse on night videos. Therefore, the performance for original night video is worse than for the same video converted to a day video in terms of autonomy and position precision. The comfort deteriorates due to the inconsistency of consecutive frames in the converted video, i.e., the videos are converted frame-by-frame and we do not apply any post-processing to ensure smooth transition between frames. The results for InvAuto and CycleGAN are comparable.

6. Conclusions

We proposed a novel architecture that we call invertible autoencoder, which, as opposed to the common deep learning architectures, allows the layers of the model performing opposite operations (like encoder and decoder) to share weights. This is achieved by enforcing orthonormal mappings in the layers of the model. We demonstrate the applicability of the proposed architecture to the problem of domain adaptation and evaluate it on benchmark datasets and an autonomous driving task. The performance of the proposed approach matches state-of-the-art methods and requires less trainable parameters.