CycleGAN-Based Singing/Humming to Instrument Conversion Technique

Abstract

In this research, singing/humming to instrument conversion techniques are proposed. In humming to instrument, two models based on cycle-consistent adversarial networks (CycleGAN) on viola are experimented. From the objective and subjective evaluations conducted, the converted audio is more similar to viola compared to humming, and the quality of the converted sound is fair to listeners. In singing to instrument, to fix the problem of the gap between singing and instrument, a dual conversion model consisting of singing to humming and humming to instrument is proposed. The objective and subjective experimental results show that the dual conversion has better converted audio quality than conversion by singing to instrument directly.

1. Introduction

Voice conversion (VC) is a technique for converting one voice into another voice with a different timbre under the condition of keeping linguistic information. In recent years, there have been many applications based on voice conversion, such as vocal conversion [1,2,3,4,5,6,7,8,9,10,11,12,13], singing voice conversion [14,15,16], emotion conversion [17,18], speech style conversion [19,20], conversion of whispers to normal voices [21,22], conversion of singing skills [23], voice correction [24], and so on. However, there are few studies on the conversion of human voices to musical instruments.

The structure and resonance principle of the human vocal tract is very similar to that of musical instruments. Musical instruments have resonance boxes to produce complex timbres; similarly, human beings use vocal cord vibration and vocal tracts such as the nose, pharynx, mouth, and larynx to produce sound. This allows humans to simulate timbres close to the sound of musical instruments, and provides the possibility and interest in building a singing/humming to instrument conversion system.

The techniques of voice conversion can be divided into parallel and non-parallel methods. The feature of the parallel voice conversion system is that the linguistic contents of source and target speech are the same, and the voices are collected as training data. For example, Chen et al. [1] and Toda et al. [2] built parallel voice conversion systems based on the Gaussian mixture model. Others use neural networks [25,26] to build parallel voice conversion systems. For example, Desai et al. used an artificial neural network (ANN) [3], Chen et al. [4] used a deep neural network (DNN), and Nakashika et al. [5] used a recurrent neural network (RNN).

On the other hand, non-parallel voice conversion is based on source-target non-parallel data, and is therefore more challenging. However, it will be more advantageous, because the non-parallel voice conversion system does not need to rely on parallel data, which reduces the difficulty of collecting data. Non-parallel voice conversion can be divided into feature disentangle and direct transformation. Feature disentangle extracts features of audio to convert. The content of the input voice is preserved, and the timbre is changed. In this way, the target speaker can speak the same content as the source speaker. For example, Sun et al. [6] combined automatic speech recognition (ASR) with RNN to design a non-parallel voice conversion system, Liu et al. [7] used WaveNet vocoder for voice conversion, and Saito et al. [8] proposed a non-parallel voice conversion using variational autoencoders conditioned by phonetic posteriorgrams and d-vectors.

The other non-parallel voice conversion system is based on direct transformation, which does not need feature extraction. For example, Kaneko et al. combined cycle-consistent adversarial network (CycleGAN) with gated convolutional neural network (gated CNN) to design a non-parallel voice conversion system—CycleGAN-VC [9]. They also proposed CycleGAN-VC2 [10], an improved version of CycleGAN-VC with an improved objective, generator, and discriminator. CycleGAN is also combined with the linear conversion of fundamental frequency by Fang et al. [11] and Wang et al. [12] added zero-centered gradient penalties on it. Aside from CycleGAN-based techniques, Serrà et al. [13] built a non-parallel conversion system based on WaveNet.

In this research, we use a CycleGAN-based technique to build a non-parallel singing/humming to instrument conversion system. Two systems of CycleGAN-VC and CycleGAN-VC2 based humming to viola conversion are experimented. In addition, in order to improve the naturalness of the converted audio in singing to viola, a dual conversion model consisting of singing-to-humming and humming-to-viola is experimented. These proposed methods and the conventional CycleGAN-VC and CycleGAN-VC2 will be compared.

The conventional CycleGAN-VC and CycleGAN-VC2 will be introduced in Section 2. The framework of the proposed methods will be presented in Section 3. The training and testing datasets, the objective and subjective evaluation methods, and the evaluation results will be shown in Section 4. Finally, the conclusion and future perspectives are conveyed in Section 5.

2. CycleGAN-VC and CycleGAN-VC2

The voice conversion systems, CycleGAN-VC and CycleGAN-VC2, are based on CycleGAN [27] and learn input-to-output mapping without relying on parallel data. Different from a generative adversarial network (GAN), their architecture can be defined as an auto-encoder. They use adversarial loss, cycle-consistency loss, and identity-mapping loss to learn the mapping from source to target, thereby achieving the goal of smoothly converting source acoustic features into target acoustic features.

In the training process of CycleGAN-VC, three types of losses, namely adversarial loss, cycle-consistency loss, and identity-mapping loss, are used to control the process of conversion. Assume $X$ is the source and $Y$ is the target. In forward-inverse and inverse-forward mapping, two generators and two discriminators are used. $G_{X\to Y}$ is the forward generator, which converts $X$ to $Y$; $G_{Y\to X}$ is the inverse generator, which converts $Y$ to $X$. $D_X$ and $D_Y$ are the discriminators of $X$ and $Y$, respectively. Both forward-inverse and inverse-forward mappings contain adversarial loss and cycle-consistency loss, and their main work is to find the optimal pseudo pair from the unpaired audio data. Forward and inverse mapping, which map converted and original $Y$ and $X$, on the other hand, use identity-mapping loss to retain linguistic information.

CycleGAN-VC uses a one-dimensional CNN in the generator. The discriminator is designed as a two-dimensional CNN to focus on spectral texture. CycleGAN-VC refers to the work of Johnson et al. [28], and adds down-sampling, residual, and up-sampling layers to the generator and discriminator. Pixel shuffler for up-sampling is used. In addition, like Cycle-GAN, CycleGAN-VC also uses instance normalization.

To deal with data with time order such as speech, one way is to use RNNs. However, RNNs are computationally demanding, making them difficult to achieve. Gated CNNs [29] can parallelize sequential data, and they have been successfully used in language and speech modeling. Therefore, it was added to CycleGAN-VC.

CycleGAN-VC2 is an improved version of CycleGAN-VC and it adopts two-step adversarial loss. To fix the over-smoothing problem of cycle-consistency loss, an additional discriminator was added, and an additional adversarial loss was also added to the circularly converted feature for inverse-forward mapping.

Unlike CycleGAN-VC, the CycleGAN-VC2 generator is not fully convolutional. The generator was upgraded from 1D (dimension) to 2-1-2D CNN to fix the problem of degradation from down-sampling and up-sampling. In addition, CycleGAN-VC2 uses a 1 × 1 convolution before or after reshaping the feature map to rescale the channel dimension.

The discriminator was changed to PatchGAN [30] in CycleGAN-VC2. PatchGAN uses convolutions in the last layer and determines the authenticity on the basis of the patch, which allows GAN to be trained stably while reducing parameters.

3. Proposed Methods

For singing/humming to instrument conversion, we present two proposed methods based on CycleGAN-VC and CycleGAN-VC2, namely CycleGAN-IC [31] and CycleGAN-IC2, respectively, and a dual conversion concatenation model consisting of singing to humming and humming to instrument, namely CycleGAN-ICd.

3.1. CycleGAN-IC and CycleGAN-IC2

CycleGAN-IC and CycleGAN-IC2 are based on and share the same generators and discriminators with CycleGAN-VC and CycleGAN-VC2. Suppose we want to convert a source into a target and we have acoustic feature sequences $x\in {ℝ}^{Q\times T_x}$ and $y\in {ℝ}^{Q\times T_y}$ belonging to audio source X and target Y. Q is the feature dimension and $T_x$ and $T_y$ are the length of the source and target sequences, respectively. The data distribution is denoted as $x~P_{X\left(x\right)}$ and $y~P_{Y\left(y\right)}$. The objective is to learn mapping $G_{X\to Y}$, which is a non-parallel conversion from $x\in X$ to $y\in Y$.

In CycleGAN-IC, the adversarial losses, cycle-consistency loss, and identity-mapping loss are defined as in Equations (1)–(4). Equations (1) and (2) are adversarial losses, which aim to make $G_{X\to Y}$(x) close to y, and $G_{Y\to X}$(y) close to x.

$${ℒ}_{adv}\left(G_{X\to Y},D_Y\right)={\mathbb{E}}_{y~P_{Y\left(y\right)}}\left[log{D}_Y\left(y\right)\right]+{\mathbb{E}}_{x~P_{X\left(x\right)}}\left[\log \left(1-D_Y\left(G_{X\to Y}\left(x\right)\right)\right)\right].$$

(1)

$${ℒ}_{adv}\left(G_{Y\to X},D_X\right)={\mathbb{E}}_{x~P_{X\left(x\right)}}\left[log{D}_X\left(x\right)\right]+{\mathbb{E}}_{y~P_{Y\left(y\right)}}\left[\log \left(1-D_X\left(G_{Y\to X}\left(y\right)\right)\right)\right].$$

(2)

In Equation (1), the discriminator $D_Y$ aims not to be deceived by maximizing the loss, and $G_{X\to Y}$ aims to generate indistinguishable audio $G_{X\to Y}\left(x\right)$ by minimizing the loss. On the other hand, in Equation (2), the discriminator $D_X$ aims not to be deceived by maximizing the loss, and $G_{Y\to X}$ aims to generate indistinguishable audio $G_{Y\to X}\left(y\right)$ by minimizing the loss.

Equation (3) is the cycle-consistency loss ${ℒ}_{cyc}$, where $G_{X\to Y}$ is the forward generator and $G_{Y\to X}$ is the inverse generator. The purpose of cycle-consistency loss is to measure the difference between the input audio and the audio after forward-inverse or inverse-forward, and to further regularize the mapping.

$${ℒ}_{cyc}\left(G_{X\to Y},G_{Y\to X}\right)={\mathbb{E}}_{x~P_{X\left(x\right)}}\left[\Vert G_{Y\to X}\left(G_{X\to Y}\left(x\right)\right)-x{\Vert }_1\right]+{\mathbb{E}}_{y~P_{Y\left(y\right)}}\left[\Vert G_{X\to Y}\left(G_{Y\to X}\left(y\right)\right)-y{\Vert }_1\right].$$

(3)

Equation (4) is the identity-mapping loss ${ℒ}_{id}$ of $G_{X\to Y}$ and $G_{Y\to X}$.

$${ℒ}_{id}\left(G_{X\to Y},G_{Y\to X}\right)={\mathbb{E}}_{y~P_{Y\left(y\right)}}\left[\Vert G_{X\to Y}\left(y\right)-y{\Vert }_1\right]+{\mathbb{E}}_{x~P_{X\left(x\right)}}\left[\Vert G_{Y\to X}\left(x\right)-x{\Vert }_1\right].$$

(4)

The identity-mapping loss is the sum of the difference between $y$ and $G_{X\to Y}\left(y\right)$ plus the difference between $x$ and $G_{Y\to X}\left(x\right)$. It encourages the generator to preserve the composition between the input and output.

The full objective loss of CycleGAN-IC is shown in Equation (5) as follows.

$$\begin{gathered} {ℒ}_{I{C}_{full}}={ℒ}_{adv}\left(G_{X\to Y},D_Y\right)+{ℒ}_{adv}\left(G_{Y\to X},D_X\right)+ \\ {\lambda }_{cyc}{ℒ}_{cyc}\left(G_{X\to Y},G_{Y\to X}\right)+{\lambda }_{id}{ℒ}_{id}\left(G_{X\to Y},G_{Y\to X}\right). \end{gathered}$$

(5)

In Equation (5), the full loss of CycleGAN-IC, ${ℒ}_{I{C}_{full}}$, is the overall sum of adversarial loss, cycle-consistency loss, and identity-mapping loss. The weighting of cycle-consistency loss ${ℒ}_{cyc}$ and identity-mapping loss ${ℒ}_{id}$ are controlled by ${\lambda }_{cyc}$ and ${\lambda }_{id}$, respectively. CycleGAN-IC uses adversarial loss once in each cycle, which is called one-step adversarial loss. The optimum solution of the mapping X to Y and Y to X equals

$$\arg \underset{G_{X\to Y},G_{Y\to X}}{\min }\underset{D_X, D_Y}{\max }{ℒ}_{I{C}_{full}}$$

(6)

On the other hand, CycleGAN-IC2 uses adversarial losses twice in each cycle, which are two-step adversarial losses and are defined as in Equations (7) and (8). $D_X^{\prime }$ and $D_Y^{\prime }$ are additional discriminators.

$$\begin{gathered} {ℒ}_{adv2}\left(G_{X\to Y},G_{Y\to X},D_X^{\prime }\right)={\mathbb{E}}_{x~P_{X\left(x\right)}}\left[log{D}_X^{\prime }\left(x\right)\right]+ \\ {\mathbb{E}}_{x~P_{X\left(x\right)}}\left[\log \left(1-D_X^{\prime }\left(G_{Y\to X}\left(G_{X\to Y}\left(x\right)\right)\right)\right)\right]. \end{gathered}$$

(7)

$$\begin{gathered} {ℒ}_{adv2}\left(G_{Y\to X},G_{X\to Y},D_Y^{\prime }\right)={\mathbb{E}}_{y~P_{Y\left(y\right)}}\left[log{D}_Y^{\prime }\left(y\right)\right]+ \\ {\mathbb{E}}_{y~P_{Y\left(y\right)}}\left[\log \left(1-D_Y^{\prime }\left(G_{X\to Y}\left(G_{Y\to X}\left(y\right)\right)\right)\right)\right]. \end{gathered}$$

(8)

Finally, the full objective loss of CycleGAN-IC2 is shown in Equation (9) as follows.

$$\begin{gathered} {ℒ}_{IC{2}_{full}}={ℒ}_{adv}\left(G_{X\to Y},D_Y\right)+{ℒ}_{adv}\left(G_{Y\to X},D_X\right)+ \\ {L}_{adv2}\left(G_{X\to Y},G_{Y\to X},D_X^{\prime }\right)+L_{adv2}\left(G_{Y\to X},G_{X\to Y},D_Y^{\prime }\right)+ \\ {\lambda }_{cyc}{ℒ}_{cyc}\left(G_{X\to Y},G_{Y\to X}\right)+{\lambda }_{id}{ℒ}_{id}\left(G_{X\to Y},G_{Y\to X}\right). \end{gathered}$$

(9)

In conventional CycleGAN-VC and CycleGAN-VC2, ${\lambda }_{cyc}$ = 10 and ${\lambda }_{id}$ = 5. When the training steps reach 10,000, ${\lambda }_{id}$ changes to 0. Since singing/humming to instrument is different to human voice conversion, and the cycle-consistency loss aims to encourage $G_{X\to Y}$ and $G_{Y\to X}$ to find (x, y) pairs with the same contextual information, if the weighting of cycle-consistency loss is increased, the generator may fail to preserve the composition between the input and output. Therefore, unlike the original design of CycleGAN-VC and CycleGAN-VC2, which adopts larger weights of cycle-consistency loss, the weights of cycle-consistency loss in CycleGAN-IC and CycleGAN-IC2 are lower than the weights of identity loss, so that the identity loss is more important in the process of loss convergence, and the composition and timbre of the instrument can be preserved in the human voice to instrument conversion. In this research, we use ${\lambda }_{cyc}$ = 1 and ${\lambda }_{id}$ = 5 in CycleGAN-IC and CycleGAN-IC2, and the value of ${\lambda }_{id}$ does not change during the training.

RNN is generally used in sequential signals such as speech. However, it is computationally demanding. Considering singing, humming, or instrument sounds are also sequential, gated CNNs containing activation function—gated linear units (GLUs)—are also used in CycleGAN-IC and CycleGAN-IC2. Assume $O_l$ is the output of the l-th layer; $U_l$, $V_l$ are learned convolutional model parameters; $b_l$, $c_l$ are learned biases.

$$O_{l+1}=(O_l\ast U_l+b_l)\otimes \sigma (O_l\ast V_l+c_l),$$

(10)

where $\otimes$ is the element-wise product and $\sigma$ is the sigmoid function. The element-wise product, also called Hadamard product, of $m\times n$ matrices A and B is defined as ${\left[A\right]}_{ij}{\left[B\right]}_{ij}$, for $1\le i\le m, 1\le j\le m$. Sigmoid function is formulated as $\sigma \left(x\right)=1/\left(1+e^{-x}\right)$.

3.2. Dual Conversion Model

Since singing contains lyrics besides melody, it is very different from the sound of the viola, which makes the converted audio in singing to instrument conversion remain the feature of singing and unable to resemble the instrument sound. To fix the problem, a dual conversion model, CycleGAN-ICd, is proposed. The dual conversion model is as shown in Figure 1. We propose converting human singing into human humming first and then converting it into musical instrument. Since humming does not contain lyrics, it is more similar to the instrument sound. The dual conversion makes the converted quality better than direct conversion from singing to instrument.

In addition to the generators $G_{X\to Y}$ and $G_{Y\to X}$, and discriminators $D_X$ and $D_Y$ in CycleGAN-IC2, CycleGAN-ICd adds one generator $G_{Y\to Z}$ and one discriminator $D_z$. The architecture of the generators and discriminators of CycleGAN-ICd is the same as CycleGAN-IC2. As shown in the left part of Figure 1, we will first convert singing to humming. The full objective loss is as Equation (9). The weighting of cycle-consistency loss ${ℒ}_{cyc}$ and identity-mapping loss ${ℒ}_{id}$ are controlled by ${\lambda }_{cyc}$ and ${\lambda }_{id}$, respectively. In this research, we used ${\lambda }_{cyc}$ = 1, ${\lambda }_{id}$ = 5. The value of ${\lambda }_{id}$ remains unchanged during the training. After the singing to humming is finished, humming to instrument is processed. As shown in the right part of Figure 1, both the original and converted humming are input to the generator $G_{Y\to Z}$ to create a converted viola to fool the discriminator $D_z$. Two additional adversarial losses, ${ℒ}_{adv3}$ and ${ℒ}_{adv3c}$ relating to original and converted humming, and identity-mapping loss ${ℒ}_{id3}$ are added as in Equations (11)–(13). The full objective loss of the humming to instrument conversion in the dual model is the sum of ${ℒ}_{adv3}$, ${ℒ}_{adv3c}$, and ${ℒ}_{id3}$.

$${ℒ}_{adv3}\left(G_{Y\to Z},D_Z\right)={\mathbb{E}}_{z~P_{Data\left(z\right)}}\left[log{D}_Z\left(z\right)\right]+{\mathbb{E}}_{y~P_{Data\left(y\right)}}\left[log\left(1-D_Z\left(G_{Y\to Z}\left(y\right)\right)\right)\right],$$

(11)

$$\begin{gathered} {ℒ}_{adv3c}\left(G_{Y\to Z},D_Z\right)={\mathbb{E}}_{z~P_{Data\left(z\right)}}\left[log{D}_Z\left(z\right)\right]+ \\ {\mathbb{E}}_{x,y~P_{Data\left(x,y\right)}}\left[log\left(1-D_Z\left(G_{Y\to Z}\left(G_{X\to Y}\left(x\right)\right)\right)\right)\right], \end{gathered}$$

(12)

$${ℒ}_{id3}\left(G_{Y\to Z}\right)={\mathbb{E}}_{z~P_{Data\left(z\right)}}\left[\parallel G_{Y\to Z}\left(z\right)-z{\parallel }_1\right],$$

(13)

3.3. Theoretical Analysis of Convergence and Complexity

Game theory is the study of the predicted and actual strategic behavior of rational agents in the game and building the mathematical models for finding the optimal strategies. Each player aims to maximize his utility by choosing the action maximizing the payoff. The action of a player is picked regarding other players’ actions. Among the game types, zero-sum games, such as poker, Go, and chess, grab a lot of attention. Zero-sum games are games in which decisions by players cannot change the sum of the players’ utilities. Actually, the total benefit of all the players equals zero all the time, for each kind of strategy. In games, how to deal with finding optimal equilibrium from multiple equilibria and avoiding undesirable equilibrium is a critical issue in game theory.

For GAN-based technology such as the proposed methods, the generator and the discriminator are two players playing against each other in a repetitive zero-sum game. The generator model is parameterized by $\varphi$, and the discriminator is parameterized by $\theta$. In this study, deep neural networks are used for the generator model G and discriminator model D. The cost functions of the proposed CycleGAN-IC, CycleGAN-IC2, and CycleGAN-ICd are listed in Equations (5) and (9), and the sum of Equations (11)–(13).

The alternating gradient updates procedure (AGD) is generally used to reach an equilibrium in GAN-based problems. Convergence can happen based on the assumption that one player, the discriminator, is playing optimally at each step, such as in Wasserstein GAN [32], which can continuously estimate the Earth-Mover (EM) distance (or Wasserstein-1) by optimizing the discriminator. The EM distance is defined as

$$\underset{\gamma \in \mathsf{\Pi }\left({\mathbb{P}}_r, {\mathbb{P}}_g\right)}{inf}{\mathbb{E}}_{\left(x,y\right)~\gamma }\left[\parallel x-y\parallel \right].$$

(14)

$\mathsf{\Pi }\left({\mathbb{P}}_r, {\mathbb{P}}_g\right)$ is the set of all joint distributions $\gamma \left(x,y\right)$. Real distribution ${\mathbb{P}}_r$ and transformed distribution ${\mathbb{P}}_g$ are the marginals. EM distance is continuous and differential, so the Wasserstein GAN critic can be trained to optimality. However, the assumption is strong and unrealistic. Wasserstein GAN becomes unstable at times.

Convergence can also be analyzed by studying GAN training dynamics as a repeated game in which both the players are utilizing no-regret algorithms [33]. From game theory, it yields a proof for the asymptotic convergence for convex-concave case as follows. Sion’s minimax theorem [33] states that if $\mathsf{\Phi }\subset {ℝ}^m$, $\mathsf{\Theta }\subset {ℝ}^n$, and they are compact and convex, and the function $\mathrm{J}:\mathsf{\Phi }\times \mathsf{\Theta }\to ℝ$ is convex and concave in the first and second, then

$$\underset{\varphi \in \mathsf{\Phi }}{\min } \underset{\theta \in \mathsf{\Theta }}{\max } J\left(\varphi ,\theta \right)=\underset{\theta \in \mathsf{\Theta }}{\max } \underset{\varphi \in \mathsf{\Phi }}{\min }J\left(\varphi ,\theta \right),$$

(15)

and an equilibrium exists. Sion’s minimax theorem can be proven [34] by Helly’s theorem, which is a statement in combinatorial geometry on the intersections of convex sets, and the KKM theorem of Knaster, Kuratowski, and Mazurkiewicz, which is a result in mathematical fixed-point theory. Therefore, if the generator and discriminator parameters are compact and convex, and the cost function is convex and concave in the first and second, there exists an equilibrium according to Sion’s minimax theorem. The procedure of getting such an equilibrium can be achieved in game theory if both players update their parameters by no-regret algorithms. The definition of no-regret algorithm is as follows [33]. Given convex loss function sequences $L_1$, $L_2$, …: $K\to ℝ$, to select a sequence $k_t$’s, which depends only on the previous $L_1$, … $L_{t-1}$, if $\frac{R\left(T\right)}{T}=o\left(1\right)$, the selection algorithm is no regret, where

$$R\left(T\right)≔ {{\displaystyle \sum }}_{t=1}^T{L}_t\left(k_t\right)-\underset{k\in K}{\min }{{\displaystyle \sum }}_{t=1}^T{L}_t\left(k\right).$$

(16)

However, the convergence does not hold for the practical non-convex case, which is the general case for deep learning. In practice, the generator and discriminator are deep neural networks, and the cost function is not necessarily convex-concave. In non-convex games, AGD can keep cycling or converge to local equilibrium. Local regret [35] is introduced, which shows that the game converges to local equilibrium in a non-convex case if a smoothed variant of online gradient descent (OGD) and mild assumptions are used. $\omega$-local regret of an online algorithm by fixing some $\eta >0$ is defined as follows.

$${\Re }_{\omega }\left(T\right)\stackrel{\text{def}}{=}{{\displaystyle \sum }}_{t=1}^T\Vert {\nabla }_{\mathcal{K}, \eta }{F}_{t,\omega }\left(x_t\right){\Vert }^2,$$

(17)

where $\omega$ is window size; $t$ is iteration; $T$ is total round; and $\mathcal{K}$ is a convex set. From [35], for any $T\ge 1$, 1$\le \omega \le T$, and $\eta \le 1$, a distribution $\mathcal{D}$ on 0-smooth, 1-bounded cost functions on $\mathcal{K}=\left[-1,1\right]$ exists, such that for any online algorithm,

$$\underset{\mathcal{D}}{\mathbb{E}}\left[{\Re }_{\omega }\left(T\right)\right]\ge \frac{1}{4\omega }\lfloor \frac{T}{2\omega }\rfloor .$$

(18)

Hence, in online learning, time smoothing truly captures non-convex optimization. The convergence of the proposed methods will further be shown through observing the convergence curve in the experiment in Section 4.

The computational complexities of CycleGAN-IC and CycleGAN-IC2 are the same as CycleGAN-VC and CycleGAN-VC2 because CycleGAN-IC and CycleGAN-IC2 share the same architecture of the generators and discriminators with CycleGAN-VC and CycleGAN-VC2. The computational complexity of CycleGAN-ICd is 1.5 times that of CycleGAN-IC2 because CycleGAN-ICd has three generators and three discriminators and shares the same architecture of the generators and discriminators with CycleGAN-IC2, while CycleGAN-IC2 has two generators and two discriminators. FLOPs (floating-point operations) [36] of the generators and discriminators are used for computational complexity analysis.

For each convolutional kernel, FLOPs equal

$$2HW\left(C_{in}{K}^2+1\right)C_{out}.$$

(19)

$H$, $W$, $C_{in}$, $K$, and $C_{out}$ are height, width, the number of channels, kernel width, and the number of output channels, respectively. The parameters of the generator and discriminator architectures are presented in

Table 1. (a) The generator architecture. (b) The discriminator architecture.

(a)
State	Amount	Operation	Architecture
		Input	$\mathrm{height}=35$	$\mathrm{width}=\mathrm{length} T$	$\mathrm{number} \mathrm{of} \mathrm{channels}=1$
		Conv	$\mathrm{kernel} \mathrm{size}=5\times 15$	$\mathrm{number} \mathrm{of} \mathrm{channels}=128$	$\mathrm{stride} \mathrm{size}=1\times 1$
		GLU	-	-	-
down-sample (2D)	1	Conv	$\mathrm{kernel} \mathrm{size}=5\times 5$	$\mathrm{number} \mathrm{of} \mathrm{channels}=256$	$\mathrm{stride} \mathrm{size}=2\times 2$
		Instance norm	-	-	-
		GLU	-	-	-
		Conv	$\mathrm{kernel} \mathrm{size}=5\times 5$	$\mathrm{number} \mathrm{of} \mathrm{channels}=512$	$\mathrm{stride} \mathrm{size}=2\times 2$
		Instance norm	-	-	-
		GLU	-	-	-
2D → 1D	1	Reshape	$\mathrm{height}=1$	$\mathrm{width}$$=\mathrm{length} T/4$	$\mathrm{number} \mathrm{of} \mathrm{channels}=2304$
		$1\times 1$ Conv	$\mathrm{kernel} \mathrm{size}=1\times 1$	$\mathrm{number} \mathrm{of} \mathrm{channels}=256$	$\mathrm{stride} \mathrm{size}=1\times 1$
		Instance norm	-	-	-
residual blocks (1D)	6	Conv	$\mathrm{kernel} \mathrm{size}=1\times 3$	$\mathrm{number} \mathrm{of} \mathrm{channels}=512$	$\mathrm{stride} \mathrm{size}=1\times 1$
		Instance norm	-	-	-
		GLU	-	-	-
		Conv	$\mathrm{kernel} \mathrm{size}=1\times 3$	$\mathrm{number} \mathrm{of} \mathrm{channels}=256$	$\mathrm{stride} \mathrm{size}=1\times 1$
		Instance norm	-	-	-
		Sum	-	-	-
1D → 2D	1	$1\times 1$ Conv	$\mathrm{kernel} \mathrm{size}=1\times 1$	$\mathrm{number} \mathrm{of} \mathrm{channels}=2304$	$\mathrm{stride} \mathrm{size}=1\times 1$
		Instance norm	-	-	-
		Reshape	$\mathrm{height}=9$	$\mathrm{width}$$=\mathrm{length} T/4$	$\mathrm{number} \mathrm{of} \mathrm{channels}=256$
up-sample (2D)	1	Conv	$\mathrm{kernel} \mathrm{size}=5\times 5$	$\mathrm{number} \mathrm{of} \mathrm{channels}=1024$	$\mathrm{stride} \mathrm{size}=1\times 1$
		Pixel Shuffler	-	-	-
		Instance norm	-	-	-
		GLU	-	-	-
		Conv	$\mathrm{kernel} \mathrm{size}=5\times 5$	$\mathrm{number} \mathrm{of} \mathrm{channels}=512$	$\mathrm{stride} \mathrm{size}=1\times 1$
		Pixel Shuffler	-	-	-
		Instance norm	-	-	-
		GLU	-	-	-
		Conv	$\mathrm{kernel} \mathrm{size}=5\times 15$	$\mathrm{number} \mathrm{of} \mathrm{channels}=35$	$\mathrm{stride} \mathrm{size}=1\times 1$
		Output	$\mathrm{height}=35$	$\mathrm{width}=\mathrm{length} T$	$\mathrm{number} \mathrm{of} \mathrm{channels}=1$
(b)
State	Amount	Operation	Architecture
		Input	$\mathrm{height}=35$	$\mathrm{width}=128$	$\mathrm{number} \mathrm{of} \mathrm{channels}=1$
		Conv	$\mathrm{kernel} \mathrm{size}=3\times 3$	$\mathrm{number} \mathrm{of} \mathrm{channels}=128$	$\mathrm{stride} \mathrm{size}=1\times 1$
		GLU	-	-	-
down-sample (2D)	1	Conv	$\mathrm{kernel} \mathrm{size}=3\times 3$	$\mathrm{number} \mathrm{of} \mathrm{channels}=256$	$\mathrm{stride} \mathrm{size}=2\times 2$
		Instance norm	-	-	-
		GLU	-	-	-
		Conv	$\mathrm{kernel} \mathrm{size}=3\times 3$	$\mathrm{number} \mathrm{of} \mathrm{channels}=512$	$\mathrm{stride} \mathrm{size}=2\times 2$
		Instance norm	-	-	-
		GLU	-	-	-
		Conv	$\mathrm{kernel} \mathrm{size}=3\times 3$	$\mathrm{number} \mathrm{of} \mathrm{channels}=1024$	$\mathrm{stride} \mathrm{size}=2\times 2$
		Instance norm	-	-	-
		GLU	-	-	-
		Conv	$\mathrm{kernel} \mathrm{size}=3\times 3$	$\mathrm{number} \mathrm{of} \mathrm{channels}=1024$	$\mathrm{stride} \mathrm{size}=1\times 1$
		Instance norm	-	-	-
		GLU	-	-	-
		Conv	$\mathrm{kernel} \mathrm{size}=1\times 3$	$\mathrm{number} \mathrm{of} \mathrm{channels}=1$	$\mathrm{stride} \mathrm{size}=1\times 1$
		Real/Fake	-	-	-

. For detailed operations in the tables, please refer to [10].

4. Experiment

Two experiments, including humming to viola and singing to viola, were conducted. For the first experiment, CycleGAN-IC and CycleGAN-IC2 were applied and compared with the conventional CycleGAN-VC and CycleGAN-VC2. For the second experiment, CycleGAN-ICd was used. Objective and subjective measures were evaluated on the converted results.

4.1. Corpus

The MIR-QBSH database [37] was used to perform experiments on singing/humming to viola. MIR-QBSH collects MIDI (Musical Instrument Digital Interface) and vocals including humming and singing. A total of 48 MIDI songs ranging from 11 s to 201 s, and 4431 8-s-clips of humming and singing are included. Four subsets—MIR-QBSH-1, MIR-QBSH-2, MIR-QBSH-3, and MIR-QBSH-4, recorded by four male singers, were used in our experiments. The first singer recorded 17 humming and 5 singing 8-s-clips, with a total length 2 min and 56 s, the second singer recorded 20 singing 8-s-clips, with a total length 2 min and 40 s, the third singer recorded 20 humming 8-s-clips, with a total length 2 min and 40 s, and the fourth singer recorded 20 singing 8-s-clips, with a total length 2 min and 40 s. In the experiment of humming to viola, MIR-QBSH-1 was used as a training set and MIR-QBSH-3 was used as a testing set. In the experiment of singing to viola, for CycleGAN-IC2, MIR-QBSH-2 was used as a training set and MIR-QBSH-4 was used as a testing set. For CycleGAN-ICd, both MIR-QBSH-1 and MIR-QBSH-2 were used as training sets and MIR-QBSH-4 was used as a testing set. The corresponding viola was created by the SONAR synthesizer [38] developed by Cakewalk from MIDI. The sampling rate of all the audio was 16 kHz.

4.2. Objective and Subjective Measures

In this research, both objective and subjective measures were applied. Root-Mean-Square Error (RMSE) of the Mel-Frequency Cepstrum Coefficients (MFCCs) and Mel-Cepstral Distortion (MCD) were used as objective measures to compare the converted audio with the original viola and the original humming or singing. In calculating the RMSE of MFCCs, the number of MFCC was 34, the analysis frame length was 1024 samples, and the frame shift was 256 samples. Since the RMSE must be calculated at the same length, the original viola was cut to fit the length. The smaller the RMSE value, the more similar the converted audio was to the original audio.

MCD was calculated by dynamic time warping (DTW), which aligns the signal to the time to calculate the difference. The smaller the value of the MCD, the more similar the converted audio is to the original audio. In MCD, the all pass constant value was set as 0.35, the Fast Fourier Transform (FFT) size was 512, and the order of mel-cepstrum size was 34.

In addition to objective measures, the mean opinion score (MOS) [39] and comparison mean opinion score (CMOS) [39] were used as subjective measures. The MOS scores were 1 to 5, meaning ‘Bad’, ‘Poor’, ‘Fair’, ‘Good’, and ‘Excellent’. CMOS compares two sounds by giving scores of −3 (much worse) to 3 (much better). To simplify the evaluation process, in this research, we used three score levels: −1 (worse), 0 (equal), and 1 (better).

4.3. Experiments Results of Humming to Viola

In this subsection, we present the objective and subjective experiment results of humming-to-viola conversion by CycleGAN-IC and CycleGAN-IC2, and compare them with the conventional CycleGAN-VC and CycleGAN-VC2. MIR-QBSH-1 and MIR-QBSH-3 were used in the experiment of humming to viola as training and testing sets. The convergence of the generator and discriminator losses of CycleGAN-VC, CycleGAN-VC2, CycleGAN-IC, and CycleGAN-IC2 are shown in Figure 2.

The RMSE and MCD between the converted and original viola in the testing set are shown in the box plots of Figure 3. The average RMSEs and MCDs between the converted and original viola are shown in

Table 2. The average RMSE of the converted viola with the original viola/humming.

	vs. Original Viola	vs. Original Humming
CycleGAN-VC	1.3193	1.7613
CycleGAN-VC2	1.2566	1.7171
CycleGAN-IC	1.3085	1.7206
CycleGAN-IC2	1.2264	1.6883

and

Table 3. The average MCD of the converted viola with the original viola/humming.

	vs. Original Viola	vs. Original Humming
CycleGAN-VC	4.7665	7.4907
CycleGAN-VC2	4.3093	7.6802
CycleGAN-IC	4.5214	7.8334
CycleGAN-IC2	4.2884	7.7419

, respectively. The average RMSEs and MCDs between the converted viola and original humming are also presented. In CycleGAN-IC, CycleGAN-IC2, CycleGAN-VC, and CycleGAN-VC2, the RMSE and MCD with the original viola were smaller than the RMSE and MCD with the original humming, meaning the converted viola was more similar to the original viola than the original humming. Therefore, the conversion does go in the right direction. From Table 2 and Table 3, the average RMSE and MCD of CycleGAN-IC2 were the smallest compared with other methods. Hence, the converted audio by CycleGAN-IC2 was the most similar to the original viola.

In addition to objective evaluation, MOS and CMOS subjective evaluations were also performed. For each humming to viola method, ten converted viola sounds were used and 10 listeners attended. The 10 listeners included four men and six women. Among them, three listeners have music backgrounds. Listeners were asked to rate the audio based on its clarity, timbre, and similarity to viola.

The percentage distribution results of MOS scores of CycleGAN-VC, CycleGAN-VC2, CycleGAN-IC, and CycleGAN-IC2 are presented in

Table 4. The percentage distribution of MOS in humming to viola.

	Bad	Poor	Fair	Good	Excellent
CycleGAN-VC	0%	58%	42%	0%	0%
CycleGAN-VC2	0%	25%	73%	2%	0%
CycleGAN-IC	0%	46%	54%	0%	0%
CycleGAN-IC2	0%	10%	79%	11%	0%

. From Table 4, the converted viola by CycleGAN-IC2 obtained the best scores. CycleGAN-VC2 and CycleGAN-IC were the second and third best, respectively. CycleGAN-VC was the worst. Converted viola from CycleGAN-VC, CycleGAN-VC2, CycleGAN-IC, and CycleGAN-IC2 received an average MOS score of 2.42, 2.77, 2.54, and 3.01 respectively.

The voting results of CMOS are as shown in

Table 5. The voting results of CMOS in humming to viola.

	CycleGAN-VC	CycleGAN-VC2	CycleGAN-IC	CycleGAN-IC2	Equal
CycleGAN-IC: CycleGAN-VC	12%	-	18%	-	70%
CycleGAN-IC: CycleGAN-VC2	-	39%	24%	-	37%
CycleGAN-IC2: CycleGAN-VC	12%	-	-	60%	28%
CycleGAN-IC2: CycleGAN-VC2	-	24%	-	50%	26%
CycleGAN-IC: CycleGAN-IC2	-	-	13%	60%	27%

. In the pair comparison, the listeners voted for the preferred audio. For example, in CycleGAN-IC vs. CycleGAN-IC2 pair, 13% preferred CycleGAN-IC, 60% preferred CycleGAN-IC2, and 27% voted equal. From the table, the viola converted by CycleGAN-IC2 was the most preferred, the viola converted by CycleGAN-VC2 was the second preferred, and the viola converted by CycleGAN-IC and CycleGAN-VC were the third and the worst. This result is consistent with the results of RMSE, MCD, and MOS. From the results of RMSE, MCD, MOS, and CMOS, the conversion from humming to viola by CycleGAN-IC2 was successful.

MIR-QBSH-1 includes singing and humming clips and was used as a training set in this experiment, because, in the beginning, we wished to train a model which could convert both singing and humming to viola. However, the quality of the converted viola from singing by CycleGAN-VC, CycleGAN-VC2, CycleGAN-IC, and CycleGAN-IC2 was not decent. We, therefore, proposed CycleGAN-ICd and perform a singing to viola experiment in the next subsection.

4.4. Experiments Results of Singing to Viola

In this section, we use CycleGAN-ICd and CycleGAN-IC2 for singing-to-viola conversion. Since CycleGAN-ICd uses dual conversion of singing to humming and humming to viola, both singing and humming data are applied. The objective and subjective evaluations are performed and compared.

MIR-QBSH-4, including twenty singing clips, was used in the RMSE and MCD evaluation. The box plots of the RMSE and MCD of the converted and original viola are presented in Figure 4. The average RMSEs and MCDs are shown in

Table 6. The average RMSE for MIR-QBSH-4 in singing to viola.

CycleGAN-IC2	1.1697
CycleGAN-ICd	1.0473

and

Table 7. The average MCD for MIR-QBSH-4 in singing to viola.

CycleGAN-IC2	4.0608
CycleGAN-ICd	3.9803

. From Figure 4, Table 6 and Table 7, CycleGAN-ICd performed better than CycleGAN-IC2 in objective evaluation.

In addition to objective evaluation, subjective evaluations were also performed. Ten listeners and ten converted audio clips were used in MOS and CMOS. From

Table 8. The percentage distribution of MOS in singing to viola.

	Bad	Poor	Fair	Good	Excellent
CycleGAN-IC2	0%	17%	73%	11%	0%
CycleGAN-ICd	0%	10%	68%	22%	0%

, the percentage distribution of MOS, the quality of the audio converted by CycleGAN-IC2 and CycleGAN-ICd were considered Fair by most of the voters. Converted audio by CycleGAN-IC2 received an average MOS score of 2.97, while converted audio by CycleGAN-ICd received an average MOS score of 3.12. The performance of conversion quality of CycleGAN-ICd was better than CycleGAN-IC2.

In addition to MOS, the CMOS results are shown in

Table 9. The voting results of CMOS in singing to viola.

	CycleGAN-IC2	CycleGAN-ICd	Equal
CycleGAN-IC2: CycleGAN-ICd	23%	56%	21%

. In CycleGAN-IC2 vs. CycleGAN-ICd pair, 23% preferred CycleGAN-IC2, 56% preferred CycleGAN-ICd, and 21% voted equal. From the MOS and CMOS results, listeners confirmed that the converted audio by CycleGAN-ICd sounded closer to the original viola than CycleGAN-IC2. Therefore, the dual conversion by singing to humming and humming to viola had subjectively better converted audio quality than conversion by singing to viola directly.

5. Conclusions

In this research, we built CycleGAN-based singing/humming to instrument conversion techniques. CycleGAN-IC and CycleGAN-IC2 are based on CycleGAN-VC and CycleGAN-VC2. In the experiment of humming to viola, objective and subjective evaluations show that CycleGAN-IC2 performs better than CycleGAN-IC and the conventional CycleGAN-VC and CycleGAN-VC2, and the converted audio is closer to viola than to humming, and the quality of the converted sound is fair to listeners.

Moreover, we proposed a dual model CycleGAN-ICd, which accomplishes singing to instrument by converting singing to humming and humming to instrument. The objective and subjective tests of singing to viola show that the CycleGAN-ICd outperformed CycleGAN-IC2 and the converted sound received an average MOS score of 3.12.

In the future, in addition to further improving the conversion quality, we will apply this research to other musical instrument conversions.