Wav2wav: Wave-to-Wave Voice Conversion

Abstract

Voice conversion is the task of changing the speaker characteristics of input speech while preserving its linguistic content. It can be used in various areas, such as entertainment, medicine, and education. The quality of the converted speech is crucial for voice conversion algorithms to be useful in these various applications. Deep learning-based voice conversion algorithms, which have been showing promising results recently, generally consist of three modules: a feature extractor, feature converter, and vocoder. The feature extractor accepts the waveform as the input and extracts speech feature vectors for further processing. These speech feature vectors are later synthesized back into waveforms by the vocoder. The feature converter module performs the actual voice conversion; therefore, many previous studies separately focused on improving this module. These works combined the separately trained vocoder to synthesize the final waveform. Since the feature converter and the vocoder are trained independently, the output of the converter may not be compatible with the input of the vocoder, which causes performance degradation. Furthermore, most voice conversion algorithms utilize mel-spectrogram-based speech feature vectors without modification. These feature vectors have performed well in a variety of speech-processing areas but could be further optimized for voice conversion tasks. To address these problems, we propose a novel wave-to-wave (wav2wav) voice conversion method that integrates the feature extractor, the feature converter, and the vocoder into a single module and trains the system in an end-to-end manner. We evaluated the efficiency of the proposed method using the VCC2018 dataset.

1. Introduction

Voice conversion aims to convert the speaker-specific characteristics of input speech into the target speaker’s characteristics while preserving the linguistic content of the input speech. The application areas of voice conversion include entertainment [1], education [2], and medicine [3] domains. With advances in deep learning, many studies on voice conversion using deep neural network (DNN) models have been attempted. In particular, generative models, such as variational autoencoders (VAEs) [4] and generative adversarial networks (GANs) [5], have shown promising results in many voice conversion tasks [6,7,8,9,10,11]. Recently proposed flow-based voice conversion [12] and diffusion model-based voice conversion [13] also showed good performance.

Most of these works on voice conversion generally consist of three steps: analysis, mapping, and reconstruction [14]. In the analysis step, feature vectors are extracted, which are easy to process while retaining relevant information from the input waveform. The mapping step is the actual voice conversion process that changes only the identity of the speaker while preserving the linguistic content of the input feature vectors. Finally, the reconstruction step uses a vocoder to synthesize a waveform using the converted feature vectors from the previous step. Figure 1 illustrates this process.

The analysis steps of the conventional approaches use fixed feature extraction methods, such as Fourier transform and mel-scale filtering. Traditionally, these feature vectors have been widely adopted in various speech-processing areas, such as speech recognition, speaker identification, and text-to-speech. However, it could be further optimized [15,16] for voice conversion tasks, as the importance of each frequency can vary across conversion pairs. Additionally, most of the existing approaches focus on improving the feature converter model itself, as it performs the actual voice conversion task and provides its output to a pre-trained vocoder. This approach may lead to poor performance because the vocoder model is not explicitly trained to process the output feature vectors from the feature converter model. Therefore, some unnatural speech characteristics that arise from the feature converter model are not sufficiently compensated for by the vocoder model, resulting in poor speech quality. Recent studies [17,18,19,20,21,22] that proposed to jointly train feature converter models and vocoder models showed promising results. In this study, we extend the work in [22] and propose an architecture that integrates all three stages of analysis–mapping–reconstruction to optimize the quality of converted speech.

Table 1. Summary of the conventional voice conversion methods.

Authors	Year	Refs.	Dataset	Method	Model
P. L. Tobing et al.	2019	[6]	VCC2018	VAE with cycle loss	RNNT
D. Yook et al.	2020	[7]	VCC2018	Ref. [6] with multiple decoders	CNN
T. Kaneko et al.	2018	[8]	VCC2016	GAN with cycle loss	CNN
T. Kaneko et al.	2019	[9]	VCC2018	Improved [8] by modifying loss and models	CNN
T. Kaneko et al.	2020	[10]	VCC2018	Improved [9] by applying TFAN norm	CNN
T. Kaneko et al.	2021	[11]	VCC2018	Improved [9] by masking input feature	CNN
M. Proszewska et al.	2022	[12]	In-house	Flow-based model	LSTM
K. Kobayashi et al.	2016	[17]	VCC2016	Direct waveform modification	Diff VC
Y. Kurita et al.	2019	[18]	In-house	Applied [17] to a singing voice conversion task	Diff VC
K. Kobayashi et al.	2018	[19]	VCC2018	Open-source implementation of [17]	Diff VC
J.-W. Kim et al.	2020	[20]	TIDIGITS	Translation-based method using a transformer	Transformer
B. Nguyen et al.	2022	[21]	VCTK	Content and speaker disentanglement	CNN

summarizes the existing voice conversion methods.

The proposed method enables wave-to-wave (wav2wav) voice conversion by jointly training the entire analysis–mapping–reconstruction steps in an end-to-end manner. This allows the model to flexibly adjust model parameters to achieve high-quality speech output. However, training from scratch consumes too much training time and data, as it is difficult to find optimal feature representations. Furthermore, training the GAN-based models used in the mapping and reconstruction steps is known to be inherently difficult. Our proposed method overcomes these difficulties in two ways. The first is that the analysis step uses two-layer convolutional networks (CNNs) initialized with a discrete Fourier transform (DFT) matrix and mel-filterbank coefficients. This allows the analysis step to start from the traditional mel-spectrogram extraction method and gradually optimize the parameters, thereby achieving efficient learning and flexible parameter adjustment. Second, the difficulties of training the GAN models in the proposed wav2wav method are addressed using pre-training and two-phase training techniques. This allowed us to train the proposed wav2wav model reliably on small datasets, such as VCC2018 [23]. To verify the performance of the proposed method, objective and subjective evaluations were performed using the mel-cepstral distance (MCD) metric and the mean opinion score (MOS) metric, respectively.

The contributions of this paper are as follows:

• We propose a novel wave-to-wave voice conversion architecture that jointly trains analysis–mapping–reconstruction modules for high-quality voice conversion.
• We provide an efficient training algorithm so that the proposed GAN-based integrated model can be reliably trained with very small amounts of training data, such as VCC2018.
• The supervised learning process of standalone vocoders were modified to accommodate unsupervised learning in the end-to-end learning of the integrated model.
• We demonstrate the usefulness of the proposed method using both objective and subjective measures.

The rest of the paper is organized as follows. In Section 2, relevant previous studies are reviewed. Section 3 describes the proposed wav2wav model. Section 4 analyzes the experimental results. Finally, Section 5 concludes the paper.

2. Related Works

In this section, we describe the CycleGAN-based voice conversion and a HiFi-GAN-based vocoder. CycleGAN-based algorithms are used as the mapping stage in many voice conversion systems due to the excellent quality of the converted speech. Recently, HiFi-GAN vocoders have been widely adopted because of the high quality of the synthesized speech and stability in training, while many other deep neural network-based vocoders exhibit high quality but are difficult to train.

2.1. CycleGAN-Based Voice Conversion

One way of training voice conversion systems requires the use of parallel training data consisting of identical transcription utterances from different speakers. However, collecting such data is expensive. The CycleGAN-based voice conversion algorithm uses two pairs of generator and discriminator for training without such parallel data. One pair converts the spectral feature from a source speaker to that of a target speaker and the other pair converts vice versa. Using both generators, it is possible to convert the spectral feature from the source speaker into that of the target speaker and then to convert it back to that of the original source speaker. Two constraints are given in the training process so that the speaker identity of the spectral feature is changed to the target speaker while the linguistic content of the utterance is preserved. The first constraint is that the spectral features before and after the cyclic conversion must have the same linguistic content, which is enforced by a cycle consistency loss [24]. The second constraint is that the speaker identity of the resulting spectral features from each generator must be indistinguishable from that of the target speaker, which is enforced by an adversarial loss from the discriminators [5].

Figure 2 illustrates the training process of the CycleGAN-based mapping model. Source speaker $A$’s waveform $w^A$ is converted to spectral feature $s^A$ using spectral analysis, and the spectral feature is used as input into the generator model:

$$s^A=F\left(w^A\right) ,$$

(1)

where $F$ represents the spectral analysis model. The resulting spectral feature $s^A$ is converted into $s^{A\to B}$ through the generator model where the identity of the speaker is changed from speaker $A$ to speaker $B$:

$$s^{A\to B}=G_{A\to B}\left(s^A\right) ,$$

(2)

where $G_{A\to B}$ represents the generator model that converts the speaker identity of the input spectral feature from speaker $A$ to speaker $B$. The spectral feature $s^A$ of the source speaker $A$ is converted to $s^{A\to B}$; then, it is converted back to $s^{A\to B\to A}$. The cycle consistency loss is measured between $s^A$ and $s^{A\to B\to A}$ to ensure the linguistic content preservation. Similarly, spectral feature $s^B$ from the target speaker $B$ is converted to $s^{B\to A}$; then, it is converted back again to $s^{B\to A\to B}$, which provides another cycle consistency loss. The discriminators $D_A$ and $D_B$ provide the adversarial losses of $s^{B\to A}$ and $s^{A\to B}$, respectively.

The recently proposed MaskCycleGAN-VC [11], which is a variant of CycleGAN-based voice conversion, achieved a state-of-the-art performance by using masked spectral features. The generator learns to restore the masked spectral parts, which improves the robustness of the generator. However, it should be noted that MaskCycleGAN-VC is a standalone mapping method that is not jointly optimized with the analysis stage nor with the reconstruction stage.

2.2. HiFi-GAN Vocoder

A vocoder reconstructs waveforms by synthesizing them from input spectral features. Rule-based vocoders, such as Griffin-Lim [25] and WORLD [26] vocoders, were widely used because of their simplicity, but the quality of the resulting speech was not satisfactory. Recently, many deep learning-based neural vocoders with excellent sound quality have been proposed. One such neural vocoder, WaveNet [27], produces high-quality speech signals, but is slow to train and inference due to its autoregressive structure. Many works attempted to minimize the computational overhead of WaveNet while maintaining high-quality outputs [28,29,30,31,32].

The recently proposed MelGAN [33] and HiFi-GAN [34] are GAN-based vocoders that can obtain high-quality speech at a much faster speed due to their non-autoregressive characteristics. HiFi-GAN uses two types of discriminators: multi-scale discriminator (MSD) [33] and multi-period discriminator (MPD) [34]. These two discriminators enable HiFi-GAN to produce state-of-the-art performance when generating high quality speech signals. However, it should be noted that the spectral feature conversion model and the vocoder model are trained independently in the conventional voice conversion approaches, which means that the vocoder is not trained to handle any residual characteristics of the source speakers that may remain in the outputs of the spectral conversion model, leading to degradation of the output waveform for voice conversion.

3. Proposed Method

As discussed in the previous section, traditional voice conversion approaches train the spectral feature conversion models and vocoder models separately and use a fixed feature extraction scheme, leading to the degraded quality of converted speech. The proposed method, depicted in Figure 3, adopts a wave-to-wave approach, which uses waveforms directly as the input and output of the system and jointly trains the analysis, mapping, and reconstruction modules in an end-to-end manner, enabling more flexible feature analysis and natural speech generation.

However, training the feature extraction module from scratch in this end-to-end learning may require too much data and training time. Since the mel-spectral features are widely used in various speech-processing tasks, we decided to improve their performance further by learning detailed parameters suitable for voice conversion based on them. We first reproduced the mel-spectral feature extraction process with a two-layer CNN initialized with a DFT matrix and mel-filterbank coefficients; then, each weight was adjusted during training. Figure 4 shows the feature extraction network. The output of the first convolution layer, which initially consisted of real and imaginary components extracted using conventional feature extraction methods, was converted to magnitude vectors via square, summation, and square root operations. The resulting magnitude vectors were fed into the second convolutional layer, followed by rectified linear unit (ReLU) and logarithm operations. The CNN parameters were shared between the two generators and updated during training.

By including the feature-extracting CNN and vocoder into a generator, the proposed voice conversion model enables waveform inputs and waveform outputs, resulting in wave-to-wave voice conversion, shown as follows:

$$w^{A\to B}=G_{A\to B}\left(w^A\right) ,$$

(3)

where $G_{A\to B}$ is the generator model that converts the speaker identity of the input speech from speaker $A$ to speaker $B$, and $w^{A\to B}$ is the converted waveform from speaker $A$’s waveform $w^A$ to match speaker $B$’s voice characteristics.

The loss function of the proposed method consists of adversarial loss [5], cycle consistency loss [35], identity-mapping loss [36], and feature-matching loss [34], similar to other CycleGAN-based methods, as shown below:

$${\mathcal{L}}_{\mathrm{a}\mathrm{d}\mathrm{v}1}\left(G_{A\to B},D_B\right)={\mathbb{E}}_{w^B}\left[\log D_B\left(F\left(w^B\right)\right)\right]+{\mathbb{E}}_{w^A}\left[\log \left(1-D_B\left(F\left(G_{A\to B}\left(w^A\right)\right)\right)\right)\right] ,$$

(4)

$${\mathcal{L}}_{\mathrm{a}\mathrm{d}\mathrm{v}2}\left(G_{A\to B},D_B\right)={\mathbb{E}}_{w^B}\left[\log D_B\left(F\left(w^B\right)\right)\right]+{\mathbb{E}}_{w^{B\to A}}\left[\log \left(1-D_B\left(F\left(G_{A\to B}\left(w^{B\to A}\right)\right)\right)\right)\right] ,$$

(5)

$${\mathcal{L}}_{\mathrm{c}\mathrm{y}\mathrm{c}}\left(G_{A\to B}\right)={\mathbb{E}}_{w^B}\left[{‖F\left(G_{A\to B}\left(G_{B\to A}\left(w^B\right)\right)\right)-F\left(w^B\right)‖}_1\right] ,$$

(6)

$${\mathcal{L}}_{\mathrm{i}\mathrm{d}}\left(G_{A\to B}\right)={\mathbb{E}}_{w^B}\left[{‖F\left(G_{A\to B}\left(w^B\right)\right)-F\left(w^B\right)‖}_1\right] ,$$

(7)

$${\mathcal{L}}_{\mathrm{f}\mathrm{m}}\left(G_{A\to B}\right)={\mathbb{E}}_{w^B}\left[{\sum }_i{‖D_B^i\left(F\left(G_{A\to B}\left(G_{B\to A}\left(w^B\right)\right)\right)\right)-D_B^i\left(F\left(w^B\right)\right)‖}_1\right] ,$$

(8)

where $\mathbb{E}$ stands for the expectation operation, $G_{B\to A}$ is the generator model that converts the speaker identity of the input speech from speaker $B$ to speaker $A$, and $D_B^i$ represents the $i$-th layer of the discriminator. Since the output of the generator model is waveforms, they need to be converted to spectral features to calculate the losses. The losses for the other conversion direction, i.e., ${\mathcal{L}}_{\mathrm{a}\mathrm{d}\mathrm{v}1}\left(G_{B\to A},D_A\right)$, ${\mathcal{L}}_{\mathrm{a}\mathrm{d}\mathrm{v}2}\left(G_{B\to A},D_A\right)$, ${\mathcal{L}}_{\mathrm{c}\mathrm{y}\mathrm{c}}\left(G_{B\to A}\right)$, ${\mathcal{L}}_{\mathrm{i}\mathrm{d}}\left(G_{B\to A}\right)$, and ${\mathcal{L}}_{\mathrm{f}\mathrm{m}}\left(G_{B\to A}\right)$, are similarly defined.

In the conventional analysis–mapping–reconstruction approaches, such as in Figure 1, the discriminator of the spectral feature conversion model and the vocoder model serve different purposes. The discriminator in the spectral feature conversion model aims to distinguish the speaker’s identity from the target speaker’s spectral feature and the converted spectral feature. On the other hand, the discriminator in the vocoder model aims to distinguish the original and synthesized waveforms. Since the generator in the proposed method includes the function of a vocoder, the loss in the proposed method is calculated by using the discriminator of the spectral feature conversion model and the vocoder model together. Through this, the proposed voice conversion model can generate waveforms similar to those from the target speaker while maintaining the quality of the original waveforms. We used a modified version of HiFi-GAN as the generator of the proposed method. Therefore, the discriminators of the proposed method utilize the HiFi-GAN discriminator, as well as the conventional GAN discriminator.

One important limitation of GAN-based models is that they are difficult to train, especially for complex models like the proposed one, where the HiFi-GAN-style vocoder is used as the generator in the CycleGAN-style training. Since it is not easy to stabilize the training process and to avoid the collapsing problem, we applied two techniques to alleviate this problem. The first technique was to pre-train the generators using the speech data of the source and target speakers in the training data. Since the supervised training of the generator was relatively easy to converge, we first trained them to bootstrap the training in the CycleGAN-style training process. The second technique was to use a phase-wise model update method. The generator and discriminator models were trained in two alternating phases. The first phase used the following loss function:

$${\mathcal{L}}_{\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{e}1}={\mathcal{L}}_{\mathrm{a}\mathrm{d}\mathrm{v}1}\left(G_{A\to B},D_B\right)+{\mathcal{L}}_{\mathrm{a}\mathrm{d}\mathrm{v}1}\left(G_{B\to A},D_A\right)+{\lambda }_{\mathrm{i}\mathrm{d}} \left({\mathcal{L}}_{\mathrm{i}\mathrm{d}}\left(G_{A\to B}\right)+{\mathcal{L}}_{\mathrm{i}\mathrm{d}}\left(G_{B\to A}\right)\right) ,$$

(9)

where ${\lambda }_{\mathrm{i}\mathrm{d}}$ is the weight for the identity-mapping loss. The second phase utilized the following loss function:

$$\begin{array}{l}{\mathcal{L}}_{\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{e}2}={\mathcal{L}}_{\mathrm{a}\mathrm{d}\mathrm{v}2}\left(G_{A\to B},D_B\right)+{\mathcal{L}}_{\mathrm{a}\mathrm{d}\mathrm{v}2}\left(G_{A\to B},MS{D}_B\right)+{\mathcal{L}}_{\mathrm{a}\mathrm{d}\mathrm{v}2}\left(G_{A\to B},MP{D}_B\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\mathcal{L}}_{\mathrm{a}\mathrm{d}\mathrm{v}2}\left(G_{B\to A},D_A\right)+{\mathcal{L}}_{\mathrm{a}\mathrm{d}\mathrm{v}2}\left(G_{B\to A},MS{D}_A\right)+{\mathcal{L}}_{\mathrm{a}\mathrm{d}\mathrm{v}2}\left(G_{B\to A},MP{D}_A\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\lambda }_{\mathrm{c}\mathrm{y}\mathrm{c}}\left({\mathcal{L}}_{\mathrm{c}\mathrm{y}\mathrm{c}}\left(G_{A\to B}\right)+{\mathcal{L}}_{\mathrm{c}\mathrm{y}\mathrm{c}}\left(G_{B\to A}\right)\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\lambda }_{\mathrm{f}\mathrm{m}}\left({\mathcal{L}}_{\mathrm{f}\mathrm{m}}\left(G_{A\to B}\right)+{\mathcal{L}}_{\mathrm{f}\mathrm{m}}\left(G_{B\to A}\right)\right)\end{array}$$

(10)

where ${\lambda }_{\mathrm{c}\mathrm{y}\mathrm{c}}$ and ${\lambda }_{\mathrm{f}\mathrm{m}}$ are the weights for the cycle consistency loss and feature-matching loss, respectively; $MS{D}_A$, $MP{D}_A$, $MS{D}_B$, and $MP{D}_B$ are the MSD and MPD of speaker $A$ and speaker $B$, respectively.

Algorithm 1 summarizes the alternating phase-wise training process of the proposed method. The parameters of the two generators are initialized with the pre-trained vocoder using the speech data from the source and target speakers. The parameters of the feature-extracting CNN are also initialized with the coefficients of the DFT matrix and mel-filterbanks. In the first training phase, speaker $A$’s waveform was fed into the generator $G_{A\to B}$ to generate speaker $B$’s speech. Using this converted waveform $w^{A\to B}$, the generator $G_{A\to B}$ and discriminator $D_B$ were trained. Also, the generator $G_{B\to A}$ and discriminator $D_A$ were trained in a similar fashion by swapping the source and target speakers.

In the second phase, the converted waveform $w^{B\to A}$ from the first phase was fed into the generator $G_{A\to B}$ to generate waveform $w^{B\to A\to B}$. Then, the generator $G_{A\to B}$, discriminator $D_B$, $MS{D}_B$, and $MP{D}_B$ were trained. Similarly, the generator $G_{B\to A}$, discriminator $D_A$, $MS{D}_A$, and $MP{D}_A$ were trained by swapping the source and target speakers. These two phases were repeated until the model converged. It should be noted that the first phase was completely unsupervised learning using non-parallel data, that is, $w^A$ and $w^B$ were not the same transcription utterances. On the other hand, since the second phase required parallel data to compute the feature-matching loss, the pseudo-parallel data, that is, $w^A$ and $w^{A\to B}$ (as well as $w^B$ and $w^{B\to A}$), which were the same transcription utterances from different speakers, were utilized.

Algorithm 1 wav2wav

1       Initialization: load pretrained parameters. 2       repeat 3$\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\mathrm{Select} w^A$$\mathrm{and} w^B$ from the training data randomly.                /* Phase 1 */ 4$\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{w}^{A\to B}\leftarrow G_{A\to B}\left(w^A\right)$ 5$\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{w}^{B\to A}\leftarrow G_{B\to A}\left(w^B\right)$ 6            Compute $\nabla {\mathcal{L}}_{\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{e}1}$ using $w^A$, $w^{B\to A}$, $w^B$, and $w^{A\to B}$. 7$\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{G}_{A\to B}\leftarrow G_{A\to B}-\eta \nabla {\mathcal{L}}_{\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{e}1}$ 8$\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{D}_B\leftarrow D_B+\eta \nabla {\mathcal{L}}_{\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{e}1}$ 9$\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{G}_{B\to A}\leftarrow G_{B\to A}-\eta \nabla {\mathcal{L}}_{\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{e}1}$ 10          $D_A\leftarrow D_A+\eta \nabla {\mathcal{L}}_{\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{e}1}$                /* Phase 2 */ 11          $w^{B\to A\to B}\leftarrow G_{A\to B}\left(w^{B\to A}\right)$ 12          $w^{A\to B\to A}\leftarrow G_{B\to A}\left(w^{A\to B}\right)$ 13          Compute $\nabla {\mathcal{L}}_{\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{e}2}$ using $w^A$, $w^{A\to B}$, $w^{A\to B\to A}$, $w^B$, $w^{B\to A}$, and $w^{B\to A\to B}$. 14          $G_{A\to B}\leftarrow G_{A\to B}-\eta \nabla {\mathcal{L}}_{\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{e}2}$ 15          $D_B\leftarrow D_B+\eta \nabla {\mathcal{L}}_{\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{e}2}$ 16          $MS{D}_B \leftarrow MS{D}_B + \eta \nabla {\mathcal{L}}_{\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{e}2}$ 17          $MP{D}_B \leftarrow MP{D}_B + \eta \nabla {\mathcal{L}}_{\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{e}2}$ 18          $G_{B\to A}\leftarrow G_{B\to A}-\eta \nabla {\mathcal{L}}_{\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{e}2}$ 19          $D_A\leftarrow D_A+\eta \nabla {\mathcal{L}}_{\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{e}2}$ 20          $MS{D}_A \leftarrow MS{D}_A + \eta \nabla {\mathcal{L}}_{\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{e}2}$ 21          $MP{D}_A \leftarrow MP{D}_A + \eta \nabla {\mathcal{L}}_{\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{e}2}$ 22      until convergence

4. Experiments

In the experiments, we used a subset of the VCC2018 [35] dataset used in many existing voice conversion studies to show that the proposed method can be effective, even with a small amount of data. Two female speakers (SF3, TF1) and two male speakers (SM3, TM1) were used as both source speakers and target speakers. We denote these speakers as F1, F2, M1, and M2, respectively. We used 81 training sentences and 35 test sentences for each speaker. A preliminary experiment was conducted to decide the optimal values of the hyperparameters ${\lambda }_{\mathrm{i}\mathrm{d}}$, ${\lambda }_{\mathrm{c}\mathrm{y}\mathrm{c}}$, and ${\lambda }_{\mathrm{f}\mathrm{m}}$, which were found to be 30, 45, and 0.5, respectively. We used MaskCycleGAN [11] as a baseline for comparison since it was the state-of-the-art among the CycleGAN-based voice conversion methods. For fair comparison, we trained a MelGAN to synthesize the waveforms from the output of MaskCycleGAN using the two speakers, i.e., the source and target speakers, for each conversion direction.

4.1. Objective Evaluation

We used MCD [37] as a measure for the objective evaluation. MCD measures the distance between a pair of spectral features. MCD is calculated as follows:

$$\mathrm{MCD}=\frac{10}{\ln 10}\sqrt{2{\sum }_{i=1}^N{\left(m_{k,i}^t-m_{k,i}^c\right)}^2} ,$$

(11)

where $N$, $m^t$, $m^c$, and $k$ denote the mel-cepstral coefficient (MCC) dimension, target speech MCC, converted speech MCC, and the frame index, respectively. We used a 16-dimensional MCC. Since the converted spectral features may have different lengths from the target, the dynamic time warping (DTW) algorithm was applied to compensate for the length difference. A lower MCD value between the converted speech and the target speech indicates a better voice conversion performance.

Table 2. Average MCDs and 95% confidence intervals of the speeches converted by MaskCycleGAN and wav2wav.

	Conversion Direction	MaskCycleGAN	wav2wav
Intra-gender	F1→F2	7.68 ± 0.29	6.08 ± 0.26
	F2→F1	7.44 ± 0.23	6.01 ± 0.26
	M1→M2	7.96 ± 0.27	6.04 ± 0.22
	M2→M1	7.04 ± 0.17	5.88 ± 0.18
	Average	7.53 ± 0.13	6.01 ± 0.11
Inter-gender	F1→M2	8.48 ± 0.21	5.86 ± 0.16
	M2→F1	8.58 ± 0.21	5.75 ± 0.16
	M1→F2	8.70 ± 0.28	6.47 ± 0.24
	F2→M1	8.36 ± 0.23	6.38 ± 0.22
	Average	8.53 ± 0.12	6.12 ± 0.11
Average		8.03 ± 0.10	6.06 ± 0.08

summarizes the average MCDs and 95% confidence intervals of the converted speeches by MaskCycleGAN and the proposed wav2wav. The proposed method outperformed MaskCycleGAN in both intra-gender and inter-gender cases, with an average MCD of 6.06 and a 95% confidence interval of ±0.08, demonstrating superior performance and more stable learning than the baseline method.

An ablation experiment was performed to analyze the proposed method further. First, we replaced the MelGAN with the HiFi-GAN for the baseline MaskCycleGAN, which we call “spec2spec” in

Table 3. Average MCDs and 95% confidence intervals for MaskCycleGAN, spec2spec, wav2spec, spec2wav, and wav2wav.

Method	Feature Extractor	Vocoder	MCD
MaskCycleGAN	DFT	MelGAN	8.03 ± 0.10
spec2spec	DFT	HiFi-GAN	7.88 ± 0.12
wav2spec	CNN	HiFi-GAN	7.52 ± 0.11
spec2wav	DFT	N/A	6.82 ± 0.10
wav2wav	CNN	N/A	6.06 ± 0.08

. The same amount of training data was used to train both vocoders. Using the HiFi-GAN vocoder improved the MCD over the baseline MelGAN vocoder. Next, we replaced the analysis module of the “spec2spec” with the feature extracting CNN (Figure 4), which is called “wav2spec” in the table. That is, “wav2spec” took waveforms as input and outputs spectral features; then, it reconstructed waveforms using a separately trained HiFi-GAN vocoder. By integrating the analysis and mapping modules, the voice conversion performance was improved further. The third variant, called “spec2wav”, did not use the feature extracting CNN, but used the vocoder as the generator. That is, only the mapping and reconstruction modules were integrated, leaving the analysis module standalone. It can be seen in the table that “spec2wav” showed a better performance than “wav2spec”, indicating that merging the mapping and reconstruction modules was more effective than merging the analysis and mapping modules. The last row of Table 3 represents the proposed wav2wav that integrates all three modules: analysis, mapping, and reconstruction, which achieved the best performance among all the variants.

To analyze the spec2wav and wav2wav further, we looked into the learned weights of the feature-extracting CNN. The weights of the first CNN layer were not changed much from the initial weights. However, the second CNN layer, which initially corresponded to the mel-filterbanks, changed a lot. Figure 5 shows the learned weights of the second CNN layer after the training was completed. The learned weights were quite different from the original filterbank coefficients, where they covered wider frequency bands for each filter. Also, they were different from speaker to speaker.

4.2. Subjective Evaluation

Subjective evaluations of the sound quality and similarity for the proposed wav2wav and the baseline MaskCycleGAN were conducted using the MOS and XAB tests, respectively. A total of 80 utterances (4 utterances × 8 conversion directions × 2 methods + 16 original utterances) were used for the sound quality test, and 96 utterances (8 utterances × 4 conversion directions × 2 methods + 32 original utterances) were used for the similarity test. A total of 10 listeners participated in the experiment. For the sound quality test, the participants were asked to rate each utterance on a scale of 1 to 5, with 1 being “very bad” and 5 being “very good.” For the similarity test, the participants were asked to indicate which of the two converted speeches from the two methods sounded more similar to the target speaker’s utterance. Each time, a target speaker’s utterance was played first, then the two converted speeches were played in random order.

Figure 6 summarizes the sound quality test results. It shows that the proposed method significantly improved the sound quality of the converted voice compared with the baseline method, regardless of the conversion directions and conversion pairs. The proposed method achieved an average MOS of 3.82, which was significantly higher than the baseline score of 2.76. Figure 7 shows how similar the converted voice was to the target speaker’s voice. The numbers in the graph indicate the proportion chosen as more similar to the target speaker’s voice. On average, 92% of the utterances converted by the proposed method were selected as being more similar to the target speakers’ voices. Some sample speech waveforms produced by wav2wav are available online at https://wav2wav.github.io/wav2wav (accessed on 20 April 2024).

5. Conclusions

In this paper, we introduce wav2wav, a wave-to-wave voice conversion approach enabling end-to-end training, which overcomes the limitations of traditional analysis–mapping–reconstruction methods. By incorporating a feature-extracting CNN into the existing vocoder and by modifying it to act as the generator of the CycleGAN-style voice conversion, the proposed method enables direct conversion from waveform to waveform. Since the proposed algorithm does not depend on any language-specific characteristics, it can be used for voice conversion for languages other than English. Moreover, the proposed method demonstrates the ability to learn effectively, even with a small-sized dataset of just 10 min. Both the objective and subjective evaluations confirmed that the proposed approach significantly improved the quality of converted speech and generated speech that closely resembled the target speech. Due to these characteristics, the proposed method can be used in a variety of applications, such as entertainment [1], education [2], and medicine [3] domains.

In future research, the proposed method may be extended to singing voice conversion. The method proposed in this study extracts global characteristics encompassing the entire voice of each speaker. Although this method is effective for converting general everyday conversational speech, it is expected that performance may deteriorate in cases where the pitch and rhythm change dynamically within a sentence, such as in a singing voice. Therefore, it will be possible to improve the performance of singing voice conversion by developing a new modeling method that takes into account the fact that speech characteristics can change even in a single sentence.