Zero-Shot Cross-Lingual Japanese Voice Cloning to Dub Chinese Short Dramas: A Four-System Benchmark with Human Evaluation

Abstract

Chinese short-form dramas (duanju) drive a growing demand for automated Japanese dubbing that preserves each character’s voice. The decisive component is zero-shot cross-lingual voice cloning: synthesising Japanese speech from a short Chinese reference clip. We benchmark four neural text-to-speech systems—GPT-SoVITS v3, F5-TTS, CosyVoice2-0.5B, and the non-cloning commercial baseline EdgeTTS—on an annotated short-drama corpus (79 episodes, 3692 segments, 12 speakers) with native-speaker-verified Japanese references, combining automatic intelligibility (Whisper re-transcription) and speaker-embedding similarity with a blind listening study by five JLPT-N1 raters. Among cloning systems, only GPT-SoVITS attains usable intelligibility (character error rate 0.215, against 1.152 for F5-TTS and 4.603 for CosyVoice2), while EdgeTTS is most intelligible overall (0.030). Critically, the automatic and perceptual measures dissociate: F5-TTS scores marginally higher on embedding similarity (0.729 vs. 0.722), yet raters preferr GPT-SoVITS three to one. Naturalness shows no detectable gap between GPT-SoVITS and EdgeTTS (mean opinion score 3.08 versus 3.12, $p=0.93$), a null result at a resolution of about 0.6 points. In a single-speaker probe, speaker similarity rises from 0.64 to 0.71 as the reference lengthens from 3 to 10 s. Embedding similarity alone is therefore an unreliable proxy for perceived speaker identity.

1. Introduction

The global rise of Chinese short dramas (duanju)—3- to 15-min episodes produced in an industrial cadence and watched by hundreds of millions across China, North America, Southeast Asia, Japan, and Korea—has created strong demand for cross-lingual dubbing. Subtitles alone are increasingly insufficient: on mobile, vertically framed short drama, viewers watch while the screen competes with the actors’ faces, and localised audio is what makes a series consumable in a new market. Dubbing in this cadence cannot rely on hiring a voice cast per language, which makes automated synthesis the only economically viable route.

Why cross-lingual voice cloning is the hard component. A dubbing pipeline can tolerate an imperfect translation far more easily than an unusable voice. The decisive step is therefore zero-shot cross-lingual voice cloning: given a short reference clip of an actor speaking Chinese, the task is to synthesise Japanese speech that is both intelligible and recognisably the same character. This setting stresses current text-to-speech (TTS) systems along two axes that are usually evaluated separately. The model must transfer timbre to a language on which it was not conditioned, and it must do so from references that are short, emotionally charged, and recorded over background music—conditions under which cloning systems are known to degrade, but exactly the conditions of real short-drama audio. Character identity must also stay stable across dozens of episodes; thus, a system that clones well on average but fails unpredictably on some speakers is unusable in production.

Why this is not solved by better models alone. It would be natural to assume that the rapid progress of zero-shot synthesis has already settled the matter: several systems now report cloning a speaker from a three-second prompt, and their published numbers are strong. Two things complicate that reading. The first is domain. Published zero-shot results are overwhelmingly obtained on read speech recorded in clean conditions, whereas short-drama references are short, emotionally charged, frequently overlaid with music, and delivered at conversational speed. The second is measurement. Progress in this literature is tracked largely by two automatic proxies—ASR-based intelligibility and speaker-embedding cosine—and it is not obvious that either remains faithful once systems are pushed outside the conditions in which those proxies were validated. A benchmark that reports only these numbers can therefore be simultaneously correct and misleading.

Research questions. We address three questions. (i) Among openly available zero-shot systems, which—if any—produce usable Japanese speech from Chinese short-drama references? (ii) How does the best of them compare against a commercial fixed-voice service that performs no cloning at all, which is the realistic fallback a production team would otherwise adopt? (iii) Do the standard automatic proxies rank these systems the same way human listeners do? The third question turns out to have a negative answer, and that answer is the main contribution of this paper.

This study. We benchmark four neural TTS systems on a real short-drama corpus under a single matched protocol: three zero-shot voice-cloning systems—GPT-SoVITS v3 [1], F5-TTS [2], and CosyVoice2-0.5B [3]—together with EdgeTTS [4], a commercial API with a fixed Japanese voice and no cloning, included as an upper reference for intelligibility and naturalness. Each system synthesises Japanese from subtitle text that was produced and verified by native Japanese speakers, using the corresponding actor’s original Chinese audio as the timbre reference, so that synthesis quality is measured without being confounding with translation quality. We evaluate both automatically (Whisper re-transcription for intelligibility, speaker-embedding cosine for similarity) and perceptually (via a blind listening study with five JLPT-N1 raters).

Contributions.

• We release a controlled benchmark for cross-lingual Japanese dubbing of Chinese short dramas, built on a manually annotated corpus of 79 episodes (3692 segments, 12 speaker identities) with native-speaker-verified Japanese references, and we specify a reproducible evaluation protocol (NFKC normalisation, punctuation stripping, MeCab-tokenised WER) for a language in which the whitespace-based word error rate is ill defined.
• We benchmark four TTS systems under matched conditions and show that only one of the three cloning systems reaches usable intelligibility: GPT-SoVITS attains CER 0.215 (median 0.091) compared to 1.152 for F5-TTS and 4.603 for CosyVoice2-0.5B, whose failure mode we trace to degenerate repetition loops under cross-lingual prompting.
• We show that automatic speaker-embedding similarity and human-perceived speaker identity dissociate: F5-TTS scores marginally higher on embedding cosine (0.729 vs. 0.722) yet loses similarity votes to GPT-SoVITS three to one—evidence that embedding cosine alone should not be used to rank cloning systems.
• We quantify naturalness against a commercial reference with a blind five-rater study, detecting no naturalness gap between GPT-SoVITS and EdgeTTS (MOS 3.08 vs. 3.12, Wilcoxon $p=0.93$), while the former additionally preserves speaker identity—a null result whose resolution we quantify—and we report inter-rater reliability rather than MOS alone.
• We provide a single-speaker reference-duration probe in which speaker similarity improved from 0.64 to 0.71 between ∼3 s and ∼10 s references, consistent with prior reports that longer prompts improve identity preservation.

The remainder of this paper is organised as follows. Section 2 reviews related work on media localization and neural speech synthesis. Section 3.1 describes the corpus. Section 3 details the synthesis setup and evaluation metrics. Section 4 presents the benchmark, the human listening study, an intra-lingual reference condition, and the reference-duration probe. Section 5 discusses the reliability and generalisability of the findings and their limitations. Section 6 concludes the paper.

2. Related Work

2.1. Neural Text-to-Speech

Modern TTS replaced the concatenative and statistical parametric pipelines with learned sequence models. Tacotron 2 [5] established the dominant two-stage decomposition—an attention-based encoder–decoder predicting mel spectrograms, followed by a neural vocoder—and reported naturalness close to that of recorded speech. Two limitations of that design drove most subsequent work. First, autoregressive spectrogram prediction is slow and prone to attention failures (skipping, repetition); FastSpeech 2 [6] removed the autoregression with explicit duration, pitch, and energy predictors, making synthesis parallel and more robust. Second, vocoding was the quality bottleneck; HiFi-GAN [7] showed that modelling the periodic structure of speech with multi-period discriminators yields high-fidelity waveforms far faster than real time.

A parallel line collapsed the two stages entirely. VITS [8] couples a conditional variational auto-encoder with adversarial training and a normalising-flow prior, learning text-to-waveform conversion end to end and modelling the one-to-many nature of prosody through a stochastic duration predictor; GPT-SoVITS [1], one of the systems we evaluate, descends from this family. NaturalSpeech 2 [9] instead formulates synthesis as latent diffusion over a neural audio codec, scaling to expressive multi-speaker speech. The failure modes we observe in Section 4.1—repetition loops and garbled cross-lingual output—are recognisably the attention- and stability-related pathologies this literature has been addressing, resurfacing when systems are pushed outside their training conditions.

2.2. Zero-Shot Voice Cloning

Per-speaker training is not affordable in the context of dubbing, and so the relevant capability is zero-shot cloning: conditioning synthesis on a few seconds of reference audio from an unseen speaker. Azzuni and El Saddik [10] survey this area and organise it around speaker adaptation, distinguishing few-shot, zero-shot and multilingual regimes; we work in the hardest cell of that taxonomy, zero-shot and cross-lingual at once. YourTTS [11] extended VITS with speaker embeddings and multilingual training, demonstrating cloning from short references. VALL-E [12] reframed the task as conditional language modelling over discrete neural-codec tokens and showed that in-context learning from a three-second prompt preserves not only timbre but emotion and acoustic environment. Voicebox [13] generalised this to a non-autoregressive flow-matching model trained with a text-guided infilling objective, and F5-TTS [2] and CosyVoice 2 [3]—both evaluated here—are flow-matching systems in the same lineage, with the latter adding explicit language tokens.

A recurring finding in this line is that prompt length matters: Mega-TTS 2 [14] introduces a multi-reference timbre encoder and a prosody language model trained with arbitrary-length prompts, reporting that identity preservation improves as the prompt lengthens. Our single-speaker reference-duration probe in Section 4.5 is consistent with that trend for cross-lingual cloning.

Multilinguality remains the weak point. XTTS [15] is motivated by the observation that most zero-shot systems support one language and that even the multilingual ones cover only a handful of high-resource languages; it is trained across sixteen languages to close that gap.Our setting is a harder case for which these systems are not optimised: the reference is Chinese, the target is Japanese, and the reference clips are drawn from dramatic delivery recorded over background music rather than clean read speech. Section 4.1 shows this gap is not incremental—two of the three cloning systems we test produce output that is essentially unusable.

Table 1 situates the systems discussed above and marks the four we evaluate.

Table 1. Representative neural TTS systems. “Zero-shot” means a previously unseen speaker can be cloned from a short reference without per-speaker training. The last column marks the systems benchmarked in this paper.

System	Modelling Approach	Zero-Shot	Multiling.	Tested Here
Tacotron 2 [5]	AR seq2seq + neural vocoder	—	—
FastSpeech 2 [6]	Non-AR, explicit duration/pitch	—	—
HiFi-GAN [7]	GAN vocoder (waveform stage)	n/a	n/a
VITS [8]	End-to-end CVAE + adversarial + flow	—	—
NaturalSpeech 2 [9]	Latent diffusion over audio codec	✓	—
YourTTS [11]	VITS + speaker encoder	✓	✓
VALL-E [12]	Codec language model, in-context	✓	—
Voicebox [13]	Flow matching, text-guided infilling	✓	✓
XTTS [15]	Multilingual zero-shot (16 langs.)	✓	✓
GPT-SoVITS v3 [1]	GPT acoustic model + SoVITS vocoder	✓	✓	✓
F5-TTS [2]	Flow matching	✓	✓	✓
CosyVoice 2-0.5B [3]	LLM + flow matching, language tokens	✓	✓	✓
EdgeTTS [4]	Commercial API, fixed voices	—	✓	✓

✓ indicates that the property is supported; — indicates that it is not; n/a indicates that the property does not apply to that system.

2.3. Cross-Lingual Dubbing and Media Localization

Audiovisual translation is constrained in ways ordinary text translation is not: Matusov et al. [16] customise neural machine translation for subtitling by learning where to break subtitle lines, subject to the length and display-duration limits the subtitling industry imposes—the output must fit the screen and the time available, not merely be accurate. Dubbing tightens these constraints further: beyond translating content, the synthesised speech must fit the original timing and remain identifiable as the same character. Federico et al. [17] formalise this as isochrony—aligning the translated speech to the original sequence of phrases and pauses—and show that prosodic alignment, not translation quality alone, governs perceived dubbing quality. Their pipeline couples length-controlled translation, prosodic alignment, duration-tuned TTS, and audio rendering.

Our study isolates one stage of that pipeline. By synthesising from human-verified Japanese references rather than machine output, we measure voice quality without confounding it with translation error, and we deliberately do not address isochrony: the question here is whether a usable target-language voice can be produced at all for this content, which is logically prior to fitting it to the original timing.

2.4. Speech Recognition for Media Content

Recognition and synthesis are the two directions of the same conversion, and recognition enters our work twice—as the object of the broader localization problem, and as the instrument by which we measure intelligibility. Whisper [18] established that large-scale weakly supervised training yields multilingual recognition robust enough to be used off the shelf, which is why it serves here as the automatic transcriber. Media content nonetheless stresses recognition in characteristic ways: Liu et al. [19] adapt end-to-end recognition to produce readable subtitles rather than raw transcripts, and their modular acoustic-to-word framework [20] targets the vocabulary mismatch that proper nouns create. Yang et al. [21] show that recognition on TV series benefits from video-guided post-correction, evidence that drama audio alone is often insufficient. Karakanta et al. [22] study automatic subtitling from the practising subtitler’s perspective, asking how professionals actually experience post-editing machine output rather than scoring it automatically—a methodological stance we adopt here by putting a human panel at the centre of the evaluation.

2.5. Evaluating Synthetic Speech

Synthesis quality is conventionally assessed along two axes. Intelligibility is measured by transcribing the synthesised audio with an ASR system and computing character or word error rate against the intended text. Speaker similarity is measured as the cosine similarity between speaker embeddings of the synthesised and reference audio; Resemblyzer [23] provides a lightweight offline d-vector encoder widely used for this purpose; stronger verification architectures such as ECAPA-TDNN [24], which adds channel attention and multi-layer feature aggregation to the TDNN backbone, define the current state of the art for the underlying speaker-verification task from which these encoders are borrowed. This automatic pairing is the protocol adopted in the F5-TTS [2] and CosyVoice 2 [3] papers themselves.

Both proxies are contested. Subjective mean opinion score remains the reference standard for naturalness. Automatic substitutes have been pursued for a decade—MOSNet [25] showed that a convolutional-recurrent predictor correlates well with human ratings at the system level while correlating only fairly at the utterance level—and the VoiceMOS challenge line of work, with UTMOS [26] among the strongest entries, exists precisely because predicting human judgements from the signal is difficult enough to warrant a dedicated task. That system-versus-utterance gap is itself instructive: automatic predictors are most trustworthy exactly where differences are large and least trustworthy where systems are close, which is the regime our two leading systems occupy. For speaker identity, the situation is less examined: the embedding cosine is routinely reported as though it stood in for perceived identity, but the encoders were trained for verification, not for judging whether a listener would accept a cloned voice as the same character. Our results bear directly on this. In Section 4.2, the system with the higher embedding similarity is the one listeners reject three to one, meaning that the two measures do not merely differ in scale—they order the systems differently. We therefore report both and treat the human study as the arbiter where they disagree.

2.6. Security and Dual Use of Cloned Speech

The same capability that makes dubbing viable also makes impersonation cheap, and the research community has treated this as a first-class problem rather than an afterthought. The ASVspoof challenge series [27] has run since 2015 specifically to develop countermeasures against synthetic and converted speech presented to speaker-verification systems; its 2021 edition added a dedicated speech-deepfake track in which the spoofed material is generated by exactly the class of TTS and voice-conversion systems we benchmark here. Azzuni and El Saddik [10] likewise frame their survey around generation and detection, arguing the two must advance together; recent work also examines how well human listeners distinguish imitated from natural speech [28] and whether detectors trained on synthesised speech generalise to human-imitated speech [29].

Our setting is comparatively low-risk but not risk-free. The voices cloned here belong to actors whose performances were licensed for this purpose, the cloned output reproduces the same characters’ lines from the same series, and we release no mapping between reference and cloned audio. We nonetheless note the asymmetry our results expose: the embedding-based similarity measure on which a verification system would rely rated F5-TTS output as more speaker-faithful than the output human listeners overwhelmingly preferred. A detection pipeline calibrated on embedding distance alone would inherit the same blind spot we document for evaluation.

3. Materials and Methods

3.1. Dataset

3.1.1. Source Material and Annotation

We build our evaluation corpus from a Chinese short drama sourced from a commercial entertainment company. The source material is a complete Chinese short drama series comprising 82 video clips. We selected 79 episodes that have complete parallel subtitle annotations, each consisting of three components: a video file (.mp4), a Chinese subtitle file (.srt) containing speaker IDs, and a corresponding Japanese subtitle file (.txt).

All Chinese subtitles were manually annotated by native Chinese-speaking researchers, while the Japanese translations were produced by native Japanese researchers and subsequently verified for cultural appropriateness and translation accuracy. The inclusion of speaker-ID annotations provides essential context for character-specific dialogue analysis and enables speaker-conditioned voice synthesis in the TTS stage.

3.1.2. Dataset Characteristics

The dataset comprises 79 paired episodes (124.91 min, 3692 subtitle segments averaging 46.7 per episode, 12 speaker IDs; full statistics in Table 2). Episode length is close to symmetric—mean 46.7 against a median of 47—indicating that no small group of unusually long episodes dominates the corpus. The speaker-ID annotation is what makes the corpus usable for synthesis: it lets every subtitle segment be traced to the actor who delivered it, so that reference audio and target text can be paired by character rather than by episode.

Table 2. Statistical information of the proposed dataset.

Dataset Composition		Subtitle Statistics
Video clips (total)	82	Duration of paired episodes (min)	124.91
Paired episodes	79	Subtitle segments	3692
Speaker IDs	12	Segments/episode, mean	46.7
		Segments/episode, median	47
		Segments/episode, range	22–79

The dataset reflects distinctive short-drama traits—high colloquialism (discourse particles, ellipsis, idioms), minimal sentence-final punctuation, short segments (median 6 Chinese characters, interquartile range 4–9, with limited local context), and dense character-specific proper nouns (names, nicknames, honorifics)—and, for synthesis specifically, reference clips that are short, emotionally charged, and frequently overlaid with background music, making it a realistic benchmark for cross-lingual dubbing.

3.1.3. Construction of the Evaluation Set

Figure 1 summarises how the evaluation material was derived from the source series, and Table 3 lists the per-speaker quantities. (1) Segmentation. Each subtitle segment of the 79 annotated episodes was cut from the episode audio at its .srt start and end timestamps with ffmpeg and stored as mono, 44.1 kHz, 16-bit PCM, together with its Chinese subtitle line. No denoising or music separation was applied, and so background music and effects present in the drama are retained in the clips. This yields 3692 clips. (2) Speaker grouping. Clips were grouped by their annotated speaker ID. The distribution is highly skewed: speakers 1 and 2 account for 1040 and 848 clips, whereas speakers 10–12 have seven or fewer; two clips without a speaker label were discarded. (3) Reference preparation. For each speaker, the longest clip (2.0–4.9 s) was used as the cloning reference, with its Chinese subtitle line as the prompt transcript; the same reference clip was given to all three cloning systems. For the speaker-similarity metric, a separate reference embedding was computed from the speaker’s clips (in file order) concatenated up to at least 10 s to reduce the dependence of the metric on a single short clip. (4) Target text. Three short test sentences in the register of the series’ dialogue were used as synthesis targets, each in a Chinese and a Japanese version (Table 4). (5) Evaluation sets. Every speaker was paired with every sentence, giving 36 items per system for the automatic benchmark (18 for GPT-SoVITS, which produced output for six speakers); the same items with Chinese targets form the intra-lingual reference condition of Section 4.3; and eight of the items form the listening-study subset (Section 3.4).

Figure 1. Construction of the evaluation material from the source series, from segmentation of the episode audio to the three evaluation sets.

Table 3. Per-speaker clip counts and cloning-reference durations. GPT-SoVITS output: speakers for which GPT-SoVITS produced usable synthesis; Listening: speakers whose items appear in the listening study.

Speaker	1	2	3	4	5	6	7	8	9	10	11	12
Clips	1040	848	696	546	253	160	70	52	15	7	1	2
Reference (s)	4.1	3.6	3.9	3.1	2.8	2.9	4.9	3.1	2.5	2.5	2.0	2.0
GPT-SoVITS output	✓	✓	✓	✓			✓	✓
Listening	✓	✓	✓	✓			✓	✓

✓ indicates that the system produced usable output for that speaker; a blank cell indicates that it did not.

Table 4. The three test sentences used as synthesis targets.

ID	English Gloss
S1	“This is my résumé; please have a look.”
S2	“The interview with the Song Group is very important to me.”
S3	“I will surely uncover the truth about my mother’s death.”

3.1.4. Dataset Scope and Planned Extensions

The current evaluation dataset covers a single drama series (Flash Marriage Lucky Grass/Destined). Cross-series and cross-genre evaluation would require additional annotated dramas with comparable speaker coverage, which are not yet available to us; collecting a second annotated series with a larger and more diverse cast is planned as follow-up work.

3.2. Speech Synthesis

Synthesis takes the native-speaker-verified Japanese subtitle text and, for the cloning systems, each of the 12 speakers’ original Chinese audio as a timbre reference. We evaluate four TTS systems: GPT-SoVITS [1] (v3, GPT acoustic model + SoVITS vocoder for reference-based cross-lingual cloning); F5-TTS [2] (flow-matching zero-shot synthesis); CosyVoice2-0.5B [3] (Alibaba’s LLM-based flow-matching system with language tokens); and EdgeTTS [4] (Microsoft’s API with a fixed Japanese voice and no cloning, an off-the-shelf fallback). Figure 2 summarises the evaluation design: every system receives the same Japanese text, the three cloning systems additionally receive a reference clip of the target speaker, and all outputs pass through one common evaluation stage.

Figure 2. Evaluation design. All four systems synthesise the same native-verified Japanese subtitle text; the three zero-shot systems additionally receive a reference clip of the target speaker’s original Chinese audio, while EdgeTTS uses a fixed Japanese voice and performs no cloning. Synthesised speech is scored for intelligibility (Whisper-medium re-transcription), speaker similarity (Resemblyzer, cloning systems only), and perceptual quality (blind five-rater listening study).

Evaluation sample. Although the corpus spans 79 episodes, the synthesis benchmark is deliberately narrower; Section 3.3 specifies the items, the coverage control, and the scoring protocol.

3.3. Experimental Setup

The study comprises three experiments on the same corpus: a four-system synthesis benchmark scored automatically (Section 4.1), a blind human listening study on a subset of the same audio (Section 4.2), and a reference-duration probe (Section 4.5).

Systems and variants. We evaluate GPT-SoVITS v3 [1], F5-TTS [2], and CosyVoice2-0.5B [3] as zero-shot cloning systems, and EdgeTTS [4] with the fixed neural voice ja-JP-NanamiNeural as a commercial reference that performs no cloning. All four receive identical Japanese input text; the three cloning systems additionally receive a reference clip of the target speaker’s original Chinese audio, so that any difference between them is attributable to the system rather than to the conditioning material.

Test material. For each of the 12 speaker identities we synthesise three sentences, giving $n=36$ speaker–sentence items per system. GPT-SoVITS produced usable audio for 6 of the 12 speakers, and so its averages are computed over $n=18$ items. Because this coverage difference could in principle manufacture the ranking, we repeat the comparison on the common 6-speaker subset on which every system produced output; the ordering and the size of the gaps are essentially unchanged (GPT-SoVITS CER 0.215, F5-TTS 1.269, CosyVoice2 6.514), so the conclusions do not rest on differing coverage.

Automatic scoring. Synthesised audio is transcribed with the Whisper-medium checkpoint [18] under language=ja. Reference and hypothesis are NFKC-normalised and stripped of punctuation before scoring; CER is computed at the character level and WER over sequences tokenised with the ja-mecab tokeniser, both via jiwer. Speaker similarity is the cosine between 256 and dimensional Resemblyzer [23] d-vectors of the synthesised clip and the speaker’s reference audio, and is reported only for the cloning systems.

Reference-duration probe. To isolate the effect of conditioning length from the effect of system choice, we hold the system fixed (F5-TTS) and vary only the reference clip, comparing a short (∼3 s) against a long (∼10 s) reference of one speaker for the same target sentence.

Statistical treatment. Three analyses are reported, each specified here so that it can be recomputed from the values in Table 5 and Table 6.

Table 5. TTS evaluation. Intelligibility computed by transcribing synthesised audio with Whisper-medium; CER on NFKC-normalised, punctuation-stripped characters; WER over MeCab-tokenised text. We report both mean and median CER because the mean is inflated by a small number of degenerate outputs (e.g., CosyVoice2’s repetition loops). Speaker similarity (Spk-Sim) is Resemblyzer embedding cosine (higher is better); N/A for EdgeTTS (fixed neural voice, no cloning). 12 speakers × 3 sentences per system ($n=36$); GPT-SoVITS on the 6 speakers with available synthesis ($n=18$). Best per column in bold; best among voice-cloning systems underlined. ^† EdgeTTS uses a single fixed neural voice (ja-JP-NanamiNeural) without speaker cloning; speaker similarity is not applicable. ^‡ GPT-SoVITS averaged over the 6 speakers with available synthesis audio.

Model	Method	Avg. WER ↓	Avg. CER ↓	Med. CER ↓	Spk-Sim ↑
EdgeTTS [4]	API (fixed voice)	0.056	0.030	0.000	— †
GPT-SoVITS v3 [1]	Zero-shot cloning	0.248 ‡	0.215 ‡	0.091	0.722
F5-TTS [2]	Flow matching (ZS)	0.998	1.152	1.000	0.729
CosyVoice2-0.5B [3]	Flow matching (ZS)	6.673	4.603	1.000	0.671

↓ indicates that a lower value is better. ↑ indicates that a higher value is better.

Table 6. Human listening study results ($n=5$ JLPT-N1 raters, 8 items per system, blind presentation). MOS = naturalness mean opinion score (1–5); Sim. votes = share of “most similar to reference” selections (40 votes total; 35% of votes were “none”). EdgeTTS uses a fixed voice and received no similarity votes. Best per column in bold. ^† EdgeTTS uses a fixed neural voice (ja-JP-NanamiNeural); cloning not applicable.

System	MOS ↑	Sim. Votes ↑	n
GPT-SoVITS v3 [1]	$3.08 \pm 1.14$	18 (45%)	40
EdgeTTS [4]	3.12$\pm 1.14$	0 (0%) †	40
F5-TTS [2]	$1.80 \pm 1.02$	6 (15%)	40
CosyVoice2-0.5B [3]	$1.52 \pm 0.88$	2 (5%)	40

↑ indicates that a higher value is better.

(i) Naturalness. System pairs are compared on the listening-study ratings with the Wilcoxon signed-rank test paired by rater–item, which requires neither normality nor equal variances and respects the fact that every rater scored every system.

(ii) Resolution of the naturalness comparison. Because the leading pair yields a null result, we report what that null can support. Because every rater scored both systems on every item, we use the $n=40$ paired rater–item differences: their standard deviation is $1.78$, so the standard error of the mean difference is $1.78/\sqrt{40}\approx 0.28$, and a t-based 95% interval on the MOS difference is $[-0.62,0.52]$, a half-width of about $0.6$ points. Differences smaller than this are not detectable at this panel size.

(iii) Similarity preference. Similarity votes are a forced choice with an abstention option, and so we test only the two leading systems against each other, discarding abstentions and the third system’s votes. This leaves 18 votes for GPT-SoVITS against 6 for F5-TTS, which we evaluate with a two-sided exact binomial test against the null $p=0.5$.

Panel reliability is reported as ICC(2, 1) and ICC(2, k) together with Krippendorff’s ordinal $\alpha$, rather than as an MOS alone, so that the precision of the panel mean can be judged alongside its value.

Environment. Synthesis and evaluation ran on NVIDIA A100 80 GB GPUs under Python 3.11.7, using PyTorch 2.3.1 with Hugging Face Transformers 4.51.3 for GPT-SoVITS, F5-TTS, Whisper transcription and Resemblyzer similarity, and PyTorch 2.10.0 with Transformers 4.57.6 for CosyVoice2. Synthesis and evaluation scripts will be released on publication.

3.4. Listening Study Protocol

Because the automatic measures defined below are proxies, the study is anchored by a human panel. The design follows the absolute category rating procedure of ITU-T P.800 [30], the reference methodology for subjective speech quality, adapted to a small expert panel rather than the large crowdsourced pools that ITU-T P.808 [31] targets; we prefer expert raters here because judging speaker identity across a language boundary is not a task naive listeners perform reliably.

Five raters holding JLPT-N1 certification each evaluated eight speaker–sentence items per system, giving 160 naturalness judgements in total. Figure 3 shows the full procedure, following the level of procedural detail recommended for perceptual experiments on synthetic and imitated speech [28]. Recruitment. The online questionnaire stated its eligibility criteria (native Japanese or JLPT N1/N2 proficiency; headphones or earphones recommended); five adult volunteers took part, all reporting JLPT N1 and an age range of 18–25 years. Stimulus selection. The eight items were drawn from the six speakers for which all four systems had produced output (Table 3), with each test sentence appearing two or three times; restricting the subset in this way makes every comparison within an item fully paired. For each item the reference was a short clip of the speaker’s original Chinese speech (0.7–2.8 s), which was not the clip used for cloning. Presentation. Items were presented in the same fixed order to every rater. Each item was presented on a single page containing the reference clip and the four systems’ renditions, labelled by blind letter codes (A–D) whose assignment to systems varied across items, so raters saw neither the system names nor a consistent ordering across items. The letter assignment was the same for all raters, and clips could be replayed freely. Raters supplied two responses. The first is a five-point naturalness rating in the P.800 absolute category rating sense (1 = very unnatural, 5 = very natural), aggregated as the mean opinion score. The second is a forced-choice similarity judgement: given the original Chinese reference clip for that speaker, the rater selects which synthesised rendition sounds most like the same person, with an explicit “none of them” option. Allowing abstention matters—35% of similarity votes were cast as “none”, which is itself a finding about how hard cross-lingual identity transfer is from short references, and which a forced binary choice would have concealed.

Figure 3. Procedure of the listening study, from recruitment and stimulus selection to the rating tasks and analysis.

Two properties of this design bear on interpretation. The panel is small, and so single-rater reliability is correspondingly modest; we report it explicitly rather than present MOS alone. Furthermore, the raters are proficient non-native rather than native speakers, which we expect to make naturalness judgements slightly more permissive than a native panel would be; Section 5.6 returns to both points.

3.5. Evaluation Metrics

Let y denote the reference Japanese text of an item and $\widehat{y}$ the Whisper transcription of the corresponding synthesised clip, both NFKC-normalised with punctuation stripped.

Intelligibility. Character error rate is the Levenshtein distance between the two character sequences, normalised by reference length:

$$\mathrm{CER}(\widehat{y},y)={\displaystyle \frac{S+D+I}{\left|y\right|}},$$

(1)

where S, D and I count the substitutions, deletions and insertions in a minimum-cost alignment. Note that CER is not bounded above by 1: insertions are unbounded, so a system that emits long spurious output can exceed 1, which is exactly what we observe for the degenerate cases in Section 4.1. The word error rate uses the same expression over MeCab-tokenised sequences, since whitespace-based tokenisation is undefined for Japanese. We report the median alongside the mean, because a small number of degenerate outputs dominates the mean while the median reflects typical behaviour.

Speaker similarity. Let $\varphi (·)\in {\mathbb{R}}^{256}$ be the Resemblyzer d-vector encoder [23]. For a synthesised clip a and the speaker’s reference clip r,

$$\mathrm{SpkSim}(a,r)={\displaystyle \frac{⟨\varphi (a),\varphi (r)⟩}{{\parallel \varphi (a)\parallel }_2{\parallel \varphi (r)\parallel }_2}}\in [-1,1].$$

(2)

This is defined only for the cloning systems; EdgeTTS synthesises a fixed voice unrelated to r, so no meaningful value exists.

Perceptual quality. Each of $k=5$ raters scores each of n items on a five-point scale, giving ratings $x_{ij}$. Naturalness is the mean opinion score $\mathrm{MOS}={\displaystyle \frac{1}{nk}}{\sum }_{i,j}{x}_{ij}$, reported with its standard deviation. Because the same rater scores every system, system pairs are compared with the Wilcoxon signed-rank test on differences paired by rater–item, which assumes neither normality nor equal variances.

Rater agreement is quantified by the two-way random-effects intraclass correlation with absolute agreement. Writing $M{S}_R$, $M{S}_C$ and $M{S}_E$ for the item, rater and residual mean squares,

$$\begin{array}{cc}\hfill \mathrm{ICC}(2, 1)& ={\displaystyle \frac{M{S}_R-M{S}_E}{M{S}_R+(k-1)M{S}_E+\frac{k}{n}(M{S}_C-M{S}_E)}},\hfill \\ \hfill \mathrm{ICC}(2, k)& ={\displaystyle \frac{M{S}_R-M{S}_E}{M{S}_R+\frac{1}{n}(M{S}_C-M{S}_E)}}.\hfill \end{array}$$

(3)

ICC(2, 1) describes the reliability of a single rater and ICC(2, k) that of the panel mean; the latter is the quantity relevant to a MOS reported as a panel average. We additionally report Krippendorff’s $\alpha =1-D_o/D_e$ with the ordinal difference function, where $D_o$ and $D_e$ are observed and expected disagreement, because $\alpha$ handles ordinal scales and missing judgements without assuming interval-scaled responses.

Together, Equations (1)–(3) define two automatic quantities and one human quantity. Section 4.2 shows that they are not redundant: the ordering induced by Equation (2) disagrees with the ordering induced by the human judgements, which is the central empirical finding of this study.

4. Results

4.1. Text-to-Speech Comparison

The TTS evaluation (WER, CER, Resemblyzer speaker similarity) is reported in Table 5.

We treat Spk-Sim as an automatic proxy that may not fully align with human-perceived voice similarity, and take CER as the primary intelligibility metric. EdgeTTS [4], a commercial system with a fixed professionally tuned Japanese voice, is the most intelligible overall (CER 0.030, WER 0.056), as expected for a production service that performs no voice cloning. GPT-SoVITS v3 [1] is by far the most intelligible of the three voice-cloning systems (CER 0.215, median 0.091)—roughly five times better than F5-TTS and over $20\times$ better than CosyVoice2 on mean CER—while its speaker similarity of 0.722 indicates strong voice identity preservation under cross-lingual cloning. GPT-SoVITS produced usable synthesis for six of the twelve speakers, and so its average is computed over those six; the common-subset control reported in Section 3.3 confirms the ranking does not follow from this difference. F5-TTS [2] attains the highest speaker similarity (0.729) but produces largely unintelligible cross-lingual output (CER 1.152): transcriptions frequently contain garbled multilingual fragments, indicating that it preserves timbre but fails to produce fluent Japanese from Chinese references. CosyVoice2-0.5B [3] exhibits the weakest intelligibility (mean CER 4.603, median 1.0); inspection of the transcriptions reveals a degenerate repetition failure mode on a subset of inputs, where the model emits long vowel loops instead of speech—likely a capacity limitation of the 0.5B variant under cross-lingual prompting. The dissociation between intelligibility and speaker similarity highlights an important trade-off: F5-TTS captures timbre marginally better than GPT-SoVITS (0.729 vs. 0.722) but at a catastrophic intelligibility cost.

Overall, the intelligibility ranking is EdgeTTS > GPT-SoVITS ≫ F5-TTS > CosyVoice2, while the speaker-similarity ranking among cloning systems is F5-TTS ≈ GPT-SoVITS > CosyVoice2; GPT-SoVITS offers the best overall balance and is the only cloning system with usable intelligibility.

Regarding evaluation methodology: we follow the automatic evaluation protocol adopted in the F5-TTS [2] and CosyVoice2 [3] papers, where ASR-based intelligibility and speaker-embedding cosine similarity jointly characterise zero-shot TTS quality. To validate the automatic ranking perceptually, we additionally conducted a blind human listening study, reported in Section 4.2; human naturalness follows the intelligibility ordering, whereas perceived similarity does not follow embedding similarity.

4.2. Human Listening Study

We complement the automatic metrics with a blind listening study: five JLPT-N1 raters scored eight speaker–sentence items per system (160 naturalness ratings) on a five-point MOS scale under blind letter codes, and chose the clip most similar to the reference (Table 6).

GPT-SoVITS (MOS $3.08\pm 1.14$) shows no detectable difference from commercial EdgeTTS ($3.12\pm 1.14$; Wilcoxon paired by rater–item, $z=-0.09$, $p=0.93$); the paired 95% interval on the difference is $[-0.62,0.52]$ MOS points, so the data exclude a large gap but cannot establish equivalence, and both far exceed F5-TTS (1.80) and CosyVoice2 (1.52; both $p<0.001$)—performing zero-shot cloning without a detectable naturalness deficit relative to a fixed commercial voice is a strong result. On perceived similarity GPT-SoVITS dominates (45% of votes vs. 15% F5-TTS, 5% CosyVoice2; EdgeTTS 0%); although F5-TTS has marginally higher automatic similarity (0.729 vs. 0.722; Table 5), raters preferred GPT-SoVITS 3:1, showing embedding similarity alone misses perceptual identity, and the 35% “none” votes reflect the difficulty of cross-lingual cloning from short references. Inter-rater reliability is ICC(2,1) = 0.30/ICC(2,k) = 0.68 (Krippendorff’s ordinal $\alpha$ = 0.31), which is low (Section 5.6). The human study thus corroborates the intelligibility ranking: GPT-SoVITS uniquely combines commercial-grade naturalness with effective cloning.

Figure 4 makes the divergence concrete. Read only through panel (a), a practitioner choosing on embedding similarity would be indifferent between GPT-SoVITS and F5-TTS, or would mildly prefer F5-TTS; panel (b) shows that listeners are not indifferent at all. The reason is visible in the same figure: the two systems differ by roughly a factor of five in character error rate, and naturalness follows intelligibility. A voice that preserves timbre but garbles the words is not perceived as the same character speaking—it is perceived as a failure. This is why we the treat embedding cosine as a screening statistic rather than a ranking criterion, and why we regard the listening study as the arbiter where the two disagree.

Figure 4. The two evaluation axes order the cloning systems differently. (a) Automatic metrics: GPT-SoVITS and F5-TTS sit within 0.007 of each other on Resemblyzer speaker similarity despite a fivefold gap in character error rate, so embedding cosine alone separates them hardly at all. (b) The blind listening study: raters chose GPT-SoVITS as most similar to the reference three times as often as F5-TTS, and naturalness (MOS, printed under each system) tracks intelligibility rather than embedding similarity. EdgeTTS performs no cloning and therefore has no Spk-Sim value and received no similarity votes, but is included in (b) as the commercial naturalness reference.

4.3. Intra-Lingual Reference Condition

To separate failures caused by crossing the language boundary from failures of the systems themselves, we also scored the same speakers, references and sentences with Chinese targets (Chinese reference → Chinese speech), using an identical protocol: Whisper-medium transcription with language=zh, NFKC normalisation, conversion of traditional to simplified characters, punctuation stripping, character error rate, and the same Resemblyzer reference embeddings (Table 7).

Table 7. Intra-lingual (Chinese → Chinese) versus cross-lingual (Chinese → Japanese) synthesis for the three cloning systems, with the same references and sentences. CER: mean character error rate; Spk-Sim: Resemblyzer cosine. GPT-SoVITS on its 6 speakers ($n=18$), other systems on all 12 ($n=36$).

System	Chinese → Chinese		Chinese → Japanese
	CER ↓	Spk-Sim ↑	CER ↓	Spk-Sim ↑
GPT-SoVITS v3	0.175	0.674	0.215	0.722
F5-TTS	0.169	0.708	1.152	0.729
CosyVoice2-0.5B	2.463	0.673	4.603	0.671

↓ indicates that a lower value is better; ↑ that a higher value is better.

The contrast is diagnostic. F5-TTS is as intelligible as GPT-SoVITS when it speaks the language of its reference (CER 0.169 versus 0.175; median 0 for both) but becomes unintelligible when asked to speak Japanese (1.152), and so its failure in the main benchmark is specifically a failure of cross-lingual transfer. CosyVoice2-0.5B fails in both conditions (CER 2.463 intra-lingually, with the same repetition loops), so its failure is not primarily cross-lingual. GPT-SoVITS degrades only modestly across the language boundary (0.175 to 0.215). Embedding similarity changes by at most 0.05 between conditions for any system, which again shows that it does not register the loss of intelligibility.

4.4. Failure Modes

Aggregate error rates compress qualitatively different failures into one number. Inspecting the transcriptions and the synthesised audio separates three distinct modes, only one of which is a matter of degree.

(i) Language-transfer failure. F5-TTS reproduces the reference timbre well—its embedding similarity, 0.729, is the highest we measure—yet its transcriptions frequently contain garbled multilingual fragments rather than Japanese. The model carries the voice across the language boundary but not the phonology: conditioned on a Chinese reference, it does not reliably produce fluent Japanese. This is the most instructive failure for our purposes, because it is invisible to speaker similarity by construction. A metric that asks, “Does this sound like the same person?” returns a high score for audio that a listener would not accept as speech in the target language at all.

(ii) Degenerate generation. CosyVoice2-0.5B exhibits a different pathology: on a subset of inputs the model enters a repetition loop, emitting long vowel sequences in place of speech. Because insertions are unbounded in Equation (1), these items push the mean CER to 4.603 while the median stays at 1.0—the gap between the two statistics is the signature of this mode. We attribute it to the capacity of the 0.5B variant under cross-lingual prompting rather than to the flow-matching formulation itself, since F5-TTS shares the latter without showing the same behaviour.

(iii) Coverage failure. GPT-SoVITS, the system we ultimately recommend, produced usable synthesis for only six of the twelve speakers. This is not a quality deficit on the items it does produce—its CER and MOS are computed on those items and are strong—but a reliability deficit, and for production dubbing it is arguably the more serious of the two. A system that silently declines half a cast requires a human in the loop for every character, which removes much of the economic case for automation. We verified that this coverage difference does not manufacture the ranking: restricting every system to the common 6-speaker subset leaves the ordering and the gaps essentially unchanged (Section 3.3).

These modes are not ordered on a single quality axis. A practitioner choosing between them faces qualitatively different risks—unusable output, unpredictable output, or absent output—and no single scalar, automatic or human, expresses that choice.

4.5. Reference-Duration Probe

Zero-shot cloning quality depends on the reference clip. Comparing F5-TTS under an ∼3 s vs. an ∼10 s reference of the same speaker for the same sentence, speaker similarity improves $0.64\to 0.71$ (Table 8). This probe covers a single speaker and sentence, so it illustrates the expected trend rather than establishing its size.

Table 8. TTS reference audio duration probe using F5-TTS (one speaker, one target sentence). Speaker similarity measured via cosine similarity of speaker embeddings.

Reference Duration	Speaker Similarity ↑
Short (∼3 s)	0.64
Long (∼10 s)	0.71

↑ indicates that a higher value is better. Bold marks the better of the two reference durations.

5. Discussion

5.1. Why Intelligibility Governs Perceived Identity

Our central result is that two systems can be nearly indistinguishable on speaker-embedding similarity (0.722 vs. 0.729) and yet be preferred three to one by listeners. The explanation is not that the embedding is wrong about timbre—F5-TTS does reproduce the reference voice colour, which is what a verification encoder is trained to detect—but that timbre is only one component of what a listener uses to judge “the same character speaking”. When the phonetic content is degraded (F5-TTS reaches CER 1.152, i.e., more errors than reference characters), the listener is no longer comparing two renditions of the same sentence; they are comparing a sentence with a failure. Perceived identity appears to be conditional on intelligibility rather than additive with it.

This has a concrete methodological consequence. Speaker encoders such as the one behind Equation (2) were trained for verification, a task in which the utterance is assumed intelligible and only the speaker is in question. Using such an encoder to rank synthesis systems silently imports that assumption. Where it fails—precisely in the cross-lingual, out-of-domain regime where zero-shot cloning is hardest—the metric is least informative exactly when it is most needed. We therefore report embedding similarity as a screening statistic: useful for detecting gross timbre failure (CosyVoice2’s 0.671), but not for ordering systems that have already cleared that bar.

5.2. Implications for Dubbing Pipelines

Three practical points follow. First, for Chinese-to-Japanese short-drama dubbing, only one of the three open cloning systems we tested is currently deployable: GPT-SoVITS is the only one for which we detect no naturalness gap against a commercial fixed voice ($p=0.93$), while additionally preserving speaker identity. Second, reliability matters as much as mean quality—GPT-SoVITS produced usable audio for only six of twelve speakers, and a system that fails unpredictably on half a cast cannot be used unattended regardless of its average score. Third, reference length is likely a cheap lever: in our single-speaker probe, extending the reference from ∼3 s to ∼10 s raised speaker similarity from 0.64 to 0.71, consistent with prior work [14], so collecting longer clean references per character is worthwhile, although the size of the benefit in this domain remains to be measured.

More broadly, our results argue for keeping a human panel in the loop when selecting among cloning systems, even a small one. The listening study reported here involved five raters and 160 naturalness judgements, yet it reversed the ranking that automatic embedding similarity would have produced.

5.3. Reliability and Generalisability of the Perceptual Findings

The listening panel is small (five raters) and homogeneous: all raters were proficient non-native listeners holding JLPT N1, all aged 18–25 years, recruited as volunteers. Figure 5 shows where this matters. Four of the five raters rated F5-TTS and CosyVoice2 below GPT-SoVITS (the exception, R3, rated F5-TTS at 3.12 against 3.00 for GPT-SoVITS), but the raters disagreed far more sharply about the commercial EdgeTTS voice, whose per-rater mean ranges from 1.88 to 5.00; this disagreement, rather than disagreement about the cloning systems, drives the low single-rater reliability (ICC(2, 1) = 0.30). Accounting for between-rater variation by resampling whole raters (rater-clustered percentile bootstrap, ${10}^4$ replicates) widens the 95% interval on the GPT-SoVITS minus EdgeTTS naturalness difference from $[-0.62,0.52]$ to $[-1.52,1.00]$. We therefore draw two conclusions with different strength. The separation of GPT-SoVITS and EdgeTTS from F5-TTS and CosyVoice2 holds for four of the five raters and in the pooled paired tests ($p<0.001$), and is the more robust finding. The relative naturalness of GPT-SoVITS and EdgeTTS, and the absolute MOS values, are not: they depend on the individual rater, and a native-speaker panel, or listeners of other ages and backgrounds, may judge them differently. Two features of the design further limit the similarity judgements: the reference clips heard by raters were short (0.7–2.8 s), and items were presented in the same order to all raters, so order effects cannot be excluded.

Figure 5. Per-rater mean naturalness for each system (8 items per cell; raters anonymised as R1–R5). The two weaker cloning systems are rated below GPT-SoVITS by four of the five raters, whereas ratings of the fixed commercial voice vary from 1.88 to 5.00 across raters.

5.4. Generalisation to Other Dramas, Speakers, and Recording Conditions

The benchmark covers one series, twelve speakers and three test sentences, and so the absolute values in Table 5 and Table 6 should not be expected to transfer: they depend on the voices, the amount and quality of reference audio per speaker (Table 3), the background music in the clips, and the sentence content. We expect the qualitative findings to transfer more readily because they follow from the systems’ behaviour rather than from properties of this corpus: the intra-lingual condition shows that F5-TTS’s failure arises from crossing the language boundary and CosyVoice2-0.5B’s from degenerate generation (Section 4.3), and the dissociation between embedding similarity and perceived identity follows from what verification encoders are trained to measure. Cleaner recordings, longer references, or genres with less background music would likely raise absolute quality for all systems; whether they would change the ranking is an empirical question that a multi-series benchmark with more speakers and sentences must answer.

5.5. Other Language Pairs and the Reverse Direction

Nothing in the evaluation protocol is specific to Chinese and Japanese except the text normalisation and tokenisation (MeCab for Japanese, character-level scoring for Chinese); the same pipeline applies to any pair for which an ASR model and a tokeniser exist. Whether the systems’ rankings would hold is less clear. Japanese shares many characters with Chinese, but its phonology and prosody differ sharply; a target language closer to the reference in phonology, or one far better represented in the systems’ training data (e.g., English), might reduce the language-transfer failure we observe for F5-TTS, while lower-resource targets would likely increase it. The reverse direction, Japanese reference to Chinese speech, would test whether the asymmetry lies in the target language or in the direction of transfer. A tandem (round-trip) design—Chinese speech converted to Japanese and back to Chinese—would additionally allow the final output to be compared directly with the original voice and text, giving a cycle-consistency measure of how much identity and content each conversion loses; improving that loop would plausibly improve single-direction dubbing as well. The intra-lingual condition reported here is a first step in this direction, as it isolates the cost of crossing the language boundary. Our corpus contains no Japanese-speaking source audio, so the reverse and round-trip settings remain future work.

5.6. Limitations

First, our corpus is a single short-drama series with 12 speaker identities, which may limit cross-genre and cross-speaker generalisability; the genre-intrinsic traits (fast colloquial delivery, emotionally charged lines, background music under the reference audio) make it a representative but not exhaustive benchmark, and multi-series evaluation is key future work. Second, and most consequentially, the human panel is small and its agreement is weak. With five raters we obtain ICC(2, 1) = 0.30, ICC(2, k) = 0.68 and Krippendorff’s ordinal $\alpha$ = 0.31; all three fall below the thresholds conventionally regarded as acceptable (roughly 0.5, 0.75 and 0.667, respectively). We report them rather than a MOS alone precisely because they qualify how much weight the perceptual findings can carry. The forced-choice similarity result is the more robust of the two human measures—eighteen votes against six between the two leading systems, a two-sided exact binomial test against $p=0.5$ giving $p\approx 0.023$—whereas the naturalness comparison is a null result at a resolution of about 0.6 MOS points (Section 3.3). The 35% of “none” similarity votes further indicates the task sits near the limit of what raters can resolve from short cross-lingual clips. A larger, native-speaker panel is required before the perceptual ordering reported here should be treated as settled. Third, GPT-SoVITS produced usable synthesis for only six of the twelve speakers, and so its averages rest on that subset; although restricting every system to the common six-speaker subset leaves the ranking unchanged, the coverage gap is itself a production-relevant reliability limitation that we do not resolve here. Fourth, the reference-duration probe covers a single speaker and sentence and should be read as illustrative, and the benchmark uses only three test sentences; accordingly, sentence-specific effects cannot be separated from system effects. Fifth, synthesis is offline; real-time dubbing would require further optimisation.

5.7. Dual-Use Considerations

This research is limited to multimedia localisation of licensed short-drama content. Voice-cloning experiments use the actors’ voices only to dub the same characters’ lines from the same series, mirroring the licensed use; the released samples are short, content-neutral, and labelled synthetic. The authors acknowledge the dual-use potential of zero-shot voice cloning and do not release reference-to-cloned-voice mappings. All necessary precautions have been taken to prevent potential misuse.

6. Conclusions

This paper presents a controlled benchmark of zero-shot cross-lingual voice cloning for Japanese dubbing of Chinese short drama, evaluating three cloning systems and one commercial fixed-voice baseline on a manually annotated 79-episode corpus with native-verified Japanese references.

Our key findings: among cloning systems, only GPT-SoVITS reaches usable intelligibility (CER 0.215, median 0.091), roughly five times better than F5-TTS and over twenty times better than CosyVoice2-0.5B, whose errors stem from degenerate repetition under cross-lingual prompting; GPT-SoVITS shows no detectable naturalness gap against commercial EdgeTTS (MOS 3.08 vs. 3.12, $p=0.93$, a null result at a resolution of about 0.6 MOS points) while additionally preserving speaker identity; automatic embedding similarity and perceived identity dissociate, with F5-TTS scoring higher on embedding cosine yet losing similarity votes three to one, implying that the embedding cosine alone should not be used to rank cloning systems; and, in a single-speaker probe, speaker similarity improved from 0.64 to 0.71 as the reference clip grew from ∼3 s to ∼10 s; an intra-lingual reference condition shows that F5-TTS’s failure is specific to cross-lingual transfer, whereas CosyVoice2-0.5B fails in both conditions.

Future Work

Four directions follow directly from what we found. Corpus breadth is the most obvious: a single series with twelve speakers cannot separate genre effects from speaker effects, and an additional annotated series with a larger and more diverse cast would let the coverage failure of Section 4.4 be attributed to one or the other. Panel scale and composition is the second: a larger native-speaker panel would tighten the MOS estimates and test whether our proficient non-native raters were systematically more permissive.

The third is methodological and, we think, the most valuable. Our results show that embedding cosine and human judgement can order systems differently, but they do not explain the discrepancy quantitatively. A predictor of perceived identity—trained the way MOSNet [25] and UTMOS [26] are trained for naturalness, but targeting similarity judgements and conditioned on intelligibility—would let this ordering be estimated without a panel. The data needed to build one is exactly the kind of paired automatic/human annotation this study produces.

Fourth, the reliability gap deserves direct study: we observe that cloning fails on particular speakers, but we cannot say which acoustic properties of a reference predict failure. A model that could reject a reference clip before synthesis, rather than after, would convert an unpredictable failure into a controllable one.