Addressing Structural Asymmetry: Unsupervised Joint Training of Bilingual Embeddings for Non-Isomorphic Spaces

Abstract

Bilingual Word Embeddings (BWEs) are crucial for multilingual NLP tasks, enabling cross-lingual transfer. While traditional joint training methods require bilingual corpora, their applicability is limited for many language pairs, especially low-resource ones. Unsupervised methods, relying on the isomorphism assumption, suffer from performance degradation when dealing with non-isomorphic embedding spaces, which are common in distant language pairs. This structural asymmetry challenges conventional approaches. To address these limitations, we propose a novel unsupervised joint training method for BWEs. We leverage monolingual corpora and introduce a dynamic programming algorithm to extract bilingual text segments, facilitating concurrent BWE training without relying on explicit bilingual supervision. Our approach effectively mitigates the challenge posed by asymmetric, non-isomorphic spaces by jointly learning BWEs in a shared space. Extensive experiments demonstrate the superiority of our method compared to existing approaches, particularly for distant language pairs exhibiting significant structural asymmetry

1. Introduction

The advent of large language models (LLMs) has profoundly reshaped the landscape of natural language processing (NLP), demonstrating remarkable capabilities in multilingual understanding and generation [1,2]. A cornerstone of their multilingual prowess lies in the ability to represent words and concepts from different languages within a shared or alignable semantic space, facilitating cross-lingual transfer for various downstream tasks [3,4,5]. Despite the remarkable achievements of LLMs, the quality of their underlying cross-lingual representations, particularly for low-resource or typologically distant languages, remains critically dependent on effective methods for aligning semantic spaces [6]. A fundamental challenge obstructing robust cross-lingual transfer, especially in unsupervised or low-resource scenarios, is the pervasive issue of structural asymmetry (non-isomorphism) between the embedding spaces of different languages [7]. This asymmetry arises not only from inherent linguistic differences but also from disparities in training data quality and domain, which are amplified for underrepresented languages. However, as we probe the geometric properties of these high-dimensional representations, the interplay between symmetry and asymmetry emerges as a critical factor influencing model performance, particularly in unsupervised settings [8,9].

Historically, cross-lingual representation methods like Bilingual Word Embeddings (BWEs) rely on the isomorphism assumption [10]. This posits structural symmetry: Word geometries (e.g., semantic distances and angles) are preserved across languages, typically via an orthogonal transformation. This presumed symmetry simplifies the learning problem, allowing for effective mapping strategies, particularly using offline methods that align pre-trained monolingual embeddings [11,12]. While this assumption holds reasonably well for closely related languages with similar linguistic structures and substantial data overlap, its validity diminishes significantly when dealing with distant language pairs [13,14]. Consequently, applications relying on cross-lingual alignment, such as unsupervised machine translation for low-resource pairs, bilingual lexicon induction for specialized domains, or zero-shot cross-lingual transfer in critical tasks (e.g., public health information access), face significant performance degradation or even failure when the underlying isomorphism assumption is violated [15]. This performance gap highlights the urgent need for unsupervised methods specifically designed to handle structural asymmetry.

Increasing evidence points toward inherent structural asymmetry, or non-isomorphism, between embedding spaces, especially for languages that are typologically distant or culturally divergent [16]. This asymmetry arises from fundamental differences in linguistic structures (syntax, morphology), variations in cultural conceptualizations reflected in language use, and disparities in the size and domain distribution of the monolingual corpora used for training [17,18]. Such asymmetry manifests as inconsistencies in the geometric neighborhoods and relational structures within the embedding spaces, rendering the simple symmetric mapping assumption inadequate. Consequently, unsupervised methods predicated on isomorphism often experience a sharp decline in performance when confronted with these asymmetric realities [19]. Addressing this structural asymmetry is crucial for building truly robust multilingual systems, especially in low-resource scenarios where parallel data, the traditional remedy via supervised joint training [20,21], is scarce.

Existing attempts to mitigate this asymmetry challenge in unsupervised settings often involve iterative refinement [22] or leveraging synthetic parallel data generated by unsupervised machine translation systems for joint training [23]. While the latter approach moves toward joint learning, potentially better accommodating asymmetry, it introduces a dependency on the quality of the synthetic data, which can be noisy and error-prone, particularly for distant language pairs where translation itself is challenging [24]. The approaches, whether reliant on the fragile isomorphism assumption or the noisy output of unsupervised MT systems, severely constrain the practical deployment of robust multilingual NLP solutions, particularly for the vast majority of language pairs lacking abundant parallel resources. This motivates the need for alternative unsupervised joint training paradigms that can directly handle structural asymmetry without relying on potentially flawed synthetic supervision or the overly strong symmetry assumption.

In this paper, we propose a novel unsupervised joint training methodology specifically designed to navigate the complexities of asymmetric, non-isomorphic embedding spaces. Our core idea is to bypass the need for explicit parallel corpora or synthetic data by mining weak bilingual signals directly from large monolingual corpora. We introduce a dynamic programming algorithm, incorporating nearest-neighbor search within the embedding space, to identify and extract parallel phrase segments across the two monolingual datasets. These mined phrases, while not perfectly parallel sentences, provide valuable anchor points for concurrently training BWEs in a shared space. We jointly optimize embeddings using monolingual context (e.g., skip-gram) and cross-lingual phrase alignments. This fosters shared representations respecting cross-lingual correspondences, even under structural asymmetry. This method directly confronts the asymmetry challenge by learning the alignment implicitly during training, rather than assuming symmetry beforehand or relying on noisy external components.

Our contributions are as follows: (1) We introduce an unsupervised joint training framework for BWEs that operates solely on monolingual corpora, directly addressing the low-resource challenge. (2) We propose a dynamic programming-based phrase-mining technique to extract bilingual signals, offering a novel alternative to synthetic data generation for unsupervised joint training. (3) Through extensive experiments on both closely related and distant language pairs (representing varying degrees of structural asymmetry), we demonstrate empirically that our method significantly outperforms existing unsupervised approaches, particularly in challenging non-isomorphic scenarios, thereby effectively mitigating the negative impact of structural asymmetry in cross-lingual representation learning. We believe this approach offers valuable insights into handling inherent data and structural asymmetries, a pertinent challenge highlighted by this special issue.

2. Related Work

The pursuit of effective cross-lingual representations, particularly BWEs, has been a vibrant area of NLP research. Methodologies have evolved significantly, grappling implicitly and explicitly with the inherent structural symmetries and asymmetries between languages.

2.1. Supervised and Semi-Supervised Approaches

Early successful approaches often relied on explicit bilingual supervision to guide the alignment process. Joint training methods learn embeddings for two languages concurrently by optimizing objectives that leverage parallel corpora [20]. For instance, models might extend monolingual objectives (like skip-gram) to predict context words across languages within aligned sentences. While effective, these methods require substantial parallel data, a resource unavailable for many language pairs, especially low-resource ones [25,26]. By directly using translation pairs, these methods effectively force an alignment, implicitly handling potential structural asymmetries through strong supervision, but their applicability is limited. Other approaches use smaller seed dictionaries [27] to learn linear mapping, reducing the supervision requirement but still needing some parallel signal.

2.2. Unsupervised Mapping Approaches and the Isomorphism Assumption

To overcome the reliance on parallel data, unsupervised methods emerged, predominantly focusing on offline mapping. These methods first train high-quality monolingual embeddings independently (e.g., using FastText [28]) and then learn a transformation, typically assumed to be orthogonal, to map one embedding space onto the other [29]. This orthogonality constraint is key, as it preserves the internal geometry (word distances and angles) of the monolingual spaces, embodying the isomorphism assumption—a belief in fundamental structural symmetry between the source and target embedding spaces. Prominent unsupervised mapping techniques include adversarial training [22], where a discriminator tries to distinguish between mapped source embeddings and target embeddings, while the mapping function tries to fool it, encouraging distributional alignment. Another successful line employs iterative self-learning [30], starting with a weak initial dictionary (e.g., based on identical strings or numerals) and refining the mapping and dictionary iteratively using techniques like Procrustes alignment. Robustness has been improved by incorporating techniques like frequency-based vocabulary selection, CSLS (cross-domain similarity local scaling) retrieval [22], and leveraging subword information [27].

However, the foundational isomorphism (symmetry) assumption has been increasingly challenged [16,31]. Studies suggest that embedding spaces, especially for distant language pairs, often exhibit significant structural asymmetry (non-isomorphism) [21,32,33,34]. This asymmetry can stem from linguistic typology, cultural differences reflected in language use, or artifacts of the training data (size, domain). When this symmetry breaks down, unsupervised mapping methods relying on it suffer performance degradation [18,35,36]. The challenge then becomes how to learn meaningful cross-lingual mappings in the face of this inherent asymmetry without supervision.

2.3. Addressing Asymmetry in Unsupervised Settings

Recognizing the limitations of the strict isomorphism assumption, researchers have explored ways to create more robust unsupervised cross-lingual links, implicitly or explicitly trying to accommodate asymmetry. Some methods attempt to improve the initial mapping by incorporating richer linguistic information beyond distributional similarity, such as morphological similarities or cognates [37], or by using more robust alignment techniques [38]. A significant direction involves moving toward unsupervised joint training. The goal is to leverage monolingual data more deeply while inferring cross-lingual links without parallel text. One prominent approach utilizes unsupervised machine translation (UNMT) [39,40]. First, a UNMT system is trained using only monolingual corpora. Then, this system generates synthetic parallel sentences via back-translation, which are subsequently used as supervision for jointly training BWEs [8,39]. This approach inherently allows for more complex, non-linear alignments than simple orthogonal mapping and can potentially adapt better to structural asymmetries. However, performance depends critically on UNMT quality. For distant languages, UNMT generates noisy synthetic data, harming BWE training. Ref. [8] proposed a fusion approach that combines offline mapping with joint training on synthetic data, showing improved robustness.

Another related line of work focuses on mining parallel or comparable data directly from large web corpora or sources like Wikipedia [41,42]. Recent work [9] specifically mined parallel sentences from the internet using multi-view knowledge distillation for low-resource pairs. While powerful, these often aim to build large parallel corpora for MT, rather than extracting the finer-grained signals needed specifically for BWE joint training from purely monolingual, potentially non-comparable sources.

3. Methodology

Our objective is to learn high-quality BWEs in an unsupervised manner, effectively navigating the challenges posed by structural asymmetry (non-isomorphism) between language embedding spaces, particularly in low-resource settings. Our approach operates solely on monolingual corpora from a source language $L_s$ and a target language $L_t$, denoted as $C_s$ and $C_t$ respectively. The overall process, depicted in Figure 1, involves three key stages: unsupervised initialization, parallel phrase mining, and unsupervised joint training.

3.1. Unsupervised Initialization of BWEs

Before joint training can commence, we need an initial estimate of the cross-lingual alignment. This initial step provides a starting point for the nearest-neighbor search crucial for our phrase-mining algorithm. We employ a state-of-the-art unsupervised offline mapping technique to establish this initial foundation. Let $X\in {\mathbb{R}}^{|V_s|\times d}$ and $Y\in {\mathbb{R}}^{|V_t|\times d}$ be the pre-trained monolingual word embedding matrices for the source and target languages, respectively, where $|V_s|$ and $|V_t|$ are the vocabulary sizes and d is the embedding dimension. These are typically trained using algorithms FastText [28] on $C_s$ and $C_t$.

We adopt the robust self-learning approach, which aims to find transformation matrices $W_s\in {\mathbb{R}}^{d\times d}$ and $W_t\in {\mathbb{R}}^{d\times d}$ that align the spaces. The core idea often involves optimizing a criterion:

$$W^{*}=\underset{W_x,W_y}{\mathrm{argmax}}\sum _{i=1}^{|V_s|}\sum _{j=1}^{|V_t|}{D}_{ij}\mathrm{sim}({\left(X{W}_x\right)}_i,{\left(Y{W}_y\right)}_j)$$

(1)

where ${\left(X{W}_x\right)}_i$ is the i-th row of the transformed source matrix, ${\left(Y{W}_y\right)}_j$ is the j-th row of the transformed target matrix, $D_{ij}$ indicates likely translation pairs, and $\mathrm{sim}(·,·)$ is a similarity function measuring the similarity between two vectors ${\left(X{W}_x\right)}_i$ and ${\left(Y{W}_y\right)}_j$, commonly, cosine similarity $cos({\left(X{W}_x\right)}_i,{\left(Y{W}_y\right)}_j)={\displaystyle \frac{{\left(X{W}_x\right)}_i·{\left(Y{W}_y\right)}_j}{\parallel {\left(X{W}_x\right)}_i\parallel \parallel {\left(Y{W}_y\right)}_j\parallel }}$ is used. The summation ${\sum }_{i=1}^{|V_s|}{\sum }_{j=1}^{|V_t|}$ iterates over all possible pairs of words in the source and target vocabularies. A common solution involves the SVD of $X^{T}DY=U\Sigma V^T$, yielding orthogonal transformations $W_x=U$ and $W_y=V$. Let $E_0^s=X{W}_x$ and $E_0^t=Y{W}_y$ represent the initial d-dimensional embeddings aligned in a common space.

3.2. Parallel Phrase Mining via Dynamic Programming

This stage extracts weak bilingual signals by identifying continuous text segments in $C_s$ and $C_t$ that are likely translations, based on word similarity in the “current” embedding space. Consider segments $s=(w_1^s,\dots ,w_N^s)$ from $C_s$ and $t=(w_1^t,\dots ,w_M^t)$ from $C_t$. Let $E^s$ and $E^t$ be the current embeddings (initially $E_0^s,E_0^t$).

Similarity Criterion: We define a binary similarity function $\mathrm{similar}(w_i^s,w_j^t)$ using nearest neighbors:

$$\mathrm{similar}(w_i^s,w_j^t)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}{e}_j^t\in {\mathrm{NN}}_k(e_i^s,E^t)\mathrm{or}{e}_i^s\in {\mathrm{NN}}_k(e_j^t,E^s)\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(2)

where $w_i^s$ is the i-th word in a source language text segment, $w_j^t$ is the j-th word in a target language text segment, $e_i^s$ is the embedding vector of the source word $w_i^s$ from the current source embedding matrix $E^s$, $e_j^t$ is the embedding vector of the target word $w_j^t$ from the current target embedding matrix $E^t$, ${\mathrm{NN}}_k(e,E^{\prime })$ is the set of the k nearest neighbors of e within $E^{\prime }$ based on cosine similarity $\cos (\mathit{u},\mathit{v})={\displaystyle \frac{\mathit{u}·\mathit{v}}{\parallel \mathit{u}\parallel \parallel \mathit{v}\parallel }}$, and k is a parameter. The $\mathrm{similar}(w_i^s,w_j^t)=1$ indicates that the words $w_i^s$ and $w_j^t$ are considered potential mutual translation candidates based on their current embedding proximity.

Dynamic Programming: We compute a matrix $m[i,j]$ that stores the length of the longest contiguous similar sequence ending at $(w_i^s,w_j^t)$. The transition is as follows:

$$m[i,j]=\left\{\begin{array}{cc}m[i-1,j-1]+1\hfill & \mathrm{if}\mathrm{similar}(w_i^s,w_j^t)=1\hfill \\ 0\hfill & \mathrm{if}\mathrm{similar}(w_i^s,w_j^t)=0\hfill \end{array}\right.$$

(3)

with base cases $m[i,0]=0$ and $m[0,j]=0$. An illustration is in Figure 2.

Phrase Extraction: We find entries $m[i,j]=L\ge L_{\min }$ (minimum length, e.g., 3) and extract the corresponding source phrase $p^s=(w_{i-L+1}^s,\dots ,w_i^s)$ and target phrase $p^t=(w_{j-L+1}^t,\dots ,w_j^t)$. These pairs form the set $P=\left\{(p_k^s,p_k^t)\right\}$.

Discussion on Validity of Mined Phrases: A critical challenge in mining bilingual phrase segments from monolingual corpora lies in distinguishing valid cross-lingual alignments from coincidental similarities. Due to the high dimensionality and potential structural asymmetry of embedding spaces, especially between typologically distant languages, isolated word-level nearest neighbor matches can be unreliable. To address this, our method enforces two criteria, as follows: (1) a minimum length threshold $L_{\min }$ ensures that only sufficiently long continuous segments are extracted, filtering out spurious matches; and (2) a mutual nearest neighbor check guarantees that word pairs exhibit consistent similarity in both directions. These constraints significantly reduce noise in the mined bilingual signals, ensuring that the resulting phrases provide meaningful supervision for unsupervised joint training.

3.3. Unsupervised Joint Training Framework

We jointly optimize $E^s$ and $E^t$ using monolingual corpora ($C_s,C_t$) and mined phrases, P. The combined objective is as follows:

$$L_{\mathrm{total}}=L_{\mathrm{mono}}(E^s;C_s)+L_{\mathrm{mono}}(E^t;C_t)+\lambda L_{\mathrm{cross}}(E^s,E^t;P)$$

(4)

where $\lambda$ balances the cross-lingual signal, $L_{\mathrm{total}}$ is the combined loss function minimized during joint training, $L_{\mathrm{mono}}(E^s;C_s)$ is the monolingual loss for the source language, $L_{\mathrm{mono}}(E^t;C_t)$ is the monolingual loss for the target language, $L_{\mathrm{cross}}(E^s,E^t;P)$ is the cross-lingual loss function (defined below) evaluated using source embeddings $E^s$, target embeddings $E^t$, and the mined phrase pairs P.

Monolingual Objective ($L_{\mathrm{mono}}$): We use skip-gram with negative sampling (SGNS):

$$L_{\mathrm{mono}}(E;C)=-\sum _{w_i\in C}\sum _{w_c\in \mathrm{Context}\left(w_i\right)}\left[log\sigma \left(e_i·e_c\right)+\sum _{n=1}^{N_{\mathrm{neg}}}{\mathbb{E}}_{w_n\sim P_n\left(w\right)}[log\sigma (-e_i·e_n)]\right]$$

(5)

where $e_i,e_c,e_n$ are embeddings for the center, context, and negative sample words, $\sigma \left(x\right)=1/(1+e^{-x})$ is the sigmoid function, $N_{\mathrm{neg}}$ is the number of negative samples, and $P_n\left(w\right)$ is the noise distribution.

Cross-lingual Objective ($L_{\mathrm{cross}}$): For each mined phrase pair $k\in P$, let $A_k=\left\{(w^s,w^t)\right\}$ be the set of implicitly aligned word pairs identified during DP. The loss maximizes their similarity:

$$L_{\mathrm{cross}}(E^s,E^t;P)=-\sum _{k=1}^{\left|P\right|}\sum _{(w^s,w^t)\in A_k}log\sigma \left(\cos (e^s,e^t)\right)$$

(6)

This pulls aligned word embeddings closer.

Optimization and Normalization: Minimize $L_{\mathrm{total}}$ using SGD (or Adam). Apply embedding normalization and mean centering periodically during training [39] for stability.

Handling Structural Asymmetry: Unlike traditional unsupervised methods that assume isomorphic embedding spaces, our approach explicitly addresses non-isomorphism through two design choices, as follows: (1) The dynamic programming mining step identifies bilingual segments without relying on symmetric neighborhood structures, avoiding biases toward language-pair-specific isomorphism. (2) The joint training framework iteratively refines embeddings in a shared space, allowing asymmetric adjustments to accommodate linguistic divergences. This eliminates the need for restrictive orthogonality constraints common in mapping-based methods.

This joint framework learns alignment dynamically from mined signals combined with monolingual context, making it adaptable to structural asymmetries without assuming strict isomorphism. The complete unsupervised training procedure integrates the initialization, mining, and joint learning steps.

4. Experimental Settings

This section details the datasets, implementation parameters, evaluation protocols, and baseline methods used to assess the performance of our proposed unsupervised joint training approach for BWEs.

4.1. Datasets and Preprocessing

Monolingual Corpora: We utilized large monolingual corpora extracted from Wikipedia for our primary experiments, ensuring comparability with many previous studies. Additionally, to test robustness across different data sources and potential domain mismatch (reflecting real-world asymmetry), we also conducted experiments using corpora derived from Common Crawl. Following common practice [22,30], we used the “WikiExtractor” tool to obtain plain text from Wikipedia dumps. For each language, we used approximately 100 million words (or a comparable amount based on typical dataset sizes in the literature) for training the embeddings.

Language Pairs: We evaluated our method on six language pairs, categorized by linguistic distance, to investigate performance under varying degrees of expected structural asymmetry:

• Closely related: English–German (En-De), English–Italian (En-It), German-Italian (De-It). These pairs are expected to exhibit relatively higher structural symmetry.
• Distant: English–Russian (En-Ru), English–Turkish (En-Tr), English–Chinese (En-Zh). These pairs are typologically more distant and are expected to exhibit greater structural asymmetry (non-isomorphism).

All experiments involving English use it as the source language unless otherwise specified (e.g., De-It).

Preprocessing: Standard text preprocessing steps were applied, including tokenization (using the Moses tokenizer scripts), lowercasing, and the removal of rare words (e.g., keeping words occurring at least five times). We constructed vocabulary sizes appropriate for each language pair, ensuring coverage of evaluation dictionaries.

4.2. Implementation Details

Monolingual Embeddings: We used the FastText [28] implementation to train the initial 300-dimensional monolingual embeddings on the preprocessed corpora. We used a window size of 5, as well as 10 negative samples, training for 5 epochs.

Initialization for Our Method: The initial alignment for our unsupervised joint training framework (Stage 1 in Section 3.1) was obtained using the robust self-learning unsupervised mapping method [30], specifically using their publicly available implementation with default settings. We also experimented with other initializations as reported in the ablation studies.

Phrase-Mining Parameters: For the dynamic programming phrase mining (Section 3.2), we set the number of nearest neighbors $k=10$ for the similarity check (Equation in Section 3.2) and the minimum phrase length threshold $L_{\min }=3$.

Joint Training Parameters: Our joint training framework (Section 3.3) was implemented by building upon standard embedding training procedures. We used the Adam optimizer with a learning rate tuned on a held-out validation set (or a standard default like 0.001). The skip-gram objective within the joint training used a window size of 5 and 5 negative samples. The weight for the cross-lingual loss term $\lambda$ was set based on preliminary experiments (e.g., $\lambda =1.0$). The joint training ran for a fixed number of iterations or epochs (e.g., 5 epochs over the effective combined data). Normalization and mean centering were applied periodically as suggested in [39]. The primary results reported use a single pass of phrase mining followed by joint training, unless specified as iterative.

4.3. Evaluation Tasks and Metrics

We evaluated the learned BWEs on intrinsic and extrinsic tasks:

1.

Bilingual Lexicon Induction (BLI): This is the primary intrinsic evaluation.

• Dataset: We used the standard test dictionaries provided by MUSE [22], which contain 1500 source words and their translations for training and 5000 for testing (we used the test split).
• Protocol: For a source word embedding, we retrieved its nearest neighbor in the target embedding space using cosine similarity. We evaluated using Accuracy@1 (P@1), i.e., the percentage of source words whose retrieved nearest neighbor is the correct translation according to the dictionary. We report average accuracies across all test words.

2.

Downstream Task Evaluation: We assessed the utility of the BWEs in three cross-lingual transfer scenarios, following protocols similar to [17]:

• Cross-lingual Natural Language Inference (XNLI): We trained a classifier (e.g., a BiLSTM with attention) on the English MultiNLI training data and evaluated its zero-shot performance on the translated XNLI test sets for the target languages, using the learned BWEs as input features.
• Cross-lingual Document Classification (CLDC): We trained a document classifier (e.g., averaging BWEs + linear SVM) on the English portion and tested its zero-shot performance on target language documents.
• Cross-lingual Information Retrieval (CLIR): We represented queries (English) and documents (target languages) by averaging the embeddings of their constituent words (after stopword removal). We ranked documents based on cosine similarity to the query.

3.

Unsupervised Machine Translation (UMT): We initialized the embedding layer of a standard Transformer model with the learned BWEs and trained a UNMT system following [39].

4.4. Baseline Methods

We compared our proposed method against representative state-of-the-art approaches:

Supervised Mapping:

• Mapper [43]: Learns a linear projection using a seed dictionary (we used 1k or 5k pairs for supervised runs).
• VecMap (Supervised) [27]: The supervised variant of VecMap, using a seed dictionary.

Unsupervised Offline Mapping: These methods rely on the isomorphism assumption.

• VecMap (Unsupervised) [30]: Robust self-learning approach. (Used for our initialization).
• MUSE (Adversarial + Refinement) [22]: Adversarial training followed by Procrustes refinement.

Unsupervised Joint Training:

• Fusion [44]: Combines offline mapping with joint training on synthetic data from UNMT.

For all baselines, we utilized publicly available code and default parameters where possible. Crucially, all methods (including ours and baselines) were trained using the “same” preprocessed monolingual corpora and vocabularies for each language pair to ensure fair comparison. For supervised baselines requiring seeds, we used standard bilingual dictionary pairs.

5. Experiments and Results

5.1. Main Results

We evaluate the quality of the learned BWEs intrinsically using the BLI task, measuring the Accuracy@1 (P@1, %) on standard test dictionaries. The results using Wikipedia corpora are presented in

Table 1. BLI Accuracy@1 (%) on Wikipedia Corpora. Higher asymmetry metrics (GH, $\parallel I-W^{T}W\parallel$) indicate greater deviation from isomorphism. The best unsupervised results are in bold. “Sup.” represents supervised and “Unsup.” represents unsupervised.

Methods	En-De	En-It	De-It	En-Ru	En-Tr	En-Zh
Asymmetry Metrics
$\parallel I-W^T{W\parallel }_F$	0.13	4.81	5.13	5.32	7.63	13.86
GH Distance	0.18	0.41	0.43	0.46	0.80	0.92
Methods with Bilingual Supervision (using 1k seeds)
Mapper [43]	60.7	58.32	53.69	32.15	33.2	32.63
VecMap (Sup.) [27]	72.8	70.57	64.72	41.6	40.65	38.3
Methods without Bilingual Supervision (Offline Unsupervised Mapping)
MUSE [22]	68.49	70.86	65.32	29.13	41.66	37.5
VecMap (Unsup.) [30]	73.02	73.42	64.52	45.66	47.6	40.6
Unsupervised Joint Training
Fusion [44]	73.11	74.13	66.07	48.7	49.15	47.13
Proposed Method	74.15	74.33	67.07	50.06	52.81	50.7

, and results using Common Crawl corpora are shown in

Table 2. BLI Accuracy@1 (%) on Common Crawl Corpora. Higher asymmetry metrics are generally observed compared to Wikipedia. The best unsupervised results are in bold. “Sup.” represents supervised and “Unsup.” represents unsupervised.

Methods	En-De	En-It	De-It	En-Ru	En-Tr	En-Zh
Asymmetry Metrics
$\parallel I-W^T{W\parallel }_F$	0.15	6.31	7.03	7.16	8.83	15.62
GH Distance	0.23	0.61	0.62	0.68	0.88	0.98
Methods with Bilingual Supervision (using 1k seeds)
Mapper [43]	43.16	39.33	38.42	33.8	39.65	30.1
VecMap (Sup.) [27]	48.37	42.56	44.13	36.18	38.12	33.4
Methods without Bilingual Supervision (Offline Unsupervised Mapping)
MUSE [22]	53.6	53.1	48.1	29.8	33.16	28.7
VecMap (Unsup.) [30]	56.02	56.04	51.53	35.59	40.37	32.87
Unsupervised Joint Training
Fusion [44]	67.2	66.31	61.8	45.65	45.32	42.17
Proposed Method	68.1	67.23	64.83	48.18	47.66	47.91

. We also include measures of embedding space asymmetry, that is, the Gromov–Hausdorff (GH) distance and the deviation from orthogonality $\parallel I-W^T{W\parallel }_F$ for the mapping W obtained from unsupervised mapping. Higher values indicate greater asymmetry/non-isomorphism.

Performance Analysis (Wikipedia):

As shown in Table 1, our proposed unsupervised joint training method consistently outperforms all unsupervised baseline methods across all six language pairs.

• Versus Offline Mapping: Our method significantly surpasses the strong offline mapping baselines MUSE [22] and VecMap [30]. While these baselines perform well on closely related pairs (En-De, En-It), their performance drops notably for distant pairs (En-Ru, En-Tr, En-Zh). This aligns with the higher asymmetry metrics observed for these distant pairs. Our method shows much greater resilience, achieving substantial gains, e.g., +4.4 P@1 over VecMap for En-Ru, +5.2 P@1 for En-Tr, and +10.1 P@1 for En-Zh. This confirms our phrase-guided joint training better handles structural asymmetry in distant pairs. Here, the isomorphism assumption fails.
• Versus Unsupervised Joint Baseline: Our method also outperforms the unsupervised joint training baseline across all pairs. While their method already improves over offline mapping by incorporating joint training on synthetic data, our approach, using directly mined phrases, yields further improvements. This suggests that the weak bilingual signal extracted by our DP mining process might be more effective or less noisy than synthetic parallel sentences generated by UNMT, especially for challenging pairs. Gains are noticeable across both close (+1.04 P@1 for En-De) and distant pairs (+1.36 P@1 for En-Ru, +3.66 P@1 for En-Tr, +3.57 P@1 for En-Zh).
• Comparison to Supervised Methods: While unsupervised methods are not expected to match fully supervised ones (which use direct translation pairs), our method closes the gap considerably, especially on distant pairs, compared to other unsupervised approaches. For instance, on En-Zh, our method (50.7) significantly improves over the best unsupervised mapping baseline (40.6) and the joint baseline (47.13), approaching the performance levels seen in some supervised settings for this challenging pair.

Performance Analysis (Common Crawl):

Table 2 presents results on the Common Crawl dataset, which is generally considered more diverse and noisy than Wikipedia, potentially exhibiting different distributional properties and, thus, greater asymmetry or domain mismatch.

• Increased Challenge: We observe that all methods generally achieve lower absolute scores on Common Crawl compared to Wikipedia, highlighting the increased difficulty. The measured asymmetry metrics (GH, $\parallel I-W^{T}W\parallel$) are also generally higher on Common Crawl for the corresponding language pairs, supporting the notion of increased structural divergence.
• Robustness of Joint Training: Despite the overall drop, the relative performance trends persist. Our proposed method employing joint training demonstrates greater robustness compared to the offline mapping methods (MUSE, VecMap (Unsup)). The performance gap between joint training methods and offline mapping methods widens on Common Crawl, particularly for distant pairs. For example, on En-Zh, our method achieves 47.91 P@1, while the best offline mapping baseline (VecMap Unsup.) reaches only 32.87 P@1.

Across both datasets, we observe a general correlation, that is, the performance of offline unsupervised methods tends to degrade more significantly as the measured asymmetry (GH distance, $\parallel I-W^{T}W\parallel$) increases (i.e., for more distant language pairs). Our joint training method demonstrates significantly less degradation, indicating its ability to better mitigate the negative impact of non-isomorphism/structural asymmetry. While asymmetry makes the task harder for all unsupervised methods, our approach appears more robust to this challenge.

5.1.1. Sensitivity to Data Divergence

Real-world scenarios often involve training on monolingual corpora from different domains or sources, introducing another layer of asymmetry beyond purely linguistic distance. To simulate this, we conducted experiments training the source language (English) embeddings on Wikipedia and the target language embeddings on Common Crawl. The results are presented in

Table 3. BLI Accuracy@1 (%) on mixed corpora (Source: Wikipedia, Target: Common Crawl). The best unsupervised results are in bold. “Unsup.” represents unsupervised.

Methods	En-De	En-It	De-It	En-Ru	En-Tr	En-Zh
Methods without Bilingual Supervision (Offline Unsupervised Mapping)
MUSE [22]	51.67	50.71	45.62	30.29	33.16	29.12
VecMap (Unsup.) [30]	58.62	57.71	54.36	38.71	43.11	33.36
Unsupervised Joint Training
Fusion [44]	65.72	54.87	63.25	47.12	48.27	43.16
Proposed Method	69.13	67.82	66.25	50.16	50.82	45.12

.

The results in Table 3 clearly show that our proposed unsupervised joint training method exhibits significantly higher robustness to this data source asymmetry compared to offline mapping baselines. The performance gap between our method and offline methods like VecMap widens considerably under data divergence. For instance, on En-De, the gap increases, and on distant pairs like En-Tr, our method achieves 50.82 P@1 while VecMap drops to 43.11 P@1. Compared to Fusion, our method also maintains a consistent advantage, suggesting that relying on directly mined phrases from the potentially divergent corpora is more effective than relying on synthetic data potentially generated from models trained on mismatched sources. This resilience highlights the adaptability of our joint training framework, which learns the alignment based on signals present “in the actual target corpora”, rather than assuming distributional similarity or relying on intermediary translation models.

While our experiments primarily utilize Wikipedia and Common Crawl corpora to simulate domain generality and noise, the robustness observed in mixed-source settings (Table 3) suggests applicability to noisy social data. For instance, Common Crawl inherently contains informal text fragments (e.g., social media snippets), and our method’s resilience to its heterogeneity—evidenced by consistent gains over baselines under data divergence—aligns with challenges in social data processing. Although explicit evaluation on domain-specific social datasets (e.g., Twitter) is beyond our scope, the proposed phrase-mining mechanism inherently filters noise by requiring contiguous semantic matches ($L_{\min }=3$), which discards spurious alignments common in noisy contexts. Future work could explicitly validate this on curated social datasets.

The proposed method’s ability to mitigate structural asymmetry is further validated by its performance in high-asymmetry scenarios. For example, in Table 3, where data source divergence introduces additional non-isomorphism, our method retains 69.13% of its baseline accuracy (En-De), whereas VecMap is 58.62%. This demonstrates that joint training with mined signals adapts more effectively to asymmetric spaces than offline mapping techniques reliant on isomorphism assumptions.

5.1.2. Sensitivity to Initial BWEs

Our method requires an initial BWE alignment (Stage 1) to bootstrap the phrase-mining process. We investigated how the choice of the unsupervised mapping method used for this initialization affects the final performance. We experimented with initializing our joint training framework using different offline unsupervised methods as the starting point ($E_0^s,E_0^t$). The results on the Common Crawl dataset are presented in

Table 4. BLI Accuracy@1 (%) on Common Crawl using the proposed method with different initializations. The best unsupervised results are in bold. “Sup.” represents supervised and “Unsup.” represents unsupervised.

Initialization Method	En-De	En-It	De-It	En-Ru	En-Tr	En-Zh
Random	0.0	0.0	0.0	0.0	0.0	0.0
Mapper [43]	67.15	67.31	63.03	47.08	47.63	47.62
VecMap (Sup.) [27]	67.23	67.51	63.32	47.68	47.72	47.28
MUSE [22]	67.18	66.85	63.72	48.16	47.75	47.69
VecMap (Unsup.) [30]	68.1	67.23	64.83	48.18	47.66	47.91

. We also include a random initialization baseline (where $W_x,W_y$ are random orthogonal matrices), which is expected to fail as no meaningful initial signal exists for phrase mining.

Table 4 demonstrates that while the choice of initialization does have a minor impact, our method achieves strong results regardless of the specific unsupervised mapping technique used initially, as long as it provides a reasonable starting point. The performance variation across different initializations (excluding random) is relatively small compared to the overall gains achieved by our joint training. For example, the standard deviation across different valid initializations for a given language pair is typically low. The random initialization fails completely, confirming the need for some initial signal to start the phrase mining. VecMap provided slightly better initialization on average for this dataset, leading to the best final results reported as our main scores, but other methods also led to competitive outcomes after our joint training phase. This relative stability suggests that our joint training procedure, driven by mined phrases and monolingual signals, can effectively refine and improve upon various initial alignments, making it less sensitive to the specific choice of the initializer compared to methods that solely rely on the initial mapping.

5.2. Ablation Studies

To understand the contribution of the core components of our method, particularly the mined parallel phrases, we performed ablation studies.

5.2.1. Impact of Bilingual Signal Type

We compared our full method (using mined parallel phrases) against two variants on the Common Crawl dataset:

1.

+Non: Joint training using only the monolingual objective ($\lambda =0$ in Equation in Section 3.3). This isolates the effect of joint optimization without any cross-lingual signal.

2.

+Word Pairs: Joint training using only aligned word pairs based on the initial nearest neighbors, without enforcing phrase continuity ($L_{\min }=1$ and extracting individual pairs). This tests the value of the phrase continuity constraint.

The results are shown in

Table 5. Ablation Study: Effect of bilingual signal types (+Non:No Cross-lingual Signal, +Word Pairs:Word Pairs Only, +Ours:Mined Phrases) on BLI Accuracy@1 (%) on Common Crawl Corpora. The best results are in bold.

Language Pair	+Non ($\mathit{\lambda }=0$)	+Word Pairs	+Ours (Phrases)
En-De	0.0	37.18	68.1
En-It	0.0	36.85	67.23
De-It	0.0	23.72	64.83
En-Ru	0.0	18.16	48.18
En-Tr	0.0	17.35	47.66
En-Zh	0.0	12.19	47.91

(corresponding to Table 5 in the original draft).

The results are stark. Removing the cross-lingual signal entirely (+Non) leads to complete failure (0.0 accuracy), confirming that unsupervised joint training requires some form of bilingual signal to bridge the languages. Using only individual nearest-neighbor word pairs (+Word Pairs) provides some signal but yields very poor results, significantly worse than even the offline mapping baselines (compare with Table 2). This highlights the critical importance of our dynamic programming approach to identify “continuous parallel phrases”. Enforcing phrase continuity acts as a strong filter against noisy or spurious nearest-neighbor matches, ensuring that the cross-lingual signal used for joint training ($L_{\mathrm{cross}}$) is of much higher quality. The substantial performance leap from “+Word Pairs” to “+Ours (Phrases)” underscores the effectiveness of the phrase-mining component.

5.2.2. Impact of Minimum Phrase Length

We investigated the effect of the minimum phrase length threshold $L_{\min }$ used during the mining stage. Figure 3 (corresponding to Figure 3 in the original draft) shows the BLI accuracy on the Common Crawl En-De pair as $L_{\min }$ varies.

Figure 3 indicates that performance improves significantly when moving from $L_{\min }=1$ (equivalent to “+Word Pairs” in the ablation) to $L_{\min }=2$ or $L_{\min }=3$. Setting $L_{\min }=3$ (our default) provides the best results. Increasing $L_{\min }$ further (e.g., to 4 or 5) results in a slight decrease in performance. This suggests a trade-off: longer phrases provide stronger, less noisy alignment signals, but requiring very long contiguous matches might filter out too many valid, shorter parallel segments, reducing the overall quantity of the bilingual signal. A minimum length of 3 appears to strike a good balance between signal quality and quantity for this task.

These analyses demonstrate the robustness of our method to data variations and initialization choices, and critically highlight the importance of the continuous parallel phrase-mining component for extracting high-quality bilingual signals essential for effective unsupervised joint training.

5.3. Downstream Task Performance

While intrinsic evaluation, such as BLI, assesses the direct word-level alignment quality, evaluation on downstream tasks assesses the practical utility of the learned BWEs for cross-lingual transfer. We evaluated our method and baselines on cross-lingual natural language inference (XNLI), cross-lingual document classification (CLDC), and cross-lingual information retrieval (CLIR). The results, using embeddings trained on the Common Crawl corpora (as they represent a more challenging scenario), are presented in

Table 6. Cross-lingual natural language inference (XNLI) test set accuracy (%) by using embeddings trained on Common Crawl corpora. “Sup.” represents supervised and “Unsup.” represents unsupervised. The best results are in bold.

Model	De	It	Ru	Tr	Zh
Supervised Baselines (using 1k seeds)
Mapper [43]	33.1	36.3	33.1	33.1	22.5
VecMap (Sup.) [27]	48.2	51.8	46.2	43.4	41.7
Unsupervised Offline Mapping
MUSE [22]	49.4	52.1	46.4	43.8	42.9
VecMap (Unsup.) [30]	48.7	51.8	46.1	43.3	42.3
Unsupervised Joint Training
Fusion [44]	60.2	62.3	57.2	54.6	53.8
Proposed Method	60.8	63.2	58.1	55.0	54.2

,

Table 7. Cross-lingual document classification (CLDC) micro-averaged F1 score (%) by using embeddings trained on Common Crawl corpora. “Sup.” represents supervised and “Unsup.” represents unsupervised. The best results are in bold.

Model	En-De	En-It	En-Ru	En-Tr	En-Zh
Supervised Baselines (using 1k seeds)
Mapper [43]	23.3	26.2	17.3	15.4	13.6
VecMap (Sup.) [27]	35.8	36.5	34.3	33.1	31.8
Unsupervised Offline Mapping
MUSE [22]	36.1	38.2	35.1	34.8	32.2
VecMap (Unsup.) [30]	37.2	38.6	36.3	35.3	33.0
Unsupervised Joint Training
Fusion	40.0	40.6	37.3	36.6	34.8
Proposed Method	40.8	41.2	38.1	37.5	35.3

and

Table 8. Cross-lingual information retrieval (CLIR) mean average precision (MAP) score by using embeddings trained on Common Crawl corpora. “Sup.” represents supervised and “Unsup.” represents unsupervised. The best results are in bold.

Model	De	It	Ru	Tr	Zh
Supervised Baselines (using 1k seeds)
Mapper [43]	0.331	0.363	0.331	0.331	0.225
VecMap (Sup.) [27]	0.186	0.201	0.176	0.165	0.151
Unsupervised Offline Mapping
MUSE [22]	0.188	0.213	0.181	0.173	0.158
VecMap (Unsup.) [30]	0.203	0.235	0.197	0.186	0.175
Unsupervised Joint Training
Fusion [44]	0.261	0.286	0.247	0.239	0.217
Proposed Method	0.276	0.291	0.257	0.246	0.228

.

1. Cross-lingual Natural Language Inference (XNLI): Table 6 shows the zero-shot classification accuracy on the XNLI test sets. The XNLI results clearly demonstrate the advantage of joint training methods for tasks requiring deeper semantic understanding. Both our proposed method and Fusion significantly outperform all offline mapping methods (MUSE, VecMap Unsup.) and even surpass the supervised mapping baselines on several language pairs. This suggests that the richer representations learned through joint optimization, which incorporate both monolingual context and cross-lingual signals, are more beneficial for complex transfer tasks than simply aligning static, pre-trained embeddings. Our method consistently achieves the best performance among the unsupervised approaches across all target languages, indicating that the quality of the cross-lingual alignment learned through mined phrases translates into better downstream performance for semantic inference.

2. Cross-lingual Document Classification (CLDC): Table 7 shows the micro-averaged F1 scores for zero-shot document classification. Similar trends are observed for CLDC. Joint training methods outperform offline mapping and supervised mapping baselines. CLDC relies on aggregating word embeddings to represent documents, potentially making it slightly less sensitive to fine-grained alignment issues than XNLI or BLI. Nevertheless, the improved quality of BWEs from joint training still yields better classification performance. Our method again edges out the other baselines, achieving the best unsupervised results across all language pairs.

3. Cross-lingual Information Retrieval (CLIR): Table 8 presents the mean average precision (MAP) scores for CLIR. The CLIR results, despite potential inconsistencies in the draft’s table values, generally follow the pattern observed in XNLI and CLDC. Joint training methods provide substantial improvements over offline mapping. CLIR performance is sensitive to the quality of semantic representation at both the word and document levels. The superior BWEs learned by our method enable more accurate cross-lingual query-document matching. Again, our proposed method achieves the best unsupervised performance.

6. Conclusions

In this paper, we systematically addressed the critical challenge of learning high-quality Bilingual Word Embeddings (BWEs) in unsupervised settings, with a focus on mitigating the pervasive issue of structural asymmetry between embedding spaces of typologically diverse languages. Structural asymmetry, particularly pronounced in distant language pairs and exacerbated by domain mismatches in monolingual corpora, has been shown to degrade the performance of traditional unsupervised methods reliant on the isomorphism assumption in bilingual lexicon induction (BLI) tasks. To overcome these limitations, we proposed a novel unsupervised joint training framework that circumvents the need for parallel corpora or synthetic data, operating solely on monolingual texts.

Our approach leverages a dynamic programming algorithm to mine parallel phrase segments directly from monolingual corpora. By exploiting nearest-neighbor relationships in the evolving embedding space, the algorithm identifies contiguous blocks of semantically related words, generating robust, data-driven bilingual signals. These mined phrases guide the concurrent optimization of source and target embeddings within a shared space, enabling the model to learn cross-lingual alignments dynamically while preserving monolingual structure. Extensive experiments across six language pairs (en-de, en-it, de-it, en-ru, en-tr, en-zh) demonstrated significant improvements over state-of-the-art methods. On BLI tasks, our framework achieved 74.15% accuracy@1 for English–German (en-de) and 74.33% accuracy@1 for English–Italian (en-it) on Wikipedia Corpora. For cross-lingual natural language inference (XNLI), our method attained 58.6% test accuracy on average, surpassing unsupervised offline mapping baselines by 11.82%.

The contributions of this work are threefold. First, we introduced an unsupervised joint training framework that explicitly addresses structural asymmetry without relying on synthetic data or external resources, achieving competitive performance even for distant language pairs like English–Turkish. Second, our dynamic programming-based phrase-mining technique provides a scalable alternative to conventional synthetic data generation, reducing computational overhead compared to back-translation-based methods. Third, we validated the effectiveness of our approach across diverse corpora, including clean (Wikipedia) and noisy (Common Crawl) domains, demonstrating robustness to data quality variations.

Despite these advancements, the proposed framework has limitations. The dynamic programming algorithm’s time complexity scales quadratically with sentence length, limiting its applicability to longer texts. Additionally, while our method excels in aligning lexical semantics, it does not explicitly model syntactic structures, potentially affecting performance in tasks requiring fine-grained grammatical transfer. Future work will focus on optimizing the algorithm’s efficiency via approximate nearest-neighbor search and integrating syntactic constraints into the alignment process. Furthermore, extending this framework to multilingual settings and exploring semi-supervised variants with minimal seed lexicons could further bridge the gap between unsupervised and fully supervised performance.

Meanwhile, our method’s design is extensible to multilingual and low-resource scenarios. Similar to multilingual joint training paradigms [45,46], our phrase-mining algorithm could be adapted to mine multi-view alignments across multiple monolingual corpora, leveraging shared embedding spaces for distant languages. Further, the minimal reliance on external resources (only monolingual corpora) aligns with low-resource settings [47]. For languages with extreme data scarcity, subword-enhanced initialization (e.g., FastText) combined with our phrase mining could mitigate lexical sparsity—an avenue for future exploration.

By directly addressing the challenges of structural asymmetry in cross-lingual representation learning, this work advances the feasibility of unsupervised methods for low-resource language pairs, offering a practical pathway toward equitable multilingual NLP systems.