A Contrastive Representation Learning Framework for Event Causality Identification

Abstract

To address the challenges associated with identifying causal relationships among event mentions in the event causality identification (ECI) task, ECI has emerged as a pivotal area of research for comprehending event structures. Recent studies have leveraged Transformer-based models, augmented by auxiliary components, to develop effective contextual representations for causality prediction. A critical step in ECI models involves transforming intricate event context representations into causal label representations, thereby facilitating the logical score calculations necessary for both training and inference. However, existing models frequently depend on simplistic feedforward networks for this transformation process, which often struggle to bridge the semantic gap between complex event contexts and target causal labels, particularly in linguistically nuanced scenarios. To address these limitations, we propose Contrastive Learning for Event Causality Identification (CLECI), an innovative ECI framework that enhances representation learning through the integration of contrastive learning techniques, a generator-discriminator mechanism with causal label embeddings. In contrast to traditional direct transformation methods, CLECI generates latent causal label embeddings that filter out irrelevant information while aligning with potential label representations. By incorporating contrastive learning principles, CLECI further augments the discriminative capability of event representations by constructing positive and negative pairs of events. Experimental evaluations conducted on the EventStoryLine (ESL), Causal-TimeBank (CTB), and MECI datasets demonstrate that CLECI achieves competitive performance, with F1-score improvements of 4.3%, 7.9%, and 2.5%, respectively, compared with the strongest baseline methods, while maintaining strong robustness in complex and noisy multilingual event contexts.

1. Introduction

Event causality identification (ECI), a critical task in Natural Language Processing (NLP) and information extraction [1], aims to detect causal relationships between event mentions within textual data. As illustrated in Figure 1, ECI model analyze event pairs across sentences to classify their causal status. For instance, in the sentence pair: (s1) quake $\stackrel{Cause}{⟶}$ tsunami and (s2) handed out $\stackrel{None}{⟶}$ killed, the model distinguishes causal links (e.g., earthquake → tsunami) from non-causal associations (e.g., handed out → killed). Accurate identification of such relationships is fundamental to advancing downstream NLP applications, including machine reading comprehension [2], question answering system [3,4], and event prediction [5].

Figure 1. Illustration of Event Causality Identification.

The evolution of ECI methodologies reveals distinct technological shifts. Early approaches predominantly relied on feature engineering [6,7]. However, the advent of deep learning has revolutionized this paradigm, with contemporary models achieving state-of-the-art performance through Transformer-based architectures (e.g., BERT [8], RoBERTa [9]) [10,11,12]. These models excel at generating context-aware event representations by encoding semantic dependencies across entire input texts. Modern ECI frameworks typically formulate the task as a causal binary classification problem, where supervised learning trains models on annotated datasets to distinguish causal pairs (e.g., earthquake → tsunami) from non-causal associations [13,14]. In the final phase of contemporary ECI models, representations of event contexts are transformed into representations of causal relationship labels. These label representations are subsequently employed to compute logit scores for potential causal relationship labels during both training and inference processes. Ideally, these representations should encapsulate distilled information that enhances the accuracy of label scoring and prediction capabilities.

In many ECI models, feedforward networks are commonly used to transform contextual event representations into causal label predictions [15,16,17]. While such architectures provide implementation simplicity, directly mapping contextual representations to causal labels through feedforward transformations may limit the model’s ability to capture complex semantic interactions between event contexts and causal relations. Event context representations often contain semantically rich but noisy patterns due to long-range dependencies between event triggers. Under these conditions, simple context-to-label transformations may propagate contextual noise into the prediction process, which can negatively affect causal relation identification.

This issue becomes particularly acute in ECI tasks involving long-range dependency modeling, where causal inference between event mentions separated by extended contextual spans is significantly impeded by noise accumulation. Specifically, lengthy input sequences frequently introduce extraneous syntactic constructs (e.g., descriptive clauses, peripheral entities) and non-causal discourse markers that obscure genuine causal linkages, thereby reducing the signal-to-noise ratio in representation learning. The representational capacity of Transformer-based encoders further compounds this challenge, as their high-dimensional latent spaces inadvertently preserve extraneous constructs within event context embeddings. For feedforward network, filtering out these extraneous constructs to reveal causal relationship proves to be far from trivial. For instance, Document-level ECI scenarios exemplify this issue acutely: when event mentions span multiple sentences, the intervening lexical content introduces substantial semantic noise.

To mitigate the prevalent noise influence in event context embeddings, we employ generative adversarial networks (GAN) to synthesize latent label embeddings that enhance causal recognition. Additionally, a contrastive learning methodology is implemented to refine the original event representations. This methodological refinement enables the GAN architecture to generate more accurate causal label embeddings, consequently improving ECI performance. The systematic integration of adversarial generation and contrastive representation learning constitutes a robust framework for noise-resilient feature extraction in complex event interaction environments. Inspired by recent advances in ECI, including diffusion-based approaches such as DiffusECI [18], we propose CLECI (Contrastive Learning-Enhanced Causality Identification), a novel framework that integrates contrastive learning with adversarial optimization to enhance latent causal label representation learning. Unlike diffusion-based generative paradigms, CLECI focuses on learning latent causal label representations and aligning them with contextual event representations through contrastive and adversarial training. The architecture operates through three principal phases. First, construct causal label, in which event mentions and their relational pairs are systematically extracted from sentence-level annotations to formulate a ground-truth label representation, and these relational structures are subsequently encoded using RoBERTa to generate contextualized embedding. Second, a latent causal label embedding space is derived through contrastive alignment of event pairs. By aggregating embeddings from causally linked event pairs, the model learns a refined true causal label embedding, which ensures geometric proximity between semantically congruent embeddings. Third, a global optimization strategy with GAN is implemented to enhance ECI discrimination. In one hand, the encoding and latent label generation processes are optimized to minimize the reconstruction loss. On the other hand, the training procedure of the discriminator is optimized to maximize its loss value, while the loss functions of other model components are simultaneously subjected to joint optimization. In summary, our contributions are as follows:

• We propose an innovative ECI framework that synergistically integrates context-to-label representation with contrastive learning. The framework employs two core components: noise-suppressed latent embeddings and context-aware event representations.
• A novel context-to-label transformation mechanism, which integrates contrastive learning and a simplified GAN framework, is introduced in ECI framework, and significantly enhances the mapping fidelity between event context embeddings and causal label representations.
• Extensive experiments are conducted on multiple ECI benchmark datasets. Experimental results demonstrate the superiority of our model, with state-of-the-art performance achieved across diverse datasets.

2. Related Work

2.1. Event Causality Identification

The Scholarly research of event causality focuses on two primary tasks: Event Causality Extraction (ECE) and ECI. ECE is a challenging task within the domain of Information Extraction (IE), requiring the automated identification of event descriptions and subsequent detection of causal relationships between them. The accurate discernment of such causal linkages is critical for advancing downstream applications. ECI is formally defined as the determination of causal relationships between paired event mentions within textual data, framed as a classification task. Specifically, event mentions, also referred to as triggers, may either appear within the same sentence (sentence-level ECI) or span across multiple sentences (document-level ECI).

To comprehensively capture contextual dependencies among event mentions, a context window with five sentences, which includes the sentence containing the target event along with its four adjacent sentences (two preceding and two succeeding [19,20]), was employed in [19,20] to identify causality using pre-trained language models (PLMs) [8,9]. In [21], SemDI (Semantic Dependency Inquiry Network for ECI), built on RoBERTa, analyzes word dependencies, applies a cloze-style context prediction task, and leverages multi-head attention mechanisms to evaluate causal associations. DualCor [22] proposes a dual-grid tagging framework based on BERT, which captures event correlations by integrating event type information with contextual representations to effectively distinguish cause-effect pairs. LSIN [23] further models external knowledge structures through graph neural networks to enhance event relation representation, while GCKAN [24] introduces a graph contrastive knowledge-augmented network to improve the modeling of causal dependencies between events.

Recent advances in pre-trained language models (PLMs) and prompt learning have given rise to two emerging approaches for Event Causality Identification (ECI). The first is causal knowledge pre-training, which enhances PLMs by incorporating causal knowledge during pre-training, enabling zero-shot, end-to-end ECI through the use of simple prompts. The second is prompt-based fine-tuning, which leverages labeled data to train PLMs using prompt templates and tailored learning strategies, thereby improving the accuracy of causal relation detection. Furthermore, CausalBERT [25] improves causal reasoning by incorporating multi-level causal pairs derived from unstructured text into the BERT framework. This model is developed through three stages: (1) unsupervised pre-training employing a BERT-based masked language model; (2) classification and ranking of causal pairs utilizing resources such as CausalBank [26] and ConceptNet [27]; (3) application to causal classification tasks via simple prompts or fine-tuning. Other recent approaches explore diverse strategies for improving event causal reasoning. CauSeRL [28] augments causal statements using a self-supervised framework to enrich event representations, while GenEI [29] adopts a generative paradigm to jointly produce causal relations and dependency paths between event mentions. Prompt-based and knowledge-enhanced approaches have also been proposed, including DPJL [30] and KADEE [31]. More recent studies further explore alternative learning paradigms for ECI. For instance, HOTECI [32] employs generative modeling with Optimal Transport to select salient contextual information, while iLIF [33] adopts an identify-while-learning strategy that iteratively refines event representations for causal inference. DiffusECI [18] formulates causal relation prediction as a diffusion-based generation process, where causal structures are progressively generated through an iterative denoising mechanism.

2.2. Contrastive Learning

Recent studies have demonstrated that pre-trained language models, such as BERT [8], are highly effective in capturing semantic information from text. These models excel at identifying complex semantic relationships between words, including synonymy, hypernymy, and causality. This capability facilitates the generation of sentence embeddings enriched with deeper semantic insights, which can be particularly valuable for tasks requiring nuanced understanding, such as causal relationship identification. The model proposed in [34] employs a three-dimensional tensor representation to encode and integrate event elements—specifically $(O_1,P,O_2)$—effectively capturing the interactions among these components. Building on this foundation, ref. [35] introduced tensor-based predicate embeddings to enhance the model’s flexibility and representational power. Similarly, ref. [36] extended event embeddings by incorporating event sentiments and intentions through a multi-task learning framework, thereby enriching the semantic representation of events with additional contextual information. While these advancements illustrate the potential of pre-trained models, they also expose limitations in their general encoding capabilities. Specifically, pre-trained encoders may struggle to fully capture the original semantic features pertinent to causality-related tasks. To address this challenge, contrastive learning has emerged as a promising approach aimed at enhancing the representational capacity of these models by emphasizing distinctions between semantically similar and dissimilar instances.

Supervised Contrastive Learning (SCL) [37], for instance, improves event representations by integrating causal labels while constructing positive and negative event pair triplets; it applies contrastive loss to generate embeddings that are more distinctive and causally informative. Similarly, ECLEP (Event Causal Label and Event Pair Interaction Graph) [38] enhances event representations by combining causal label information with event-pair interactions. It achieves this by constructing an event-pair interaction graph and employing Graph Attention Networks (GAT) [39] to iteratively update event embeddings. These methods underscore the importance of encoding causal relationships, yet challenges remain in extracting the most meaningful and essential features for causality identification.

A more recent approach, known as ICCL (In-Context Contrastive Learning for Event Causality Identification) [40], integrates prompt-based learning with in-context contrastive learning to effectively address these challenges. By leveraging RoBERTa [9] for masked token prediction within query templates, ICCL employs contrastive learning techniques to distinguish between positive and negative samples. This dual methodology significantly enhances the model’s capacity to differentiate causal pairs from non-causal ones, thereby advancing the representation of causality within event embeddings. To further enhance the model’s representation capability for the ECI task via contrastive learning, we integrate contrastive learning mechanisms into the encoding phase of the ECI framework.

3. Proposed Model

3.1. Problem Formulation

The ECI task focuses on determining whether a causal relationship exists between two event mentions within a given text. Specifically, for a pair of event mentions $even{t}_1$ and $even{t}_2$, we use a tuple $T=(h,t)$, where ‘h’ stands for the head index and ‘t’ stands for the tail index, to indicate the starting and ending positions of the trigger words for events $e_1$ or $e_2$ in text. The ECI task aims to predict whether there is a causal relationship between two event mentions utilizing contextualized representations of event mentions and their surrounding text, and the result is denoted as a binary label $y\in \{0,1\}$, where $y=1$ indicates that $e_1$ causes $e_2$, $y=0$ indicates no causal relationship. We expect to predict accurate y event with lots of long and noisy contexts that introduce semantic gap between complex event representations and causal labels.

3.2. Overall Framework

This section provides an in-depth description of our proposed framework, CLECI, illustrated in Figure 2. The CLECI framework is composed of three main modules: the Encoding Module, the Contrastive Learning Module, and the Prediction Module.

Figure 2. Proposed CLECI Model Architecture Diagram, which consists of encoding module, contrastive learning module, prediction module.

This framework integrates contrastive learning mechanisms during the training phase to augment the encoder’s representational capacity, enabling the extraction of semantically discriminative event features and consequently optimizing the quality of derived embeddings.

• Encoding Module: In this module, event pairs are encoded using context and label respectively, and both of them are subsequently fed into a pre-trained model (we utilize RoBERTa within our framework) to obtain representations for both events and labels.
• Contrastive Learning Module: This module facilitates the extraction of semantically discriminative event representations through contrastive feature analysis of features obtained from the Encoding Module, wherein inter-class dissimilarities and intra-class similarities are jointly optimized to refine the representational quality of derived embeddings.
• Prediction Module: The vector representations of paired input events generated by the encoding module are concatenated and processed through a Multilayer Perceptron (MLP) architecture. The computational pipeline culminates in a softmax-activated output layer that generates a probabilistic distribution over potential causal relationships.

3.3. Event Encoding Module

The encoding module is specifically engineered to address the challenge of noise propagation in event context representations, a critical impediment to accurate ECI. Given that event contexts often contain extraneous or semantically irrelevant information, direct transformation of these noisy representations into causal label embeddings result in ineffective learning. The module’s primary objective is to enhance the model’s ability to focus on causality-relevant contextual features while generating noise-suppressed label representations for improved ECI performance.

We employ the encoding module to process text containing event contexts to generate two distinct representation types, contextual event representations and label-based event Representations. Specifically, for the two events from the same sentence or document $e_1$ and $e_2$, a five-sentence window as its context $c_x$ ($x=1$ or 2), which includes the sentence containing the event along with two preceding and two succeeding sentences is constructed. These individual contexts $c_1$ and $c_2$ are concatenated to form a unified contextual representation $C_i$, serving as for modeling event-level context.

On the other hand, following an approach similar to that proposed in DiffusECI [18], both events $e_1$ and $e_2$ are additionally encoded as label representations to capture causality-specific semantic features. The implementation details of these two components are outlined below.

3.3.1. Event Representation and Label Embedding

For a given text with length L including two target events $e_1$ and $e_2$, the attention mask, which indicates the positions that should be attended to during encoding, is denoted as $M\in {\{0,1\}}^{B\times L}$, where B indicates the batch size. The trigger word positions are represented by $T\in {\mathbb{R}}^{B\times 2}$, containing indices of the two trigger words in each input sequence (specifically, the positions of the head event and tail event). This function leverages a pre-trained language model architecture, specifically employing RoBERTa as its foundational framework to perform the designated task. The input to the proposed model comprises context S that includes events $e_1$ and $e_2$, accompanied by the attention mask M; consequently, the resulting output generates token-level embeddings, capturing fine-grained semantic representations of the input sequence. The embedding of input context $S_i$, represented by the hidden states of the encoder’s last layer, is as follows:

$$Z_i=\mathrm{Encoder}(S,M,{\varphi }_0),Z_i\in {\mathbb{R}}^{B\times L\times d},$$

(1)

where ${\varphi }_0$ denotes the dropout mask following the default RoBERTa configuration.

Assume the index of paired event triggers in the i-th sample of a batch is denoted as $T_i=(h_i,t_i)$, where $h_i$ and $t_i$ represent the positions index of the head event and tail event, respectively. The embeddings for the head event $e_1$ and tail event $e_2$ are captured from the hidden state as Equations (2) and (3):

$$c_1=Z_{i,h_i},$$

(2)

$$c_2=Z_{i,t_i}.$$

(3)

For the i-th sample, the context embedding representation $C_i$ for the event pair with index $(h_i,t_i)$ is constructed in Equation (4). Then, the representations of event pairs are achieved from multiple perspectives:

$$C_i=\mathrm{stack}\left([c_1,c_2]\right), C_i\in {\mathbb{R}}^{2\times d},$$

(4)

where $\mathrm{stack}\left(\right)$ indicates the operation that concatenate the $c_i$ and $c_i$.

The relation labels associated with event pairs in the training dataset are encoded into structured label strings, incorporating both the event mentions and their corresponding causal relationships, such as “{e1} has no relation to {e2}” or “{e1} causes {e2}”. The resulting relationship label denoted as l for events $e1$ and $e2$ is subsequently encoded to obtain the label representation showed in Equation (5):

$$Z_{a1}=\mathrm{Encoder}(l,M,{\varphi }_1),Z_{a1}\in {\mathbb{R}}^{B\times L\times d},$$

(5)

where ${\varphi }_1$ denotes the dropout mask following the default RoBERTa configuration.

To enhance the model’s ability to capture semantically meaningful information from the label, the label l is further encoded to generate an additional embedding $Z_{a2}$, which similarly encapsulates the label representations for events $e1$ and $e2$ in Equation (6):

$$Z_{a2}=\mathrm{Encoder}(l,M,{\varphi }_2),Z_{a2}\in {\mathbb{R}}^{B\times L\times d},$$

(6)

where ${\varphi }_2$ denotes the dropout mask following the default RoBERTa configuration.

3.3.2. Latent Label Embedding Generation

Label representations extracted directly from textual data inherently encode explicit causal relationships, yielding semantically rich embeddings that significantly improve the model’s ability to discern and classify such relational patterns. To enhance the model’s ability to accurately discern causal relationships, we propose a dedicated module for generating latent causal label embeddings that exhibit high fidelity to the original label representations, thereby assisting event contexts in executing the ECI task.

Inspired by the architectural principles of Generative Adversarial Networks [41], we design a generator formulated in Equation (7) that integrates the event context embedding $C_i$ with random noise $z_0$ as inputs.

$$z_i=G(c_i,z_0),$$

(7)

where $z_i$ denotes the generated latent causal label embeddings, $G(·)$ signifies the generator function, and $z_0$ is a noise vector sampled from a standard Gaussian distribution.

The resulting latent causal label embeddings, whose dimensions are identical to the true causal label embeddings, are leveraged to significantly improve causality identification within the verification module. The detailed is illustrated in Figure 3.

Figure 3. GAN architecture diagram used in our framework.

The loss function is designed to incorporate adversarial loss, which is derived from the discriminator, with the objective of minimizing the divergence between the generated latent causal label embeddings and their corresponding true causal label embeddings. Assume the true causal label embeddings is $Z_a$, the generator’s loss is formulated in Equation (8):

$$L_g={\mathbb{E}}_{c_i,z_0}\left[logD\left(z_i\right)\right]+·{\mathbb{E}}_{c_i}\left[\parallel z_i-z_a{\parallel }^2\right],$$

(8)

where ${\mathbb{E}}_{c_i,z_0}$ denotes the expected value over the joint distribution of context information $c_i$ and initial noise $z_0$, $D\left(z_i\right)$ represents the output of the discriminator for the generated latent causal label embeddings $z_i$. By optimizing this loss function, the generator learns to produce latent causal label embeddings that are more representative of the underlying data distribution. These enhanced embeddings significantly improve the ability of the validation module to accurately identify causal relationships.

For the discriminator, its role is to differentiate real causal label embeddings $Z_a$ from generated latent causal label embeddings $z_i$. In other words, it can distinguish of $Z_a$ and $z_i$ with a probability score $D\left(z\right)$. The loss function is as Equation (9):

$$L_d=-{\mathbb{E}}_{Z_a}\left[logD\left(Z_a\right)\right]-{\mathbb{E}}_{C_i,z_0}\left[log(1-D\left(z_i\right))\right],$$

(9)

where ${\mathbb{E}}_{Z_a}$ denotes the expected value over the distribution of real causal label embeddings $Z_a$, and ${\mathbb{E}}_{C_i,z_0}$ denotes the expected value over the joint distribution of context information $C_i$ and initial noise $z_0$. By maximizing this loss function, the discriminator has the ability to differentiate real from generated labels.

For the generation of latent causal label embeddings, the generator and discriminator are optimized jointly, enabling the generator to produce latent causal label embeddings that closely match the distribution of real causal label embeddings.

3.4. Contrastive Learning Module

To enhance the causal label embeddings [42] for distinguish relation samples, we introduce self-supervised contrastive learning module.

For each generated latent causal label embedding, we treat it as an anchor sample and perform contrastive learning by comparing it with its positive and negative samples. Specifically, for generated anchor embedding $z_i$ in Equation (7), the previously encoded causal label embeddings $z_{a1}$ in Equation (5) and $z_{a2}$ in Equation (6) are designated as positive samples, while all other samples within the same batch are treated as negative samples. Through this approach, the proposed model effectively captures the essential features of the samples and optimizes the contrastive learning process by minimizing the contrastive loss. To formalize the contrastive learning loss function, we define the similarity between the anchor embedding $z_i$ and its positive samples, computed using Euclidean distance, as $W(z_i,z_{a1})$ in Equation (10) or $W(z_i,z_{a2})$ in Equation (11), and the similarity between anchor embedding and its negative samples as $W(z_i,z_k)$ in Equation (12):

$$W(z_i,z_{a1})=exp\left({\displaystyle \frac{z_i^T{z}_{a1}}{\tau }}\right),$$

(10)

$$W(z_i,z_{a2})=exp\left({\displaystyle \frac{z_i^T{z}_{a2}}{\tau }}\right),$$

(11)

$$W(z_i,z_k)=exp\left({\displaystyle \frac{z_i^T{z}_k}{\tau }}\right),$$

(12)

where $\tau \in \mathbb{R}$ is a positive value of temperature. Following common practice in contrastive learning [42], the temperature parameter is set to $\tau =0.3$.

Through contrastive learning, the similarity between positive samples and the generated anchor embedding is significantly enhanced. However, the original InfoNCE objective, as proposed in prior work [43], faces challenges in scenarios involving multiple positive samples. To address this limitation, we draw inspiration from the supervised contrastive learning framework introduced in [44] and adopt a similar formulation. Based on this, we define the loss ${\mathcal{L}}_{\mathrm{cl}}$ in Equation (13):

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{cl}}& =-{\displaystyle \frac{1}{2}}\left(log{\displaystyle \frac{W(z_i,z_{a1})}{W(z_i,z_{a1})+W(z_i,z_k)}}\right)\hfill \\ \hfill & +log{\displaystyle \frac{W(z_i,z_{a2})}{W(z_i,z_{a2})+W(z_i,z_k)}}.\hfill \end{array}$$

(13)

3.5. Causality Prediction Module

The enhanced latent label representations, combined with event context representations, are utilized for ECI. In this module, both the event context embedding and the causal label embedding are leveraged to predict causal relationships between events. During the model training phase, the event context representation $c_i$ and the true causal label embedding $z_{a1}$ are concatenated and projected into a new latent space through an MLP. This projection captures the essential features of event relations, and the model learns these representations by minimizing the event causality prediction loss.

During the model validation phase, the event context representation $c_i$ and the generated causal label embedding $z_i$ are concatenated and projected into a new latent space using the pre-trained MLP. The resulting representation is then utilized to estimate the probability distribution of causality, which is showed in Equation (14):

$$P\left(y\right|c_i,\mathbf{Z})=\mathrm{softmax}\left(\mathrm{MLP}(c_i+\mathbf{Z})\right),$$

(14)

where y indicates the true label associated for the event pair, $\mathbf{Z}$ is a set and defined as $\{z_{a1},z_i\}$ in the model training stage, $\mathbf{Z}$ satisfy $\mathbf{Z}=\left\{z_i\right\}$ in the model validation stage. The corresponding loss for event causality prediction is defined as Equation (15):

$${\mathcal{L}}_p=-{\displaystyle \frac{1}{2}}(logP\left(y\right|c_i,z_i)+logP\left(y\right|c_i,z_{a1})).$$

(15)

3.6. Model Optimization Strategy

To improve the proposed model’s ability to accurately identify causal relationships, we introduce a multi-optimizer strategy, shown in Figure 4, that simultaneously optimizes the latent label generation, enhances embeddings through contrastive learning, and globally refines event causality prediction. For latent label generation, the optimization process involves minimizing the generator’s loss function to produce high-fidelity embeddings, while simultaneously maximizing the discriminator’s loss to enhance its discriminative capability. To achieve this dual objective, two distinct optimizers—denoted as Optimizer A and Optimizer B—are employed to coordinate the optimization of their respective loss functions.

Figure 4. Optimizer scheduling diagram in our framework.

Optimizer A is designed to minimize the loss associated with the latent label generator, while simultaneously enhancing the embeddings with contrastive learning model and improving event causality prediction accuracy. This optimization ensures that the generated latent causal label embeddings closely approximate the true causal label embeddings, thereby facilitating the training of the event causality prediction network. The specific loss function to be optimized is formally defined in Equation (16):

$${\mathcal{L}}_{opt-a}={\mathcal{L}}_p+{\mathcal{L}}_{cl}+{\mathcal{L}}_g,$$

(16)

Optimizer B is activated subsequent to optimizer A and is employed to maximize the discriminator’s loss, thereby enhancing its ability to differentiate between true causal label embeddings and generated latent causal label embeddings. The loss function for the discriminator is formally defined as Equation (17):

$${\mathcal{L}}_{opt-b}={\mathcal{L}}_d.$$

(17)

Two optimizers, denoted as optimizer A and optimizer B, are used in this strategy, and optimizer A is employed to minize the total loss of latent label generator, enhances embeddings, and event causality prediction in

The optimizers $\mathrm{optimizer} \mathrm{A}$ and $\mathrm{optimizer} \mathrm{B}$ are optimized in stages according to task requirements. Initially, both $\mathrm{optimizer} \mathrm{A}$ and $\mathrm{optimizer} \mathrm{B}$ are established; thereafter, these optimizers are manually activated or frozen at different stages of training.

For optimizing $\mathrm{optimizer} \mathrm{A}$, we activate the optimizer described by Shazeer et al. [45] allowing us to compute pair embeddings alongside generated causal label embeddings. The primary objective of the generator is to produce realistic causal label embeddings from noise input in order to deceive the discriminator. The loss function during this stage consists of three components: prediction loss, adversarial loss (wherein the generator seeks to increase the probability that the discriminator classifies generated labels as real), and contrastive learning loss. This is detailed further in Equation (16). In the model training, two optimizers are activated alternately until the generated latent relation causal label embedding can not be distinguished from true relation embedding, that is to say, discriminator is ineffective for the generated embeddings. By this way, for model validating only context embedding $c_i$ alongside latent causal label embedding $z_i$ is used for ECI tasks accurately.

During model training, the two optimizers are alternately activated until the generated latent causal label embeddings become indistinguishable from the true relation embeddings, indicating that the discriminator can no longer effectively differentiate between them. Consequently, to validate the trained model, only context embedding $c_i$ and latent causal label embedding $z_i$ are utilized for accurate ECI tasks.

4. Experiments

In this section, we design special experiments to evaluate our model for ECI tasks. Three English benchmark datasets: EventStoryLine (ESL) [46], Causal-TimeBank (CTB) [47], and Multilingual Event Causality Identification (MECI) [48] are employed. These datasets are widely recognized in the field for tasks related to ECI. Detailed information regarding these datasets is presented in Table 1.

Table 1. Dataset statistics for Event Causality Identification. Relation count includes both causal and non-causal event pairs.

Dataset	Document Count	Event Count	Relation Count	Causality Count	Year of Release
Causal-TimeBank (CTB) [47]	184	6813	7608	318	2014
Event StoryLine (v0.9) (ESL) [46]	258	5334	5655	5519	2017
Multilingual Event Causality Identification (MECI) [48]	3591	46,000	11,000	11,000	2022

The ESL dataset [46] comprises 258 documents and 5334 event mentions. From these, 7805 intra-sentence event pairs were extracted, of which 1770 pairs are annotated with causal relationships. In accordance with the established methodology [46], the last two topics are designated as the development set, while a 5-fold cross-validation procedure is applied to the documents from the remaining topics.

The CTB dataset [47] comprises 184 documents and 6813 annotated event mentions. Among the 7608 intra-sentence event pairs extracted, 298 pairs are explicitly annotated as causal relationships. Following the methodology described in [11], a 10-fold cross-validation protocol is implemented, with performance metrics reported as the average across all folds.

The MECI dataset, introduced by [48], is specifically designed to support causal relationship modeling and research in multilingual and cross-lingual contexts. Comprising meticulously annotated sentence pairs in multiple languages (e.g., English and Chinese), MECI ensures high data quality by explicitly classifying each pair as causal or non-causal. The dataset is carefully designed to account for syntactic and semantic variations across languages, while incorporating diverse types of causal relationships, thereby presenting significant analytical challenges. As a result, MECI serves as a critical benchmark for evaluating and advancing multilingual ECI models, contributing to the broader field of multilingual natural language processing.

To evaluate the performance of the proposed model, we adopt precision (P), recall (R), and the F1-score, as defined in Equations (18)–(20), as standard evaluation metrics. Together, these metrics offer a comprehensive assessment of the model’s predictive accuracy and robustness. Precision quantifies the proportion of true positives (i.e., correctly identified causal event pairs) among all instances predicted as positive by the model. Recall measures the proportion of correctly identified positive instances relative to the total number of actual positives in the dataset. The F1-score, calculated as the harmonic mean of precision and recall, provides a balanced metric that reflects both aspects, effectively capturing the model’s overall ability to detect causal relationships. The Equations (18)–(20) present the formulas for these evaluation metrics:

$$P={\displaystyle \frac{TP}{TP+FP}} \left(\mathrm{Precision}\right),$$

(18)

$$R={\displaystyle \frac{TP}{TP+FN}} \left(\mathrm{Recall}\right),$$

(19)

$$F1=2\times {\displaystyle \frac{P\times R}{P+R}} \left(\mathrm{F}1\text{-}\mathrm{score}\right),$$

(20)

where $TP$ denotes the set of correctly identified causal event pairs, $FP$ represents the set of false positives (i.e., non-causal pairs incorrectly classified as causal), and $FN$ corresponds to the set of false negatives (i.e., causal pairs missed by the model). The F1-score, as a composite metric, simultaneously captures the model’s capability to accurately detect causal relationships and minimize both false positives and false negatives, thereby providing a balanced measure of predictive performance.

4.1. Baselines

The baselines we used for ECI mainly include Transformer-based models and others. The following baseline models, which do not utilize Transformer-based architectures, are included:

1.

LSTM: A model based on Long Short-Term Memory (LSTM) networks, proposed by [15].

2.

Seq: A sequence-based model originally introduced by [14] in 2017 for the ECI dataset and later utilized in 2019 by [15] for their study.

3.

LR+ and LIP: Document-structure-based models proposed by [15].

4.

ML: A feature-based model developed by [47] in 2014.

For transformer-based models, we consider the following baselines:

1.

KEMM [11]: A knowledge-enhanced model that incorporates triplet knowledge into sentences to leverage external information. It masks event mentions to better mine causal patterns.

2.

LSIN [23]: A latent structure induction network that models external knowledge structures using graph neural networks. This model is closely related to ours.

3.

CauSeRL [28]: A method for augmenting causal statements via a self-supervised framework that enriches event representations with causal patterns.

4.

GenEI [29]: A generative model that jointly produces causal relations and dependency paths between event mentions using reinforcement learning.

5.

DPJL [30]: A prompt-based approach leveraging derivative prompt learning techniques.

6.

KADEE [31]: A model that integrates external knowledge with internal event analogies.

7.

GCKAN [24]: A graph contrastive knowledge-augmented network designed for the ECI task.

8.

HOTECI [32]: A generative framework that uses Optimal Transport (OT) to automatically select salient sentences and words from full documents for ECI. It enhances performance by employing a T5 encoder-decoder to generate both causal relation labels and contextual highlights.

9.

iLIF [33]: An “identify-while-learning” approach that iteratively refines event representations to improve causal inference.

10.

DiffusECI [18]: A diffusion model-based method for event causality identification.

4.2. Experiment Settings

Our model utilizes the RoBERTa-large variant as the encoder, with a hidden size of $d=1024$. The development set of EventStoryLine (ESL) is used to tune all hyperparameters of the CLECI model.

For the prediction module, we employ a two-layer multilayer perceptron (MLP) with a hidden dimension of 1024. The mini-batch size is set to 32. The model is optimized using the AdamW optimizer with a learning rate of $5\times {10}^{-5}$. All hyperparameters are selected based on the best F1 score on the ESL development set [49].

4.3. Comparison

4.3.1. EventStoryLine and Causal-TimeBank

Table 2 summarizes the performance metrics—Precision (P), Recall (R), and F1 score (F1)—of our proposed CLECI model on the EventStoryLine and Causal-TimeBank datasets, alongside comparative results from other models. The experimental results demonstrate that CLECI achieves state-of-the-art performance, with F1 score improvements of 4.3% and 7.9% on the ESL and CTB datasets, respectively, compared to existing models. These findings substantiate the efficacy of the proposed contrastive learning framework in enhancing label transformation for ECI.

Table 2. Model performance on the test sets of EventStoryLine and Causal-TimeBank. Bold values indicate the best performance in each column.

Model	EventStoryLine			Causal-TimeBank
	P (%)	R (%)	F1 (%)	P (%)	R (%)	F1 (%)
LSTM (2019) [15]	34.0	41.5	37.4	-	-	-
Seq (2019) [14]	32.7	44.9	37.8	-	-	-
LR+ (2019) [15]	37.0	45.2	40.7	-	-	-
LIP (2019) [15]	38.8	52.4	44.6	-	-	-
ML (2014) [47]	-	-	-	67.3	22.6	33.9
KMMG (2021) [50]	41.9	62.5	50.1	36.6	55.6	44.1
RichGCN (2021) [16]	49.2	63.0	55.2	39.7	56.5	46.7
LSIN (2021) [23]	49.7	58.1	52.5	51.5	56.5	53.7
CauSeRL (2021) [28]	41.9	69.0	52.1	43.6	68.1	53.2
GenEI (2022) [29]	59.5	57.1	58.8	60.1	53.3	56.5
DPJL (2022) [30]	65.3	70.8	67.9	63.6	66.7	64.6
KADE (2023) [31]	61.5	73.2	66.8	56.8	70.6	66.7
GCKAN (2024) [24]	50.9	60.6	55.3	52.2	60.7	56.1
HOTECI (2024) [32]	66.1	72.3	69.1	71.1	65.9	68.4
iLIF (2024) [33]	76.8	66.3	71.2	-	-	-
DiffusECI (2024) [18]	65.3	78.3	71.4	87.7	66.1	75.4
CLECIfull	72.2	79.6	75.7	88.2	78.9	83.3

Traditional baseline models, including LSTM (2019), Seq (2019), LR+ (2019), and LIP (2019), attained F1 scores of 37.4%, 39.8%, 40.7%, and 44.6%, respectively, on the EventStoryLine dataset. These approaches, which rely on conventional feature extraction and sequence modeling techniques, exhibit limited effectiveness in capturing event causality. Similarly, on the Causal-TimeBank dataset, the ML (2014) method achieved an F1 score of 33.9%, further underscoring the challenges associated with identifying causal relationships using traditional methodologies. In contrast, knowledge-enhanced methodologies such as KMMG (2021), RichGCN (2021), and LSIN (2021) incorporated background knowledge to enhance event representations. These models achieved F1 scores of 50.1%, 55.3%, and 52.5% on the EventStoryLine dataset, demonstrating significant advancements over traditional approaches in capturing complex event semantics and relationships. Notably, their performance on the Causal-TimeBank dataset ranged from 44.1% to 53.7%, reflecting variations in efficacy across different datasets.

Further developments in event causality extraction emerged with models like CauSeRL (2021) and GenECI (2022), which leveraged self-supervised learning and event knowledge graphs to achieve F1 scores of 53.2% and 58.0% on the EventStoryLine dataset. Subsequent innovations, such as DPJL (2022) and KADE (2023), integrated multi-path information and advanced semantic relationship modeling, pushing performance to F1 scores of 60.3% and 62.7% on the same dataset. Recent methods such as iLIF (2024) and DiffusECI (2024) achieved F1 scores of 71.2% and 71.4% on the EventStoryLine dataset, while DiffusECI obtained an F1 score of 75.4% on the Causal-TimeBank dataset. Additionally, GCKAN (2024), which utilizes graph-based contrastive learning, attained F1 scores of 55.3% and 56.1% on the two datasets, highlighting the potential of structural information exploitation for causality identification. Collectively, these results illustrate a clear trajectory of performance improvements driven by evolving techniques, culminating in the superior capability of contrastive learning frameworks exemplified by CLECI.

The proposed method, denoted as $CLEC{I}_{full}$, exhibited superior performance relative to all existing models across both datasets. On the EventStoryLine dataset, it achieved Precision, Recall, and F1 scores of 72%, 79.6%, and 75.7%, respectively. Similarly, on the Causal-TimeBank dataset, it attained Precision, Recall, and F1 scores of 88.2%, 78.9%, and 83.3%. These findings substantiate significant improvements in both predictive accuracy and model robustness, thereby confirming the effectiveness of the proposed methodology.

The results presented in Table 2 demonstrate the superior performance of the proposed model for event causality extraction compared to existing methodologies. Specifically, the $CLEC{I}_{full}$ method achieved significant improvements in the F1 score—a comprehensive evaluation metric—across both datasets, with increases of 4.3% on the EventStoryLine dataset and 7.9% on the Causal-TimeBank dataset. These findings validate the effectiveness of the proposed enhancements, which integrate contrastive learning with label transformation and combine background knowledge with graph-based contrastive learning techniques.

4.3.2. Different Languages

The base model XLM-RoBERTa utilizes a RoBERTa encoder but entirely excludes the contrastive learning, generator discriminator, and causal label embedding modules. This configuration performs event relationship classification exclusively based on the raw related event pairs representations, without incorporating the concatenation of context representations for label distribution prediction.

Table 3 provides a comparative analysis of the CLECI model against existing methodologies on the test set of the multilingual MECI dataset [48]. The encoder model utilizes the multilingual variant of RoBERTa, specifically the large version of XLM-RoBERTa. The results indicate that CLECI achieves strong performance across multiple languages in the MECI dataset. The model obtains the best results on English, Turkish, and Urdu, while remaining competitive with strong baselines on Danish and Spanish. These findings demonstrate the effectiveness of the proposed model for multilingual ECI tasks. The slightly lower performance observed in Danish and Spanish may be related to differences in linguistic structure across languages, which may influence the effectiveness of representation learning for causal relation modeling. These findings further validate the effectiveness of the proposed model in addressing multilingual ECI tasks.

Table 3. Model performance on the test sets of the MECI dataset. Bold values indicate the best performance in each column.

Model	English			Danish			Spanish			Turkish			Urdu
	P (%)	R (%)	F1 (%)	P (%)	R (%)	F1 (%)	P (%)	R %)	F1 (%)	P (%)	R (%)	F1 (%)	P (%)	R (%)	F1 (%)
XLM-RoBERTa [9]	48.7	59.9	53.7	35.9	36.2	36.0	50.6	49.1	49.9	44.0	59.4	50.5	40.4	43.2	41.8
KnowBERT [11]	39.3	42.6	40.9	31.4	11.4	16.7	39.9	28.4	33.2	36.5	46.7	41.0	41.1	22.2	28.9
RichGCN [16]	50.6	68.0	58.1	31.9	50.0	38.9	50.7	55.0	52.8	50.5	64.6	56.7	37.7	56.0	45.1
DiffusECI [18]	70.1	68.3	69.2	42.7	53.3	47.4	62.9	50.2	55.8	52.6	66.5	58.7	58.1	52.5	55.2
CLECI (ours)	75.2	68.4	71.7	46.8	49.2	44.6	54.2	56.8	55.5	69.4	51.4	59.5	59.6	53.1	56.1

4.4. Ablation Study

To rigorously evaluate the contributions of individual modules within the CLECI model to causality identification, a comprehensive series of ablation experiments was conducted, as summarized in Table 4. Key components—including the contrastive learning module, the generator-discriminator module, and the causal label embedding module—were systematically removed to analyze their respective impacts on model performance. Throughout these experiments, the core architecture of CLECI was maintained, and multiple model variants were evaluated to isolate the effects of each module.

Table 4. Ablation study on EventStoryLine. Bold values indicate the best performance in each column.

Model	P (%)	R (%)	F1 (%)
RoBERTa [9]	39.3	45.5	42.2
GAN+CL w/o [component]	54.2	59.1	56.5
CL w/o [component]	71.9	74.6	73.2
CLECI (full)	72.2	79.6	75.7

Contrastive Learning Module (w/o CL). In this variant, the contrastive learning module is excluded, and training is conducted solely using event pair representations combined with a classification head. The contrastive learning module plays a pivotal role in enhancing the discriminative capability of event representations by constructing triplets consisting of anchors, positive samples, and negative samples. The removal of this module resulted in a significant reduction with 2.5% for the F1 score, highlighting essential contribution of contrastive learning module to improving the quality of event representations.

Generator-Discriminator Module (w/o GAN+CL). This variant eliminates the generator–discriminator module and relies directly on event representations for classification through a simple feedforward transformation without incorporating latent causal label embedding. The generator-discriminator module is essential for effectively learning latent causal label embeddings; thus its removal led to an alarming drop of 19.2 percentage points in F1 score demonstrating that latent label representations effectively improve the performance of causality classification.

4.5. Embedding Visualization

Figure 5 visualizes the learned embeddings of true labels and generated latent embeddings during the training process using t-distributed stochastic neighbor embedding (t-SNE) [51]. The embeddings are analyzed across multiple training epochs, with the corresponding F1 scores provided beneath each subplot. In the visualization, blue points correspond to true causal label embeddings, while red points represent generated latent embeddings.

Figure 5. ESL dataset embedding visualization.

During the training process, a pronounced clustering effect is observed in the embeddings. In the initial epochs (e.g., Figure 5a), a distinct separation between two types of embeddings is evident, characterized by minimal overlap. As training progresses—demonstrated in Figure 5b–d, the clusters exhibit increasing cohesion, with the generated embeddings (red points) progressively converging toward the true causal label embeddings (blue points). This trend is accompanied by corresponding improvements in F1 scores, indicating that the model successfully learns representations that align more accurately with the true label space.

This visualization demonstrates the model’s efficacy in improving the alignment between generated and true causal label embeddings for event pairs, highlighting both the progression of training and the impact of contrastive and adversarial objectives on representation learning.

4.6. Model Complexity and Training Cost

All models are trained using the same dataset and under the same GPU environment to ensure a fair comparison of computational cost. The backbone encoder used in CLECI is RoBERTa-large, which contains approximately 355 M parameters. The additional components introduced in the proposed framework, including the generator, discriminator, and projection layers for contrastive learning, add only a relatively small number of parameters compared with the backbone model. To provide a clearer view of the computational requirements, we report the model size and training time of CLECI together with representative transformer-based baselines.

Table 5 presents the parameter scale and training time comparison. Since most compared methods adopt similar pretrained transformer encoders, the computational cost is largely dominated by the backbone model. Although CLECI introduces adversarial training and contrastive learning modules, the overall parameter scale and training time remain comparable to existing transformer-based approaches.

Table 5. Model complexity and training cost comparison.

Model	Backbone	Parameters	Training Time
iLIF	T5-base	∼336 M	∼2.5 h
DiffusECI	RoBERTa-large	∼433 M	∼3.6 h
CLECI (ours)	RoBERTa-large	∼376 M	∼3 h

5. Conclusions and Future Work

In this paper, an ECI model, denoted as CLECI, is proposed. In this model, the event causality is predicted through event relation representation, relation latent causal label embedding generation and latent causal label embedding enhancement. The experimental results demonstrate that proposed CLECI outperforms both non-Transformer baseline models and several Transformer-based variants, confirming its effectiveness in capturing complex causal relationships. More importantly, CLECI introduces a context-to-label transformation mechanism that explicitly learns latent causal label embedding and aligns them with contextual event representations, thereby effectively reducing the semantic gap between event context representations and causal labels. The contrastive learning module significantly improves the model’s ability to generate latent causal label embeddings by enhancing the similarity between latent and true label representations. The generator-discriminator mechanism plays a critical role in learning these latent label representations, while the causal label embedding mechanism further strengthens the model’s ability to distinguish among diverse types of causal relationships.

Furthermore, CLECI demonstrates robust adaptability in multilingual contexts, primarily due to the advanced language understanding capabilities of the XLM-RoBERTa encoder and the generalization capacity inherent in contrastive learning. The model achieves significant performance improvements on publicly available datasets, including F1-score improvements of 4.3%, 7.9%, and 2.5% on ESL, CTB, and MECI, respectively, compared with the state-of-the-art methods, highlighting its applicability not only to single-language causality identification but also to multilingual and cross-lingual tasks. This highlights its potential for extensive applicability across a wide range of linguistic environments. By addressing critical challenges in ECI through the synergistic integration of multiple modules, CLECI advances novel methodologies for modeling causal relationships and lays a robust foundation for future research in this domain.