A Hybrid Transformer-Ensemble Framework for Precise Election Poll Analysis

Abstract

The prediction of election outcomes is a critical area in political analysis and decision-making. Accurate forecasting models can significantly influence electoral strategies and policy formulation. Existing models, however, face challenges in handling complex, dynamic, and high-dimensional election data. This paper addresses these issues by utilizing the Election Polls Dataset, which includes structured and unstructured data from multiple polling agencies such as YouGov (U.K. and NY, USA), Morning Consult (Washington, DC, USA), and Harris Insights (Chicago, IL, USA). We propose a novel hybrid approach combining Transformer models with ensemble learning techniques, including Random Forest, XGBoost, and Gradient Boosting, to enhance prediction accuracy. The novelty of this approach lies in the integration of Transformer’s attention mechanism with ensemble methods, improving both prediction accuracy and model stability. The performance of the model is evaluated using metrics such as accuracy, F1-Score, precision, recall, and AUC-ROC. Experimental results show that the proposed model outperforms existing methods, achieving a 93.4% accuracy, surpassing the baseline model by 1.4%.

1. Introduction

Election polls have historically been invaluable tools for forecasting political races, but the reliability of these polling results is often criticized—especially when voter composition is diverse and local politics move rapidly [1,2]. Standard models, like a simple linear regression or basic machine learning model may not be enough to grab the complexity and non-linearity of what we see in electoral statistics, as we have seen there is complexity to the datasets (polling trends, demographic undercurrents, etc.) [3]. So, there is a need to develop methods that can effectively help us find a way for more complex methods to work well with such complexities. Statistical models exhibit a 20–30% error in predicting electoral outcomes with highly volatile polling data, and even deep learning mechanisms, such as neural networks, struggle to capture long-term dependencies and context from both structured (i.e., polls) and unstructured sources [4]. Inspired by the success of model family Transformer in sequence prediction and empowerment of attention mechanism, we found that even though its predictive power when it comes to sequential data is significantly improved, the emergence of ensemble learning techniques that combine various model abilities can help reduce overfitting and enhance robustness [5].

Recent developments in machine learning, especially the advent of Transformer models, have led to a paradigm shift in processing complex and high-dimensional data from a wide range of applications [6]. Originally developed for NLP, Transformers have become useful in a wide spectrum of prediction tasks other than text, such time series forecasting, sentiment analysis, and even trading. Ensemble learning techniques, including Random Forest, Gradient Boosting algorithms, and stacking models have the potential to enhance single model accuracy by aggregating the predictions of a number of weaker models to form a stronger aggregate prediction. But no one has combined the two general approaches of Transformer models and ensemble methods for election poll analysis where time trends are essential and the subtleties of public sentiment matter a lot [7].

Despite their widespread adoption, conventional election forecasting models—including linear and logistic regression, ARIMA, and standard machine learning classifiers—exhibit several structural limitations when applied to real-world polling data:

• House effects and pollster bias:

Polling agencies often introduce systematic biases due to methodology, sampling design, and weighting strategies. Traditional regression and aggregate polling models assume independent observations and require extensive manual adjustments to account for pollster-specific offsets.

• Time-varying electorate and late-stage momentum shifts:

Voter preferences frequently change in response to debates, scandals, or strategic voting, especially near the election day. Static models and classical autoregressive approaches struggle to capture such non-stationary and abrupt temporal dynamics.

• Sparse and irregular polling intervals:

Polls are collected at uneven time intervals and with varying frequencies across election types, violating the regularity assumptions of classical time-series models and leading to information loss when interpolation or aggregation is applied.

• Heterogeneous covariates:

Election datasets combine numerical indicators (e.g., vote share), categorical metadata (e.g., pollster, methodology), and occasional unstructured text (e.g., polling notes). Traditional machine learning pipelines rely heavily on manual feature engineering to integrate these modalities, often reducing robustness.

• Target ambiguity and calibration challenges:

Election forecasting may involve predicting vote share or binary win–loss outcomes. Many existing approaches inadequately distinguish between these objectives, resulting in poorly calibrated predictions and inconsistent evaluation.

• Distribution shifts across election types:

Polling dynamics differ substantially across Senate, House, Governor, and Presidential races, limiting the generalization ability of models trained under stable distribution assumptions.

These challenges motivate the need for modeling frameworks capable of handling heterogeneous inputs, irregular temporal structures, and distributional variability without extensive manual intervention.

The key difficulty in the standard election poll forecasting techniques is to try and model complex inter-relationships between time series data (poll trends) and sentiment analysis (from social networks, speech, etc.), as well as across features such as demographics, geography, and past voting behavior [6,7]. These variables are not consistently captured by classical polling models and distorted forecasts, or an overly simplistic view might emerge. This paper introduces an original hybrid approach that combines the Transformer models and ensemble strategies to tackle this issue. By combining the capability of Transformer model to handle sequential data and its attention mechanism with the strength of ensemble learning, which can help reduce bias and variance, a more robust framework for predicting election outcomes can be built.

The novelty of this work originates from incorporating Transformer networks, which are popular for sequential data-dependent tasks, with ensemble models for election forecasting. While each of these technologies has been studied independently, combining them into a hybrid model is a novel concept. This model takes into account not only sequential data progression, but also external variables related to sentiment and demographic trends, as well as regional factors like incumbency. This study addresses several unresolved challenges in election poll forecasting and makes the following three key contributions:

• Modeling heterogeneous and temporally evolving polling data:

Traditional election forecasting models struggle to represent irregular, time-dependent polling data composed of mixed numerical, categorical, and textual attributes. To address this limitation, this paper proposes a Transformer-based temporal encoding framework that serializes structured polling features into sequential representations, enabling effective learning of temporal dependencies and evolving polling trends.

• Enhancing robustness against polling noise and systematic bias:

Forecasting performance is often degraded by pollster bias, sampling noise, and instability inherent to single predictive models. To mitigate these effects, the proposed framework integrates a multi-model ensemble layer, combining Random Forest, Gradient Boosting, XGBoost, and LightGBM through stacking and voting mechanisms applied to Transformer-derived embeddings, thereby improving prediction stability and generalization.

• Providing transparent component-level performance analysis:

Prior studies frequently lack clarity regarding the relative contribution of individual modeling components. This work addresses this gap by conducting systematic comparisons with established temporal baselines (LSTM, GRU, and Temporal CNN) and performing ablation and ensemble contribution analyses to quantify the impact of each component on overall forecasting performance.

2. Literature Review

Election prediction and polling analyses have come a long way following the advent of machine learning and deep learning methods. With the increasing complexity of polling data and its alignment with non-structured data sources such as social media, demographic changes, and historic voting patterns, more advanced models must be used to generate the best possible prediction. This work will briefly review the development of models predicting election outcomes and the adoption of Transformer based models, with special attention to the role that ensemble learning plays in increasing prediction power.

2.1. Election Polling Models: Traditional Approaches

The classic models for forecasting electoral results were based on statistical methods (e.g., regression techniques, time-series models, or aggregate poll average) [8,9]. Regression models such as linear and logistic regression have been heavily applied on predicting election results from polling data, as well as demographic information [10,11]. However, these models are limited in their ability to address pollster-specific bias, irregular polling intervals, and abrupt temporal shifts in voter preferences, which are common characteristics of real-world election polling data [12]. Time-series techniques like ARIMA and exponential smoothing have been used to understand patterns as time progresses, but they are limited in their ability to capture evolution in public sentiment or new socio-political situations [13]. Furthermore, aggregated polling techniques (which combine different polls to make predictions) are often biased due to small sample sizes and varying quality of poll methodology.

2.2. Deep Learning Approaches in Election Prediction

The predictive accuracy of election polls has the potential to be enhanced through deep learning methods such as neural networks [14,15]. Models such as FNN and RNN have been applied to model non-linearity in polling data. LSTM-RNNs have also been demonstrated as effective in capturing time-based dependencies for polling data [16]. Such models can represent relationships among data and, for example, be helpful in investigating how votes are subject to change over time [17]. Despite their improved capacity to model temporal dependencies, RNN- and LSTM-based approaches remain sensitive to irregular polling frequencies and struggle to scale effectively across heterogeneous election types and feature modalities.

2.3. Transformer Models and Their Application in Election Forecasting

‘Transformers’ which revolutionized the field of Natural Language Processing (NLP), have been utilized for various prediction tasks [18]. Transformer’s self-attention is stronger in learning the long-term dependency and are thus capable to handle sequence-based problems, such as time-series prediction. In election forecasting, Transformer models have been applied to pre-process textual data (e.g., campaign speeches, debates, and social media posts) and have been able to provide sentiment analysis as well [19,20]. In addition, they are quite capable of making sense and mining through large volumes of unstructured data, which makes them especially valuable when common methods cannot apply well. According to numerous research papers, Transformers are proven to be superior in terms of accuracy and computational efficiency compared to RNNs/LSTMs, especially for complex, high-dimensional data sets [21].

2.4. Ensemble Learning Techniques in Election Prediction

Ensemble learning algorithms utilize a combination of models to enhance prediction precision and are designed to minimize the variance and bias across individual models [22,23]. Algorithms such as Random Forests, Gradient Boosting, and stacking have been effectively used to enhance the robustness of election prediction models. Ensemble methods are especially advantageous when base models have complementary strengths and weaknesses, because they provide a holistic solution rather than a model-centric one [24,25]. For example, Random Forests amalgamate decision trees to prevent overfitting and uncover complex structures in data. Gradient Boosting techniques progressively approach the desired model by adjusting for the mistakes of prior models—a useful advantage while designing around noisy or unbalanced electioneering. Ensembles that stack the predictions of multiple coordinating base learners have been especially useful in improving the predictive accuracy of machine learning models for election forecasting.

2.5. Hybrid Approaches: Transformer Ensemble Learning

Although both the Transformer and ensemble models have been used to predict election outcomes individually, combining these approaches is a relatively under explored research area [26,27]. The hybridization of Transformers with ensemble learning has great potential to advance election result predictions. The capability of Transformer model for modeling complex sequential data and its attention mechanism can be well supplemented by the reduction in overfitting and high-generalization ability provided by ensemble learning. For instance, by employing Transformer-based models for feature extraction and applying an ensemble learning algorithm to make the final prediction, one can potentially enhance the accuracy of predictions [28,29]. Such hybrid models could overcome the limitation of each well-established tool by taking advantage of their individual strengths to make less error-prone and downtime-resistant prediction.

2.6. Challenges in Election Prediction Models

Despite the progress in prediction, a few problems remain [30]. One key problem is that it is difficult to combine different sets of data, such as polls and social media sentiment, as well as demographic facts and campaign finance reports. These heterogeneous datasets are difficult to combine in standard models. Furthermore, it is the changing/back-tracking/wavering/confounding nature of political campaigns and voter mood over time that jumble predictions, particularly in unpredictable election years. One important challenge is the bias that might exist in conducting polls, a problem that can be confounded with certain models of machine learning that are not particularly resilient to outliers and/or unbalanced data. The question of model interpretability remains and much of machine learning, including the types that are Transformer-based, acts like a black box, making it difficult to understand how predictions are determined—something that is important for trust in election forecasting models.

2.7. Summary and Positioning of the Present Study

Existing election forecasting studies have primarily focused on either statistical aggregation of polling data, classical machine learning models, or deep learning approaches applied in isolation. Traditional regression- and time-series-based methods struggle to accommodate heterogeneous polling attributes and irregular temporal structures, while recent deep learning models often focus on a single modality or rely on standalone architectures without addressing model instability caused by polling noise and bias.

Although Transformer models have demonstrated strong performance in learning long-range dependencies from sequential data, their application in election forecasting has largely been limited to textual sentiment analysis or isolated time-series modeling [31]. Similarly, ensemble learning techniques have been employed to improve robustness and reduce variance, but they typically operate on manually engineered features and do not leverage learned temporal representations.

As a result, there remains a clear research gap in developing a unified framework that (i) learns temporal representations directly from heterogeneous polling data, (ii) mitigates pollster bias and sampling noise through ensemble modeling, and (iii) provides transparent, component-level performance evaluation. The present study addresses this gap by integrating Transformer-based temporal feature learning with ensemble learning strategies, supported by comparative baselines and ablation analyses to quantify the contribution of each modeling component.

We reviewed the progression of election prediction models in this section, from classical statistical approaches to advanced deep learning. The use of Transformer models has significantly improved the modeling of sequential data and unstructured content (e.g., social media or speech). Moreover, it has been widely demonstrated that ensemble methods could offer sound frameworks for improving prediction accuracy and decreasing model bias. Although Transformer ensemble modeling and application field integration is still its infancy, early research efforts have shown great potential for enhancing the interpretability and scalability of election outcome predictions.

In the following section, this work will investigate the utilization of Transformer models in combination with ensemble learning strategies to overcome such issues, which represents a new direction for improving prediction accuracy and robustness of election forecast models.

3. Dataset Description

The dataset includes several election polls, ranging from both current and historical data, across various types of the elections, including U.S. Senate, House, Governor, and Presidential races [32]. The data contains many polling features, including the polling firm, methodology, sample size, and estimated support percentages for political parties. The dataset is extensive, covering multiple election cycles and offering not only real-time but also historical contexts. The columns in the polling data and their values are described in detail in

Table 1. Detailed overview of election polling dataset attributes and variable descriptions.

Column Name	Description	Example Values
question_id	Unique identifier for each poll question	148,863, 77,170
poll_id	Identifier for the specific poll	8990, 9000
cycle	Election cycle (year)	2020, 2022
state	State in which the poll was conducted	California, Georgia
pollster_id	Unique identifier for the polling agency	568, 1302
pollster	Name of the polling agency	YouGov, Morning Consult
sponsor_ids	Identifier for the sponsor(s) of the poll	352, 538
sponsors	Organization or entity sponsoring the poll	Economist, Politico
display_name	Name displayed for the pollster or polling organization	YouGov, Morning Consult
pollster_rating_id	Rating identifier for the pollster	391, 218
pollster_rating_name	Rating assigned to the pollster	B+, B
fte_grade	Pollster’s grade based on their accuracy and reliability	B+, A−
sample_size	Number of respondents in the poll	1303, 2000
population	Type of population sampled (e.g., registered voters, likely voters)	rv (registered voters), lv (likely voters)
methodology	Method of polling (e.g., online, live phone)	Online, live phone
office_type	Type of election office being polled (e.g., U.S. House, Governor)	U.S. House, U.S. Senate
seat_number	Electoral seat or district number	10, 35
start_date	Date when the poll started	10/30/2021, 10/29/2021
end_date	Date when the poll ended	11/02/2021, 11/01/2021
sponsor_candidate	Political candidate or party sponsoring the poll	Democratic party, Republican party
tracking	Whether the poll is tracking trends over time	TRUE, FALSE
nationwide_batch	Whether the poll is a nationwide poll	TRUE, FALSE
created_at	Timestamp of when the poll data was recorded or created	11/03/2021 09:37, 11/02/2021 09:36
notes	Additional notes related to the poll	N/A, Tracking data not available
stage	Election stage (e.g., general election, primary election)	general, runoff
race_id	Unique identifier for the election race	8990, 8989
candidate_id	Unique identifier for the candidate being polled	21,377, 14,567
candidate_name	Name of the political candidate	Terry R. McAuliffe, Jon Ossoff
candidate_party	Political party of the candidate	DEM, REP, OTH
pct	Percentage of votes support for the candidate	47.0, 50.8

, which describes unique ids, polling methods used for each pollster’s information, such as sample size and date range, and candidate-specific information. The final dataset consists of approximately 48,000 polling observations after preprocessing, comprising Senate (≈14,200), House (≈18,500), Governor (≈7300), and Presidential (≈8000) races.

Although the dataset includes multiple election types (Senate, House, Governor, and Presidential), all records are standardized into a unified tabular schema with consistent feature definitions. Election-specific differences are captured implicitly through contextual metadata rather than separate preprocessing pipelines. In this study, the primary prediction task is formulated as a binary classification problem, where the objective is to predict whether a candidate wins or loses a given election race. The binary target variable is derived from the final election outcome for each (race_id, candidate_id) pair. The polling percentage (pct) is treated as an input feature reflecting voter support rather than as a regression target.

4. Proposed Methodology

This work introduces a hybrid methodology for the prediction of election results by using Transformer models and ensemble learning methods. Here, we describe the methodology, including mathematical and statistical calculations of data preprocessing, model design, training, and evaluation. The mathematical expressions corresponding to several stages of our approach are given. The voting strategy aggregates base learner predictions using soft voting, while the stacking strategy employs a meta-learner to learn optimal combinations of base model outputs. Figure 1 presents the full pipeline for election prediction consisting of input data encodings (words), embedding layer, positional encoding, self-attention layers, and a feed-forward subnetwork.

4.1. Data Preprocessing

Data preprocessing is a critical step in this study to ensure data quality, consistency, and suitability for effective learning by the Transformer and ensemble models.

4.1.1. Data Cleaning and Transformation

Data cleaning involves handling missing values and duplicates, as well as ensuring correct data formats. For numerical columns such as sample size and pct (percentage of support), missing values are imputed using the mean. Let $x_1,x_2,\dots ,x_n$ be the observed values of a column with missing values and $\bar{x}$ the mean of the non-missing values. The imputed value for a missing entry is given by Equation (1).

$${\widehat{x}}_i=\bar{x} \mathrm{for} \mathrm{missing} \mathrm{values} \mathrm{of} x_i$$

(1)

Equation (1) is used to impute missing values in numerical polling attributes such as sample_size and pct, ensuring a complete numerical feature matrix prior to categorical encoding and sequence construction. Categorical columns, such as pollster and methodology, are encoded using one-hot encoding. For a categorical variable with $k$ possible values, one-hot encoding generates a binary vector, $\mathrm{v}\in {\mathbb{R}}^k$, where each entry is 1 if the corresponding category is present and 0 otherwise.

4.1.2. Text Preprocessing

The unstructured text data in columns, such as notes and URLs, is processed using standard NLP techniques.

• Tokenization: A text, $t$, is split into tokens, $t_1,t_2,\dots ,t_m$.
• Stop Word Removal: A stop word list, $S=\{s_1,s_2,\dots ,s_p\}$, is used to filter out common words that do not contribute to the analysis. The resulting set of words, $T^{\prime }$, is given by Equation (2).

  $$T^{\prime }=\{t_i\mid t_i\in T \mathrm{and} t_i\notin S\}$$

  (2)
• TF-IDF Vectorization: For each term, $t$, in the document set, $D=\{d_1,d_2,\dots ,d_n\}$, the term frequency (TF) and inverse document frequency (IDF) are computed as in Equations (3) and (4), respectively.

  $$\mathrm{TF}(t,d_i)={\displaystyle \frac{\mathrm{count} \mathrm{of} t \mathrm{in} d_i}{\mathrm{total} \mathrm{terms} \mathrm{in} d_i}}$$

  (3)

  $$\mathrm{IDF}(t)=\mathrm{l}\mathrm{o}\mathrm{g}\left({\displaystyle \frac{\mid D\mid }{\mid \{d_j\mid t\in d_j\}\mid }}\right)$$

  (4)

The TF-IDF score for a term, $t$, in document, $d_i$, is then computed with Equation (5).

$$\mathrm{TF}\text{-}\mathrm{IDF}(t,d_i)=\mathrm{TF}(t,d_i)\times \mathrm{IDF}\left(t\right)$$

(5)

The resulting TF-IDF vectors provide a lightweight numerical representation of textual polling notes, which are concatenated with structured polling features prior to Transformer-based temporal encoding.

4.1.3. Feature Engineering

We compute new features, such as sentiment scores, for each text entry using sentiment analysis models. Let $S\left(t\right)$ represent the sentiment score for text, $t$, where $S\left(t\right)\in [-\mathrm{1,1}]$ (negative to positive sentiment).

4.2. Model Design

4.2.1. Transformer Model for Sequence Learning

The Transformer model is employed to learn temporal dependencies in the election polling data. The key mathematical operation in the Transformer is the self-attention mechanism, which computes attention scores between different tokens in the sequence. Given a sequence of time-indexed polling tokens, the query (Q), key (K), and value (V) matrices are computed as Equation (6):

$$Q=X{W}_Q,K=X{W}_K,V=X{W}_V$$

(6)

where $W_Q,W_K,W_V$ are learned weight matrices. The attention scores, $\mathrm{Attention} (Q,K,V)$, are computed as Equation (7):

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V$$

(7)

where $d_k$ is the dimension of the key vector, and the softmax function ensures the scores are normalized. Equations (6) and (7) define the self-attention mechanism used to model dependencies across polling time steps rather than linguistic tokens.

In the proposed framework, each token corresponds to a time-indexed polling snapshot represented by a concatenated feature vector comprising numerical polling attributes, encoded categorical variables, and TF-IDF text features. The self-attention mechanism therefore models dependencies across polling time steps rather than across words.

4.2.2. Transformer Input Representation and Temporal Serialization

In the proposed framework, the Transformer model operates on sequential representations derived from structured polling data rather than raw textual tokens. To enable temporal learning, tabular polling records are explicitly serialized into time-ordered sequences as described below.

The Definition of Temporal Units: Each input sequence is constructed for a unique (race_id, candidate_id) pair. Polling records associated with the same race and candidate are chronologically ordered based on the poll end_date, which serves as the temporal index. This ordering reflects the evolution of voter preference over time for a given electoral contest.

The Definition of Time Steps and Tokens: Each time step corresponds to a single polling snapshot (or aggregated polling instance when multiple polls share the same end date). At each time step, t, a fixed-length feature vector is constructed by concatenating the numerical attributes (e.g., pct, sample_size), encoded categorical variables (e.g., pollster, methodology, population), and TF-IDF-based representations of available textual notes.

This concatenated vector represents one token in the Transformer input sequence.

Sequence construction: The resulting input to the Transformer is a sequence, $X=[x_1,x_2,\dots ,x_T],$ where T denotes the number of available polling observations for a given race–candidate pair, and each x_t captures the full polling context at time t.

Handling Irregular Polling Intervals: Because polling data are collected at irregular time intervals, the Transformer relies on positional encodings to preserve temporal order without assuming uniform spacing between observations. This allows the model to learn temporal dependencies directly from the observed sequence structure.

Through this serialization strategy, the Transformer is able to attend across polling time steps, capturing both short-term fluctuations and longer-term trends in voter preference while operating entirely on structured tabular representations.

4.2.3. Ensemble Learning for Robust Predictions

Figure 2 shows the workflow of Transformer and ensemble models for election poll forecasting.

Preprocessing and Embedding: The input (i.e., structured/text data) is preprocessed and embedded. The feature vectors are obtained from Transformer model and then given to ensemble models, such as Random Forest and Gradient Boosting. The final model output is averaged or majority voted for predicting the election results. We use ensemble learning to fuse different models together.

Random Forest: A collection of decision trees $\left\{T_1,T_2,\dots ,T_m\right\}$ is used. Each tree, $T_i$, predicts the outcome, $y_i$, and the final prediction is the average of the individual predictions, as given by Equation (8):

$${\widehat{y}}_{\mathrm{RF}}={\displaystyle \frac{1}{m}}\sum _{i=1}^m{T}_i(X)$$

(8)

This aggregation captures nonlinear interactions among Transformer-derived features while reducing variance introduced by individual decision trees. Equations (8)–(10) formalize how Transformer-derived representations are aggregated and re-weighted through ensemble learning.

Gradient Boosting: The Gradient Boosting method sequentially trains models to correct the errors of previous models. The update for model $i$ is denoted by Equation (9):

$$F_i\left(X\right)=F_{i-1}\left(X\right)+\eta \nabla L\left(F_{i-1}\right(X),y)$$

(9)

where $\eta$ is the learning rate, $\nabla L$ is the gradient of the loss function, and $F_{i-1}\left(X\right)$ is the previous model’s prediction. Gradient Boosting iteratively refines predictions on Transformer embeddings by emphasizing difficult polling instances affected by noise or bias.

Stacking: In stacking, the outputs of base models are used as input for a meta-learner, which makes the final prediction. The final model $F_{\mathrm{stack}}\left(X\right)$ is learned as per Equation (10):

$$F_{\mathrm{stack}}\left(X\right)=\mathrm{MetaLearner}\left({\mathrm{BaseModel}}_1\left(X\right),{\mathrm{BaseModel}}_2\left(X\right),\dots ,{\mathrm{BaseModel}}_m\left(X\right)\right)$$

(10)

where $\mathrm{MetaLearner}$ is typically a linear model or another machine learning algorithm. The stacking mechanism integrates outputs from heterogeneous ensemble learners using a meta-learner, allowing adaptive weighting of models based on their predictive reliability.

4.3. Model Training and Evaluation

4.3.1. Training the Model

The data is divided as training (80%) and test (20%) data. We use the training set for fitting the Transformer and ensemble models. Since the primary objective of this study is election outcome classification (win/loss), the Transformer and ensemble models are trained using cross-entropy loss. Regression-based error metrics are reported only as auxiliary indicators to assess prediction confidence and deviation.

The loss function for the Transformer model is given by Equation (11).

$${\mathcal{L}}_{\mathrm{Transformer}}=-\sum _{i=1}^N{y}_i\mathrm{l}\mathrm{o}\mathrm{g} {\widehat{y}}_i$$

(11)

The input, $X$, is passed through two linear layers with a ReLU activation in between, as given by Equation (12):

$$\mathrm{FFN}\left(X\right)=\mathrm{ReLU}(X{W}_1+b_1)W_2+b_2$$

(12)

where $W_1,W_2$ are the weight matrices, $b_1,b_2$ are the biases, $y_i$ is the actual label, and ${\widehat{y}}_i$ is the predicted probability.

For ensemble models, the loss function is adjusted based on the type of ensemble (e.g., for Gradient Boosting, MSE or log-loss is used).

4.3.2. Hyperparameter Tuning

Hyperparameters such as learning rate, $\eta$, the number of layers, $L$, and the number of attention heads in the Transformer model are tuned using grid search or random search. The optimal hyperparameters are those that minimize the cross-validation error.

4.3.3. Evaluation Metrics

Model performance is primarily evaluated using classification metrics, including accuracy, F1-Score, precision, recall, and AUC-ROC, which directly reflect the correctness and reliability of election outcome predictions. Mean Absolute Error (MAE) is reported as a supplementary measure to capture deviations in predicted confidence scores but is not used as a primary optimization objective. The following evaluation metrics are selected to assess both classification accuracy and error sensitivity of election outcome predictions, with primary emphasis on classification performance. The models are evaluated using several metrics, including:

Accuracy: The proportion of correct predictions is calculated through Equation (13):

$$\mathrm{Accuracy}={\displaystyle \frac{1}{N}}\sum _{i=1}^N\mathbb{I}(y_i={\widehat{y}}_i)$$

(13)

where $\mathbb{I}$ is the indicator function that returns 1 if $y_i={\widehat{y}}_i$ and 0 otherwise.

F1-Score: The harmonic mean of precision and recall will be calculated as per Equation (14):

$$\mathrm{F}1\text{-}\mathrm{Score}=2\times {\displaystyle \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

(14)

where precision and recall are defined as Equations (14) and (15).

$$\begin{gathered} \mathrm{Precision}={\displaystyle \frac{\mathrm{True} \mathrm{Positives}}{\mathrm{True} \mathrm{Positives}+\mathrm{False} \mathrm{Positives}}}, \\ \hspace{1em}\hspace{1em}\mathrm{Recall}={\displaystyle \frac{\mathrm{True} \mathrm{Positives}}{\mathrm{True} \mathrm{Positives}+\mathrm{False} \mathrm{Negatives}}} \end{gathered}$$

(15)

Mean Absolute Error (MAE): It measures the average absolute difference between the predicted and actual values, and it is calculated as per Equation (16).

$$\mathrm{MAE}={\displaystyle \frac{1}{N}}\sum _{i=1}^N\mid y_i-{\widehat{y}}_i\mid$$

(16)

AUC-ROC: It measures the model’s ability to distinguish between classes, and it is used for binary classification problems.

4.4. Hybrid Approach Integration

The Transformer model is used to extract features from both structured (e.g., polling data and demographic information) and unstructured (e.g., social media sentiment and news articles) data. The output of the Transformer model is fed into the ensemble models, which aggregate the predictions to produce a final election forecast. The mathematical combination of predictions from the Transformer and ensemble methods can be expressed as Equation (17):

$${\widehat{y}}_{\mathrm{final}}={\displaystyle \frac{1}{m}}\sum _{i=1}^m\left(T_i\left(X\right)+F_{\mathrm{stack}}\left(X\right)\right)$$

(17)

where $T_i\left(X\right)$ are the predictions from the Transformer and ensemble models, and $F_{\mathrm{stack}}\left(X\right)$ is the meta-learned prediction.

When trained, the model will be used to predict future elections by updating the input with real-time polling and sentiment data. The model produces a probabilistic estimate for the likelihood of each candidate winning.

5. Experimental Setup

The experimental setting is as follows: we are tasked with obtaining election polling data from sources like YouGov, Morning Consult, and Harris Insights, which include structured information (e.g., numbers such as sample size and polling percentages), as well as unstructured information (e.g., English textual content that includes social media posts or candidate speeches). Preprocessing operations include handling missing values, one-hot encoding of categorical features, and tokenization and vectorization of text. The models are written in Python 3.8, using packages such as TensorFlow, Keras Scikit-learn, and XGBoost, and the training is carried out on a system equipped with NVIDIA Tesla P100 GPU and 64 GB RAM. Our core model is the Transformer, which is able to handle both structured and unstructured data through the recursive application of attention (and multi-head attention) layers. The ensemble models including Random Forest, Gradient Boosting, Xgboost, LightGBM, and stacking are utilized for better predictive performance. The models are trained on 80% of the data and hyperparameters were selected with grid search and random search. We apply accuracy, F1-Score, Mean Absolute Error (MAE), and AUC-ROC as evaluation metrics. The last predictions are obtained by integrating results generated by Transformer and ensemble models using average or vote strategy. The important elements of the experimental setup and model configurations used for election prediction experiment are summarized in

Table 2. Experimental setup and model configuration.

Component	Specification
Dataset	Election polls data (structured + unstructured)
Data Source	YouGov, Morning Consult, Harris Insights, etc.
Data Preprocessing	- Impute missing values (mean for continuous, mode for categorical) - One-hot encoding for categorical variables - Tokenization and stop word removal for text data - Text vectorization (TF-IDF or BERT embeddings)
Computational Environment	Programming Language: Python 3.8 Libraries: TensorFlow, Keras, Scikit-learn, XGBoost, LightGBM, CatBoost Hardware: NVIDIA Tesla P100 GPU, Intel Xeon CPU RAM: 64 GB IDE: Jupyter Notebook 6.5.4, Visual Studio Code 1.85.0 OS: Ubuntu 20.04 LTS
Model Architecture	Transformer model: embedding, self-attention, multi-head attention, feed-forward; Neural network ensemble models: Random Forest, Gradient Boosting, XGBoost, LightGBM, stacking, Voting
Training	Data Split: 80% training, 20% testing Optimizer: Adam (for Transformer), gradient descent for ensemble methods Hyperparameter Tuning: grid search/random search
Evaluation Metrics	Accuracy, F1-Score, Mean Absolute Error (MAE), AUC-ROC Statistical Parameters: $\text{-} \mathrm{Precision}:{\displaystyle \frac{TP}{TP+FP}}$ $\text{-} \mathrm{Recall}:{\displaystyle \frac{TP}{TP+FN}}$ $\text{-} \mathrm{F}1\text{-}\mathrm{Score}:2\times {\displaystyle \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$ $\text{-} \mathrm{Mean} \mathrm{squared} \mathrm{error} (\mathrm{MSE}):{\displaystyle \frac{1}{N}}{\sum }_{i=1}^N(y_i-{\widehat{y}}_i{)}^2$
Hyperparameter Tuning	Transformer: number of layers, attention heads, learning rate, batch size Random Forest: number of trees, max depth, min samples split Gradient Boosting: learning rate, number of estimators, max depth XGBoost: learning rate, max depth, subsample rate LightGBM: learning rate, max depth, min child samples Stacking: base models, meta-learner type Voting: Bbse models (Random Forest, XGBoost, etc.)
Ablation Study	- Transformer-only: evaluate Transformer model in isolation for feature extraction. - Ensemble-only: evaluate ensemble models (e.g., Random Forest, XGBoost, etc.) in isolation. - Transformer–ensemble: evaluate combined Transformer and ensemble methods for final predictions. - Transformer–stacking: evaluate performance improvement with stacking. - Transformer–voting: evaluate performance improvement with voting models.
Training Process	Train Transformer and ensemble models using backpropagation, cross-validation, and hyperparameter optimization

.

To provide a rigorous evaluation of the proposed Transformer ensemble framework, we compare its performance against widely used deep learning models for temporal sequence modeling, namely Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), and Temporal Convolutional Network (Temporal CNN).

For a fair comparison, all baseline models are trained on the same serialized polling sequences used by the Transformer, where each time step corresponds to a time-indexed polling snapshot represented by structured feature vectors. The LSTM and GRU models are implemented with stacked recurrent layers, while the Temporal CNN employs one-dimensional causal convolutions to capture temporal dependencies.

All baseline models use identical training–testing splits, feature preprocessing steps, and optimization settings as the proposed approach. Performance is evaluated using the same classification metrics to ensure comparability across models.

6. Results and Discussion

In this section, we report the experiment results of our proposed model Transformer ensemble and compare it with individual models, as well as the base paper. Model performance is primarily assessed using classification metrics (accuracy, F1-Score, precision, recall, and AUC-ROC), while MAE is reported as an auxiliary indicator of confidence deviation. Finally, we also statistically confirm performance improvement through statistical tests, such as paired t-tests, Chi-Square test, confidence interval, and effect size. The distribution and outlier chart for important poll features, such as the Party A share, Party B share, and Party C share percentages, are seen in Figure 3. The left column shows the distribution of individual features, and the right column contains outlier analysis, depicted using boxplots. The distribution histograms show how the poll data is spread out, while the boxplots reveal the existence of possible outlying values for each party’s share.

The AIC score analysis of various model combinations is shown in Figure 4, where the first plot on the left plots AIC scores for different models, such as Transformer, Random Forest, XGBoost, Gradient Boosting, and LightGBM, as well as Voting. The right panel shows the relative AIC differences (ΔAIC), corresponding to the differences between each model’s AIC and that of the best model. The Voting model has the smallest AIC and hence is best-performing at this point metric. The optimal AIC value of 460 is marked in red on the left graph. AIC is used here as a relative model-fit indicator to complement classification metrics. AIC is reported as a complementary model-fit indicator to assess relative information efficiency and robustness rather than as a primary classification objective.

6.1. Performance of Individual Models

The following table shows how the Transformer model performed among other models such as Random Forest, XGBoost, Gradient Boosting, LightGBM, and voting ensemble. The evaluation metrices are accuracy, F1-Score, precision, recall, AUC-ROC, and MAE.

Table 3. Performance of Individual Models.

Model	Accuracy (%)	F1-Score	Precision	Recall	AUC-ROC	MAE
Transformer	86.8	0.84	0.86	0.82	0.93	0.21
Random Forest	81.6	0.79	0.81	0.77	0.89	0.27
XGBoost	84.5	0.81	0.83	0.79	0.91	0.23
Gradient Boosting	83.2	0.8	0.82	0.78	0.9	0.25
LightGBM	85.3	0.82	0.84	0.8	0.92	0.24
Voting	83.9	0.8	0.82	0.79	0.9	0.26

reports performance of each model (Transformer, Random Forest, XGBoost, Gradient Boosting, LightGBM, and Voting) as evaluated using accuracy, F1-Score, precision, recall, AUC-ROC, and MAE.

MAE is reported as an auxiliary diagnostic metric, reflecting deviation in predicted confidence scores; classification metrics constitute the primary evaluation criteria.

6.2. Ensemble Component Contribution Analysis

To quantify the contribution of individual ensemble learners, we conduct a leave-one-out ablation analysis. Starting from the full ensemble (RF + XGBoost + Gradient Boosting + LightGBM), we re-train the ensemble while removing one component model at a time and measure the change in classification performance. The performance drop (Δ) relative to the full ensemble is used as an indicator of each learner’s contribution. In the stacking configuration, Random Forest, XGBoost, Gradient Boosting, and LightGBM serve as base learners, whose prediction outputs are combined using a meta-learner based on logistic regression.

The leave-one-out analysis indicates that removing XGBoost results in the largest performance degradation, suggesting that XGBoost contributes most strongly to the ensemble’s predictive performance, particularly in correcting difficult polling instances. This is shown in

Table 4. Ensemble component contribution (leave-one-out ablation).

Ensemble Variant	Accuracy (%)	AUC-ROC	F1-Score	ΔAccuracy vs. Full	ΔAUC vs. Full
Full (RF + XGB + GB + LGBM)	93.4	0.94	0.88	—	—
Full − RF	92.7	0.938	0.875	−0.7	−0.002
Full − XGBoost	91.9	0.93	0.868	−1.5	−0.010
Full − Gradient Boosting	92.4	0.935	0.872	−1.0	−0.005
Full − LightGBM	92.1	0.932	0.87	−1.3	−0.008

.

6.3. Ablation Study Results

Ablation study compares different network settings—Transformer-only, ensemble-only, and Transformer ensemble. This is useful to understand the effectiveness of individual model component and the combination of Transformer models with ensemble. Performance by models (Transformer-only, ensemble-only, and Transformer ensemble) in terms of accuracy, F1-Score, precision, recall, AUC-ROC, and Mean Absolute Error (MAE) are shown in

Table 5. Performance comparison of individual models and hybrid model (Transformer ensemble).

Model	Accuracy (%)	F1-Score	Precision	Recall	AUC-ROC	MAE	Training Time (s)
Transformer-only	86.8	0.84	0.86	0.82	0.93	0.21	210
Ensemble-only	82	0.8	0.82	0.78	0.9	0.26	145
Transformer ensemble (proposed)	93.4	0.88	0.9	0.85	0.94	0.19	295

. Figure 5 shows the details of the ablation study in a 3D model. Unless explicitly stated otherwise, “Transformer” refers to the standalone Transformer encoder without ensemble aggregation, while the proposed framework integrates Transformer-derived representations with ensemble learning.

All experiments were conducted on a single NVIDIA GPU. The proposed Transformer ensemble framework requires approximately 295 s for training, compared to 210 s for the standalone Transformer model and 145 s for the ensemble-only configuration, reflecting the additional computational cost of ensemble aggregation. The combination of Transformer and ensemble achieves the highest performance, with 93.4% accuracy and 0.94 AUC-ROC, showing significant improvements compared to the individual models.

6.4. Statistical Significance of the Results

To determine whether the observed improvements are statistically significant, we conducted paired t-tests comparing the performance of the base paper and the proposed model. The p-values indicate that the gains are not purely accidental.

Table 6. Statistical significance comparison between base paper and proposed model (Transformer ensemble).

Metric	Base Paper Mean	Proposed Model Mean	t-Statistic	p-Value	95% CI for Difference (Lower Bound, Upper Bound)	Effect Size (Cohen’s d)
Accuracy (%)	92	93.4	6.67	0.0001	(0.9, 1.8)	0.75
F1-Score	0.87	0.88	2.48	0.015	(0.01, 0.03)	0.48
Precision	0.85	0.9	4.15	0.0007	(0.03, 0.05)	0.62
Recall	0.8	0.85	3.62	0.002	(0.03, 0.05)	0.56
AUC-ROC	0.91	0.94	4.15	0.0007	(0.02, 0.05)	0.61
MAE	0.25	0.19	5.91	0.0001	(0.04, 0.08)	0.8

shows the statistical significance of improvement between the base paper and the proposed model (Transformer ensemble).

• p-value: all p-values are below 0.05, confirming that the differences observed are statistically significant.
• We computed the 95% confidence intervals (CI) for the difference in performance metrics, which gives an estimate of the range within which true differences lie.
• These intervals indicate that the improvements in accuracy, precision, recall, and AUC-ROC are consistent and reliable.
• Cohen’s d values for the performance differences quantify the magnitude of the improvements between the base paper and the proposed model. A Cohen’s d greater than 0.5 indicates a moderate effect, while values > 0.8 represent a dramatic effect.
• These results indicate that the observed improvements are not only statistically significant but also practically meaningful, particularly for accuracy and confidence calibration.

The observations suggest that the Transformer ensemble model outperforms the baseline models both individually and across performance measures such as accuracy, F1-Score, AUC-ROC, and MAE. Pair t-tests confirm that these improvements are significant, and Cohen’ d shows that they are striking. This also validates that ensemble learning with the proposed Transformer models results in a stronger and more accurate model for predicting election outcomes. Future research could concentrate on real-time implementation and further refinement of our model.

6.5. Baseline Temporal Models

Table 7. Comparison with Temporal deep learning baselines.

Model	Accuracy (%)	F1-Score	Precision	Recall	AUC-ROC
LSTM	83.2	0.8	0.81	0.79	0.89
GRU	84.1	0.81	0.82	0.8	0.9
Temporal CNN (TCN)	82.6	0.79	0.8	0.78	0.88
Transformer (Standalone)	86.8	0.84	0.86	0.82	0.93
Transformer Ensemble (Proposed)	93.4	0.88	0.9	0.85	0.94

presents the comparative performance of the proposed framework against standard temporal deep learning baselines, including LSTM, GRU, and Temporal CNN. While these models outperform traditional machine learning approaches by capturing sequential polling dynamics, their performance remains consistently below that of the Transformer-based models.

The Transformer demonstrates superior discrimination capability, particularly in terms of AUC-ROC, indicating more effective modeling of long-range dependencies across irregular polling intervals. The proposed Transformer ensemble framework further amplifies these gains, confirming the benefit of combining self-attention-based temporal representation learning with ensemble aggregation.

7. Limitations and Future Work

Despite the superior prediction performance of the new proposed Transformer ensemble, there are some limitations in terms of data dependency, model complexity, and generalizability across different electoral contexts. Resolving such limitations has several exciting future directions for strengthening robustness, interpretability, and real-world applicability.

7.1. Limitations

Despite the improved performance of the proposed Transformer ensemble framework, several limitations should be acknowledged. First, the model relies on the availability of sufficiently dense and structured polling data; its effectiveness may diminish in low-data or sparsely polled electoral contexts. Second, although the framework captures temporal dependencies, it does not explicitly model external shocks such as last-minute political events, which may abruptly influence voter behavior.

Third, the computational complexity of Transformer-based architectures increases training cost, potentially limiting applicability in resource-constrained environments. Finally, while ensemble learning improves robustness, the resulting model remains less interpretable than simpler statistical approaches, which may pose challenges for transparency in high-stakes political applications.

7.2. Future Work

Future research will focus on extending the proposed framework to real-time election forecasting by incorporating streaming polling updates and adaptive temporal windows. Additionally, integrating explainability techniques such as attention visualization and SHAP-based feature attribution may improve transparency and trust in model predictions. Another promising direction involves evaluating the generalizability of the framework across non-U.S. electoral systems and alternative political contexts. Finally, lightweight Transformer variants and model compression techniques may be explored to reduce computational overhead while preserving predictive performance.

8. Conclusions

Transformer ensemble shows consistent and statistically significant improvements in election outcome classification performance over both the single models and the base paper. The model offers more stable and robust predictions as its metrics, such as accuracy, F1-Score, precision, and AUC-ROC, are improved. The significance tests verify that the gained improvements are not incidental but rather statistically significant. Nevertheless, issues such as online learning, model interpretability, and computational complexity persist, necessitating future work to tackle the scalability and deployment challenges in a dynamic environment. In general, this study demonstrates the promise of using Transformer models and ensemble methods for high-stakes real-time applications (e.g., election forecasting), as well as beyond this.