GemSP: An Ensemble Model for User Story Point Estimation Using Gemini Embeddings

Abstract

Accurately estimating story points in Agile Scrum environments remains a challenging task, as traditional models often struggle to capture the complex relationships between user stories and their corresponding effort estimations. In this study, we leverage Gemini’s embedding representations to enhance the modeling of user stories within a story point estimation dataset. To improve prediction performance, we propose GemSP, an ensemble regression model that integrates two complementary regression techniques applied to the Gemini embeddings. Our approach aims to exploit the rich semantic representations of user stories while benefiting from the robustness of ensemble learning. Experimental results show that, when instantiated with Gemini embeddings, the proposed GemSP framework achieves lower prediction error than selected baseline models (GPT-2, Deep-SE, and GPT2SP) under cross-project evaluation on JIRA datasets. These results illustrate the practical benefit of decoupling semantic representation learning from regression, enabling effective integration of stronger embedding models within lightweight ensemble predictors.

1. Introduction

Estimating the size of a software project and the effort of its development is a crucial part of software project management, especially during the planning phase. In agile development, and more specifically in the Scrum framework [1], software development is organized into functional units called user stories which are implemented through time-boxed iterations known as sprints. Each sprint typically lasts between two and four weeks, during which the development team is expected to complete a selected set of user stories.

In fact, User Stories serve as the foundation for prioritizing tasks, estimating effort, and aligning development with user expectations and business objectives [2,3,4]. Consequently, unlike traditional approaches where the effort for the entire project must be estimated at the beginning, the focus of agile development is placed on estimating the effort required for each individual user story in the product backlog. In this context, story points are defined as a unit of measure mainly used to estimate the effort required for implementing a user story. Despite being handled by experienced team members, story point estimation remains challenging, time-consuming, and prone to imprecision and inconsistencies across different estimates.

Manually shifting through this data increases the chances of project delays, thereby impacting the success of the development effort. These challenges are addressed by automated user story point detection, which facilitates the efficient and accurate identification of user stories from textual data. This accelerates project planning and ensures that the team remains aligned with user expectations and needs [5]. Machine learning techniques [6,7,8] can serve as a promising alternative by exploiting historical project data to improve estimation accuracy and reduce human errors [9]. In this sense, several works have been presented. These later approaches rely on the extraction of features such as word length, ticket type, priority level, title, description, and subsequently predict the corresponding story point estimation. Early studies used traditional machine learning methods, employing handcrafted features for this purpose [8,10]. Deep learning approaches aim to improve prediction by automatically learning representations from raw text, using feature engineering [11,12]. Lastly, transformer-based models provide a deeper understanding of the global context of user stories and enhance generalization and transferability across projects [13]. In this context, large language models (LLMs), which are advanced transformer-based architectures trained on massive corpora, offer a promising approach to user–story detection, providing significant advantages over traditional methods. Motivated by this potential, we introduce GemSP ( Gemini-enhanced Story Pointing), an ensemble regression framework that leverages LLM-derived embeddings together with complementary regressors to deliver more accurate and robust story–point estimation. As presented in [14], large language models (LLMs)—such as BERT [14], Gemini, and the GPT family—can capture complex semantics and analyze the linguistic structure of text. These abilities make LLMs perfect for tasks requiring higher levels of contextual understanding, such as user story point detection [15]. The proposed approach does not attempt to define a universal scale for story points. Instead, it learns the relationship between the semantic content of user stories and their historically assigned story points within Agile projects. As such, the model captures relative estimation behavior rather than enforcing a fixed or objective effort scale. Nevertheless, works such as GPT2SP utilize LLMs for both the embedding and inference phases, which require significant resource consumption. In this sense, hybrid approaches have been introduced, using transformers exclusively for semantic embedding and then applying a structured regression model [16]. We propose GemSP, a hybrid method that integrates Gemini embeddings with lightweight ML regressors to produce more accurate and robust estimates. Accordingly, GemSP exploits Gemini embeddings to construct a robust feature representation, enabling more accurate and comprehensive user–story detection [17,18]. Furthermore, it allows us to better capture the semantics and contextual meaning of texts [7,19]. By combining Gemini embeddings with machine-learning models, our model GemSP achieves state-of-the-art performance in user-story detection. This integration improves both accuracy and efficiency. The combination of these two strengths allows us to enhance both the accuracy and efficiency of the detection process.

To present our analysis and insights more clearly, we address the following research questions: RQ1: How can Gemini embeddings improve the accuracy and reliability of user story point detection in Agile development? RQ2: What machine learning techniques best complement Gemini embeddings for robust story point estimation? RQ3: How does the proposed approach compare with traditional models in terms of efficiency, scalability, and generalization across different datasets?

Motivation: As Agile development scales, the manual estimation of user story points becomes increasingly inefficient, inconsistent, and prone to subjectivity. Traditional models, including rule-based and conventional machine learning approaches, struggle to accurately capture the variability and contextual nuances of user stories, leading to unreliable estimations across different projects. With the growing demand for automation in software development, leveraging Large Language Models (LLMs) like Gemini embeddings presents a transformative opportunity. These embeddings offer a deep semantic understanding of user stories, enabling more precise and consistent estimations. By integrating Gemini embeddings with advanced machine learning techniques, our approach aims to overcome the limitations of existing methods, providing a scalable and automated framework for user story point detection.

The main contributions of this paper can be summarized as follows:

1.

As far as we are aware, this study is one of the earliest to apply Gemini embeddings to story point detection.

2.

Our proposed method demonstrates superior performance compared to selected state-of-the-art baselines, including GPT-2, Deep-SE, and GPT2SP, under cross-project evaluation on JIRA datasets

3.

We also present the open research questions that guide our study and provide a detailed analysis of the proposed algorithm. The representations generated by Gemini embeddings can be precomputed and reused across multiple downstream tasks, including regression, classification, similarity search, clustering, ranking, and retrieval. This property enables efficient deployment of the proposed framework, as the embedding extraction step does not need to be repeated during model retraining or inference. Such general-purpose embeddings have been shown to transfer effectively across tasks and domains [20].

2. Literature Survey

Several studies have applied traditional machine learning techniques to story point estimation. In [21], the authors introduced a machine learning classifier specifically designed for issue reports, aimed at accurately estimating the number of story points required to address each issue. Their research was based on a dataset comprising 5607 filtered issues from eight open-source projects and one industrial project. By utilizing the Term Frequency-Inverse Document Frequency (TF-IDF) method to extract crucial textual features, the team compared various techniques, including Support Vector Machine (SVM), k-Nearest Neighbours (KNN), Decision Trees, and Naive Bayes. The results demonstrated that SVM models offer significant accuracy. In [10], the authors proposed neural network models using a dataset of 21 projects, including story points and velocity as input variables. They selected three machine learning techniques—Adaptive Neuro-Fuzzy Modeling, Generalized Regression Neural Networks, and Radial Basis Function Networks—to predict software effort, aiming to adapt estimates based on project dynamics. In [8], the authors focused on estimating effort for Agile Scrum projects using regression-based techniques like Support Vector Machine (SVM), Random Forest(RF), Multi-Layer Perceptron (MLP), and Gradient Boosting Machines (GBM). They estimated effort at the user story level and aggregated it for iterations, phases, and projects, demonstrating that the GBM model, applied to a MongoDB dataset achieved an accuracy rate of 99.8%. In [22], the authors presented a method that combines story points with the k-Nearest Neighbors (KNN) algorithm to estimate total effort and completion time.

The approaches based on traditional machine learning models, rely on handcrafted features whether through vector representations such as TF-IDF or using manually extracted attributes. However, the identification of relevant variables remains largely human-driven process, time-consuming and requires deep domain knowledge. Additionally, these models struggle with low transferability to different contexts and often fail to capture complex semantic relationships in natural language, which can affect estimation quality.

These challenges have led researchers to turn to Deep Learning models in order to automatically generate richer and more abstract representations of user stories.In this context, the Deep-SE model [12] is proposed to predict story points, automatically extracting features from JIRA; specifically the title and description of user stories. The model converts this information into vector representations using natural language processing techniques and analyzes them with LSTM (Long Short-Term Memory) and RHWN (Recurrent Highway Network) architectures to capture issue semantics. Deep-SE was evaluated on 23,313 issues from 16 open-source projects and reported a mean absolute error (MAE) of 2.08, showing improved performance relative to several traditional machine learning baselines.

According to [13], the Deep-SE model has major limitations. It is trained from scratch for each project based on specific user stories, taking 2 to 7 h to build a pre-trained model, which makes scaling difficult. Deep-SE does not use generic language models and its vocabulary is unique to each project which limits its transferability. The model also relies on unidirectional LSTMs that read text from left to right, restricting its ability to capture global dependencies and rich meaning, and this architecture is not easy to interpret, making it hard to understand predictions. Zhang et al. [23] emphasize the role of human-AI collaboration in bridging gaps between incomplete user requirements and software generation. Their AgileGen framework highlights how acceptance criteria, expressed in Gherkin language, can align user stories with semantic consistency in generated code, which directly influences story point detection reliability. Complementing this perspective, Almalki [24] explores AI-driven decision support systems in Agile project management, showing how intelligent risk mitigation and resource allocation can strengthen the accuracy of effort estimation processes. A broader overview is offered by dos Santos et al. [3], who provide a systematic literature review on automatic user story generation. Their findings underscore the evolving research trend of enhancing user story quality and point estimation through machine learning and natural language processing techniques. Parallel to this, Islam and Sandborn [25] propose a multimodal generative AI model for story point estimation, showing that integrating textual and contextual features can yield improved performance compared to several traditional regression-based approaches.

In terms of model innovation, Younas et al. [26] introduce SPERT, a reinforcement learning-enhanced transformer for Agile story point estimation. Their study illustrates how adaptive learning mechanisms can refine estimation accuracy compared to static deep learning baselines. This line of work aligns with dos Santos’ doctoral research [27], which focuses on leveraging text generation to improve user story quality, reinforcing the link between enhanced requirement articulation and precise story point prediction. Further advancements in tool support have been presented by Hallmann et al. [28], who developed USeR, a web-based user story reviewer designed to assist quality optimization. By embedding AI-driven feedback loops, USeR supports practitioners in refining user stories before point estimation, thereby reducing ambiguity and inconsistency in the process.

In [29], the authors propose a model combining Bidirectional Long Short-Term Memory (BiLSTM) with Recurrent Neural Network (RNN) and Convolutional Neural Network (CNN) architectures to enhance feature representation for story point estimation. The main advantage of this approach is the use of bi-directional encoding, which allows the model to capture dependencies from both directions (forward and backward) in the text, ultimately improving the accuracy of story point estimation. However, similar to Deep-SE, it still faces challenges regarding transferability and interpretability, which can limit its application across different contexts and obscure the reasoning behind the predictions made by the model.

In contrast, another approach uses Graph Neural Networks (GNN) [30] for estimating story points. TextLevelGNN represents user stories as graphs to be able to learn information using graph data structure. The aim of the proposed mode is to classify story points into four levels, capturing relationships between words. Initial results show accuracy comparable to TF–IDF with Random Forest. However, there are limitations: because story point estimation is fundamentally a regression problem, a classification formulation restricts the granularity of the predictions.

Faced with the limitations of traditional Deep Learning approaches, GPT2SP [13] is proposed as an innovative method for estimating story points based on Transformers, and especially on the pre-trained GPT-2 model. The primary contribution of GPT2SP is its use of transformers to capture word-level semantic relationships, allowing the pretrained model to be fine-tunied for new projects without complete retraining each time. Lastly, it assures interpretability by using self attention mechanism [31]. Still, the solution based on transformers for both embedding and prediction is computationally expensive. Indeed, it demands a high use of memory and high computational power. In addition, despite using pre-trained model, both training and inference phases remain expensive.

Unlike GPT2SP, which fine-tunes and utilizes the full GPT-2 architecture end-to-end for regression tasks, this approach [16] decouples semantic understanding from prediction. It proposes a method that integrates Deep Learning, Machine Learning, and natural language processing techniques. Specifically, it employs SBERT (Sentence-BERT) for understanding the context of user stories and Gradient Boosted Trees for predictive modeling. Notably, this approach uses Transformers solely for semantic embedding and then applies a structured regression model (LightGBM), thereby avoiding reliance on resource-intensive, end-to-end Transformer-based architectures. This hybrid design achieves both high accuracy and practical scalability.

3. Proposed Framework

In this section, we present the formal methodology used to extract and process data from user stories and subsequently apply a predictive ensemble learning model as shown in Figure 1. The framework consists of two primary steps, outlined as follows.

3.1. Step 1: Extracting Gemini Embeddings from User Stories

Let the user stories be represented as a set of vectors $\mathbf{S}=\{{\mathbf{s}}_1,{\mathbf{s}}_2,\dots ,{\mathbf{s}}_n\}$, where each ${\mathbf{s}}_i$ corresponds to the i-th user story in the corpus. The first step is to extract the Gemini embeddings ${\mathbf{G}}_i$ for each user story ${\mathbf{s}}_i$. This is accomplished using a transformation function $\mathcal{T}$, defined as

$${\mathbf{G}}_i=\mathcal{T}\left({\mathbf{s}}_i\right)\forall i\in \{1,2,\dots ,n\}$$

(1)

where $\mathcal{T}:{\mathbb{R}}^d\to {\mathbb{R}}^m$ is a mapping from the original feature space of the user story to a higher-dimensional embedding space ${\mathbb{R}}^m$.

The resulting Gemini embeddings ${\mathbf{G}}_i$ are then normalized via a transformation function $\mathcal{N}$ to ensure that the embeddings exhibit consistent scale and variance across the corpus:

$${\mathbf{G}}_i^{\mathrm{norm}}=\mathcal{N}\left({\mathbf{G}}_i\right)\forall i\in \{1,2,\dots ,n\}$$

(2)

where $\mathcal{N}:{\mathbb{R}}^m\to {\mathbb{R}}^m$ performs standard normalization (e.g., z-score normalization) on the embeddings.

3.2. Step 2: Predictive Ensemble Learning Model

After the embeddings have been extracted and normalized, the second step involves applying an ensemble learning model $\mathcal{E}$ though voting regressor, to make predictions based on the transformed user stories. The ensemble model combines predictions from multiple base learners ${\mathcal{L}}_j$, $j\in \{1,2,\dots ,k\}$, and produces a final prediction ${\widehat{y}}_i$ for each user story ${\mathbf{s}}_i$:

$${\widehat{y}}_i=\mathcal{E}\left({\left\{{\mathcal{L}}_j\left({\mathbf{G}}_i^{\mathrm{norm}}\right)\right\}}_{j=1}^k\right)$$

(3)

where ${\mathcal{L}}_j$ represents the j-th base learner and $\mathcal{E}$ is the ensemble function that aggregates the individual predictions. For instance, the ensemble function could take the form of a weighted average:

$${\widehat{y}}_i=\sum _{j=1}^k{w}_j·{\mathcal{L}}_j\left({\mathbf{G}}_i^{\mathrm{norm}}\right)\mathrm{where}\sum _{j=1}^k{w}_j=1,w_j\ge 0$$

(4)

Here, $w_j$ represents the weight of the j-th base learner’s prediction, determined via cross-validation or other optimization methods.

3.3. Final Prediction

After the model is trained, the final prediction for a new user story ${\mathbf{s}}_{\mathrm{new}}$ is computed as

$${\widehat{y}}_{\mathrm{new}}=\mathcal{E}\left({\left\{{\mathcal{L}}_j\left({\mathbf{G}}_{\mathrm{new}}^{\mathrm{norm}}\right)\right\}}_{j=1}^k\right)$$

(5)

where ${\mathbf{G}}_{\mathrm{new}}^{\mathrm{norm}}=\mathcal{N}\left(\mathcal{T}\left({\mathbf{s}}_{\mathrm{new}}\right)\right)$ represents the normalized Gemini embedding for the new user story ${\mathbf{s}}_{\mathrm{new}}$. $\left\{{\mathcal{L}}_j\right\}$ and the ensemble function $\mathcal{E}$ are optimized through gradient-based methods or other optimization techniques to minimize the overall loss.

Gemini is used solely to generate deterministic semantic embeddings of user story text. All numerical story point predictions are produced by traditional regression models, thereby avoiding hallucination or nondeterministic outputs associated with generative LLMs.

4. Experimental Setup

4.1. Studied Datasets

To ensure a fair comparison between the GPT2SP model and the state-of-the-art Deep-SE approach, we utilize the same benchmark datasets as those used by Choetkiertikul et al. [12]. These datasets were collected from JIRA, a widely-used issue tracking system for Agile software development, including story point estimation via the JIRA (Atlassian, Sydney, Australia; https://www.atlassian.com/software/jira, accessed on 5 January 2026). Agile plugin. Choetkiertikul et al. collected the dataset by extracting key information—such as issue ID, title, descriptions, and story points—through the JIRA REST API, upto 8 August 2016. The statistical details of the datasets are provided in

Table 1. Comparative analysis of story point estimation approaches.

Approach	Text Representation	Prediction Model	Cross-Project	Handling of Subjectivity	Key Limitations
TF-IDF + ML	Handcrafted features (TF-IDF, metadata)	Classical ML regressors (RF, SVM)	No	None	Limited semantic understanding; weak generalization
Deep-SE	Learned word embeddings per project	LSTM + RHWN	No	Implicit (project-specific)	Retraining required; high computational cost
BiLSTM/CNN-based	Learned sequence representations	Deep neural networks	Limited	Implicit	Limited transferability; low interpretability
TextLevelGNN	Graph-based word representations	GNN classifier	No	None	Classification-based formulation; coarse granularity
GPT2SP	GPT-2 (small) embeddings	Transformer-based regressor	Limited	Implicit	Computationally expensive; fixed foundation model
SBERT + LightGBM	Sentence-level embeddings (SBERT)	Gradient-boosted trees	Yes	Partial (normalization)	Embedding quality bounded by SBERT
GemSP (Ours)	Gemini embeddings	Ensemble regression (RF + XGBoost)	Yes	Explicit (per-project normalization)	Interpretability of ensemble models

. Our experiments were conducted using a total of 23,313 issues across 16 different projects.

In this work, no explicit dependency or relational structure between software tasks or user stories is assumed. Each user story is treated as an independent textual unit, and story point estimation is performed solely based on its semantic content and historical story point assignments. It is important to emphasize that GPT2SP relies on GPT-2, a smaller 2019-era transformer intentionally chosen for computational feasibility at the time of its proposal. In contrast, GemSP employs Gemini embeddings derived from a more recent foundation model. Consequently, our goal is not to isolate architectural superiority over GPT2SP, but rather to assess the effectiveness of a decoupled embedding–regression design under contemporary representation learning capabilities.

4.2. Model Implementation

In order to develop our model, we leveraged the Gemini API provided by Google Cloud. This API enables us to generate high-quality embeddings, which serve as dense vector representations of data. These embeddings capture meaningful semantic relationships and contextual information, making them highly effective for various machine learning tasks. Once the embeddings are extracted, they are passed as input features to machine learning models for further processing and analysis. These models are implemented using the Scikit-learn library (version 1.2.2), a popular open-source toolkit in Python (version 3.9)

Known for its efficient and user-friendly machine learning algorithms. In this context we propose an improved algorithm by combining Google Cloud’s Gemini API for embedding generation with Scikit-learn’s ML models. Algorithm 1 outlines the complete GemSP workflow, including embedding generation, dimensionality reduction, ensemble regression, and final prediction.

Algorithm 1: GemSP Algorithm

Input: $X_{\mathrm{text}}$ represents User Story Text, where $X_{\mathrm{text}}\in {\mathbb{R}}^{N\times D}$, N is the number of             user stories, and D is the dimensionality of text features Output: Predicted User Story Points $\widehat{Y}\in {\mathbb{R}}^{N\times 1}$ Step 1: Gemini Embeddings Generation  $i=1$ to N$\phantom{(}$      Gemini API to generate text embeddings for user story $x_i$      ${\mathbf{e}}_i=\mathrm{GeminiAPI}\left(x_i\right)$ Step 2: Apply PCA for Dimensionality Reduction $${\mathbf{e}}_i^{\mathrm{pca}}=\mathrm{PCA}\left({\mathbf{e}}_i\right),\mathrm{where}\mathrm{PCA}\mathrm{reduces}\mathrm{the}\mathrm{embedding}\mathrm{to}{\mathbb{R}}^K$$       Apply PCA to the embeddings to reduce dimensionality to K. Step 3: Ensemble Random Forest and XGBoost Regressors $i=1$ to N$\phantom{(}$       Train Random Forest Regressor on ${\mathbf{e}}_i^{\mathrm{pca}}$: $${\widehat{y}}_i^{\mathrm{rf}}=\mathrm{RFRegressor}\left({\mathbf{e}}_i^{\mathrm{pca}}\right)$$        Train XGBoost Regressor on ${\mathbf{e}}_i^{\mathrm{pca}}$: $${\widehat{y}}_i^{\mathrm{xgb}}=\mathrm{XGBRegressor}\left({\mathbf{e}}_i^{\mathrm{pca}}\right)$$ Step 4: Ensembling Combine the predictions of RF and XGB regressors through   voting regressor for final prediction: $${\widehat{y}}_i=\alpha ·{\widehat{y}}_i^{\mathrm{rf}}+(1-\alpha )·{\widehat{y}}_i^{\mathrm{xgb}},\mathrm{where}\alpha \in [0,1]$$ $\widehat{Y}$, the predicted user story points for each input user story text.

4.3. Hyperparameters

The performance of the ensemble regression model is influenced by carefully selected hyperparameters for the Gradient Boosting Regressor (GBR) and Random Forest Regressor (RFR). For GBR, 300 estimators were used to enable learning over multiple boosting stages, while a learning rate of 0.05 ensures gradual updates to prevent overfitting. The maximum tree depth was limited to 5, balancing complexity and generalization, and a random state of 2 was set to ensure reproducibility. Similarly, the RFR model is configured with 300 estimators for stability, a maximum depth of 10 to prevent overfitting, and a minimum of 5 samples per split, ensuring meaningful node divisions. The random state of 42 is maintained for consistency across runs. We have implemented Principal Component Analysis (PCA) with 100 components to reduce the dimensionality of the embeddings while preserving key information. The data is also normalized using StandardScaler, ensuring that all features have a mean of zero and a unit variance. The dataset is split into 80% training and 20% testing, maintaining a random state of 42 for reproducibility. Finally, a Voting Regressor combines GBR and RFR, leveraging their complementary strengths to enhance accurate prediction and accuracy by reducing individual model biases.

4.4. Metrics

We evaluate model performance using both Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) to assess average deviation and sensitivity to larger errors. MAE is a measure that quantifies the average magnitude of errors in a set of predictions. It provides an understanding of prediction accuracy by directly representing the deviation between predicted and actual values. Additionally, using MAE enables a direct comparison with findings from other studies in the literature, facilitating a meaningful assessment and comparison of performance relative to existing research. In addition to MAE, we report RMSE to capture sensitivity to larger errors. Statistical significance between models is assessed using a paired Wilcoxon signed-rank test at the 0.05 level.

4.5. Handling the Subjectivity of Story Points

Story points are inherently subjective and team-dependent, reflecting relative effort rather than absolute development cost. Consequently, this work does not aim to predict true effort, but rather to learn historical estimation behavior as practiced in Agile projects. To mitigate scale inconsistencies across projects, we apply per-project normalization to story point values before model training. This normalization ensures that estimation patterns can be learned across projects without conflating incompatible scales. In addition to being team-dependent, story points are also time-dependent. The same developer may assign different values to similar user stories at different points in their career as estimation skill and domain familiarity evolve. Moreover, Agile methods do not prescribe a universal scale for assigning story points; rather, teams establish their own internal heuristics. These two factors introduce inconsistency and noise into datasets, as identical or comparable user stories may receive different story point assignments across projects and over time. To mitigate these effects, we apply a per-project z-score normalization of story point labels prior to model training. This transforms each project’s story points into a relative scale centred around that project’s historical mean. As a result, the model learns relationships between textual semantics and relative estimation patterns rather than absolute numeric values. While normalization cannot completely remove temporal subjectivity, it substantially reduces inter-project scale distortion and allows the model to generalize estimation behaviour more effectively across heterogeneous teams.

5. Experimental Results

In this study, we aim to evaluate and compare the performance of different machine learning models in predicting user story points. The models under consideration include GPT-2, Deep-SE, GPT2SP, and our proposed model GemSP. The results of our experiments are reported in

Table 2. The Descriptive Statistics of the Story Point Datasets Provided by Choetkiertikul et al. [12].

Repository	Project	#Issues	${\mathbf{SP}}_{\mathbf{min}}$	${\mathbf{SP}}_{\mathbf{max}}$	${\mathbf{SP}}_{\mathbf{mean}}$	${\mathbf{SP}}_{\mathbf{median}}$	${\mathbf{SP}}_{\mathbf{var}}$	${\mathbf{SP}}_{\mathbf{std}}$
Apache	Mesos	1680	1	40	3.09	3	5.87	2.42
	Usergrid	482	1	8	2.85	3	1.97	1.4
Appcelerator	Appcelerator Studio	2919	1	40	5.64	5	11.07	3.33
	Aptana Studio	829	1	34	8.02	8	25.97	5.1
	Titanium SDK/CLI	2251	1	34	6.32	5	25.97	5.1
Dura Space	DuraCloud	666	1	6	3.12	3	4.12	2.03
Atlassian	Bamboo	521	1	20	4.2	2	4.6	2.14
	Clover	384	1	49	3.59	1	42.95	6.55
	JIRA Software	352	1	20	4.43	3	12.35	3.51
Moodle	Moodle	1166	1	155	15.34	5	468.53	21.66
Lsstcorp	Data Management	4667	1	100	9.57	4	275.71	16.61
Mulesoft	Mule	889	1	30	5.08	5	12.24	3.5
	Mule Studio	732	1	34	6.4	5	19.94	4.46
Spring	Spring XD	3526	1	3.7	3.7	3	10.42	3.23
Talendforge	Talend Data Quality	1381	1	40	5.92	5	26.96	5.19
	Talend ESB	868	1	13	2.16	2	2.24	1.5
	Total	23,313	-	-	-	-	-	-

, which presents the Mean Absolute Error (MAE) for each model, alongside the improvements over baseline models.

The results indicate that all models exhibit a noticeable reduction in MAE when compared to the baseline GPT-2 model. The GPT-2 model, which serves as the initial benchmark, achieved an MAE of 5.306. This result reflects the challenges in accurately predicting story points using a generic transformer-based model. While GPT-2 demonstrates reasonable performance, the reported improvement margin of 3.326 indicates scope for further refinement, motivating the exploration of more task-adapted modeling approaches. The Deep-SE model, a deep learning-based approach, achieved a lower MAE of 3.5, with an improvement of 1.52 over GPT-2. This result highlights the benefits of using deep learning techniques tailored to the task of story point prediction. The Deep-SE model has the ability to integrate deeper layers and potentially contextualize. The relationships between story points and other project attributes seem to improve the performance compared to the baseline, though still not at the level of the most refined models.

The GPT2SP model, a variant specifically adapted to predict story points, demonstrated a further improvement, achieving an MAE of 2.14. The improvement here, though smaller in magnitude (0.16), still reflects a step forward in adapting GPT-2 for this specialized task. The relatively minor improvement compared to Deep-SE suggests that GPT2SP benefits from domain adaptation. However, it still has limitations in capturing the intricate patterns associated with story point estimation. Finally, the GemSP framework achieved the lowest mean absolute error (MAE) of 1.98 an RMSE of 2.22, indicating both low average and large-error sensitivity. Relative to GPT-2 and Deep-SE, this corresponds to reported MAE reductions of 3.326 and 1.52, respectively. This result demonstrates the effectiveness of our proposed approach, which integrates additional contextual and domain-specific features to refine the prediction process. The consistent reduction in MAE across models indicates that GemSP more effectively captures the complexity of story point estimation, offering a more reliable and efficient method for predicting user story points. To address variability across projects, we applied per-project normalization to story point labels before training. This approach significantly reduced variance in prediction errors and improved generalization across datasets. Without normalization, MAE increased by up to 25% on small projects, confirming its importance for stable cross-project estimation. Figure 2 presents the scatter plot of the prediction results: true values versus predicted values. Figure 3 presents a comprehensive evaluation of the regression model using six complementary diagnostic visualizations. The true-versus-predicted scatter plot indicates a moderate linear correspondence between ground-truth and predicted values, achieving an $R^2$ score of $0.228$ with a root mean square error (RMSE) of $0.025$, demonstrating that the model captures a meaningful portion of the underlying trend while maintaining low absolute prediction error. The residuals histogram exhibits an approximately symmetric, near-Gaussian distribution centered around zero, suggesting the absence of strong systematic bias in the prediction errors. This behavior is further confirmed by the residuals-versus-predicted plot, where residuals remain evenly scattered around the zero reference line without visible heteroscedastic patterns, indicating stable variance across the prediction range.

The two-dimensional density heatmap of binned true–predicted pairs reveals a concentrated mass along the identity line, confirming that the majority of predictions remain closely aligned with their corresponding ground-truth values despite moderate dispersion. The calibration curve further shows close agreement between the mean predicted and mean true values across bins, indicating satisfactory calibration and minimal expectation mismatch. Finally, the principal component analysis (PCA) variance plot demonstrates that the first four components explain nearly all of the variance in the feature space, with the first two components contributing the dominant share, confirming that the learned representation is effectively captured within a low-dimensional subspace. Collectively, these results indicate that while the model exhibits moderate explanatory power, it maintains strong calibration, stable error characteristics, and efficient feature representation (

Table 3. Performance comparison across models using MAE and RMSE (↓ indicates that lower values are better).

Model	MAE ↓	RMSE ↓
GPT-2	5.306	5.83
Deep-SE	3.50	3.93
GPT2SP	2.14	2.45
GemSP (Ours)	1.98	2.29

).

6. Discussion

6.1. Interpretation of Cross-Project Evaluation

Cross-project evaluation is used to assess whether semantic representations learned from user stories generalize across projects, rather than to imply absolute comparability of story point values. Since story points are inherently team-specific, cross-project results should be interpreted as measuring relative estimation transferability, not universal effort prediction. Per-project normalization is applied to reduce scale bias, but we emphasize that real-world adoption remains project-specific. Normalization ensures comparability by aligning effort scales across projects, helping the model learn relative rather than absolute effort patterns. This directly improves performance consistency and was crucial to GemSP’s robustness. It is therefore important to interpret our results as modelling estimation practice rather than objective development effort. In practical use, teams may fine-tune the model on their own backlog history so that predictions align with their internal story-point culture. The model is best viewed as an assistive estimation aid that supports planning discussions rather than replacing Agile team judgement.

6.2. RQ1

How Does the Proposed Approach Using Gemini Embeddings Improve the Accuracy of User Story Point Classification Compared to Existing Models Like GPT2SP?

Unlike GPT2SP, which primarily relies on transformer-based encoding, our proposed model GemSP leverages Gemini embeddings. These embeddings yield richer, more context-aware representations of user stories, improving accuracy. Additionally, by applying regression models to the embeddings, the approach allows for a more fine-grained mapping of user stories to story points, reducing overfitting tendencies often seen in deep learning models. This structured pipeline facilitates improved generalization across different software projects and yields lower prediction error than GPT2SP under the evaluated experimental setting.

6.3. RQ2

Which Features of the Gemini Embeddings Contribute Most to the Improved Performance in the Classification Task?

The improved performance in the classification task can be due to several key features of Gemini embeddings. One of the primary factors is the improved contextual semantic representation, which enables the model to understand the meaning of user stories beyond mere keyword matching. By capturing deeper semantic relationships between words and phrase, the embeddings enable the classification model to distinguish subtle differences in user story descriptions, leading to more accurate predictions. Additionally, the high dimensional nature of Gemini embeddings encodes both syntactic and semantic properties, allowing better understanding of user story variations. The adaptability of Gemini embeddings across different software development domains further strengthens their contribution, ensuring that the model remains effective on diverse datasets.

6.4. RQ3

Can the Proposed Model Be Effectively Scaled for Large-Scale Software Projects with Varying Complexity Levels?

GemSP is designed to be highly scalable, making it suitable for large-scale software projects with varying complexity levels. The integration of embeddings ensures that the entire model does not need to be retrained as the dataset grows, since we only extract the embeddings from the Gemini model. This significantly reduces computational overhead compared to Deep Learning models that require complete retraining when new data is introduced. Unlike traditional classification methods that struggle with increasing dataset size, our approach efficiently adapts to evolving datasets by leveraging regression rather than relying on static training models. Additionally, the computational efficiency of regression models compared to Deep Neural Networks makes our method more practical for real-time applications, such as Agile project management tools, where user stories are frequently updated and classified. The Gemini being LLM can input more tokens compared to other models such as BERT and other transformers, this helps to handle the detection of very large user stories as well.

6.5. RQ4

What Are the Advantages of GemSP over Other Algorithms? The following outlines the advantages of the proposed algorithm, GemSP, over traditional methods, emphasizing scalability, accuracy, the pre-trained nature of Gemini embeddings, and efficiency in processing large datasets, as shown in Figure 4.

1.

Scalability: GemSP demonstrates a high degree of scalability, making it particularly suitable for large-scale software projects of varying complexity. Its embedding generation mechanism allows for processing of relevant user stories, eliminating the need for full retraining of models. Indeed as the dataset size increases, we only extract only Gemini representations thereby avoiding the need to retrain the Gemini model. This modular architecture, coupled with efficient regression models, ensures that GemSP can manage increasing data volumes without much computational costs.

2.

Gemini is LLM: The Gemini embeddings, pre-trained on a large-scale corpus, endow GemSP with exceptional contextual and semantic comprehension of textual inputs. This pre-training enables the model to capture deep meanings, subtle interrelationships, and linguistic nuances in user stories, providing a more robust and contextually aware representation compared to traditional feature extraction techniques.

3.

Efficiency in Processing Large Volumes of Data Advantage: GemSP excels in efficiently processing large volumes of textual data, benefiting from the combination of Gemini embeddings and optimized regression models implemented using tools such as Scikit-learn. This automated pipeline accelerates the user story point estimation process, making it ideal for fast-paced development environments where rapid and reliable analysis is crucial.

4.

Accuracy: GemSP achieves superior accuracy in user story point estimation through the integration of Gemini embeddings and regression approaches, further optimized by regression models. Empirical results indicate a 5–7% improvement in performance (0.17 score increase) over models such as GPT2SP, with validation from human evaluators showing a high degree of agreement with expert-labeled story points. The observed performance gains should therefore be interpreted as a combination of improved semantic representations and robust ensemble regression, rather than as evidence of inherent superiority of the regression architecture alone.

7. Limitations

Interpretability remains a significant challenge when applying the GemSP algorithm, particularly because it utilizes ensemble models such as Random Forest and XGBoost in conjunction with complex embeddings like Gemini. These models are inherently non-linear and involve numerous decision trees or boosting rounds, making it difficult to trace how individual features in the user story contributed to the final prediction of story points. This lack of transparency can be problematic, particularly in environments where understanding the reasoning behind a prediction is critical for stakeholders, including project managers, product owners, and team members. In Agile environments, where continuous feedback and trust are crucial, the inability to explain why a certain story point was assigned can lead to skepticism and resistance toward the model. Teams may question the accuracy of the estimations if they do not understand the reasoning behind them. Moreover, the reliance on high-dimensional embeddings derived from complex models such as Gemini further complicates the interpretability, as it is challenging to map from the embedding space back to the original textual content of the user stories in a clear and understandable manner. Without proper interpretability, users of the GemSP algorithm may struggle to trust the system, which can hinder its effective adoption and integration into the Agile estimation process. Due to the use of ensemble models such as Random Forest and XGBoost operating on high-dimensional Gemini embeddings, interpretability remains a challenge. Tracing the contribution of individual user story features to final story point predictions is non-trivial, as the models involve multiple decision paths and non-linear interactions. Improving transparency through explainable AI techniques constitutes an important direction for future work. The datasets originate from multiple open-source projects, reflecting heterogeneous estimation practices. While this introduces variability, it enables evaluation of the model’s ability to generalize estimation behavior across diverse project contexts. The model predicts assigned story points rather than verified development effort. Any discrepancy between story points and actual effort is inherited from the dataset and represents a limitation shared by all data-driven story point estimation approaches. The combination of ensemble regression models and high-dimensional embeddings introduces interpretability challenges, which may affect practitioner trust. Future work will explore explainable AI techniques to provide transparent justifications for predictions.

8. Conclusions and Future Work

In conclusion, this paper introduces a novel algorithm that leverages Gemini, a powerful new framework that has shown remarkable performance improvements compared to existing models in the field of user story point detection. One of the key highlights of the proposed algorithm is its performance, as it achieved a Mean Absolute Error (MAE) score decrease of 0.17, which translates to a 5–7% improvement over GPT2SP, the leading model in this domain. This indicates a significant reduction in prediction error, showcasing the potential of Gemini to more accurately predict user story points. Moreover, the paper addresses several critical research questions that enhance our understanding of the algorithm’s abilities and limitations. The findings indicate that the model is scalable and capable of handling large datasets and a variety of tasks with minimal computational overhead. This makes it suitable for diverse environments, including real-time applications. Despite the inherent subjectivity of story point estimation, our results show that meaningful predictive structure exists once project-level normalization is applied. This suggests that empirical estimation support tools such as GemSP can complement Agile practice while still respecting the locally defined nature of story points.

For future work, we plan to improve the explainability of our algorithm, particularly in the context of user story point detection. Although our current model performs well, incorporating explainable AI techniques will provide transparency in the decision-making process, enabling end-users to better understand how the model arrives at its predictions. This is crucial for trust and ensuring that the model’s outputs are interpretable. Moreover, we aim to explore with advanced techniques such as attention mechanisms and transformer-based models to further improve both the accuracy and the interpretability of the algorithm, making it more accessible for real-world applications. Furthermore, we intend to extend the model’s applicability to different domains, languages, and datasets. Adapting the algorithm to handle specialized terminology and multilingual data will help increase its versatility in diverse software development environments.