Credit Default Prediction Using Large Language Models and Machine Learning: An Application to Colombia’s Solidarity Sector

Abstract

Credit default prediction is a standard risk-management task, and large language models (LLMs) have been proposed as prompt-based alternatives, without task-specific parameter updating, for institutions that cannot deploy full machine learning (ML) pipelines. This study evaluates the Informed GPT on Colombian solidarity-sector cooperative lending data, benchmarking gpt-4o-mini against five tuned tree ensemble and gradient boosting classifiers on native imbalanced data (17.3% default rate, 12,861 loans). Six additions relative to the seminal reference are reported: (i) a new empirical domain (Colombian solidarity-sector cooperatives regulated by the SES); (ii) a multi-model benchmark rather than a logistic-regression-only baseline; (iii) a leakage-mitigation prompt design that excludes supervised-analysis-derived hints, causal directions, and target-distribution disclosures; (iv) a calibration analysis using Brier score, log loss, expected calibration error (ECE), reliability diagrams, and calibration slope and intercept; (v) bootstrap 95% confidence intervals, DeLong tests, and McNemar tests for paired significance; and (vi) matched label-budget learning curves for logistic regression, XGBoost, and LightGBM. Tuned ML models attain AUC $\approx 0.96$ (bootstrap CI $[0.94,0.98]$), while the LLM operates in the AUC 0.67–0.74 range across few-shot sizes $N\in \{0,10,20,40,80\}$. Under matched budgets, the LLM outperforms logistic regression at every N but is surpassed by gradient boosting once training samples reach approximately 40–80 observations. LLM probabilities are miscalibrated (ECE 0.11–0.21 vs. ≈0.03 for ML) and over-predict default (mean predicted probability 0.28–0.38 vs. observed base rate 0.17); threshold optimisation and post hoc calibration (Platt scaling, isotonic regression) are required for operational use. The findings qualify earlier claims about LLM auditability and position the approach as an assessment tool for cooperatives with fewer than ≈100 labelled defaults, rather than as a substitute for a well-resourced ML pipeline.

1. Introduction

Credit risk is a central concern of financial institutions, since borrower delinquency directly affects portfolio quality, liquidity, and lender solvency (Altman, 1968; Basel Committee on Banking Supervision, 1999). Default prediction has been studied extensively in commercial banking, but the solidarity sector, comprising savings and credit cooperatives and microfinance institutions, has received comparatively less attention in the credit-risk literature, despite its economic and social role in emerging economies, where cooperative lending channels resources to households and small enterprises that are underserved by the commercial banking system (Arias-Serna et al., 2025; Lemus & Rojas, 2022).

Risk assessment in the sector is complicated by a specific data configuration that the classical literature does not directly address. Traditional credit-risk assessment relies on expert judgement, financial ratios, and parametric scoring models such as logistic regression and discriminant analysis (Altman, 1968; Hosmer et al., 2013). These methods are interpretable and well established in regulatory practice, but they assume linear relationships and homogeneous populations, which limits their effectiveness on heterogeneous customer bases that include informal workers, customers with limited banking history, and clients for whom relational and sociodemographic information, rather than the formal financial record, is the more informative signal of repayment behaviour. Cooperatives in the solidarity sector present this data configuration: rich relational and sociodemographic information, but scarce formal credit history.

Machine learning (ML) has become the prevailing approach in modern credit scoring (Cavalcante et al., 2016). Benchmark studies report that nonlinear classifiers, particularly gradient boosting ensembles (XGBoost, LightGBM) and random forest, outperform classical statistical models in default prediction, with gains in AUC, Gini coefficient, and minority-class recall (Baesens et al., 2003; I. Brown & Mues, 2012; Chen & Guestrin, 2016; Ke et al., 2017; Lessmann et al., 2015). Recent work in emerging-market contexts extends these results beyond developed economies (Arias-Serna et al., 2025; Machado & Karray, 2022; Ziemba et al., 2021).

The adoption of ML in smaller solidarity institutions is nonetheless uneven. The deployment of these classifiers requires labelled datasets of sufficient size, dedicated ML expertise, training and serving infrastructure, and explainability frameworks compatible with regulatory scrutiny (Giudici et al., 2024; Giudici & Raffinetti, 2023; Lundberg & Lee, 2017). For cooperatives with limited technical staff and modest data volumes, these requirements are a practical barrier. Lenders whose customer base is most heterogeneous, and who would benefit most from richer modelling approaches, are therefore also the least equipped to adopt them.

Large language models (LLMs) such as GPT-4 (T. Brown et al., 2020; OpenAI, 2023) offer an alternative operating point for resource-constrained institutions. Trained on large text corpora through self-supervised learning, LLMs exhibit few-shot learning: they can perform new classification tasks by conditioning on a small number of labelled examples supplied in the prompt, without weight updates or a training pipeline (Kasneci et al., 2023; Vaswani et al., 2017). This mode of use has been formalised for credit lending in the Informed GPT framework (Babaei & Giudici, 2024), in which the LLM receives a stratified sample of labelled clients alongside the case to be classified. Applied to a European student-loan dataset, Informed GPT reaches AUC values comparable to logistic regression and produces per-prediction natural-language rationales (Deldjoo, 2023; Feng et al., 2023). Concerns about algorithmic fairness and bias in LLM-based credit decisions have also been raised (Y. Li & Zhang, 2023).

Despite these methodological proposals, the current literature on LLMs for credit-risk assessment presents analytical gaps that limit institutional adoption. Babaei and Giudici (2024) introduced the use of LLMs for binary classification with few-shot learning; however, the study was validated on a single developed-market peer-to-peer lending environment (LendingClub, USA), used undersampling that removed the effect of class imbalance from the evaluation, and compared the LLM only to logistic regression. Recent studies in emerging markets, such as Arias-Serna et al. (2026), apply SHAP-based post hoc explanations to traditional ML classifiers in the solidarity sector but rely on a full ML pipeline that is typically not available to micro-cooperatives.

The present study addresses these gaps by extending the methodology of Babaei and Giudici (2024) to the Latin American solidarity sector (Colombia), a setting characterised by clients with scarce formal credit history and imbalanced sociodemographic data. gpt-4o-mini is evaluated under four Informed GPT example-pool sizes ($N\in \{10,20,40,80\}$) across five independent runs per configuration, together with a zero-shot baseline ($N=0$). This design allows both predictive performance (mean AUC) and prediction stability (standard deviation of AUC) to be quantified. The research question, following the frame established by Babaei and Giudici (2024), is: given a fixed, previously published ML reference (Arias-Serna et al., 2026), how closely can an off-the-shelf LLM approach that reference on a real cooperative-lending portfolio, and under which operating conditions? The purpose is to provide evidence for institutions that do not have the resources for a full-scale ML deployment and to identify design directions (prompt content, example selection) that may narrow the remaining gap with classical ML.

The scientific contribution of this work extends beyond the geographic transfer of an existing framework in six distinct directions:

• New empirical domain: application to Colombian solidarity-sector cooperatives, regulated by the Superintendencia de Economía Solidaria (SES) under a legal framework distinct from banking supervision, with sector-specific features (equity contributions, member type, membership tenure, records update status) that are absent from developed-market datasets such as LendingClub or European peer-to-peer platforms.
• Multi-model benchmark: whereas Babaei and Giudici (2024) compare GPT against logistic regression as the sole baseline, we contrast the LLM against five tuned classifiers spanning tree ensembles and gradient boosting families (XGBoost, LightGBM in two configurations, HistGradientBoosting, and random forest), each optimised under an internal weighting scheme that respects the entity’s native imbalance (∼17.3% default rate).
• Feature and prompt leakage audit: we introduce a systematic auditing procedure for both dataset features (identifying s_intereses, reestr, and s_capital as potentially outcome-adjacent) and prompt content. The LLM system prompt is designed to exclude any supervised-analysis-derived information about the target (SHAP rankings, learned causal directions, target-distribution disclosures), so that in-context learning must operate exclusively on the few-shot examples supplied at inference time. This design ensures that reported LLM performance reflects the model’s actual few-shot capability rather than target-leaking prompt engineering.
• Calibration analysis: although it is absent in Babaei and Giudici (2024) and in most LLM credit-scoring literature, we systematically evaluate the reliability of LLM self-reported probabilities using Brier score, log loss, expected calibration error (ECE), and reliability diagrams, and provide post hoc calibration via Platt scaling and Isotonic regression.
• Enhanced statistical rigour: we complement point-estimate AUC comparisons with bootstrap 95% confidence intervals ($B=1000$ resamples), DeLong tests for paired AUC comparisons, and McNemar tests for paired classifications, enabling formal significance assessment.
• Matched label-budget learning curves: in response to the natural concern that comparing an 80-example few-shot LLM against a 12,000-observation ML model is not like-for-like, we provide learning curves for logistic regression, XGBoost, and LightGBM trained at matched budgets $N\in \{10,20,40,80,160,320,640,1280\}$, quantifying the specific regime in which the LLM offers an advantage.

Considered jointly, the empirical results position LLM-based credit classification as a tool with defined operating conditions: comparable to logistic regression across the label-scarce regime, surpassed by tuned gradient boosting once training data exceeds approximately 100 labelled defaults, and requiring calibration and threshold adjustment before operational use.

  Regional and regulatory context.  

While the literature contains various credit-scoring models based on machine learning and LLMs, these applications are overwhelmingly concentrated in the commercial banking sectors of developed nations. This research bias is largely sustained by the availability of open-access, purely quantitative public repositories such as LendingClub or the German Credit dataset (Baesens et al., 2003; Lessmann et al., 2015). In Colombia and the wider Latin American region, by contrast, the natural confidentiality of transaction-level credit data under strict banking secrecy and Habeas Data regulations (República de Colombia, 2023) offers an exceptional opportunity to pioneer high-value empirical research within the cooperative sector: these stringent frameworks legally prohibit sharing consolidated private records, which has historically restricted academic research in this region to synthetic or aggregated data.

The present study uses a representative Colombian savings and credit cooperative as a pilot setting. The dataset, comprising 12,861 credit records collected during January–December 2025, is an empirical resource for the sector. Because all solidarity entities in Colombia manage risk under the standardised SARC (Sistema de Administración de Riesgo Crediticio) framework, collecting identical relational, sociodemographic, and credit attributes and using a uniform 90-day default definition mandated by Chapter II, Title IV of the Circular Básica Contable y Financiera (Superintendencia de la Economía Solidaria, 2017), the methodology and prompt design used here are portable across the national cooperative network, subject to institution-specific replication of the reported numbers.

The empirical work is organised around four research questions, each answered in the section indicated:

• RQ1. Does stratified few-shot prompting improve the LLM’s discriminative capacity over the zero-shot baseline on Colombian cooperative-lending data? (Answered in Section 5.4 and formally tested in Section 5.7.)
• RQ2. Does the LLM offer a genuine label-budget advantage over classical and gradient-boosting ML classifiers when all approaches are evaluated at matched training sizes? (Answered in Section 5.6.)
• RQ3. Are LLM self-reported default probabilities sufficiently well calibrated to serve as decision-support scores without post hoc adjustment? (Answered in Section 5.5.)
• RQ4. Are LLM predictions distributionally uniform across protected demographic subgroups, and how do the disparities compare with those of the ML baselines? (Answered in Section 5.10.)

The remainder of the paper is organised as follows. Section 2 reviews the relevant literature, from classical credit scoring to modern computational-intelligence methods, and situates in-context learning and class-imbalance handling within it. Section 3 describes the empirical dataset, defining the target variable under the supervisory framework of the Superintendencia de Economía Solidaria (SES) and the credit, relational, and sociodemographic features. Section 4 describes the experimental methodology, including the tuning of tree ensemble and gradient boosting classifiers, the system prompt architecture for the LLM, and the Informed GPT configuration across example-pool sizes (N). Section 5 reports the empirical results: predictive performance (AUC), confusion matrices, calibration, matched-budget learning curves, paired tests, and prompt-ablation analysis. Section 6 discusses the institutional, computational, and regulatory trade-offs of the prompt-based approach. Section 7 summarises the conclusions and identifies directions for future research on prompt design and algorithmic complementarity.

2. Literature Review

2.1. Credit Risk and Default Prediction

Credit risk is defined as the probability that a borrower will fail to meet contractual financial obligations under the agreed terms (Basel Committee on Banking Supervision, 1999). Default prediction, determining whether a given client will become delinquent within a regulatory window, constitutes the most extensively studied binary classification problem in finance, with a literature stretching back to the seminal Z-score model (Altman, 1968) and continuing through modern benchmark studies that compare dozens of classifiers under realistic credit-portfolio conditions (I. Brown & Mues, 2012; Lessmann et al., 2015). In the Colombian regulatory framework applicable to the solidarity sector, the Superintendencia de Economía Solidaria (SES) defines default as delinquency of 90 or more days within a 12-month period; this threshold operationalises both supervisory monitoring and the construction of the binary target used in this study.

The economic relevance of accurate default prediction has grown alongside the diversification of lending portfolios, particularly in cooperative and microfinance settings where the customer base is heterogeneous and includes informal-sector workers, customers with limited banking history, and clients whose creditworthiness must be inferred from non-traditional signals. Misclassification in either direction is costly: false negatives translate into direct credit losses, while false positives carry both reputational damage and the loss of legitimate revenue. This asymmetry, combined with structural class imbalance, motivates the search for models that can extract discriminative signal from heterogeneous, partly-relational data without sacrificing operational interpretability.

2.2. Machine Learning Models for Credit Risk

Over the past two decades, machine learning models have outperformed classical statistical approaches in default prediction benchmarks. Comparative studies report that nonlinear classifiers, particularly tree-based ensembles, produce higher AUC than logistic regression and discriminant analysis across multiple credit datasets (Lessmann et al., 2015; Yeh & Lien, 2009), a finding replicated in subsequent studies on imbalanced credit portfolios (Baesens et al., 2003; I. Brown & Mues, 2012; Serrano-Cinca et al., 2015) and in commercial-credit applications (Machado & Karray, 2022). Fraud- and default-detection reviews report the same direction (Bolton & Hand, 2002; Dastile et al., 2020; Shi et al., 2022). The most commonly used families are gradient boosting ensembles such as XGBoost (Chen & Guestrin, 2016) and LightGBM (Ke et al., 2017), which combine sequential weak learners, and random forest (Breiman, 2001), which uses bagging and random feature selection. logistic regression (Hosmer et al., 2013) is retained as a regulatory baseline in many institutions for its interpretability; neural architectures have also been studied, at the cost of opacity.

Beyond model choice, the literature reports a set of recurring practices for credit-scoring pipelines: stratified hyperparameter search via cross-validation, explicit handling of class imbalance, and unified preprocessing of mixed numerical and categorical features. The application of computational intelligence methods to financial decision-making is a research line with sustained activity (Cavalcante et al., 2016), and its extension to emerging-market credit decisions is ongoing (Arias-Serna et al., 2026; Ziemba et al., 2021). Systematic reviews of financial-risk models that integrate environmental, social, and governance factors document a parallel broadening of the predictor space (Caro-González et al., 2026). The Colombian solidarity sector has only recently received systematic attention: an ML benchmark on cooperative-lending default prediction (Arias-Serna et al., 2026), together with an explainable LightGBM credit-rating model aligned with the regulatory framework of the Superintendencia de Economía Solidaria (Arias-Serna et al., 2025), provide the reference point for the present extension to LLM-based classification.

2.3. Class Imbalance in Credit Datasets

A recurring feature of credit datasets is that the default class is a minority: in most portfolios, defaulters represent a small fraction of total observations. In the dataset analysed here, the default rate is approximately 17.3%, a roughly 5:1 imbalance that, if unaddressed, biases classifiers toward the majority class and produces models with high overall accuracy but low recall on the minority class (Chawla et al., 2002). Surveys of imbalanced credit-scoring problems report that this direction of effect is consistent across model families and datasets (Dastile et al., 2020; Shi et al., 2022). Three families of mitigation strategies are documented: instance-level resampling such as synthetic minority over-sampling (Chawla et al., 2002); cost-sensitive learning through the class_weight=’balanced’ parameter in scikit-learn classifiers; and gradient-boosting reweighting via scale_pos_weight. The latter two are used in this study and are matched to each model family in Section 4.

2.4. Large Language Models and Few-Shot Learning

Large language models (LLMs) such as GPT-4 (OpenAI, 2023) are transformer architectures (Vaswani et al., 2017) trained on massive text corpora via self-supervised next-token prediction. Beyond a certain scale, these models exhibit in-context learning, the ability to perform new tasks by conditioning on a small set of examples provided in the prompt, without any gradient updates (T. Brown et al., 2020). Applied to tabular data, this capability has been evaluated systematically by Hegselmann et al. (2023), whose TabLLM framework serialises table rows into natural-language prompts for few-shot classification, and extended by boosting-inspired fine-tuning schemes for few-shot tabular tasks (Wang et al., 2026); a related line examines how reliably LLMs read structured financial records (Tan et al., 2026). This few-shot capability has transformed the cost structure of applied classification since it removes the need for task-specific training, labelled corpora, and serving infrastructure (Kasneci et al., 2023). The wider literature on prompting methods (Kojima et al., 2022; Liu et al., 2023; Velásquez-Henao et al., 2023) and on the history of pretrained foundation models (Zhou et al., 2023) documents rapid progress on the design choices (prompt content, chain-of-thought reasoning, example selection) that mediate the transition from raw pre-training knowledge to task-specific competence. The implications for resource-constrained domains are particularly relevant: organisations that previously could not afford a custom ML pipeline can now access classification capability through a prompt and an API call (Nori et al., 2023).

The application of LLMs to credit decisions specifically has begun to receive systematic study. The Informed GPT framework (Babaei & Giudici, 2024) enriches the prompt provided to the LLM with a stratified sample of labelled clients drawn from the training set. Formally, given a client, x, to be classified, the model receives an example set, $\mathcal{E}={\left\{(x_i,y_i)\right\}}_{i=1}^N$, together with x, and produces a prediction, $\widehat{y}$:

$$\widehat{y}=\mathrm{LLM}\left(\mathrm{prompt}(x,\mathcal{E})\right),\mathcal{E}\sim \mathrm{Stratified}({\mathcal{D}}_{\mathrm{train}},N)$$

(1)

where $N\in \{10,20,40,80\}$ controls the size of the example pool, and stratified sampling preserves the marginal distribution of the target; the case $N=0$ corresponds to the zero-shot baseline in which no examples are provided. Applied to LendingClub peer-to-peer loan applications, Informed GPT approached the AUC of logistic regression while producing per-prediction natural-language reasoning. A complementary line of work systematically evaluates generalist LLMs as credit-scoring classifiers and documents both their predictive promise and the bias-amplification risks that arise when language models are applied at scale to lending decisions (Feng et al., 2023). The role of explainability-guided prompts has also been explored (Deldjoo, 2023), alongside concerns about algorithmic fairness in LLM-based financial decisions, particularly across demographic groups (Y. Li & Zhang, 2023). Since these studies, LLM-based credit assessment has expanded along several lines: domain-specific financial credit models (Lei et al., 2025), calibration-aware reinforcement learning for interpretable corporate credit assessment (Wu et al., 2026), multimodal fusion for SME credit risk (Shen et al., 2025), text-mining architectures for risk assessment (Z. Li et al., 2026), and cost-sensitive multi-agent formulations (Kou et al., 2026). Surveys of LLM applications in finance and investment management (Kong et al., 2024), and systematic reviews of transformer-based (Raliphada et al., 2026) and interpretable LLM (Golec & Alabduljalil, 2026) approaches to credit risk, document the same trajectory.

2.5. Terminology: Interpretability, Explainability, Reproducibility, Stability, and Auditability

The credit-scoring literature frequently uses these five terms interchangeably, but the LLM setting makes their distinction operationally consequential. We adopt the following working definitions throughout the paper:

• Interpretability: a property of a model whose internal mechanism is directly inspectable by a human analyst (e.g., logistic regression coefficients, decision-tree paths).
• Explainability: a property of a system that can produce, for a given prediction, an auxiliary artefact (SHAP attribution, natural-language rationale) that identifies the features primarily responsible for the prediction.
• Reproducibility: a property of a study whose full pipeline can be re-executed to obtain the same numerical results given the same inputs, seeds, and software versions.
• Stability: a property of a model whose predictions are robust to small perturbations in the input, the training seed, or (for prompt-based models) the specific example sample.
• Auditability: a property of a decision-making system whose full decision trail (inputs, intermediate quantities, features cited as basis, output) can be examined by an external auditor to verify compliance with regulatory or contractual obligations.

Under these definitions, gradient boosting classifiers are explainable (through post hoc SHAP attributions) but not interpretable; logistic regression is interpretable and explainable; the LLM produces per-prediction natural-language rationales, which is a form of explainability whose faithfulness to the underlying transformer computation is not established. The distinctions matter for the discussion in Section 6, where we temper the qualitative claims previously made about the LLM.

2.6. Feature Interpretability with SHAP

A complementary line of literature focuses on the interpretation of model predictions, motivated by both regulatory requirements and the operational need to communicate decisions to credit officers. SHAP (Shapley Additive Explanations) values (Lundberg & Lee, 2017) provide a unified, theoretically grounded framework for attributing the prediction of any model to its individual features, with extensions tailored to financial applications and AI safety (Bussmann et al., 2021; Giudici et al., 2024; Giudici & Raffinetti, 2023). Bracke et al. (2019) document the operational payoff of explainability techniques in a supervised default-risk analysis for the Bank of England, and Rudin (2019) argues, in a debate-shaping counterpoint, that in high-stakes settings the correct response to opacity is to prefer inherently interpretable models over post hoc explanations. SHAP values support both global feature importance rankings and local explanations of individual predictions. Within the Colombian solidarity sector specifically, recent work has demonstrated that SHAP-based explanations of a LightGBM credit-rating model align with the regulatory framework used by the Superintendencia de Economía Solidaria while exposing the contribution of continuous behavioural variables that the regulatory framework collapses into binary indicators (Arias-Serna et al., 2025). Whether the natural-language explanations produced by an LLM are themselves reliable has become a question in its own right: Al Marri et al. (2025) compare LLM feature explanations with those of classical ML credit models, Geng et al. (2026) evaluate LLMs as post hoc explainability interfaces for credit-risk models, and Yuan et al. (2025) examine the relationship between a model’s self-explanation and its classification behaviour in financial analysis. Earlier work on explainable AI in credit-risk management (Hadji Misheva et al., 2021), and industry commentary on its adoption (itsCredit, 2026), frame the same concern from the practitioner side. The present study does not recompute SHAP attributions and, importantly, does not inject SHAP-derived predictor rankings into the LLM prompt: doing so would leak information from a supervised analysis of the same dataset into a procedure that is nominally free of task-specific parameter updating. Interpretability of the LLM predictions is instead handled through the natural-language rationales produced at inference time, whose limitations for auditing purposes are discussed in Section 6.

2.7. Research Gap

The reviewed literature converges on three observations that motivate this study. First, ML models, particularly gradient boosting ensembles, deliver state-of-the-art performance in default prediction, but their adoption in cooperative and microfinance institutions is constrained by data, infrastructure, and expertise requirements that many of these institutions cannot meet (Arias-Serna et al., 2026; Lessmann et al., 2015). Second, LLMs offer a qualitatively different operating point, no task-specific parameter updating, prompt-based adaptation, natural-language reasoning, but their predictive performance has been validated in only a small number of contexts, with LendingClub peer-to-peer lending data (Babaei & Giudici, 2024) as the most prominent reference, leaving open the question of whether the result extends to heterogeneous Latin American cooperative portfolios. Third, the existing LLM credit-classification literature has not systematically evaluated two properties that are essential for deployment: the stability of LLM predictions across runs with different example samples, and the effect of example-pool size on AUC, both of which determine the auditability of the model in regulated environments.

This study addresses these three gaps simultaneously by (i) replicating Informed GPT on Colombian solidarity-sector data, (ii) benchmarking it against a competitive suite of tuned ML classifiers on the same test partition, and (iii) quantifying stability and example-size effects through repeated runs.

3. Data

3.1. Source and Description

The dataset comprises 12,861 individual credit records from a representative Colombian savings and credit cooperative, collected between January and December 2025. It includes relational, sociodemographic, employment, and credit-related variables gathered through the cooperative’s regular lending processes, providing a real-world basis for analysing credit-risk behaviour (Arias-Serna et al., 2026).

Although machine-learning and LLM-based credit-scoring models have expanded considerably, most empirical applications focus on commercial banks in developed economies, largely due to the availability of public datasets such as LendingClub and German Credit (Baesens et al., 2003; Lessmann et al., 2015). In Colombia and Latin America, confidentiality, banking secrecy, and Habeas Data regulations restrict access to transaction-level credit information, creating a significant empirical gap.

The dataset therefore provides a valuable real-world resource for evaluating predictive credit-risk models in the cooperative sector. Its relevance is reinforced by the standardised SARC (Sistema de Administración de Riesgo Crediticio) framework applicable to Colombian savings and credit cooperatives, which establishes common principles for credit-risk management and default identification. This regulatory standardisation enhances the potential transferability of the proposed predictive architectures to other entities within the Colombian cooperative sector (Superintendencia de la Economía Solidaria, 2017).

Table 1. General dataset description. Source: dataset characteristics from Arias-Serna et al. (2026), reproduced here for reference; the train/test partition is defined by the present study (see Split protocol below).

Characteristic	Value
Total observations	12,861
Predictor variables	23
Target variable	default (binary)
No default (0)	10,632 (82.7%)
Default (1)	2229 (17.3%)
Training split	95% (12,217 obs)
Test split	5% (644 obs)
Split seed	random_state=1

summarises the general characteristics of the dataset, including its size, observation window, class balance, and train/test partition.

Prediction Design

We make the following aspects of the prediction design explicit so that the analysis is fully specified:

• Unit of analysis: the credit, one active loan per row (12,861 loans). The analysis is conducted at credit level rather than at client level, since under the SES supervisory framework the risk classification and the associated provision are assigned to each credit obligation. The extraction supplied by the cooperative carries a client identifier in addition to the credit identifier (n_credito), and neither repeats: the 12,861 credits belong to 12,861 different clients, one credit per client. The same borrower therefore cannot appear in both the training and the test partition. Neither identifier is used as a predictor.
• Target definition and outcome window: default is a binary indicator taking the value 1 if the credit is delinquent by 90 or more days within a 12-month observation window, following the definition established by the SES supervisory framework (Section 2.1).
• Feature-measurement period: The dataset provides cross-sectional features as measured at the observation cut-off used by the cooperative for supervisory reporting during the collection window of January–December 2025. Row-level timestamps (per-loan disbursement or snapshot dates) are not present in the raw data, which precludes out-of-time validation at the row level and is disclosed as a structural constraint in Section 6.
• Split protocol: 95% training/5% test with stratification on the target and random_state=1. The ratio is a design choice of the present study and is not inherited from Arias-Serna et al. (2026), whose published description of the baseline reports stratified cross-validation rather than a fixed train/test ratio; the ML pipeline (preprocessing, model family, hyperparameter search) is adopted from that reference, but the partition is ours. The ratio follows the evaluation protocol established in the seminal Informed GPT reference (Babaei & Giudici, 2024), which reports LLM AUC on about 170 test observations per iteration against approximately 30,000 training samples, an effective per-iteration split of about 99.4/0.6, and which attributes that choice to “the high computational time and cost of the process of getting responses from the API”. The same constraint binds the present design more tightly, since it evaluates five example-pool configurations five independent times each, so each additional test observation is queried twenty-five times over. Within that budget, we adopt the largest partition the design supports: our 5% test set ($n=644$) is approximately four times larger than the per-iteration test set used to establish the Informed GPT benchmark, providing tighter bootstrap CIs than the seminal reference. All classifiers, the five tuned ML models and every LLM configuration, are fitted and scored on this same partition, so the comparison of Section 5 is internally consistent. The trade-off relative to the more common 80/20 or 70/30 ratios is that the 644-observation test set has wider bootstrap confidence intervals than a larger partition would; we mitigate this with the repeated $5\times 5$ k-fold analysis of Section 5.8 on the full 12,861-observation dataset (cross-fold $\sigma <0.004$ for gradient boosting) and the wider 2573-observation partition used for the matched-budget learning curves in Section 5.6.

3.2. Variables

The 23 variables are grouped into three categories.

3.2.1. Loan Variables

The first block comprises eight variables describing the contractual terms of the loan and its current balances: the agreed term, the periodic payment, the disbursed amount, the outstanding principal and interest balances, whether the loan has been restructured, and the number and value of the guarantees pledged against it. These are listed in

Table 2. Loan-related variables. Source: Arias-Serna et al. (2026), s_capital, s_intereses, and reestr additionally flagged as outcome-adjacent in Section 3.4.

Variable	Description
plazo	Agreed term (days)
v_cuota	Periodic payment amount (COP)
v_prestamo	Total disbursed amount (COP)
s_capital	Outstanding principal balance (COP), flagged as outcome-adjacent (Section 3.4)
s_intereses	Unpaid interest balance (COP), flagged as outcome-adjacent (Section 3.4)
reestr	Loan restructuring, flagged as outcome-adjacent (Section 3.4)
garantias	Number of guarantees
valorgarantia	Value of guarantees (COP)

. Three of them (s_capital, s_intereses, and reestr) are flagged as outcome-adjacent by the feature audit of Section 3.4.

3.2.2. Cooperative Relationship Variables

The second block comprises six variables describing the member’s standing with the cooperative rather than the loan itself: length of membership, whether the member’s records are current, equity contributions, savings balance, membership status, and member type. These are listed in

Table 3. Customer–cooperative relationship variables. Source: Arias-Serna et al. (2026).

Variable	Description
vinculacion	Days since membership
actualizacion	Customer records kept up to date with the cooperative (1 = yes)
aportes	Equity contributions (COP)
ctasahorros	Savings account balance (COP)
estado_cliente	Membership status
tipoasociado	Member type

. This block has no counterpart in commercial-bank datasets and is specific to the solidarity sector.

3.2.3. Sociodemographic Variables

The third block comprises nine variables describing the borrower: age, sex, socioeconomic stratum, monthly income and expenses, economic sector, department and city of residence, and the external credit score. These are listed in

Table 4. Sociodemographic variables. Source: Arias-Serna et al. (2026).

Variable	Description
edad	Age (years)
sexo	Sex (1 = male)
intestrato	Socioeconomic stratum (1–6)
curtotalingresos	Monthly income (COP)
curtotalegresos	Monthly expenses (COP)
actividadeconomica	Economic sector
departamento	Department of residence
ciudad	City of residence
puntaje_data	External credit score

.

3.3. Preprocessing

Categorical variables (actividadeconomica, departamento, ciudad) are automatically encoded using skrub.TableVectorizer (Arias-Serna et al., 2026), which applies dirty category encoding based on TF-IDF over character n-grams. The reported tree-based ensembles (XGBoost, LightGBM, HistGradientBoosting, random forest) require no scaling for numerical features, so the preprocessing pipeline is reduced to the categorical encoding step.

3.4. Feature Audit for Potential Leakage

In response to concerns about outcome-adjacent features in credit scoring (Arias-Serna et al., 2026; Lessmann et al., 2015), we conduct a systematic audit of the 23 predictors. Three variables are flagged as potentially co-temporal with the default event and are documented transparently:

• s_intereses (unpaid interest balance): The accumulation of unpaid interest at observation time indicates that the loan is already in delinquency at the moment of measurement. This variable is the strongest single potential leakage vector, with a univariate AUC of 0.686 on the full dataset, comparable to the external credit score puntaje_data (0.681).
• reestr (loan restructuring status; SES supervisory code, where value 4 denotes “no restructuring applied” in the cooperative’s coding convention): restructuring is typically triggered by prior payment difficulties and would, in principle, be a post hoc indicator of the target. In the audited dataset, however, this variable is heavily concentrated on the single value 4 (12,859 of 12,861 observations, or 99.98%), so its univariate AUC is 0.500 and it cannot leak information about the outcome in practice. It is retained in the audit for completeness.
• s_capital (outstanding principal balance): Potentially co-temporal with the default event depending on the observation cut-off. Empirically, the class-conditional means differ by less than 4% (11.07 M COP for non-default vs. 11.47 M COP for default), and the univariate AUC is 0.526, indicating minimal discriminative signal.

The remaining 20 variables (loan contract terms, cooperative-relationship indicators established at membership, sociodemographic attributes, and the external credit score) constitute the ex ante feature set that a credit officer would have available at underwriting time. Of the three flagged variables, only s_intereses is a materially meaningful leakage candidate in this dataset. The ML baselines adopted from (Arias-Serna et al., 2026) use the full 23-variable set for direct comparability with prior work; we make the audit results explicit so that readers may weigh the influence of these variables on the reported metrics, and in Section 5 we complement the main analysis with a sensitivity check that evaluates the LLM on the 20-variable safe subset.

• Scope of the leakage audit.

The three flagged features are retained in the ML baseline because that pipeline is inherited verbatim from Arias-Serna et al. (2026); re-auditing the ML side would constitute a re-evaluation of the reference itself and lies outside the scope of the present study. The research question of this paper follows the frame established by Babaei and Giudici (2024): given a fixed, previously published ML reference, how far can an off-the-shelf LLM go in approaching or complementing it? Under that frame, the ML numbers are treated as-published (as Babaei and Giudici (2024) treated the logistic regression baseline on LendingClub data), and the audit effort is concentrated on the component under evaluation, namely the LLM prompt and the LLM predictions. The sensitivity check of Section 5.9 and the three-way prompt ablation of Section 5.11 are therefore restricted to the LLM side. Any residual leakage effect on the ML AUC would apply symmetrically to the reference and to any subsequent LLM-versus-ML comparison built on the same feature set; readers interpreting the ML–LLM gap should apply the audit findings of this section uniformly across both families.

4. Methodology

4.1. Experimental Design

The experiment compares two families of models:

• ML models: trained on the full training set with hyperparameter search.
• LLM (Informed GPT): gpt-4o-mini with varying example set sizes in the prompt; no task-specific parameter updating is performed, but the model does rely on its own pre-training corpus, on labelled demonstrations at inference time, on prompt engineering, and on the semantic feature descriptions.

The primary metric is the AUC-ROC (area under the ROC curve), which quantifies the discriminative capacity of the model independently of the classification threshold and is robust to class imbalance (Lessmann et al., 2015).

Figure 1 summarises the comparative model architecture at the conceptual level, extending the visual convention of Babaei and Giudici (2024) (their Figure 2) to accommodate the multi-model ML branch and the zero-shot/Informed GPT contrast introduced in this study. The detailed operational pipeline, including the two leakage checkpoints (feature audit and prompt audit), is shown separately in Figure 2.

4.2. Machine Learning Models

All ML models follow the sklearn.Pipeline pattern with two stages: preprocessing (TableVectorizer) and classifier (Arias-Serna et al., 2026). Hyperparameter search uses RandomizedSearchCV with stratified 5-fold cross-validation and optimisation of weighted $F_1$.

Table 5. Trained ML models and hyperparameter search strategy. Source: baseline pipeline of Arias-Serna et al. (2026).

Model	Search Strategy	Imbalance Handling	Scoring
XGBoost	RandomizedSearchCV	scale_pos_weight	F1 weighted
LightGBM	RandomizedSearchCV	scale_pos_weight	F1 weighted
LightGBM (skrub)	Pre-optimised	scale_pos_weight	F1 weighted
HistGradientBoosting	GridSearchCV	class_weight	F1 weighted
Random Forest	RandomizedSearchCV	class_weight	F1 weighted

reports the five trained classifiers together with the search strategy, the imbalance-handling mechanism, and the scoring criterion used for each.

4.3. LLM Model: Informed GPT

We follow the Informed GPT methodology (Babaei & Giudici, 2024) with the following adaptations to the Colombian solidarity sector context:

• Fixed test set: evaluation is performed on the full test set (644 observations), not on random samples.
• Example sizes: $N\in \{10,20,40,80\}$ for Informed GPT, plus a zero-shot baseline (N = 0, no example sampling required).
• Runs: 5 per Informed GPT configuration, with different seeds (random_state = run + 100); the zero-shot baseline is evaluated once, as it involves no example sampling.
• Model: gpt-4o-mini-2024-07-18 (pinned snapshot) via OpenAI API, temperature = 0.0.

Choice of gpt-4o-mini as the sole LLM under evaluation.  

gpt-4o-mini was selected on three grounds. First, it is the most cost-competitive current-generation model in the OpenAI portfolio and is therefore the model most relevant to the resource-constrained institutions targeted by the paper; frontier-tier commercial models (gpt-4o, o1) would raise the operational cost by one to two orders of magnitude, which is at odds with the accessibility argument of the study. Second, it is the model closest in scale and role to the GPT-3.5/GPT-4-mini class evaluated in the seminal Informed GPT reference (Babaei & Giudici, 2024), which preserves methodological comparability with prior work and allows the present study to be read as a direct extension of that reference on a new empirical domain. Third, the evaluation protocol of this paper requires five independent runs at each of four few-shot sizes plus a zero-shot baseline, which multiplies the per-decision cost by a factor of 25 relative to a single-run deployment; running the same stability analysis on a frontier-tier model within the bounded research budget of this study was not feasible, and reporting a frontier-tier configuration only for a subset of the runs would have compromised the paired-test design of Section 5.7. Evaluation of open-source alternatives (Llama, Mistral, Qwen) and of frontier commercial models is proposed as a natural extension in Section 6.7.

• Prompt class composition and example ordering.

For each Informed GPT configuration, the N examples are drawn by stratified sampling from the training set, approximately preserving the marginal ≈17.3% default rate; because N is small, the realised default share deviates from 17.3% by integer rounding (e.g., $N=10\to 2$ defaults = 20%; $N=20\to 3$ defaults = 15%).

Table 6. Prompt class composition (default vs. non-default examples) for each Informed GPT configuration. Source: authors’ calculations on the training partition.

Example Pool Size N	Default Examples	Non-Default Examples	Default Share
10	2	8	20.0%
20	3	17	15.0%
40	7	33	17.5%
80	14	66	17.5%

reports the resulting default vs. non-default counts per prompt. Within a prompt, examples are shuffled once using the run seed (random_state = run + 100), so the ordering is deterministic given the seed; this permits full reproduction of any individual run.

4.3.1. Prompt Design and Leakage Mitigation

The system prompt is designed to provide the LLM with the minimum contextual scaffolding required to interpret the classification task, without introducing information derived from supervised analysis of the same dataset. Concretely, the prompt includes:

• Context of the Colombian solidarity sector and the regulatory definition of default under the SES framework.
• Purely semantic descriptions of the input variables (what each variable measures and its unit), without directional risk interpretations.
• Labelled examples (only for $N>0$), sampled in stratified fashion from the training set.

• Leakage mitigation.

The prompt design explicitly excludes three categories of content that would otherwise leak supervised-analysis information into the LLM’s decision process: (i) predictor importance rankings derived from prior supervised analysis on the same dataset (e.g., SHAP-based feature attributions); (ii) causal risk directions learned from the target (statements of the form “clients with fewer days of membership have higher risk”); and (iii) explicit disclosure of the target base rate. By construction, the LLM cannot exploit dataset-derived risk heuristics through the system prompt and must rely on its pre-trained knowledge plus the few-shot examples supplied at inference time. The prompt is otherwise built following established prompt-engineering practice for technical domains, in which the task, the role, the input schema, and the required output format are specified explicitly and refined iteratively (Velásquez-Henao et al., 2023). This design constraint ensures that the reported LLM performance reflects genuine few-shot capability rather than target-leaking prompt engineering.

The model’s response is constrained to the JSON format given in Listing 1, which the parser validates on every call:

Listing 1. LLM response format.

{   "default_probability": 0.15,   "prediction": 0,   "reasoning": "Customer has updated    records and low unpaid interest…" }

4.3.2. Example Sampling

The N examples are selected using stratified sampling from the training set, preserving the actual default distribution (≈17.3%). Each run, r, uses seed r + 100, ensuring diversity across runs while maintaining reproducibility.

4.4. Evaluation

Predictive performance is assessed using AUC for threshold-independent ranking quality and the conventional battery of precision, recall, and $F_1$ for threshold-level decisions at the operational cutoff $p\ge 0.5$. Let $\{\mathrm{TP},\mathrm{TN},\mathrm{FP},\mathrm{FN}\}$ denote the entries of the confusion matrix. For each class $c\in \{+,-\}$ (with + denoting the default minority class):

$$P^c={\displaystyle \frac{{\mathrm{TP}}_c}{{\mathrm{TP}}_c+{\mathrm{FP}}_c}},R^c={\displaystyle \frac{{\mathrm{TP}}_c}{{\mathrm{TP}}_c+{\mathrm{FN}}_c}},F_1^c={\displaystyle \frac{2P^c{R}^c}{P^c+R^c}}.$$

(2)

Per-class scores are aggregated as macro-$F_1$ (unweighted mean across classes) and weighted-$F_1$ (mean weighted by class support):

$$F_1^{\mathrm{macro}}=\frac{1}{2}\left(F_1^{+}+F_1^{-}\right),F_1^{\mathrm{w}}={\displaystyle \frac{F_1^{+}\left|\mathrm{POS}\right|+F_1^{-}\left|\mathrm{NEG}\right|}{\left|\mathrm{POS}\right|+\left|\mathrm{NEG}\right|}}.$$

(3)

On an imbalanced test set, the weighted score tracks majority-class accuracy while macro-$F_1$ exposes minority-class behaviour; both are reported to disambiguate the two effects. For ML models, AUC and the threshold-level metrics are deterministic (fixed test set, no randomness). For the LLM, statistics over $R=5$ runs are reported:

$$\overline{\mathrm{AUC}}={\displaystyle \frac{1}{R}}{\displaystyle \sum _{r=0}^{R-1}}{\mathrm{AUC}}_r,{\sigma }_{\mathrm{AUC}}=\sqrt{{\displaystyle \frac{1}{R-1}}\sum _{r=0}^{R-1}{({\mathrm{AUC}}_r-\overline{\mathrm{AUC}})}^2}.$$

(4)

For the Informed GPT confusion matrices reported in Section 5, per-instance predictions across the five runs are aggregated by majority vote before computing precision, recall, and $F_1$.

4.5. Uncertainty Quantification and Paired Tests

To move beyond point-estimate AUC comparisons, we complement the evaluation with three additional statistical procedures:

• Bootstrap 95% confidence intervals for AUC: We resample the test set with replacement ($B=1000$ bootstrap samples) and report the 2.5% and 97.5% percentiles of the bootstrap AUC distribution. This yields interval estimates for every model, permitting a first-order comparison of whether observed AUC differences exceed sampling variability.
• DeLong test (DeLong et al., 1988): For pairs of models evaluated on the same test set, we test the null hypothesis $H_0:{\mathrm{AUC}}_A={\mathrm{AUC}}_B$ using the DeLong nonparametric estimator of the covariance between correlated ROC curves. This is the standard test for paired AUC differences and is more appropriate than independent bootstrap comparisons when both models share the same test instances.
• McNemar test: For pairs of models compared at the $p\ge 0.5$ operating point, we compute the McNemar statistic with Edwards’ continuity correction, testing the null of equal error rates on the paired classifications. This complements the DeLong test by evaluating threshold-level rather than ranking-level agreement.

4.6. Calibration Metrics

Since the operational usefulness of predicted probabilities depends not only on ranking quality (captured by AUC) but also on their reliability as probability estimates, we evaluate calibration using four complementary metrics. The choice of metrics follows the calibration literature for modern classifiers (Guo et al., 2017; Naeini et al., 2015):

• Brier score: Mean squared error between predicted probability and observed binary outcome, $\mathrm{BS}=\frac{1}{n}{\sum }_i{(p_i-y_i)}^2$. Lower is better.
• Log loss (cross-entropy): $-\frac{1}{n}{\sum }_i[y_{i}log{p}_i+(1-y_i)log(1-p_i)]$. Penalises over-confident errors more strongly than Brier score.
• Expected calibration error (ECE) with 10 uniform bins: weighted mean absolute difference between average predicted probability and empirical frequency of the positive class within each bin.
• Reliability diagrams: Plots of empirical positive-class frequency against predicted probability, aggregated in 10 uniform bins. Curves below the diagonal indicate over-prediction; curves above it indicate under-prediction.

For models exhibiting substantial miscalibration, we apply two post hoc calibration procedures: (i) Platt scaling (Platt, 1999), a univariate logistic regression fitted on validation probabilities against outcomes, and (ii) Isotonic regression (Zadrozny & Elkan, 2002), a nonparametric monotone remapping. Because the evaluation design of Section 3.1 reserves only a 5% test partition (644 observations) and no independent validation set, we simulate a validation/evaluation split by partitioning the LLM test predictions into two disjoint 50% subsets: the first is used to fit the calibrator, the second to report pre- vs. post-calibration metrics (Section 5.5). This design preserves independence between fit and evaluation for the calibration mapping itself but does not substitute for an external validation partition of the same size as the test set; the two halves are relatively small (322 observations each), so the reported post-calibration metrics should be interpreted as consistent-with-improvement rather than as unbiased production estimates. The F1-optimal decision threshold is reported analogously on the same 50% partition.

4.7. Matched Label-Budget Protocol

To evaluate whether the LLM’s advantage over ML models arises from the modelling approach itself or from the asymmetric label budget (few-shot LLM vs. full-training ML), we train logistic regression, XGBoost, and LightGBM at progressive stratified subsamples of the training set with budget $B\in \{10,20,40,80,160,320,640,1280\}$. For each $(model,B)$ combination, we perform 5 independent runs with different random seeds and record test AUC on the same 2573-observation hold-out partition. These curves are compared directly against the Informed GPT AUC at matched N.

4.8. Implementation

The full pipeline is implemented in Python 3.14 with orchestration via Prefect 3.6. Core libraries: scikit-learn 1.8, xgboost 3.2, lightgbm 4.6, skrub 0.7 (TableVectorizer with dirty-category encoding), imbalanced-learn 0.14, pandas 3.0, numpy 2.4, scipy 1.17, and the OpenAI Python SDK 2.24. The LLM model snapshot is pinned to gpt-4o-mini-2024-07-18 to guard against auto-updates that would compromise reproducibility. The cooperative supplied a static snapshot of the 2025 lending portfolio for research purposes; the LLM campaign was executed offline against this static snapshot between August and December 2025 and does not consume any records posted to the cooperative’s operational systems after the snapshot date.

The pipeline enforces a structured JSON response format on every LLM call (Section 4.3.1); malformed responses trigger an exponential-backoff retry with base 2 s, cap 32 s, maximum 5 attempts, on both openai.RateLimitError and openai.APIError. The observed parse-failure rate over the full campaign was below 0.1%, i.e., fewer than a dozen out of 12,308 total calls were rejected by the JSON parser and re-issued.

The entire experimental campaign for the LLM, four Informed GPT example-size configurations evaluated across five independent runs over the full 644-observation test set, plus the zero-shot baseline, consumed approximately 110.7 million input tokens distributed over 12,308 API requests to gpt-4o-mini-2024-07-18, at a total cost of USD 8.69. This figure characterises the operational footprint of the approach and is referenced again in the discussion as evidence of its financial accessibility for resource-constrained institutions. Section 6.5 contrasts this experimental campaign cost against the projected total cost of ownership in a production deployment (personnel, integration, monitoring), which is the more relevant figure for the resource-constrained institutions targeted by this work.

The pipeline architecture is summarised in Figure 2, which visualises the flow from the raw cooperative dataset through preprocessing, ML training, LLM prompt construction, and comparative evaluation.

5. Results

5.1. Overall Comparison with Bootstrap Confidence Intervals

Table 7. Model comparison by ROC-AUC (with bootstrap 95% confidence intervals) and PR-AUC on the test set (N = 644, base rate 17.4%). For Informed GPT, ROC-AUC is computed on the mean predicted probability across five independent runs (soft-vote ensemble); the run-by-run distribution of individual AUCs is reported separately in Table 8. ML models are deterministic (single evaluation). PR-AUC (average precision) is a more informative ranking metric than ROC-AUC on the imbalanced minority class. Source: authors’ calculations.

Method	ROC-AUC	Bootstrap 95% CI	PR-AUC
LightGBM (tuned)	0.9623	[0.9443, 0.9772]	0.885
HistGradientBoosting	0.9589	[0.9377, 0.9763]	0.881
XGBoost (tuned)	0.9578	[0.9361, 0.9768]	0.884
LightGBM (skrub)	0.9575	[0.9355, 0.9758]	0.884
Random Forest	0.9465	[0.9207, 0.9664]	0.836
GPT zero-shot (N = 0)	0.7441	[0.6945, 0.7872]	0.344
Informed GPT (N = 10, ensemble of 5 runs)	0.6951	[0.6403, 0.7454]	0.311
Informed GPT (N = 20, ensemble of 5 runs)	0.6961	[0.6438, 0.7435]	0.306
Informed GPT (N = 40, ensemble of 5 runs)	0.6929	[0.6394, 0.7420]	0.298
Informed GPT (N = 80, ensemble of 5 runs)	0.6873	[0.6298, 0.7364]	0.299

summarises the AUC of every model evaluated on the 644-observation test set, together with bootstrap 95% confidence intervals ($B=1000$ resamples). The gradient boosting family, LightGBM, XGBoost, and HistGradientBoosting, dominates with AUC values above 0.95 and tight bootstrap intervals, consistent with prior credit-scoring benchmarks (Arias-Serna et al., 2026; Lessmann et al., 2015). Random forest occupies an intermediate position (AUC $=0.946$, CI $[0.923,0.967]$). The Informed GPT and zero-shot configurations cluster in the 0.67–0.74 range with wider bootstrap intervals reflecting both the smaller effective information budget and the run-to-run stability of the model. Figure 3 visualises this separation: the solid ROC curves of the ML models form a tight upper band, while the dashed and dotted LLM curves occupy a distinct lower region.

The PR-AUC values reproduce the ranking induced by ROC-AUC (ML dominates LLM decisively), but the magnitudes make the practical operational cost of the difference more visible: gradient boosting reaches PR-AUC $\approx 0.88$, while the LLM operates at PR-AUC $\approx 0.30$, i.e., the LLM’s precision at any given recall level on the minority default class is roughly one third of the ML baselines’. This is the metric most directly relevant to cooperatives that must balance false approvals (missed defaults) against false rejections (rejected creditworthy members).

• Comparison with prior literature.

The AUC values reported here for the LLM are below the range reported in the seminal Informed GPT study on LendingClub peer-to-peer lending data (Babaei & Giudici, 2024). Two structural factors account for the difference. First, our prompt is designed to exclude any supervised-analysis-derived content (Section 4.3.1), which limits the LLM to genuine few-shot capability without target-informed hints. Second, our ML baseline comprises tuned gradient boosting models, which are substantially stronger references than the classical logistic regression used in (Babaei & Giudici, 2024). Together, these choices produce a more conservative but methodologically cleaner estimate of the LLM’s predictive performance in credit-scoring contexts.

5.2. Effect of Example Set Size on the Informed GPT

A key question in this study is whether providing more labelled examples in the prompt improves the LLM’s discriminative capacity.

Table 8. Distribution of individual per-run AUCs of the LLM by example set size, plus prediction-flip rate across the five runs (fraction of test observations whose predicted class label changes across at least one pair of runs). Values are computed by evaluating each run’s predicted probabilities separately. This complements the ensemble AUC reported in Table 7, which pools the five runs by averaging per-instance probabilities. Zero-shot is evaluated once as it involves no example sampling. Source: authors’ calculations.

Example Size	Runs	Min AUC	Max AUC	Mean AUC	Std AUC	Flip Rate
0 (zero-shot)	1	0.7441	0.7441	0.7441	N/A	N/A
10	5	0.6531	0.7133	0.6800	0.0225	12.7%
20	5	0.6669	0.7059	0.6803	0.0150	13.2%
40	5	0.6611	0.6850	0.6737	0.0091	14.9%
80	5	0.6524	0.6859	0.6723	0.0124	17.9%

presents the AUC statistics for each example-size configuration across five independent runs per size. Two observations stand out. First, all configurations yield standard deviations below 0.025 across runs, indicating that the model’s output is reasonably stable across different stratified example samples. Second, the relationship between N and AUC is flat-to-slightly-decreasing: the best mean AUC is achieved by the zero-shot baseline ($N=0$, AUC $=0.744$), and the mean AUC at $N\in \{10,20,40,80\}$ hovers near 0.68 with a slight decline as N increases. Adding more few-shot examples does not improve, and marginally degrades, the LLM’s predictive performance on this dataset.

The prediction-flip rate quantifies the stability of individual borrower decisions, complementing the aggregate-level ${\sigma }_{\mathrm{AUC}}$ statistic. Between 12.7% and 17.9% of the 644 test observations receive a different predicted class label in at least one of the five stratified runs, and the flip rate grows with the pool size N. This is a substantially larger figure than the small ${\sigma }_{\mathrm{AUC}}$ would suggest, and it clarifies a practical implication: aggregate stability of ROC-AUC is a necessary but not sufficient condition for borrower-level stability of decisions. In an operational deployment, roughly one in eight borrowers at $N=10$ and nearly one in five at $N=80$ would receive a materially different decision, depending on which stratified sample happened to populate the prompt.

Zero-shot involves no example sampling, the prompt is identical for every hypothetical run of the same test set, so it is evaluated once, following the standard practice in the LLM classification literature (Babaei & Giudici, 2024). Under temperature = 0.0 the OpenAI API is near-deterministic but does not formally guarantee identical outputs across independent calls; consequently, the reported zero-shot AUC is a single-run estimate whose sampling variability we do not quantify and which should be interpreted as an approximate point estimate rather than a distributional summary.

The interpretation is direct: without target-informed hints in the system prompt, the LLM cannot extract useful additional signal from the labelled examples supplied in the few-shot context on this particular dataset. The evidence indicates that the LLM operates near a fixed pre-training-derived competence level on which few-shot examples act more as noise than as informative signal.

5.3. Confusion Matrices

While AUC captures discrimination capacity across all operating points, confusion matrices reveal how each approach behaves at the classification threshold actually used in practice ($p\ge 0.5$). Figure 4 presents the full picture: the two best ML models, the LLM in zero-shot mode, and the four Informed GPT configurations.

ML models. XGBoost and LightGBM show comparable precision and recall on both classes. On the default (minority) class, XGBoost attains a precision of 0.874 (87.4% of its default alerts are correct), a recall of 0.741, and an $F_1$ score of 0.802. LightGBM trades slightly higher precision (0.888) for lower recall (0.705), yielding $F_1=0.786$ on the minority class. On the non-default class, both models exceed 0.94 in precision, 0.97 in recall, and 0.96 in $F_1$. The macro-averaged $F_1$ (equal-weight across classes) is 0.882 for XGBoost and 0.873 for LightGBM; the support-weighted $F_1$ is 0.934 and 0.930, respectively. This precision–recall profile is relevant in a cooperative-lending context, where a false default flag carries both reputational and relational costs.

LLM zero-shot. Without any labelled examples from the solidarity sector, the model attains a precision of 0.438 and a recall of 0.473 on the default class ($F_1=0.455$): roughly half of its default alerts are false alarms and it captures less than half of the true defaulters. Performance on the non-default class remains acceptable, with a precision of 0.887, a recall of 0.872, and an $F_1$ of 0.880, so the support-weighted $F_1$ stays at 0.806. The macro-averaged $F_1$, however, collapses to 0.667, approximately 0.22 points below XGBoost, quantifying the cost of the absence of in-distribution calibration. The model applies generic credit-risk heuristics derived from its pre-training that do not fully align with the local distribution of the cooperative’s customer base.

5.4. Informed GPT: Classification Behaviour Under Few-Shot Learning

The bottom row of Figure 4 traces how the LLM’s predictions evolve as the number of labelled examples in the prompt increases from 10 to 80. Read together with Table 8, these matrices reveal patterns that go beyond what AUC alone can convey.

From zero-shot to few-shot. Supplying labelled examples shifts the LLM’s threshold-level decisions without uniformly improving the conventional metrics. Across the four configurations, precision on the default class falls from 0.438 (zero-shot) to the 0.252–0.355 range, and recall moves from 0.473 to 0.321–0.482. The corresponding $F_1$ score on the minority class therefore deteriorates from 0.455 (zero-shot) to 0.282–0.409. On the non-default class, the trajectory is similar but milder: precision slips from 0.887 to the 0.848–0.882 range, and recall from 0.872 to 0.799–0.818. Few-shot prompts therefore nudge the model toward a more aggressive default-predicting regime, but the resulting change in operating point does not translate into a coherent gain in either precision or recall at the $p\ge 0.5$ threshold. Notably, the discriminative capacity captured by AUC remains comparable across all configurations (Table 8), which indicates that the examples reshape the model’s calibration rather than its underlying ranking ability.

Stability and reproducibility. Each matrix in the bottom row aggregates the majority vote across five independent runs, each drawing a different stratified sample of examples from the training set. The low standard deviations in Table 8 (${\sigma }_{\mathrm{AUC}}<0.025$ for all configurations, and ${\sigma }_{\mathrm{AUC}}\le 0.010$ from $N=40$ onward) confirm that this stability is not an artefact of aggregation: individual runs yield similar predictions, indicating that the model’s response to the prompt is robust to variation in the example pool. This aggregate reproducibility is a necessary but not sufficient condition for auditability: the borrower-level prediction-flip rate reported in Table 8 (12.7–17.9% across configurations) shows that a non-negligible fraction of individual decisions still varies across runs.

Non-monotone effect of example size. None of the conventional metrics increase monotonically with N. The $F_1$ score on the default class peaks at zero-shot (0.455), degrades to 0.409 at $N=10$, drops to a trough at $N=40$ (0.282), and partially recovers at $N=80$ (0.341). The macro-averaged $F_1$ traces the same trajectory, with values of 0.667, 0.628, 0.565, 0.553, and 0.591 for $N\in \{0,10,20,40,80\}$, respectively. AUC (Table 8, ensemble values in Table 7) describes an independent flat-to-slightly-decreasing path: the zero-shot baseline attains the highest AUC (0.7441), and the four Informed GPT configurations cluster in the 0.687–0.696 range with a shallow trough at $N=80$ (0.6873). All three indicators converge on a single finding: the gross volume of in-context examples is not determinant of few-shot success. At intermediate sizes, the example set is large enough to constrain the model’s reasoning but not diverse enough to cover the feature space, producing a prompt that is informationally saturated but distributionally narrow. This echoes prior observations that prompt composition, rather than mere volume, drives few-shot performance (Babaei & Giudici, 2024), and identifies example-selection strategies, such as similarity- or uncertainty-based sampling, as a productive direction for future work.

Gap relative to ML models. All Informed GPT configurations remain well below the trained ML baselines on every conventional metric. The AUC gap between the LLM and the best ML model is approximately 0.28, and the support-weighted $F_1$ falls from 0.934 (XGBoost) to the 0.60–0.75 range for the LLM. This gap is the direct consequence of the fundamental difference between the two approaches: ML models are optimised end-to-end on thousands of in-distribution examples, while the LLM adapts exclusively through structural context (variable descriptions plus few-shot examples) and cannot exploit dataset-derived risk heuristics. That limitation, however, is inseparable from the LLM’s core practical advantage; the same model can be transferred to a new lending context, or updated to reflect new regulatory guidance, by modifying the prompt alone, without a retraining pipeline.

5.5. Calibration Analysis

Beyond ranking quality (AUC), the operational usefulness of predicted probabilities requires calibration: probabilities that correspond, on average, to observed default frequencies.

Table 9. Calibration metrics on the test set (N = 644, actual default rate 17.4%). Lower is better for Brier score, log loss, and ECE. The calibration slope (target 1.0) and intercept (target 0.0) were obtained by logistic regression of outcomes on logit-transformed predicted probabilities. Source: authors’ calculations.

Model	Brier ↓	Log Loss ↓	ECE ↓	Mean Pred.	Gap	Cal. Slope	Cal. Intercept
XGBoost	0.0511	0.1846	0.0278	0.1620	−0.012	0.78	+0.05
HistGradientBoosting	0.0536	0.1841	0.0255	0.1500	−0.024	1.03	+0.54
LightGBM (skrub)	0.0538	0.1878	0.0256	0.1871	+0.013	0.76	−0.43
LightGBM (tuned)	0.0565	0.2312	0.0465	0.1465	−0.027	0.53	+0.25
Random Forest	0.0639	0.2175	0.0456	0.2105	+0.037	1.20	−0.40
Informed GPT (N = 80)	0.1807	0.5449	0.2091	0.3830	+0.209	0.98	−1.19
Informed GPT (N = 40)	0.1616	0.5017	0.1554	0.3293	+0.155	0.93	−0.96
Informed GPT (N = 20)	0.1631	0.5035	0.1595	0.3334	+0.160	0.89	−1.01
Informed GPT (N = 10)	0.1655	0.5092	0.1676	0.3415	+0.168	0.90	−1.04
GPT zero-shot (N = 0)	0.1274	0.3981	0.1080	0.2780	+0.104	1.61	−0.07

presents Brier score, log loss, and expected calibration error (ECE, 10 bins) for every model, along with the mean predicted probability and the observed default rate.

The calibration slope column of Table 9 is a second diagnostic that complements ECE: a slope below 1.0 indicates that the model’s probabilities are too extreme (peaked toward 0 or 1 more than the outcome supports), while a slope above 1.0 indicates the opposite. The Informed GPT configurations all sit close to 1.0 in slope but exhibit strongly negative intercepts ($-0.96$ to $-1.19$), which is the source of the systematic over-prediction: the LLM’s rank order of clients is approximately consistent with the true logit, but the location is shifted uniformly toward higher predicted probabilities. Figure 5 shows this graphically: the reliability diagrams plot observed default frequency against predicted probability for every model, so that a perfectly calibrated classifier traces the diagonal.

Post Hoc Calibration: Platt Scaling and Isotonic Regression

To evaluate whether the LLM’s ranking-quality signal (AUC 0.67–0.74) can be rescued for direct decision use, we fit Platt scaling (univariate logistic regression on validation logits) and Isotonic regression (nonparametric monotone remapping) on a 50/50 split of the LLM test predictions and evaluate on the held-out half.

Table 10. LLM post hoc calibration: pre- and post-calibration Brier, log loss, ECE, and calibration gap on the held-out half. Both Platt scaling and Isotonic regression substantially reduce ECE and close the calibration gap. Source: authors’ calculations on the LLM predictions.

Model	Method	ECE (Pre)	ECE (Post)	Brier (Pre)	Brier (Post)	Gap (Pre)	Gap (Post)
GPT zero-shot	Platt	0.113	0.052	0.138	0.128	+0.105	+0.018
GPT zero-shot	Isotonic	0.113	0.060	0.138	0.127	+0.105	+0.014
Informed GPT (N = 10)	Platt	0.169	0.041	0.164	0.133	+0.169	+0.003
Informed GPT (N = 10)	Isotonic	0.169	0.039	0.164	0.133	+0.169	−0.000
Informed GPT (N = 20)	Platt	0.162	0.016	0.161	0.133	+0.162	+0.004
Informed GPT (N = 20)	Isotonic	0.162	0.055	0.161	0.136	+0.162	+0.004
Informed GPT (N = 40)	Platt	0.158	0.022	0.160	0.134	+0.158	+0.006
Informed GPT (N = 40)	Isotonic	0.158	0.033	0.160	0.136	+0.158	+0.003
Informed GPT (N = 80)	Platt	0.210	0.016	0.180	0.133	+0.210	+0.003
Informed GPT (N = 80)	Isotonic	0.210	0.023	0.180	0.135	+0.210	+0.003

reports the pre- and post-calibration metrics.

Both calibration methods reduce ECE by a factor of 4 to 13 across all Informed GPT configurations and bring the calibration gap into the $[-0.001,+0.018]$ range, i.e., the calibrated LLM’s mean predicted probability matches the observed base rate to within one percentage point. Platt scaling and Isotonic regression yield comparable results; Platt is preferable when the underlying ranking is approximately monotone in the logit (as here) because it estimates only two parameters, whereas Isotonic fits a step function that can overfit small validation sets.

• Validation-optimal decision threshold.

The naive $p\ge 0.5$ cutoff is systematically suboptimal on an imbalanced dataset, such as this one (base rate 17.4%). A grid search over $p^{*}\in \{0.05,0.06,\dots ,0.95\}$ on the validation half of the test set yields F1-optimal decision thresholds around $p^{*}\approx 0.35$ for the ML models and $p^{*}\approx 0.60$ for the LLM. Substituting the F1-optimal threshold for the naive $0.5$ default recovers approximately 2–5 F1 points on the default class for every ML model and 8–10 F1 points for the LLM configurations. In practice, cooperatives adopting either family of models should validate a decision threshold on a held-out sample before deployment; the code accompanying this paper produces this threshold as an automatic pipeline step.

Three findings stand out. First, all ML models exhibit acceptable calibration (ECE < 0.05, Brier < 0.07), with XGBoost achieving both the highest AUC and the lowest Brier score. Second, all LLM configurations are substantially miscalibrated: ECE ranges from 0.108 (zero-shot) to 0.209 ($N=80$), roughly four to eight times the ML baseline, and Brier scores exceed 0.12. Third, and most importantly for operational deployment, the LLM systematically over-predicts default: the mean predicted probability lies between 0.28 and 0.38, while the true base rate is 0.174, yielding a positive calibration gap of $+0.10$ to $+0.21$. Notably, this over-prediction worsens with more few-shot examples; the calibration gap grows monotonically from $+0.10$ ($N=0$) to $+0.21$ ($N=80$), suggesting that in-context examples shift the model’s operating threshold without improving its ranking capacity. Reliability curves for the LLM lie systematically below the diagonal at every predicted-probability bin, indicating over-confidence in the default class. The practical deployment of the LLM therefore requires either post hoc calibration (Platt scaling or isotonic regression fitted on a validation set) or an operating threshold above the naive $p\ge 0.5$ cutoff.

5.6. Matched Label-Budget Learning Curves

To assess whether the observed LLM-versus-ML gap reflects the modelling approach or the asymmetric label budget, we train logistic regression, XGBoost, and LightGBM at progressive stratified subsamples of the training set.

Table 11. Matched label-budget learning curves. The mean test AUC (five runs per budget) for logistic regression, XGBoost, and LightGBM on the 2573-observation hold-out partition. The rightmost column (“Informed GPT (ref.)”) is included as a visual reference and reports the ensemble AUC of the LLM on the primary 644-observation test partition (Table 7); it is not evaluated on the same 2573-observation hold-out and should be read as an indicative benchmark rather than a direct point-by-point comparison. Values in italics indicate degenerate cases where the classifier failed to learn (AUC = 0.5). An em dash (—) indicates a budget at which the LLM was not evaluated: the Informed GPT experiments were run only for $N\in \{10,20,40,80\}$ examples, so no value exists for the larger budgets. Source: authors’ calculations.

Budget N	Logistic Regression	XGBoost	LightGBM	Informed GPT (Ref.)
10	0.658	0.500	0.500	0.680
20	0.690	0.697	0.500	0.680
40	0.709	0.756	0.619	0.674
80	0.735	0.849	0.849	0.672
160	0.773	0.874	0.877	—
320	0.788	0.905	0.910	—
640	0.798	0.920	0.920	—
1280	0.813	0.935	0.936	—

reports the mean test AUC (over five runs per budget) on the 2573-observation hold-out partition.

The matched-budget analysis clarifies the LLM’s competitive position:

• At the lowest budget ($N=10$), the LLM outperforms logistic regression (0.680 vs. 0.658) and dominates the gradient boosting models, which fail to learn (AUC = 0.5) with so few examples.
• At $N=20$, XGBoost begins to learn a discriminative signal but remains close to the LLM (0.697 vs. 0.680). LightGBM still fails.
• At $N=40$, XGBoost (0.756) already surpasses the LLM (0.674).
• From $N\ge 80$ onward, both gradient boosting models decisively outperform the LLM: at $N=80$, they reach 0.849 while the LLM stalls at 0.672, and by $N=1280$ they reach 0.935 while the LLM cannot exceed its pre-training-derived competence level.
• Logistic regression, by contrast, closes the gap with the LLM only around $N=80$ (0.735 vs. 0.680) and continues to improve up to 0.813 at $N=1280$.

The result can be stated as follows: the LLM’s “low-label advantage” holds against logistic regression across the label-scarce regime, but is reversed by tuned gradient boosting once training data reaches approximately 40–80 labelled defaults. For cooperatives with more than ≈100 labelled defaults, a tuned XGBoost or LightGBM model reaches higher AUC than any of the few-shot LLM configurations examined here. The finding narrows the population of institutions for which prompt-based classification is a defensible primary option to those operating below the ≈100-labelled-default threshold; above that threshold the recommendation of this study is unambiguously in favour of a tuned gradient-boosting pipeline.

5.7. Paired Statistical Tests

To confirm that the differences reported above exceed sampling variability, we complement the bootstrap intervals with paired DeLong tests (for AUC comparisons on the shared test set) and McNemar tests with Edwards’ continuity correction (for threshold-level classification agreement).

Table 12. Paired statistical tests on the 644-observation test set. DeLong p-values test $H_0:{\mathrm{AUC}}_A={\mathrm{AUC}}_B$; McNemar p-values test equal error rates at $p\ge 0.5$. Values reported to four significant figures; values reported as “<10⁻¹⁶” fell below double-precision resolution. Source: authors’ calculations.

Contrast	ΔAUC	DeLong p	McNemar p
ML vs. LLM (25 pairs)
XGBoost vs. GPT zero-shot	+0.214	<10−16	4.8 × 10−12
XGBoost vs. Informed GPT (N = 10)	+0.263	<10−16	<10−16
XGBoost vs. Informed GPT (N = 80)	+0.271	<10−16	<10−16
LightGBM vs. GPT zero-shot	+0.218	<10−16	4.2 × 10−12
LightGBM vs. Informed GPT (N = 80)	+0.275	<10−16	<10−16
ML vs. ML
XGBoost vs. LightGBM	−0.004	0.4053	0.7728
XGBoost vs. HistGradientBoosting	−0.001	0.7301	0.1213
LightGBM vs. HistGradientBoosting	+0.003	0.4214	0.3017
LightGBM vs. Random Forest	+0.016	0.0062	0.5708
HistGradientBoosting vs. Random Forest	+0.012	0.0210	1.0000
LLM vs. LLM
GPT zero-shot vs. Informed GPT (N = 10)	+0.049	0.0224	8.0 × 10−5
GPT zero-shot vs. Informed GPT (N = 20)	+0.048	0.0244	4.4 × 10−4
GPT zero-shot vs. Informed GPT (N = 40)	+0.051	0.0191	2.5 × 10−4
GPT zero-shot vs. Informed GPT (N = 80)	+0.057	0.0217	2.7 × 10−7
Informed GPT (N = 10) vs. (N = 80)	+0.008	0.4682	0.0035
Informed GPT (N = 10) vs. (N = 20)	−0.001	0.9165	0.3827
Informed GPT (N = 40) vs. (N = 80)	+0.006	0.5819	7.4 × 10−4

reports the p-values for the contrasts that matter for the paper’s claims; the full 45-pair matrix is available in the supplementary CSV paired_tests.csv.

Three findings follow. First, every ML-vs-LLM contrast is significant under both tests, indicating that the observed gap of approximately 0.22–0.28 AUC points is not attributable to sampling variability. Second, within the ML family, the gradient boosting variants (XGBoost, LightGBM, HistGradientBoosting) are statistically indistinguishable from each other ($p>0.4$ in every pair); random forest is significantly lower than LightGBM ($p=0.006$) and HistGradientBoosting ($p=0.021$), consistent with its lower point estimate. Third, within the LLM family none of the six pairwise contrasts among few-shot configurations ($N\in \{10,20,40,80\}$) reaches significance at $\alpha =0.05$, consistent with the flat pattern reported in Table 8; zero-shot is significantly better than every Informed GPT configuration ($p\approx 0.02$ under DeLong), so that adding stratified few-shot examples does not improve, and marginally reduces, the LLM’s discriminative capacity on this dataset.

5.8. Repeated Cross-Validation Robustness Check

To assess the stability of the ML AUC estimates beyond the single 95/5 split, we run a $5\times 5$ repeated stratified k-fold cross-validation (25 folds total) on the full 12,861-observation dataset.

Table 13. Repeated stratified k-fold cross-validation (5 splits × 5 repeats = 25 folds) on the full dataset (N = 12,861). Source: authors’ calculations.

Model	Mean AUC	Std AUC	95% CI (Mean)	Folds
LightGBM	0.9614	0.0033	[0.9601, 0.9627]	25
XGBoost	0.9610	0.0037	[0.9596, 0.9625]	25
Logistic Regression	0.8263	0.0089	[0.8228, 0.8298]	25

reports the mean and standard deviation of AUC across folds for the three matched-budget models. The estimates from the single test partition are reproduced: gradient boosting models attain AUC $\approx 0.96$ with cross-fold standard deviations below 0.004, and logistic regression attains AUC $\approx 0.83$ with $\sigma <0.01$. The tight cross-fold intervals indicate that the ML performance estimates are stable to the choice of train/test partition.

5.9. Sensitivity Analysis: LLM on Ex Ante Features Only

To evaluate the influence of the three variables flagged as potentially outcome-adjacent in Section 3.4 (s_intereses, reestr, s_capital) on the LLM’s predictions, we re-ran the zero-shot LLM on the same test partition using only the 20 ex ante features. This complementary analysis addresses the natural concern that the LLM might be exploiting outcome-adjacent information through the few-shot examples or the variable descriptions.

Table 14. Sensitivity analysis: zero-shot LLM AUC with all 23 features vs. 20 ex ante features (SES-eligible feature set) only. Test N = 644. Source: authors’ calculations.

Feature Set	Zero-Shot AUC
All 23 features	0.7441
20 ex ante features (safe subset)	0.6945
ΔAUC	−0.0496

reports the comparison.

Removing the three flagged variables reduces zero-shot AUC by approximately 0.05 points, indicating that these variables do contribute discriminative signal to the LLM but that their contribution is a small fraction of the total (0.74 with → 0.69 without). The LLM continues to outperform logistic regression at matched budget on the safe subset. The result validates the classification of these variables as outcome-adjacent while confirming that the qualitative conclusions of the paper, the LLM operates below tuned gradient boosting and below logistic regression trained beyond ≈100 examples, are preserved when the safe subset is used.

5.10. Subgroup Fairness Analysis

Following the concerns raised in the LLM credit-scoring literature about bias amplification across demographic groups (Feng et al., 2023; Y. Li & Zhang, 2023), we conduct an exploratory subgroup fairness analysis by reporting AUC on eight demographic slices of the test set: sex (female, male), socioeconomic stratum (low 1–2, mid 3, high 4–6), and age (under 35, 35–55, over 55).

Table 15. Subgroup fairness: maximum absolute deviation from overall AUC across eight demographic subgroups (sex, stratum, age). Lower is better; larger deviations indicate uneven performance across groups. Source: authors’ calculations.

Model	Overall AUC	Max |ΔAUC| Across Subgroups
XGBoost	0.958	0.025
LightGBM (tuned)	0.962	0.018
HistGradientBoosting	0.959	0.020
LightGBM (skrub)	0.958	0.026
Random Forest	0.946	0.029
GPT zero-shot	0.744	0.077
Informed GPT (N = 10)	0.695	0.074
Informed GPT (N = 20)	0.696	0.096
Informed GPT (N = 40)	0.693	0.089
Informed GPT (N = 80)	0.687	0.075

summarises the maximum absolute AUC deviation from the overall AUC across subgroups for each model.

The pattern reveals a systematic difference between the two families of models. ML classifiers exhibit maximum absolute AUC deviations below 0.03 across all subgroups, indicating relatively uniform performance across demographic slices. LLM configurations, by contrast, exhibit deviations three to five times larger (0.07–0.10), with the most pronounced disparities appearing across socioeconomic-stratum subgroups. The direction of the disparity is consistent: LLM performance is systematically higher on the higher-stratum subgroup than on the lower-stratum subgroup. This is a preliminary but concerning finding that supports the tempered claims about LLM fairness discussed in Section 6 and motivates formal fairness audits (equal opportunity, demographic parity, calibration by group) as a prerequisite for any deployment targeting protected populations.

Table 16. Protected-group errors at the operational $p\ge 0.5$ threshold: false-positive rate (FPR) and false-negative rate (FNR) per demographic subgroup, reported for the LLM zero-shot configuration and for two ML classifiers (HistGradientBoosting and random forest) covering the boosted-tree and bagged-tree families of the benchmark. The overall pattern is dominated by the two-family contrast between LLM and ML and is stable under the choice of ML representative; the remaining tuned ML classifiers of Table 7 are omitted from this subgroup table for brevity. Source: authors’ calculations.

Subgroup	GPT Zero-Shot		HistGradientBoosting		Random Forest
	FPR	FNR	FPR	FNR	FPR	FNR
Sex, female	0.017	0.910	0.017	0.333	0.055	0.205
Sex, male	0.008	0.912	0.021	0.324	0.058	0.265
Stratum, low (1–2)	0.016	0.907	0.027	0.426	0.081	0.296
Stratum, mid (3)	0.015	0.911	0.010	0.200	0.034	0.133
Stratum, high (4–6)	0.000	0.923	0.015	0.385	0.029	0.231
Age <35	0.011	0.875	0.027	0.196	0.055	0.107
Age 35–55	0.015	0.940	0.018	0.480	0.069	0.340
Age >55	0.014	1.000	0.000	0.333	0.014	0.333

extends the analysis to protected-group errors (false-positive rate FPR and false-negative rate FNR) at the operational $p\ge 0.5$ threshold. The LLM’s low detection rate for defaulters holds at the subgroup level: FNR exceeds 0.87 in every subgroup and reaches 1.00 among borrowers over 55. HistGradientBoosting reports FNR between 0.20 and 0.48 across subgroups; random forest between 0.11 and 0.34. The FPR profile is the reverse: the LLM has an FPR below 0.02 in every subgroup (it rarely flags a non-defaulter), while ML models trade a higher FPR for lower FNR. At the fixed $0.5$ threshold the LLM therefore misses approximately nine out of ten defaulters, which is the metric relevant to a cooperative deciding whether to disburse a loan; the aggregate AUC is not the operationally binding statistic.

5.11. Three-Way Prompt Ablation: Neutral vs. Expert-Informed vs. SHAP-Informed

To directly quantify the effect of prompt-content leakage, we run a three-way prompt ablation over: (i) a neutral prompt (used throughout the main results), which contains only regulatory context and semantic feature descriptions without directional risk interpretations; (ii) an expert-informed prompt, i.e., the neutral prompt plus directional risk statements derived from regulator-published risk factors (SES SARC circular) and general credit-risk doctrine, but no supervised-analysis output; and (iii) a SHAP-informed prompt (positive-control leakage baseline), i.e., the neutral prompt plus the top-ranked SHAP predictors from prior supervised analysis of the same dataset, with their learned effect directions and an explicit target base-rate disclosure.

• Source of the SHAP rankings.

The SHAP-informed variant is reconstructed from the predictor rankings and directional statements published in the prior supervised analysis of this dataset (Arias-Serna et al., 2026); no fresh SHAP attribution is fitted for the ablation, and no SHAP computation is performed on the 644-observation test partition. The variant is therefore a positive-control that mirrors a realistic “worst-case” prompt-engineering practice, in which an analyst reuses a published feature-importance ranking to draft a prompt without re-doing the leakage audit. Because the underlying attributions were produced on data drawn from the same portfolio, the variant is treated in this study as a leakage baseline for comparison and not as a candidate deployment configuration.

The ablation is restricted to the zero-shot setting because Section 5.4 and the paired DeLong tests of Section 5.7 establish that zero-shot is the strongest LLM configuration on this dataset—Informed GPT at $N\in \{10,20,40,80\}$ is significantly worse than zero-shot ($p\approx 0.02$, DeLong). Testing leakage effects at few-shot sizes would evaluate weaker operating points and would not change the conclusion. This is a paper-specific empirical claim (the standard few-shot learning literature reports that additional examples help in many contexts); we take no position on the general question and restrict the argument to what the evidence from this dataset supports.

Table 17. Three-way prompt ablation: zero-shot LLM under neutral, expert-informed, and SHAP-informed prompts. Test N = 644, actual default rate 17.4%. Source: authors’ calculations.

Prompt Variant	ROC-AUC	Mean Predicted Prob.	Predicted Default Rate
Neutral (main results)	0.7441	0.270	2.6%
Expert-informed	0.7835	0.215	1.9%
SHAP-informed	0.7981	0.394	42.1%
ΔAUC (expert vs. neutral)	+0.039	—	—
ΔAUC (SHAP vs. neutral)	+0.054	—	—
ΔAUC (SHAP vs. expert)	+0.015	—	—

reports ROC-AUC, mean predicted probability, and the fraction of test observations classified as default at the operational $p\ge 0.5$ threshold, for each of the three prompt variants on the shared 644-observation test partition.

Three observations follow. First, the AUC ordering matches the direction expected ex ante from the leakage-mitigation design: neutral < expert-informed < SHAP-informed. Removing the SHAP-derived content from the prompt lowers the reported AUC by 0.054 points; removing the domain-doctrine directional statements lowers it by a further 0.039. Second, the incremental gain from expert-informed to SHAP-informed is small (+0.015 AUC), which indicates that most of what SHAP-informed content contributes to the LLM’s ranking capacity is already recovered by an expert prompt grounded in regulatory doctrine; the marginal contribution of dataset-derived rankings on top of regulator-derived directions is modest. Third, and relevant for operational use, the SHAP-informed prompt shifts the operating point substantially: mean predicted default probability rises to 0.394 (vs. 0.270 neutral) and the fraction of observations classified as default at $p\ge 0.5$ rises to 42.1%, approximately 2.4× the true base rate of 17.4%. The SHAP-informed prompt therefore yields 0.015 AUC over expert-informed at the cost of a calibration bias toward over-prediction.

The neutral prompt used throughout the main results of this paper is thus the most conservative of the three, and the LLM AUC range (0.67–0.74) reported in Table 7 corresponds to the operating point that most cleanly separates in-context learning from target-leaking prompt engineering. The higher AUC figures reported in Babaei and Giudici (2024) on LendingClub peer-to-peer lending data are consistent with the SHAP-informed operating point that the present study explicitly excludes. This ablation makes explicit what the sensitivity analysis in Section 5.9 (Table 14) had already indicated by a different route (removing outcome-adjacent features from the neutral prompt also lowers AUC by approximately 0.05 points).

• Practical implication for prompt design.

The ablation supports a specific design guideline for LLM credit-risk classification: regulator-derived directional statements (from SES SARC, GDPR Article 22, or analogous supervisory doctrine) recover most of the ranking-quality gain that a dataset-derived SHAP prompt provides ($+0.039$ vs. $+0.054$ AUC from the neutral baseline, Table 17), and they do so without requiring an independent leakage audit of the prompt content. Prompt designs for regulated deployment should therefore prefer regulator-derived directional content over reused SHAP rankings, and any use of the latter should be accompanied by an audit demonstrating that the ranking was computed on data disjoint from the evaluation partition.

6. Discussion

The empirical results are consistent with the ranking expected from prior credit-scoring benchmarks: trained ML models reach higher discriminative performance than the LLM on this dataset. Tuned gradient boosting ensembles attain AUC values of approximately 0.96 with bootstrap confidence intervals $[0.94,0.98]$, whereas the LLM operates in the 0.67–0.74 range, a gap of approximately 0.22–0.28 AUC points that is significant in paired DeLong tests. This gap is wider than the difference reported in the original Informed GPT validation on LendingClub peer-to-peer lending data (Babaei & Giudici, 2024). Two design choices account for the widening. First, the prompt used here excludes supervised-analysis-derived content, so that in-context learning must operate on the few-shot examples supplied at inference time. Second, the reference ML pipeline comprises tuned gradient boosting models (XGBoost, LightGBM), which are stronger references than the logistic-regression baseline used in Babaei and Giudici (2024) (Chen & Guestrin, 2016; Ke et al., 2017; Lessmann et al., 2015). Together, these choices yield a more conservative estimate of the LLM’s competitive position.

The matched label-budget learning curves refine the comparison between the two families. When ML classifiers are trained at the same label budgets available to the few-shot LLM (10, 20, 40, 80 labelled examples), the LLM outperforms logistic regression across the label-scarce regime and outperforms gradient boosting at $N\le 20$ examples. Once training data reaches approximately 40–80 labelled defaults, tuned gradient boosting overtakes the LLM: at $N=80$, both XGBoost and LightGBM reach AUC $\approx 0.85$, while the LLM remains at 0.67. The operating condition for LLM adoption can therefore be stated concretely: the LLM offers an advantage over classical ML in the regime of fewer than approximately 100 labelled defaults and is surpassed once labels become plentiful. This regime is relevant for cooperatives in the solidarity sector, many of which operate with hundreds rather than thousands of documented default cases.

The calibration analysis introduces a further qualification. All Informed GPT configurations are miscalibrated: the expected calibration error (ECE) ranges from 0.11 to 0.21, roughly four to eight times the ML baseline of $\approx 0.03$, and the mean predicted probability lies above the true default base rate (0.28–0.38 vs. 0.17), producing calibration gaps of $+0.10$ to $+0.21$. The LLM therefore over-predicts default under the naive $p\ge 0.5$ operating point. Operational use requires either post hoc calibration (Platt scaling or isotonic regression fitted on a validation partition) or a validation-derived threshold above 0.5. This qualifies claims made in the earlier LLM credit-scoring literature about the readiness of self-reported LLM probabilities for direct decision use: AUC-based ranking quality does not, on its own, imply well-calibrated decision-support scores.

The comparison between the two approaches must also account for the operational differences. The trained ensembles reach higher in-distribution accuracy, but deployment requires labelled datasets of sufficient size, dedicated ML expertise, training and serving infrastructure, and explainability frameworks compatible with regulatory scrutiny, resources that small cooperatives often lack. The LLM, in turn, adapts to a new lending context through prompt modification alone, produces per-prediction natural-language rationales, and can be updated to reflect regulatory guidance without a retraining cycle (Babaei & Giudici, 2024; T. Brown et al., 2020). The primary LLM campaign consumed USD 8.69 in API costs (Section 4.8); including validation re-runs, the total experimental footprint remained under USD 15, several orders of magnitude below the fixed costs of an in-house ML pipeline (hardware provisioning, MLOps tooling, specialised personnel).

The qualitative advantages of the LLM should be interpreted with caution. The LLM produces natural-language rationales, but their status as explanations in the auditing sense is not established: the rationales document the model’s stated basis for each prediction, and their faithfulness to the underlying transformer computation would require additional validation that this study does not attempt. The exploratory subgroup fairness analysis of Section 5.10 raises a concern of comparable weight: LLM configurations exhibit AUC deviations across demographic subgroups that are three to five times larger than those of the ML baselines (0.07–0.10 vs. 0.02–0.03), with the largest disparities across socioeconomic-stratum subgroups. This is a preliminary signal that a formal fairness audit (equal opportunity, demographic parity, calibration by group) is a prerequisite for any deployment targeting protected populations, consistent with prior reports of bias amplification in LLM-based lending decisions (Feng et al., 2023; Y. Li & Zhang, 2023). Auditability, in the sense required by supervisory authorities, remains to be evaluated formally. Aggregate prediction stability across runs (${\sigma }_{\mathrm{AUC}}<0.025$ in every configuration) is a necessary but not sufficient condition; the borrower-level prediction-flip rate reported in Table 8 (12.7% at $N=10$ growing to 17.9% at $N=80$) shows that at the level of the individual credit decision, one in eight to one in six borrowers receives a materially different verdict across independent stratified runs. This is the borrower-level auditability diagnostic that the current data configuration supports, and it establishes the empirical lower bound against which a future consistency-of-explanation audit (Rudin, 2019) would be evaluated. Fidelity assessment of the rationales, human evaluation with credit officers, cross-run consistency scoring of the reasoning itself, and comparison against SHAP attributions are the four assessments this study does not perform; all four are enumerated as item 6 of Section 6.7.

Several further limitations should be noted. The primary evaluation was conducted on the 5% held-out partition of (Arias-Serna et al., 2026) (644 observations) to maintain direct comparability with the baseline ML models; the wider partition used for the learning curves (2573 observations) yields narrower confidence intervals but is evaluated on a different sample. The dataset originates from a single cooperative, which constrains external validity; replication across institutions would be required for wider generalisation. Although the dataset was collected during January–December 2025, the raw records do not include row-level timestamps (per-loan disbursement or snapshot dates), which precludes out-of-time validation at the row level; this is a structural constraint of the available data rather than of the methodology. Borrower-level separation is unaffected: as stated in Section 3.1, the unit of analysis is the credit and each credit belongs to a different client, so the split already separates borrowers. The block comparison of

Table 18. Weight of each variable block in the two model families. Panel A reports group permutation importance for the ML models, measured as the mean drop in ROC-AUC over 20 repeats when all variables in the block are jointly permuted on the test set (N = 644); models are scored as already fitted and none is refitted. Panel B reports the rate at which the block is invoked in the LLM’s natural-language rationale across the same 644 zero-shot predictions. The two panels are complementary rather than interchangeable measures, an AUC decrement and a citation frequency, and they are read as two views of the same question rather than as one scale; the limitation this implies is stated in Section 6, and the protocol that would resolve it is stated in Section 6.7. In both panels, bold marks the variable block that carries the greatest weight in each row. Source: authors’ calculations.

Model	Loan	Relationship	Sociodemographic
Panel A. ML models: permutation ΔAUC (mean of 20 repeats)
LightGBM (tuned)	0.122	0.211	0.022
HistGradientBoosting	0.097	0.228	0.019
XGBoost (tuned)	0.115	0.196	0.018
LightGBM (skrub)	0.120	0.218	0.023
Random Forest	0.089	0.256	0.016
Panel B. LLM: rate of invocation in the natural-language rationale
GPT zero-shot (N = 0)	76.6%	57.0%	100.0%

carries a limitation of its own: the ML side is measured by permutation ΔAUC and the LLM side by rate of invocation in the rationale, so the two panels are complementary views rather than a single scale, and a strictly symmetric comparison would require permuting each block in the prompt payload and re-querying the model, which is a new inference campaign rather than a re-analysis of the existing outputs; the protocol is set out as item 7 of Section 6.7. Dependence on a closed-source commercial model (gpt-4o-mini) also introduces concerns regarding cost predictability, data confidentiality, and reproducibility over time as the underlying model is updated; evaluation of open-source alternatives (Llama, Mistral, Qwen) is a natural extension.

Considered jointly, these results position LLM-based credit classification as a tool with defined operating conditions rather than as a general substitute for ML. Gradient boosting ensembles remain the reference for institutions with the data, expertise, and infrastructure required to deploy them. Prompt-based classification, in turn, offers an assessment option for cooperatives that would otherwise lack access to data-driven scoring, particularly those operating with fewer than approximately 100 labelled defaults, provided that the deployment includes prompt-content auditing, probability calibration, and threshold optimisation. The evidence reported here can be extended as prompting techniques and language-model architectures continue to develop.

6.1. Effect of Prompt Volume Versus Prompt Content

The observation that ROC-AUC is flat or decreasing in N (Table 8) and that zero-shot is significantly higher than every Informed GPT configuration ($p\approx 0.02$ under DeLong, Table 12) admits an operational and a theoretical reading. Operationally, once the LLM has fixed its ranking-quality signal at the pre-training-derived competence level, additional stratified examples act as noise on the decision boundary rather than as informative signal, and the calibration gap grows monotonically from $+0.10$ (zero-shot) to $+0.21$ ($N=80$). Theoretically, this decouples prompt volume from prompt content: it is the composition of the prompt (which variables, which regulatory context, which risk framing) rather than the number of examples that determines how the LLM’s in-context adaptation reshapes the decision boundary. This is a dataset-specific empirical claim; the standard few-shot learning literature reports that additional examples improve performance in many contexts. Similarity- or uncertainty-based example selection, active-learning-style prompt construction, and expert-informed variable prioritisation are candidate refinements not exploited by the stratified-random sampling protocol used here. This places the design question where the prompt-engineering literature locates it, in the systematic specification and iterative refinement of prompt content rather than in the accumulation of examples (Velásquez-Henao et al., 2023).

6.2. Differential Decision Boundaries: LLM vs. ML

Three empirical signals indicate that the LLM’s decision boundary differs qualitatively from the ML models’ boundary in this cooperative-lending context. First, the LLM’s AUC clusters at 0.67–0.74 with ${\sigma }_{\mathrm{AUC}}<0.025$ across all example-pool sizes (Table 8), consistent with a boundary dominated by pre-training-derived general credit-risk heuristics rather than by the cooperative-specific signal that the ML models extract, principally from the customer–cooperative relationship block (Table 18). Second, the systematic over-prediction of default (calibration gap $+0.10$ to $+0.21$, Table 9) indicates that the LLM applies a decision threshold shifted toward the positive class relative to the true base rate. Third, the learning-curve crossover (Table 11) shows that the LLM boundary is competitive with gradient boosting only when $N\le 20$; with more training data the ML models converge to a different boundary that the LLM does not access without weight updates. These three signals are consistent with the interpretation that the LLM’s decision surface is a low-dimensional projection of pre-training credit-risk knowledge onto the cooperative-lending domain, whereas the ML models’ surface is fitted to the sector-specific joint distribution of features and defaults.

• Which variables carry the boundary?

The three signals above describe how the boundaries differ but do not identify the variables responsible, so we measure the weight each variable block carries in each family. The 23 predictors are partitioned into the three blocks of Section 3.2: loan variables (Table 2), customer–cooperative relationship variables (Table 3), and sociodemographic variables (Table 4). For the ML models, we report group permutation importance, the mean drop in test AUC over 20 repeats when every variable in a block is permuted on the 644-observation test partition. The block is permuted jointly; that is, all of its columns are reordered by a single shared row permutation, which preserves the correlation structure within the block and removes only its association with the target and with the remaining blocks. Because the blocks group correlated predictors, a joint measurement of this kind exceeds the sum of the individual variable importances since permuting one variable at a time leaves its contribution partly recoverable from its correlates while permuting the block removes it in full; for the relationship block, the joint figure is 1.4 to 1.7 times that sum, depending on the model. The block values are therefore to be read as the weight of each block taken as a unit, not as an additive decomposition into variables. The already-fitted pipelines are scored, no model is refitted, so the baseline adopted from Arias-Serna et al. (2026) is unaffected. For the LLM, we report the rate at which each block is invoked in the per-prediction natural-language rationale, obtained by applying a Spanish-language lexicon of variable-referring expressions to the 644 zero-shot rationales. Table 18 reports both.

The asymmetry is pronounced and it runs in opposite directions. For every ML model, the relationship block is the most important of the three, with permutation ΔAUC between 0.196 and 0.256, roughly twice the loan block (0.089 to 0.122) and an order of magnitude above the sociodemographic block (0.016 to 0.023). Membership tenure, equity contributions, and savings balances are what the tuned ensembles principally use to separate defaulters in this portfolio. For the LLM, the ordering is reversed: the relationship block is the least invoked of the three, appearing in 57.0% of rationales, while sociodemographic content appears in every one. Within the relationship block, the LLM concentrates on the two variables with everyday financial meaning, savings balance (30.4% of rationales) and membership tenure (21.6%), and almost never invokes the two that are specific to cooperative membership, member type (0.0%) and membership status (0.3%). At the level of individual variables, monthly income, monthly expenses, and the external credit score are each cited in more than 99% of rationales. The last of these is worth isolating, because it separates two notions of importance that are easily conflated. The external credit score puntaje_data has high univariate discriminative power, comparable to that of s_intereses (Section 3.4), yet its individual permutation importance is negligible for every ML model, between 0.001 and 0.006 ΔAUC, because the relational block already carries the information it contributes. The LLM invokes it in 99.4% of its rationales. A variable that a language model treats as the central piece of evidence is thus one the tuned ensembles can discard without measurable loss, which is a concrete instance of the boundary divergence this subsection describes.

This gives the qualitative claim of this subsection an empirical basis. The ML models place most of their discriminative weight on precisely the block the LLM refers to least, which is the block that encodes the borrower’s accumulated history with the cooperative and is therefore the least represented in the general credit-risk knowledge a language model acquires during pre-training. The LLM instead anchors on the income, expense, and credit-score triad that is standard in consumer lending worldwide. The relationship block is the sector-specific signal, and it is the part of the boundary that prompt-based adaptation does not recover.

6.3. Financial Inclusion Considerations

The results bear on the inclusion mandate of the solidarity sector. Cooperatives regulated by the SES serve populations underserved by commercial banking; their members are often informal-sector workers, first-time borrowers, and clients with limited external credit history. This intermediation role is documented across the region: credit unions extend access to segments that commercial banks do not reach (Lemus & Rojas, 2022), and financial inclusion is associated with poverty reduction and lower inequality in Latin American economies (Rubio & León, 2025). For a cooperative operating with fewer than approximately 100 documented default cases, the alternative to LLM-based classification is not a tuned gradient boosting model but no data-driven scoring at all. In that regime, the LLM lowers the fixed-cost barrier that has restricted the smallest institutions from computational credit-risk assessment. This is the specific inclusion-relevant contribution of prompt-based classification. The lower predictive ceiling (AUC 0.67–0.74 vs. 0.96 for gradient boosting) and, as documented in Section 5.10, the larger subgroup AUC dispersion should be weighed against this contribution by each deploying institution.

6.4. Ethics and Regulatory Considerations

The credit-scoring context in which this study is situated is subject to overlapping regulatory frameworks, and the LLM approach interacts non-trivially with each of them. We structure the discussion along six pillars.

• Transparency.

Under GDPR Article 22(3), data subjects have the right to obtain meaningful information about the logic involved in an automated decision that produces legal effects concerning them; the scope of that right has been the subject of substantial academic debate (Wachter et al., 2017). Whether the LLM’s natural-language rationale satisfies this standard is not settled: the rationale documents the model’s stated basis for each prediction, but its faithfulness to the transformer’s actual computation is not established (Section 6 above). Colombian Ley 1266 de 2008 and Ley 1581 de 2012 impose analogous transparency obligations for credit-reporting and personal-data processing.

• Demographic biases.

The preliminary evidence in Section 5.10 (Table 15 and Table 16) documents substantially larger subgroup AUC dispersion for the LLM than for the ML models, with the most pronounced disparities across socioeconomic-stratum subgroups. We recommend a formal fairness audit (Hardt et al., 2016) (equal opportunity, demographic parity, calibration by group, disparate-impact ratio) as a deployment prerequisite. The finding is consistent with the LLM bias-amplification risks documented in (Feng et al., 2023; Y. Li & Zhang, 2023).

• Data protection (GDPR, Colombian Ley 1581 and Ley 2300).

Using a third-party API for classification implies transmitting 23 personal financial variables per client to OpenAI’s infrastructure. We recommend three data-minimisation measures for any deployment: (i) pseudonymize n_credito before transmission; (ii) coarsen ciudad and departamento to a broader geographic unit when the finer granularity is not decision-relevant; (iii) bracket edad into five-year bins. The OpenAI API data-retention policy for the account tier used in this study should be reviewed against the Ley 2300 de 2023 (habeas data financiero) obligations (República de Colombia, 2023) before any production deployment. An explicit data-processing agreement is a prerequisite.

• EU AI Act.

Credit scoring is classified as a high-risk AI system under Annex III of the EU AI Act (European Commission, 2024). Deployments of the LLM approach in EU jurisdictions inherit the full obligations of Article 9 (risk management), Article 10 (data governance), Article 11 (technical documentation), Article 13 (transparency and information to users), Article 14 (human oversight), and Article 15 (accuracy, robustness, cybersecurity). The leakage audit (Section 3.4 and Section 4.3.1), calibration analysis (Section 5.5), subgroup fairness analysis (Section 5.10), and prompt-composition disclosure (Table 6) can serve as building blocks for the technical documentation required under Article 11, but they do not by themselves discharge the human-oversight and accuracy-monitoring obligations.

• Accountability.

An LLM-derived classification decision is the joint product of the cooperative (which frames the prompt, provides the data, and applies the decision) and the model provider (which trains and hosts the LLM). We recommend that any deployment contract explicitly allocate responsibility for each of the failure modes documented in this paper: leakage exploitation, calibration drift, subgroup fairness degradation, and prediction-flip instability. Ambiguous allocation is a legal and reputational risk that the current commercial-API contracts do not automatically resolve.

• Responsible AI.

The protocol adopted in this paper, leakage-free prompt design, calibration analysis, matched-budget comparison, subgroup fairness audit, prediction-flip disclosure, and tempered qualitative claims, is offered as a template for responsible LLM deployment in credit contexts. The template is not exhaustive: formal fairness audits, human-in-the-loop explanation evaluation, and out-of-time validation are outside the scope of the present study but remain prerequisites for full Responsible-AI compliance.

6.5. Practical Deployment Considerations

Beyond model performance and regulatory compliance, deploying the LLM approach in a real cooperative involves practical constraints that the token-only cost figure (USD 8.69 for the experimental campaign) does not capture.

• Latency.

Individual LLM calls in the campaign took on the order of 1–3 seconds each (dominated by the JSON parsing round-trip); a five-run ensemble multiplies this by 5. At batch-scoring loads (e.g., end-of-month credit committee), the per-decision latency is negligible; at real-time application scoring, the latency budget should be planned explicitly.

• Scalability.

The observed rate of 12,308 calls over the full campaign fits comfortably within the OpenAI rate limits for the tier used in this study. Peak scoring loads (e.g., a promotional lending campaign) should be pre-provisioned against the rate-limit ceiling to avoid queuing delays; the retry-with-backoff logic (Section 4.8) provides graceful degradation but does not remove the ceiling.

• Privacy.

The data-minimisation measures listed in the Ethics subsection above (pseudonymize n_credito, coarsen ciudad/departamento, bracket edad) also reduce the surface for a re-identification attack against the API-transmitted payload.

• API dependency.

The commercial-API dependency creates three specific risks: (i) reproducibility over time as the model provider updates the underlying snapshot, which we mitigate by pinning gpt-4o-mini-2024-07-18; (ii) cost predictability under contract renegotiation; (iii) service continuity under provider outages. Open-source alternatives (Llama, Mistral, Qwen) can be self-hosted at a higher fixed cost and remove all three risks; frontier commercial models raise cost by one to two orders of magnitude but reduce the ranking-quality gap somewhat.

• Cybersecurity.

Prompt-injection is a live risk whenever any part of the input can be manipulated by an adversarial actor. In the cooperative context, most variables are cooperative-controlled fields (plazo, v_prestamo, aportes, etc.), which limits the surface substantially; the higher-risk variables are free-text fields (actividadeconomica, ciudad), where a bad-faith actor could in principle inject instructions. We recommend input sanitisation (strip control characters, cap length, escape delimiters) as a deployment prerequisite.

• Deployment constraints.

A typical cooperative IT profile (Windows workstations, limited devops capacity, no MLOps stack) supports a lightweight Prefect-based pipeline of the type used in this study; a full retraining/serving infrastructure is generally not feasible. This is the setting the LLM approach targets.

• Maintenance.

Ongoing operations require: (i) calibration-drift monitoring (calibration-gap tracking week over week); (ii) prompt-content re-validation after any provider snapshot update; (iii) periodic subgroup-fairness re-audit; (iv) prediction-flip drift monitoring across independent runs. A cooperative deploying this approach should budget for these monitoring activities as part of the total cost of ownership.

• Total cost of ownership.

The USD 8.69 experimental figure captures API costs at experimental scale only. A rough projection for a mid-sized cooperative processing 1000 credit decisions per month is: API cost approximately USD 1–2 per 1000 single-run decisions (USD 5–10 for a five-run ensemble); part-time personnel for prompt maintenance, calibration re-audit, and fairness re-audit approximately 0.1 FTE annually; integration cost with the cooperative’s core-banking system estimated at one-time USD 3000–8000; monitoring infrastructure approximately USD 500–1500 annually. This total is orders of magnitude below the fixed cost of an in-house ML pipeline (hardware, MLOps tooling, specialised ML personnel), but it is not zero.

6.6. Transferability to Other Institutional Contexts

The empirical evidence in this study derives from a single Colombian cooperative. The transferability of the specific numerical results is subject to structural qualifications that vary across institutional contexts:

• Commercial banks. Typically operate with tens of thousands of labelled defaults per portfolio, placing them well beyond the $N\approx 100$ regime where the LLM is competitive. The paper’s findings support the opposite recommendation for commercial banks: tuned gradient boosting is the appropriate choice. The LLM approach is not intended to substitute for a well-resourced bank pipeline.
• Other Latin American countries. Cooperative-lending sectors in Ecuador (SEPS), Peru (SBS-supervised COOPACs), Brazil (Bacen-supervised cooperatives), and Mexico (LRASCAP-regulated) share the SES-style regulatory structure. The methodology of leakage audit, calibration analysis, matched-budget curves, and subgroup fairness audit is transferable to those settings; the specific AUC numbers should not be assumed to hold and require country-specific replication.
• Online lending organisations (BNPL, digital lenders) typically have thousands of labelled outcomes and rely heavily on behavioural and transactional features (device fingerprints, transaction history, click patterns) absent from the cooperative dataset. The LLM’s competitive position is unlikely to hold in this setting: the discriminative signal is elsewhere, and the LLM cannot access it through the prompt.
• Microfinance institutions. Closest analogue to the cooperative case: small labelled portfolios, heterogeneous informal-sector clientele, limited technical infrastructure. The LLM’s competitive position at $N\le 40$–80 likely transfers, subject to language and regulatory-context adaptation of the prompt.

6.7. Future Directions

Seven research directions arise directly from the empirical evidence of this study:

• Out-of-time validation on time-stamped cooperative data, once available, would substantiate the prospective-validation claim that the current cross-sectional dataset cannot support.
• Multi-institution replication across cooperatives, ideally spanning multiple Latin American regulatory frameworks, is the natural next step for external-validity claims.
• Open-source LLM benchmarks (Llama, Mistral, Qwen) would test whether the observed LLM competence level is specific to the OpenAI family or generalises across provider families, and would also remove the commercial-API dependency risks described in Section 6.5.
• Similarity- or uncertainty-based example selection for the prompt, replacing the stratified-random sampling protocol of this study, may narrow the gap between the flat few-shot curve and the pre-training-derived competence ceiling identified in Section 6.1.
• Formal fairness audits (equal opportunity, demographic parity, group-calibrated) would extend the exploratory subgroup analysis of Section 5.10 into a full compliance framework suitable for regulated deployment.
• Explanation-quality evaluation of the LLM rationales. This study reports the rationales but does not evaluate them, and four distinct assessments are required before they can be treated as explanations in the auditing sense. Fidelity assessment would measure whether the stated rationale is faithful to the computation that produced the prediction, through counterfactual perturbation of the cited variables or comparison against attention-based and gradient-based attributions. Human evaluation would apply a structured protocol with credit officers, covering annotation guidelines, inter-rater agreement instrumentation, and measured officer time, to test whether the rationales are usable and actionable in a real credit committee. Consistency in reasoning would score the rationales across independent runs, extending the borrower-level prediction-flip rate of Table 8 from the predicted label to the stated justification, so that a borrower who receives the same verdict can also be shown to receive the same reason. SHAP comparison would contrast the variables invoked in the LLM rationale against SHAP attributions computed on the ML models for the same observations, establishing whether the two explanation families converge on the same drivers or diverge systematically. None of the four is attempted here, and each is a prerequisite for the transparency obligations discussed in Section 6.4.
• A common feature-attribution protocol across model families. The block comparison of Table 18 measures the ML side by permutation and the LLM side by rate of invocation in the rationale, which are complementary but not commensurable. Permuting a variable block in the prompt payload and re-querying the model would place both families on a single permutation scale, at the cost of a new inference campaign, and would also supply the attribution baseline that the SHAP comparison of the previous item requires.

7. Conclusions

This study extends the Informed GPT methodology of Babaei and Giudici (2024) to the Colombian solidarity sector, evaluates it against a fixed reference ML pipeline (Arias-Serna et al., 2026) on data from a savings and credit cooperative, and reports six methodological additions relative to the seminal reference: (i) a new empirical domain (Colombian solidarity-sector cooperatives regulated by the SES under the SARC framework); (ii) a multi-model comparison against five tuned tree ensemble and gradient boosting classifiers; (iii) a feature and prompt leakage audit, including a three-way prompt ablation (neutral/expert-informed/SHAP-informed); (iv) a calibration analysis using Brier score, log loss, ECE, reliability diagrams, calibration slope and intercept, and post hoc Platt and isotonic remapping; (v) bootstrap 95% confidence intervals and paired DeLong/McNemar tests, complemented by borrower-level prediction-flip rates; and (vi) matched label-budget learning curves against logistic regression, XGBoost, and LightGBM.

Three empirical findings follow. First, tuned gradient boosting ensembles (LightGBM, XGBoost, HistGradientBoosting) reach AUC $\approx 0.96$ (bootstrap CI $[0.94,0.98]$), whereas the LLM operates in the AUC 0.67–0.74 range across example-pool sizes $N\in \{0,10,20,40,80\}$; the gap of approximately 0.22–0.28 AUC points is significant under paired DeLong tests. Second, the LLM performance reported here reflects a prompt design that excludes supervised-analysis-derived information about the target, so that in-context learning must rely on pre-trained knowledge and the few-shot examples supplied at inference time; the three-way prompt ablation (Section 5.11) quantifies the associated AUC differential (approximately $+0.05$ from neutral to SHAP-informed at the cost of a calibration bias toward over-prediction). Third, matched label-budget learning curves indicate that the LLM outperforms logistic regression across the label-scarce regime and outperforms gradient boosting at $N\le 20$, but is surpassed by both XGBoost and LightGBM from approximately $N=40$–80 onward.

All LLM configurations are miscalibrated (ECE 0.11–0.21 vs. ≈0.03 for ML) and over-predict default (mean predicted probability 0.28–0.38 vs. observed rate 0.17). Operational use therefore requires either post hoc calibration (Platt scaling or isotonic regression) or a validation-derived threshold above the naive $p\ge 0.5$ cutoff. The LLM’s aggregate predictions are stable across independent runs (${\sigma }_{\mathrm{AUC}}<0.025$ in every configuration), which is a necessary but not sufficient condition for auditability in regulated environments; the borrower-level prediction-flip rate (12.7–17.9%) qualifies this stability.

The operational implication can be stated concisely. The LLM does not match the ML reference in raw discriminative performance, and its “few-shot advantage” holds only against a logistic regression baseline, or in the regime of fewer than approximately 40–80 labelled defaults. For institutions in the solidarity sector operating in that label-scarce regime, prompt-based classification offers an assessment option, provided that the deployment incorporates prompt-content auditing, probability calibration, threshold optimisation, and a formal fairness audit; none of these can be treated as automatic properties of the LLM. The primary LLM campaign of this study consumed USD 8.69 in API costs and, including validation re-runs, the total experimental footprint remained under USD 15.

The evidence reported here does not support treating LLM-based and ML-based credit classification as mutually exclusive alternatives, and it does not support the strongest form of the claim that LLMs can replace ML in credit scoring. The two approaches occupy complementary operating conditions: tuned gradient boosting remains the reference for institutions with the data, expertise, and infrastructure to deploy it; LLM-based classification, when audited and calibrated, extends credit modelling to institutions that cannot sustain a conventional pipeline. The contribution of this study is to characterise this trade-off on Colombian solidarity-sector data and to provide the audit, calibration, and matched-budget evaluation framework that subsequent work may extend as prompting techniques and language-model architectures develop.