Decision-Focused Learning-Based Optimization for Renewable Imbalance Settlement and Flexible Resource Dispatch

Abstract

High renewable penetration makes imbalance settlement inseparable from the physical decisions governing reserve procurement and flexibility activation. This paper develops a decision-focused learning-based optimization framework that trains renewable-deviation and flexible-resource deliverability representations through downstream dispatch, reliability, and settlement consequences. The mathematical contribution is a settlement-aware learning objective that couples learned uncertainty, resource-time credible-capacity certification, network-constrained multi-stage dispatch, and counterfactual marginal-contribution allocation while retaining an exact revenue-adequacy identity. The 33-node Zhangjiakou-type regional case uses 15 min intervals and comprises five resource classes: independent storage, data-center flexibility, industrial adjustable load, commercial demand response, and electric-vehicle aggregation. Relative to a fixed-ratio reserve rule, the proposed method lowers the regional balancing cost from 950 to 618 thousand USD (34.9%), achieves 97.8% renewable accommodation, limits the shortage probability to 0.7%, and attains a settlement-fairness index of 0.92. The framework solves a 500-asset instance in 118 s. External validation uses 4027 half-hour observations from the 2025 Elexon/BMRS market, including measured wind and solar output, day-ahead forecasts, load, imbalance prices, and procured-reserve prices. On the 1487-interval December test set, the proposed model reduces the replay cost from 2953.3 to 2598.2 thousand GBP (12.0%), decreases the shortage-interval frequency from 4.64% to 1.28%, and reaches 99.74% renewable accommodation. Comparisons with forecast-then-optimize, Wasserstein distributionally robust optimization, off-policy reinforcement learning, and graph-based behavioral cloning establish that the improvement comes from jointly learning which uncertainty matters for dispatch and which flexible capacity is deliverable.

1. Introduction

The rapid expansion of wind and solar generation has changed the operating logic of modern electricity markets. In conventional systems, imbalance settlement was primarily an ex post financial instrument: a renewable producer, load-serving entity, or market participant deviated from its schedule and paid a deviation charge calculated after physical balancing. Under high renewable penetration, this logic becomes incomplete because the financial settlement is inseparable from the real-time physical decision that prevents reserve shortage, wind spillage, photovoltaic curtailment, and frequency-security degradation. Forecast errors are no longer small residuals around dispatchable thermal production. They are structured, weather-driven, and often correlated across neighboring renewable plants and load zones. This creates a new research problem: imbalance management must be formulated as a coupled learning, optimization, and settlement problem rather than a pure forecasting problem or a pure market-pricing problem.

Wind power forecasting and probabilistic forecast calibration have been studied extensively, and the literature has shown that forecast value depends on the operational decision in which the forecast is used [1]. Reserve-market studies further indicate that the economic value of balancing capacity depends on the joint distribution of renewable uncertainty and system flexibility [2]. However, the practical settlement rules used in many markets still apply deterministic deviation bands, fixed penalty factors, or administrative reserve requirements. These rules are transparent but insufficiently adaptive. They do not distinguish between deviations that actually create reserve scarcity and deviations that are absorbed by already-available flexibility. They also cannot evaluate the deliverable capacity of new flexible resources whose availability depends on state of charge, computational workload, industrial production processes, or customer response fatigue.

The need for an integrated decision model has become more urgent as the power system evolves into a cyber–physical and multi-energy platform. Recent grid-resilience studies after large disturbances emphasize that inverter-rich, automated, and highly coupled systems may exhibit interaction patterns that cannot be captured by isolated reliability margins [3]. Risk-averse dispatch and safe learning methods have begun to integrate frequency-security constraints into learning-based dispatch, demonstrating that artificial intelligence can be embedded into safety-aware operating models rather than used only as an external predictor [4]. Distributionally robust operational risk assessment has also shown that high wind penetration requires uncertainty sets and risk metrics that are computationally fast enough for operational use [5]. These advances motivate a formulation in which learning models are trained for their ability to improve market and dispatch decisions.

Recent work in Energies sharpens this motivation. Off-policy reinforcement learning has been used for economic dispatch with an ensemble reward design [6], graph neural networks combined with behavioral cloning have accelerated topology-aware real-time scheduling [7], open balancing-market simulation has exposed how reserve-market and imbalance-settlement rules interact [8], and coordinated renewable–demand-response trading has shown that flexibility uncertainty directly changes imbalance exposure [9]. Safe-economic and fairness-aware renewable dispatch has also been formulated as a two-stage scheduling problem [10]. These studies establish strong learning, graph, market-simulation, and fairness baselines. However, they do not train the uncertainty representation through a dispatch model that simultaneously certifies deliverable flexible capacity and produces revenue-adequate participant-level imbalance charges.

This paper develops a decision-focused learning-based optimization framework for renewable imbalance settlement and flexible resource dispatch. The term decision-focused is important. In a forecast-then-optimize pipeline, a machine-learning model first minimizes a statistical loss, such as mean absolute error, pinball loss, or negative log-likelihood. The resulting forecast is then passed into a market-clearing or reserve-dispatch optimization model. This two-step process can be suboptimal because the forecast components that most affect downstream imbalance cost may not be the components that dominate the statistical error. For example, a small underforecast during a scarcity period can be more expensive than a large overforecast during a low-load period. Similarly, a forecast error at a node with limited flexible resources can be more consequential than an error at a node with abundant storage. Decision-focused learning directly internalizes this asymmetry by training the prediction model through the optimization loss induced by the dispatch and settlement model.

The proposed framework has three research objectives. First, it learns renewable deviation representations that are operationally meaningful for reserve procurement and real-time balancing. The learned representation describes not only the expected forecast error but also tail-risk scenarios, temporal persistence, nodal correlations, and uncertainty around flexible-resource availability. Second, it certifies the credible capacity of heterogeneous emerging flexible resources and dispatches them through a multi-stage optimization model. The model covers day-ahead reserve procurement, intraday adjustment, real-time activation, reserve shortage, renewable curtailment, and operating constraints of storage, data centers, industrial flexible loads, commercial response, and electric-vehicle aggregators. Third, it designs a responsibility-aware settlement mechanism that allocates imbalance costs between renewable-side and load-side participants according to their learned marginal contributions to system imbalance. The mechanism follows the principle that each participant should bear the marginal balancing cost imposed by its deviation, while preserving settlement budget balance and flexible-resource participation incentives.

From a market-design perspective, this shift is not simply a technical improvement; it changes the definition of accountability. In a traditional deviation-charge system, the settlement rule observes the difference between a submitted schedule and an actual metered quantity, and then applies a price. Such a rule is easy to audit but may be economically inaccurate when the system has multiple sources of flexibility and multiple drivers of reserve activation. A renewable producer may create a deviation that is absorbed by low-cost storage, while another smaller deviation may occur during a congested interval and require expensive load interruption. The two deviations should not necessarily receive the same charge. Similarly, a load-side participant may increase consumption during a scarcity interval and materially worsen the reserve requirement even if its deviation is smaller than that of a renewable plant. Therefore, the settlement design must reflect both physical causality and economic marginal contribution. The proposed formulation operationalizes this idea by making the settlement coefficient a function of counterfactual balancing cost, learned deviation patterns, and resource scarcity.

This paper also addresses a second implementation challenge: emerging flexible resources cannot be treated as firm capacity. A battery with a high power rating may have low credible upward capacity if its state of charge is depleted. A data center may have substantial electrical load but only a portion can be shifted without violating service quality. An industrial user may provide deep reduction during one event but require a recovery period before the next event. An electric-vehicle aggregator may provide fast response only when enough vehicles are connected. These characteristics imply that flexible-resource capacity must be certified dynamically. The proposed model therefore distinguishes between technical maximum capacity, learned credible capacity, certified reserve capacity, and real-time activated capacity. This distinction is essential for a marketized reserve product because it prevents over-procurement from unreliable resources and avoids undercompensation of resources that deliver high-quality response.

The contribution is not the independent invention of decision-focused learning, Wasserstein ambiguity sets, chance constraints, Shapley values, or reserve procurement. Those elements retain their established definitions. The new formulation lies in their mathematical coupling. First, the training loss contains realized dispatch cost, tail reliability, settlement fairness, and revenue-adequacy residuals, so prediction errors receive weights determined by their physical and financial consequences. Second, resource-time deliverability enters the optimization layer through learned credible-capacity bounds; the gradient therefore reflects both renewable uncertainty and the probability that committed flexibility will actually respond. Third, each participant’s settlement share is obtained from a counterfactual redispatch value function with the day-ahead commitment fixed, and the resulting non-negative Shapley weights scale an explicit balancing-revenue requirement. This construction guarantees revenue adequacy by identity rather than checking it after simulation. Fourth, the integrated formulation is evaluated against fixed-ratio, forecast-then-optimize, Wasserstein distributionally robust optimization, off-policy reinforcement learning, and graph-based dispatch baselines on both a network-constrained regional case and an actual-market replay.

Table 1 delineates the established foundations retained in the framework and the mathematical roles developed in this study.

Table 1. Originality boundary of the proposed formulation.

Component	Established Basis Retained Here	Mathematical Role Developed in This Study
Prediction loss	Pinball calibration and decision-focused optimality loss	Adds dispatch cost, tail shortage, settlement fairness, and revenue adequacy to one end-to-end objective.
Optimization layer	Differentiable convex optimization and fixed-commitment KKT differentiation	Propagates gradients through reserve procurement, network-constrained redispatch, curtailment, and heterogeneous flexible-resource activation.
Ambiguity and risk	Empirical Wasserstein ball, chance constraints, and CVaR	Validates the radius out of sample and couples it to learned resource deliverability instead of applying an exogenous uncertainty margin.
Credible capacity	Conventional derating and availability certification	Learns a resource-time deliverability probability that becomes a binding reserve-offer limit inside the decision layer.
Settlement	Counterfactual marginal contribution and sampled Shapley allocation	Re-solves physical redispatch for sampled coalitions, fixes first-stage commitments, and scales contributions to an exact revenue requirement.

The remainder of this paper is organized as follows: Section 2 reviews related work on renewable imbalance management, learning-based optimization, and emerging flexible resources. Section 3, Section 4 and Section 5 present the proposed mathematical model. Section 3 defines the decision-focused learning architecture, Section 4 formulates the multi-stage flexible dispatch and capacity-certification problem, and Section 5 develops the responsibility-aware settlement mechanism and solution approach. Section 6 describes the data, experimental design, parameters, and benchmarks. Section 7 reports the numerical results, and Section 8 concludes the paper.

2. Literature Review

2.1. Renewable Imbalance Management, Reserve Procurement, and Market Settlement

Renewable imbalance management has traditionally been studied through forecasting, reserve valuation, and market settlement. Probabilistic wind forecasting provides a foundation for scenario generation, risk-aware scheduling, and quantile-based reserve procurement [11]. Early non-parametric wind-power forecasting work showed that distributional forecasts are more useful than point forecasts when operators must decide reserve margins and imbalance penalties [12]. These ideas support the present study, but the proposed framework departs from the forecasting-centered view by making the forecast representation endogenous to the optimization objective.

The operating-reserve literature emphasizes that the value of reserve depends on scarcity, response speed, and the opportunity cost of resources. Stochastic operational planning with wind generation has shown how uncertainty can be embedded in security-constrained dispatch decisions [13]. Multi-area stochastic unit commitment studies further demonstrate that wind uncertainty creates geographically differentiated reserve needs, especially when transmission constraints limit balancing transfers [14]. Textbook treatments of renewable market integration have formalized the relationship among uncertainty, reserve scheduling, and market-clearing design [15]. Stochastic market designs for wind producers further show that imbalance exposure changes optimal bidding and reserve behavior [16]. These studies are essential, but they normally assume that the uncertainty model is exogenous or pre-estimated. The present paper instead trains the uncertainty representation according to dispatch and settlement consequences.

Recent multi-energy and virtual-power-plant studies provide direct motivation for the flexible-resource dimension of this paper. Temporally coordinated operation of green airport microgrids has demonstrated that flexible loads, climatic correlations, hydrogen storage, and stochastic programming can be integrated in a decomposed operational model [17]. Carbon-oriented collaborative pricing in multi-energy systems further shows that price and operation layers must be coordinated when energy carriers and flexible resources interact [18]. Virtual power plant models for inertia and primary frequency response show that distributed resources can be aggregated into ancillary-service products when their dynamic response characteristics are properly represented [19]. Dynamic construction strategies for virtual power plants extend this logic by recognizing that distributed-resource interactions and response recovery processes determine the feasible aggregation boundary [20]. These works support the flexible-resource pool concept adopted here, while the present paper focuses on imbalance settlement and source–load cost responsibility.

Hydropower, storage, and high-wind operational risk studies also provide relevant modeling insight. Carryover storage valuation in cascaded hydropower planning shows that intertemporal energy value must be internalized when a storage-like resource is used for both current balancing and future operation [21]. This is closely related to battery energy storage in imbalance settlement, because excessive real-time activation may reduce future reserve deliverability. Model-free voltage calculation for active distribution networks shows that data-driven methods can provide operational models when physical topology or parameters are incomplete [22]. Oscillation-source-location studies in power systems show that event attribution is necessary for post-disturbance accountability and control response [23]. The settlement layer in this paper follows the same attribution logic: imbalance costs should be traced to the sources and loads that materially contribute to reserve activation.

2.2. Learning-Based Optimization and Decision-Focused Learning

Learning-based optimization has emerged to overcome the limitations of sequential forecast-then-optimize pipelines. In classical stochastic programming, the uncertainty distribution is specified before optimization, and theoretical foundations for scenario-based decision models are well established [24]. Sample average approximation is then used to construct a tractable deterministic equivalent [25]. Decomposition methods then exploit the structure of large-scale stochastic programs and mixed-integer programs [26]. Although these methods are rigorous, they do not prescribe how a predictive model should be trained when the final goal is economic decision quality rather than statistical forecast accuracy. Decision-focused learning addresses this gap by training the predictor through the optimization objective itself.

Task-based end-to-end learning for stochastic optimization was among the earliest energy-system formulations of this idea and demonstrated that predictions should be judged by their induced operational cost [27]. The melding of data and decision pipelines extended the decision-focused view by showing how machine-learning models can be integrated with combinatorial optimization problems [28]. The smart predict-then-optimize framework provided a structured loss function for training prediction models in downstream linear optimization settings [29]. Differentiable optimization layers, such as OptNet, enabled constrained optimization problems to be embedded within neural networks [30]. Differentiable convex optimization layers later made this approach more general and practical for disciplined convex programs [31]. These methodological foundations are directly relevant to the present model, where renewable-deviation representations are trained according to the cost of dispatch and settlement decisions.

Robust and distributionally robust optimization are also central to learning-based decision models under renewable uncertainty. The price-of-robustness framework formalized how uncertainty protection can be selected without making operations excessively conservative [32]. General robust optimization theory provides the broader mathematical basis for converting uncertain constraints into tractable deterministic equivalents [33]. Moment-based distributionally robust optimization introduced ambiguity sets that preserve tractability when only partial distributional information is known [34]. Wasserstein distributionally robust optimization allows ambiguity sets to be learned from empirical samples, making it attractive for renewable-forecast deviations and flexible-resource availability [35]. Conditional value-at-risk provides a coherent way to penalize tail-cost outcomes caused by reserve shortage, curtailment, or expensive emergency activation [36]. The proposed framework combines these ideas with decision-focused learning so that the ambiguity representation, risk penalty, and reserve decisions are co-designed.

2.3. Emerging Flexible Resources and Source–Load Responsibility Allocation

Demand-side and distributed flexible resources have become central to renewable-dominated power systems. Early demand-side market analysis emphasized that electricity demand is not purely passive and that load flexibility can alter market efficiency and reliability [37]. Demand-response and smart-load surveys then classified the operational mechanisms through which customers can provide load reduction, shifting, and ancillary services [38]. The virtual power plant concept proposed a market-oriented aggregation structure for distributed generators, storage devices, and controllable loads [39]. Mid-term dispatch optimization of virtual power plants showed how aggregation can be scheduled under market and technical constraints [40]. Bidding studies for virtual power plants in energy and reserve markets further clarified that aggregated flexibility must be priced and settled according to both energy and reserve value [41]. These ideas are integrated in the reserve-pool and market-clearing layers of the proposed model. The scope of emerging flexibility is now broader than traditional demand response. Energy storage can shift renewable energy across hours and provide rapid balancing, but its opportunity cost depends on intertemporal state of charge and future scarcity. Storage valuation studies in electricity markets provide evidence that arbitrage value, reserve value, and congestion value must be jointly considered [42]. Renewable integration studies further emphasize the role of storage in reducing curtailment and improving system flexibility [43]. Data centers can provide flexible load by temporal workload shifting, cooling adjustment, or geographic load migration; classic Internet-scale system studies show that electricity cost can be reduced when computational workloads respond to energy prices [44]. Electric vehicles can act as distributed storage, and vehicle-to-grid research has long recognized their potential to provide regulation and renewable balancing services [45]. In the present paper, these resource classes are represented in a unified dispatch and certification model.

Multi-energy coupling and sector integration also influence flexible-resource dispatch. The role of hydrogen in future sustainable transport and energy systems illustrates how power, transport, storage, and industrial sectors increasingly form one flexibility ecosystem [46]. Data-model hybrid scheduling for grid-connected PEM green hydrogen production shows that multi-energy flexibility requires coordination between physical process models and data-driven scheduling decisions [47]. These works motivate the inclusion of state-dependent, energy-limited, and service-quality-constrained flexibility in renewable imbalance settlement. The key gap addressed in this paper is that existing resource models rarely connect flexible-resource dispatch with responsibility-aware imbalance settlement. A market can procure and dispatch flexible resources efficiently, but if the settlement rule fails to trace who caused the reserve activation, incentives remain distorted. The proposed model therefore combines dispatch optimization with source–load marginal contribution tracing.

The literature therefore leaves three linked gaps. First, probabilistic forecasting studies often evaluate calibration and sharpness without measuring whether the resulting distribution improves imbalance settlement. Second, stochastic and robust dispatch studies optimize operating decisions but usually assume that the uncertainty model has already been selected. Third, market-mechanism studies examine reserve pricing, virtual power plant participation, or demand response incentives, but they rarely embed responsibility allocation directly into the physical dispatch optimization. The present paper closes these gaps by co-designing the learned uncertainty representation, the reserve-dispatch optimization, and the settlement-cost allocation rule. This co-design is particularly relevant for regional pilots, where the operator must justify not only the total balancing cost but also the distribution of charges among renewable generators, load-side users, aggregators, and flexible-resource providers.

Figure 1 distinguishes the forward computation from the training feedback. Renewable forecasts, realized generation, load, resource states, and market records enter the learning layer, which produces deviation scenarios and resource-time deliverability probabilities. The probabilities impose credible-capacity limits, and both outputs enter the same network-constrained dispatch. Real-time activation then determines the counterfactual redispatch value used for settlement. Cost, shortage, curtailment, fairness, and revenue adequacy are evaluated after settlement, and the resulting weighted decision loss is differentiated back to the learning model. No arrow bypasses physical feasibility or settlement accounting.

Figure 1. Schematic of the decision-focused learning-based optimization framework. Solid arrows trace the forward market-clearing and settlement sequence; the separated dashed blue path returns the composite decision-loss gradient to the learning layer.

3. Decision-Focused Learning Architecture

This section defines the learning-to-decision architecture that connects renewable-deviation data, flexible-resource availability, optimization decisions, and imbalance settlement. Let $\mathcal{T}$ denote the set of dispatch intervals, $\mathcal{G}$ the set of renewable generators, $\mathcal{R}$ the set of flexible resources, $\mathcal{N}$ the set of network nodes, and $\mathcal{S}$ the set of scenarios. Historical data include renewable forecasts, actual generation, meteorological variables, nodal loads, resource states, market-clearing results, and settlement outcomes. The training sample is defined as

$$\mathcal{D}={\left\{({\mathbf{x}}^d,{\mathbf{p}}^{\mathrm{for},d},{\mathbf{p}}^{\mathrm{act},d},{\mathbf{l}}^d,{\mathbf{s}}^d,{\mathbf{y}}^d)\right\}}_{d=1}^D,$$

(1)

where ${\mathbf{x}}^d$ contains weather and calendar features, ${\mathbf{p}}^{\mathrm{for},d}$ and ${\mathbf{p}}^{\mathrm{act},d}$ are forecast and actual renewable outputs, ${\mathbf{l}}^d$ is load, ${\mathbf{s}}^d$ is the state of flexible resources, and ${\mathbf{y}}^d$ collects market and settlement labels. Renewable deviation is measured by

$$\Delta p_{g,t}^d=p_{g,t}^{\mathrm{act},d}-p_{g,t}^{\mathrm{for},d},g\in \mathcal{G},t\in \mathcal{T}.$$

(2)

The learning model maps observable features to a conditional distribution of uncertainty,

$${\widehat{\mathbb{P}}}_{\theta }({\mathsf{\xi }}_t\mid {\mathbf{x}}_t)={\mathsf{\Phi }}_{\theta }\left({\mathbf{x}}_t\right),$$

(3)

where $\theta$ denotes trainable parameters and ${\mathsf{\xi }}_t$ includes renewable deviations, load deviations, and flexible-resource availability shocks.

A scenario generator converts the learned distribution into optimization scenarios:

$${\mathsf{\xi }}_{t,s}=h_{\theta }({\mathbf{x}}_t,{\mathsf{\epsilon }}_s),s\in \mathcal{S},$$

(4)

where ${\mathsf{\epsilon }}_s$ is an exogenous random seed or a latent uncertainty sample. Given ${\widehat{\mathbb{P}}}_{\theta }$, the downstream optimization returns the optimal decision vector,

$${\mathbf{z}}^{★}\left(\theta \right)=\arg \underset{\mathbf{z}\in \mathcal{Z}}{\min }{\mathbb{E}}_{\mathsf{\xi }\sim {\widehat{\mathbb{P}}}_{\theta }}\left[C(\mathbf{z},\mathsf{\xi })\right],$$

(5)

where $\mathbf{z}$ contains reserve procurement, certified capacity, flexible-resource activation, curtailment, shortage, and settlement variables. The decision-focused objective extends the established predict-then-optimize loss by including the reliability and settlement consequences generated downstream:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{DFL}}\left(\theta \right)& ={\displaystyle \frac{1}{D}}{\displaystyle \sum _{d=1}^D}{\tilde{C}}_d\left(\theta \right)+{\lambda }_{\mathrm{cal}}{\mathcal{L}}_{\mathrm{cal}}\left(\theta \right),\hfill \\ \hfill {\tilde{C}}_d& =C_d+{\lambda }_{\mathrm{risk}}{\mathrm{CVaR}}_{0.95}\left({\mathsf{\ell }}_d^{\mathrm{sho}}\right)+{\lambda }_{\mathrm{fair}}{\mathcal{F}}_d^{\mathrm{fair}}+{\lambda }_{\mathrm{RA}}{\left(u_d^{\mathrm{RA}}\right)}^2.\hfill \end{array}$$

(6)

where $u_d^{\mathrm{RA}}$ is the non-negative revenue-adequacy residual defined in Section 5. The coefficients selected on the validation set are ${\lambda }_{\mathrm{cal}}=0.15$, ${\lambda }_{\mathrm{risk}}=0.20$, ${\lambda }_{\mathrm{fair}}=0.05$, and ${\lambda }_{\mathrm{RA}}=1.00$. The gradient of the composite decision loss is

$${\nabla }_{\theta }{\mathcal{L}}_{\mathrm{DFL}}={\displaystyle \frac{\partial \tilde{C}}{\partial {\mathbf{z}}^{★}}}{\displaystyle \frac{\partial {\mathbf{z}}^{★}}{\partial \theta }}+{\displaystyle \frac{\partial \tilde{C}}{\partial \mathsf{\xi }}}{\displaystyle \frac{\partial \mathsf{\xi }}{\partial \theta }}+{\lambda }_{\mathrm{cal}}{\nabla }_{\theta }{\mathcal{L}}_{\mathrm{DFL}}$$

(7)

When the downstream problem is convex after fixing integer commitment variables, implicit differentiation through optimality conditions can be used:

$${\nabla }_{\mathbf{z}}\mathcal{J}({\mathbf{z}}^{★},{\mathsf{\mu }}^{★};\theta )=\mathbf{0},{\mathsf{\mu }}^{★}\ge \mathbf{0},{\mathsf{\mu }}^{★}\odot \mathbf{g}({\mathbf{z}}^{★};\theta )=\mathbf{0}.$$

(8)

Here, $\mathcal{J}$ is the Lagrangian, $\mathsf{\mu }$ are dual multipliers, and $\mathbf{g}$ represents inequality constraints.

Calibration is maintained through a distributional scoring loss,

$${\mathcal{L}}_{\mathrm{cal}}\left(\theta \right)=\sum _{\tau \in \mathcal{Q}}\sum _{g,t}{\rho }_{\tau }\left(\Delta p_{g,t}-{\widehat{q}}_{g,t}^{\tau }\left(\theta \right)\right),$$

(9)

where ${\rho }_{\tau }\left(u\right)=u(\tau -{\mathbf{1}}_{u<0})$ is the pinball loss and ${\widehat{q}}_{g,t}^{\tau }$ is the learned quantile. To provide reliability protection, an empirical Wasserstein ambiguity set is constructed as

$${\mathcal{P}}_{\theta }=\left\{\mathbb{Q}:W_1\left(\mathbb{Q},{\widehat{\mathbb{P}}}_{\theta }\right)\le {ϵ}_{\theta }\right\},$$

(10)

where $W_1$ is the Wasserstein distance and ${ϵ}_{\theta }$ is a learned or validated radius. Reserve shortage risk is summarized by

$${\mathrm{Risk}}_t\left(\theta \right)=\underset{\mathbb{Q}\in {\mathcal{P}}_{\theta }}{\sup }\mathbb{Q}\left(\sum _g\Delta p_{g,t}^{-}+\sum _n\Delta l_{n,t}^{+}>Q_t^{\mathrm{up}}\right),$$

(11)

where $\Delta p^{-}$ is renewable underproduction, $\Delta l^{+}$ is load overconsumption, and $Q_t^{\mathrm{up}}$ is upward reserve.

For each flexible resource r, the learning model estimates a deliverability probability:

$${\rho }_{r,t}=\sigma \left({\mathbf{w}}_r^{\top }{\mathsf{\psi }}_{r,t}+b_r\right),$$

(12)

where ${\mathsf{\psi }}_{r,t}$ includes state of charge, workload, industrial operating mode, customer response history, or electric-vehicle plug-in probability. Credible capacity is then certified as

$$K_{r,t}^{\mathrm{cred}}={\rho }_{r,t}{\eta }_{r,t}{K}_{r,t}^{\max },$$

(13)

where ${\eta }_{r,t}$ is a performance derating coefficient and $K_{r,t}^{\max }$ is the technical maximum capacity. The learned activation cost is represented by

$${\widehat{c}}_{r,t}\left(u_{r,t}\right)=a_{r,t}{u}_{r,t}^2+b_{r,t}{u}_{r,t}+d_{r,t},$$

(14)

where the coefficients are learned from bids, historical activation, degradation cost, and response-recovery records.

The marginal contribution feature of participant i to imbalance is defined as

$${\chi }_{i,t}={\mathbb{E}}_{\theta }\left[C({\mathbf{z}}^{★},\mathsf{\xi })-C({\mathbf{z}}_{-i}^{★},{\mathsf{\xi }}_{-i})\right],$$

(15)

where $-i$ denotes a counterfactual system in which participant i’s deviation is removed. A source–load responsibility vector is learned as

$${\mathsf{\omega }}_t=\mathrm{softmax}\left(\mathbf{A}{\mathsf{\chi }}_t+\mathbf{B}{\mathsf{\kappa }}_t\right),$$

(16)

where ${\mathsf{\kappa }}_t$ contains contractual schedules, historical compliance scores, and nodal scarcity indicators. Online learning updates the parameter vector according to

$${\theta }^{k+1}={\theta }^k-{\alpha }_k{\nabla }_{\theta }{\mathcal{L}}_{\mathrm{DFL}}({\theta }^k),$$

(17)

where ${\alpha }_k$ is an adaptive step size. The integrated learning target is finally written as

$$\underset{\theta }{\min }{\mathcal{L}}_{\mathrm{DFL}}\left(\theta \right)\mathrm{s}.\mathrm{t}.{\mathrm{Risk}}_t\left(\theta \right)\le {\epsilon }^{\mathrm{risk}},\forall t\in \mathcal{T}.$$

(18)

The predictor used in every reported experiment is a two-hidden-layer quantile network with 64 and 32 rectified linear units. Inputs comprise 24 lagged renewable deviations, eight lagged load deviations, sine and cosine time encodings, weekday, forecast renewable output, forecast load, and the most recent resource states. The model outputs the 0.05, 0.25, 0.50, 0.75, and 0.95 conditional quantiles and the five resource-class deliverability probabilities. Training used Adam, a batch size of 256, an initial learning rate of ${10}^{-3}$, 100 epochs, early stopping after 12 validation epochs, and random seed 4446492. Each training dispatch used 64 reparameterized scenarios; final evaluation used 200 scenarios. All continuous decisions were differentiated through the KKT system in Equation (8); binary day-ahead commitment variables were fixed at their incumbent values within a gradient batch and re-optimized between batches.

The Wasserstein radius in Equation (10) is dimensionless because every uncertainty coordinate is standardized by its training-set standard deviation. The candidate set $\{0.01,0.02,0.035,0.05,0.075\}$ was evaluated chronologically on the validation period. The selected value, ${ϵ}_{\theta }=0.035$, minimized the validation balancing cost subject to shortage probability no greater than 1%. The chance-constraint form and Wasserstein ball are therefore inherited tools; their joint dependence on the decision-trained deviation distribution and learned deliverability bounds is specific to the proposed layer.

The practical interpretation of this architecture is as follows: The learning component observes historical dependencies among weather conditions, forecast errors, load deviations, and resource responses. It does not merely predict a single net imbalance value. Instead, it learns a structured representation that the optimization model can query to determine the upward reserve requirement, the location of probable reserve shortages, the flexible resources likely to be deliverable, and the participants likely to generate costly deviations. The optimization component then converts this representation into market decisions. If the learned model overstates a deviation that is inexpensive and nonbinding, the downstream loss imposes only a weak penalty. If it understates a deviation that creates scarcity, the optimization loss rises because the system incurs reserve-shortage or emergency-activation costs. This asymmetric weighting is the principal mechanism of decision-focused training.

Another important feature is interpretability. Decision-focused learning is sometimes criticized because end-to-end training may reduce transparency. In this paper, interpretability is preserved by explicitly separating the learned components from the market-clearing structure. The model learns deviation distributions, credible capacity coefficients, activation-cost parameters, and responsibility features. The reserve-balance, storage-state, service-quality, network, and settlement-budget relationships remain explicit. As a result, market operators can audit why a reserve requirement increased, why a resource received a lower certified capacity, or why a participant received a higher imbalance charge. This structure is compatible with regulatory review because the learning model supports decisions but does not replace the market-clearing logic.

The learning layer is not an external forecasting module. It is a market-facing representation engine whose parameters are trained by the cost and reliability consequences of the downstream optimization.

4. Multi-Stage Optimization for Flexible Resource Dispatch

This section formulates the multi-stage reserve procurement, capacity certification, and real-time dispatch model. The system operator minimizes the expected total cost of reserve procurement, flexible-resource activation, renewable curtailment, reserve shortage, and settlement imbalance:

$$\underset{\mathbf{z}}{\min }\sum _t\left(C_t^{\mathrm{cap}}+{\mathbb{E}}_s\left[C_{t,s}^{\mathrm{act}}+C_{t,s}^{\mathrm{cur}}+C_{t,s}^{\mathrm{sho}}\right]\right).$$

(19)

The upward reserve requirement is linked to the learned deviation quantile:

$$Q_t^{\mathrm{up}}\ge {\mathrm{Quantile}}_{1-\epsilon }\left(\sum _g\Delta p_{g,t}^{-}+\sum _n\Delta l_{n,t}^{+}\mid {\widehat{\mathbb{P}}}_{\theta }\right).$$

(20)

The certified capacity of all flexible resources must cover the procured reserve volume:

$$\sum _{r\in \mathcal{R}}{k}_{r,t}^{\mathrm{cert}}\ge Q_t^{\mathrm{up}},\forall t.$$

(21)

Each certified capacity is bounded by the credible capacity learned in Section 3:

$$0\le k_{r,t}^{\mathrm{cert}}\le K_{r,t}^{\mathrm{cred}},\forall r,t.$$

(22)

The capacity procurement cost is

$$C_t^{\mathrm{cap}}=\sum _{r\in \mathcal{R}}{\pi }_{r,t}^{\mathrm{cap}}{k}_{r,t}^{\mathrm{cert}},$$

(23)

where ${\pi }_{r,t}^{\mathrm{cap}}$ is the capacity offer price or market-clearing price. Real-time balance under each scenario is enforced by

$$\sum _g{p}_{g,t,s}^{\mathrm{ren}}+\sum _m{p}_{m,t,s}^{\mathrm{conv}}+\sum _r{u}_{r,t,s}^{\mathrm{up}}+{\mathsf{\ell }}_{t,s}^{\mathrm{sho}}=L_{t,s}+p_{t,s}^{\mathrm{cur}}+\sum _r{u}_{r,t,s}^{\mathrm{dn}}.$$

(24)

Renewable generation is determined by forecast output, learned deviation, and curtailment:

$$p_{g,t,s}^{\mathrm{ren}}=p_{g,t}^{\mathrm{for}}+\Delta p_{g,t,s}-p_{g,t,s}^{\mathrm{cur}},\forall g,t,s.$$

(25)

Battery storage energy evolves as

$$E_{r,t+1,s}=E_{r,t,s}+{\eta }_r^{\mathrm{ch}}{u}_{r,t,s}^{\mathrm{dn}}\Delta t-{\displaystyle \frac{u_{r,t,s}^{\mathrm{up}}\Delta t}{{\eta }_r^{\mathrm{dis}}}},r\in {\mathcal{R}}^{\mathrm{ES}}.$$

(26)

The storage state and power outputs satisfy

$$E_r^{\min }\le E_{r,t,s}\le E_r^{\max },0\le u_{r,t,s}^{\mathrm{up}}\le P_r^{\mathrm{dis}},0\le u_{r,t,s}^{\mathrm{dn}}\le P_r^{\mathrm{ch}}.$$

(27)

Data-center load reduction is bounded by workload elasticity:

$$0\le u_{r,t,s}^{\mathrm{up}}\le {\gamma }_{r,t}^{\mathrm{dc}}{W}_{r,t,s}^{\mathrm{base}},r\in {\mathcal{R}}^{\mathrm{DC}}.$$

(28)

The service-quality constraint for data centers is

$${\displaystyle \sum _{\tau =t}^{t+H_r^{\mathrm{dc}}}}\left(W_{r,\tau ,s}^{\mathrm{base}}-u_{r,\tau ,s}^{\mathrm{up}}\right)\ge {\mathsf{\Omega }}_{r,t}^{\mathrm{req}},r\in {\mathcal{R}}^{\mathrm{DC}}.$$

(29)

Industrial load response is bounded by process limits:

$$0\le u_{r,t,s}^{\mathrm{up}}\le {\gamma }_{r,t}^{\mathrm{ind}}{L}_{r,t}^{\mathrm{ind}},r\in {\mathcal{R}}^{\mathrm{IND}}.$$

(30)

The rebound process after industrial or commercial load reduction is modeled as

$$R_{r,t+1,s}={\phi }_r{R}_{r,t,s}+{\zeta }_r{u}_{r,t,s}^{\mathrm{up}},r\in {\mathcal{R}}^{\mathrm{FL}}.$$

(31)

Rebound power must be incorporated into load balance:

$$L_{t,s}=L_t^{\mathrm{for}}+\Delta L_{t,s}+\sum _{r\in {\mathcal{R}}^{\mathrm{FL}}}{R}_{r,t,s}.$$

(32)

Electric-vehicle aggregator energy satisfies

$$B_{r,t+1,s}=B_{r,t,s}+{\eta }_r^{\mathrm{ev},\mathrm{ch}}{u}_{r,t,s}^{\mathrm{dn}}\Delta t-{\displaystyle \frac{u_{r,t,s}^{\mathrm{up}}\Delta t}{{\eta }_r^{\mathrm{ev},\mathrm{dis}}}}+A_{r,t,s}^{\mathrm{arr}}-D_{r,t,s}^{\mathrm{dep}}.$$

(33)

EV mobility requirements are enforced by

$$B_{r,t,s}\ge B_{r,t}^{\mathrm{mob}},r\in {\mathcal{R}}^{\mathrm{EV}},\forall t,s.$$

(34)

Flexible-resource activation cannot exceed certified capacity:

$$0\le u_{r,t,s}^{\mathrm{up}}+u_{r,t,s}^{\mathrm{dn}}\le k_{r,t}^{\mathrm{cert}},\forall r,t,s.$$

(35)

Ramping limits are

$$-{\overline{R}}_r^{\mathrm{dn}}\le u_{r,t,s}^{\mathrm{up}}-u_{r,t-1,s}^{\mathrm{up}}\le {\overline{R}}_r^{\mathrm{up}},\forall r,t,s.$$

(36)

Nodal power balance under a DC network approximation is

$$\sum _{i\in {\mathcal{I}}_n}{p}_{i,t,s}-d_{n,t,s}+u_{n,t,s}^{\mathrm{up}}-u_{n,t,s}^{\mathrm{dn}}=\sum _{\mathsf{\ell }\in {\mathcal{L}}_n}{B}_{\mathsf{\ell }}({\vartheta }_{n,t,s}-{\vartheta }_{m\left(\mathsf{\ell }\right),t,s}).$$

(37)

Transmission security is enforced by

$$-{\overline{F}}_{\mathsf{\ell }}\le B_{\mathsf{\ell }}({\vartheta }_{n,t,s}-{\vartheta }_{m\left(\mathsf{\ell }\right),t,s})\le {\overline{F}}_{\mathsf{\ell }},\forall \mathsf{\ell },t,s.$$

(38)

Reserve shortage is non-negative and penalized heavily:

$${\mathsf{\ell }}_{t,s}^{\mathrm{sho}}\ge 0,C_{t,s}^{\mathrm{sho}}={\pi }^{\mathrm{VOLL}}{\mathsf{\ell }}_{t,s}^{\mathrm{sho}}.$$

(39)

Renewable curtailment is bounded by available renewable generation:

$$0\le p_{g,t,s}^{\mathrm{cur}}\le p_{g,t}^{\mathrm{for}}+\Delta p_{g,t,s},\forall g,t,s.$$

(40)

The chance constraint for reserve sufficiency is

$${\mathbb{P}}_{\theta }\left(\sum _r{u}_{r,t,s}^{\mathrm{up}}+{\mathsf{\ell }}_{t,s}^{\mathrm{sho}}\ge \sum _g\Delta p_{g,t,s}^{-}+\sum _n\Delta l_{n,t,s}^{+}\right)\ge 1-\epsilon .$$

(41)

A distributionally robust counterpart is represented by

$$\underset{\mathbb{Q}\in {\mathcal{P}}_{\theta }}{\sup }{\mathbb{E}}_{\mathbb{Q}}[{\mathsf{\ell }}_t^{\mathrm{sho}}]\le {\overline{\mathsf{\ell }}}_t^{\mathrm{sho}},\forall t.$$

(42)

Table 2 states the exact calculation sequence. In the objective, $C_{t,s}^{\mathrm{act}}={\sum }_r{\widehat{c}}_{r,t}(u_{r,t,s}^{\mathrm{up}}+u_{r,t,s}^{\mathrm{dn}})$, $C_{t,s}^{\mathrm{cur}}={\pi }^{\mathrm{cur}}{\sum }_g{p}_{g,t,s}^{\mathrm{cur}}$, and $C_{t,s}^{\mathrm{sho}}={\pi }^{\mathrm{VOLL}}{\mathsf{\ell }}_{t,s}^{\mathrm{sho}}$. Upward reserve is therefore not calculated as an unexplained residual: the validated uncertainty quantile sets the system requirement, learned deliverability bounds each offer, the master problem selects the certified capacity, and each scenario subproblem resolves resource activation subject to Equations (24)–(42).

Table 2. Implemented calculation sequence for flexible-resource dispatch.

Step	Information State	Decision Calculated	Binding Formulation
1	Day-ahead	Conditional deviation quantiles and 64 training or 200 test scenarios	Equations (3), (4) and (9)–(11)
2	Day-ahead	Resource-time credible and certified capacity	Equations (12), (13) and (21)–(23)
3	Intraday update	Updated scenario probabilities and reserve requirement	Equations (20), (41) and (42)
4	Real time	Least-cost upward/downward activation, curtailment, and shortage	Equations (24)–(40); all energy, rebound, ramp, and network limits remain active
5	Ex post	Counterfactual redispatch, participant charges, and flexible-resource payments	Equations (43)–(53)

The reported regional experiments used the scenario-based deterministic equivalent with a Benders master problem for certified capacity and one second-stage quadratic dispatch subproblem per scenario. The implementation used Python 3.11, PyTorch 2.3.1, CVXPY 1.5.3, and Gurobi 11.0.3. Gurobi used an optimality gap of ${10}^{-4}$, a feasibility tolerance of ${10}^{-6}$, eight solver threads, and a 600 s limit; no reported instance reached that limit. Benders iterations stopped when the relative upper–lower bound gap fell below ${10}^{-4}$. The computing platform comprised an Intel Xeon Gold 6338 CPU, 256 GB RAM, and an NVIDIA RTX A5000 GPU. These settings underpin the computational results reported in Section 7.

5. Responsibility-Aware Imbalance Settlement and Solution Approach

The settlement model converts the complete physical balancing requirement into auditable charges and payments. Capacity procurement is included because it is incurred to cover the realized uncertainty even when part of the reserve remains unactivated. For each scenario,

$$C_{t,s}^{\mathrm{imb}}=C_t^{\mathrm{cap}}+\sum _r{\pi }_{r,t,s}^{\mathrm{act}}(u_{r,t,s}^{\mathrm{up}}+u_{r,t,s}^{\mathrm{dn}})+{\pi }^{\mathrm{cur}}{p}_{t,s}^{\mathrm{cur}}+{\pi }^{\mathrm{VOLL}}{\mathsf{\ell }}_{t,s}^{\mathrm{sho}}, R_t^{\mathrm{bal}}={\mathbb{E}}_s\left[C_{t,s}^{\mathrm{imb}}\right].$$

(43)

The normalized settlement weight uses the sampled Shapley contribution defined in Equation (47). The equal-weight branch is invoked only when every contribution is non-positive:

$${\omega }_{i,t}=\{\begin{array}{cc}{\displaystyle \frac{\max \{{\varphi }_{i,t},0\}}{{\sum }_{j\in \mathcal{M}}\max \{{\varphi }_{j,t},0\}}},\hfill & \sum _j\max \{{\varphi }_{j,t},0\}>0,\hfill \\ 1/M,\hfill & \mathrm{otherwise},\hfill \end{array} i\in \mathcal{M}$$

(44)

The counterfactual marginal contribution of renewable generator g is

$${\chi }_{g,t}^{\mathrm{ren}}={\mathbb{E}}_s\left[C_{t,s}^{\mathrm{imb}}-C_{t,s}^{\mathrm{imb}}(\Delta p_{g,t,s}=0)\right].$$

(45)

The counterfactual marginal contribution of load participant n is

$${\chi }_{n,t}^{\mathrm{load}}={\mathbb{E}}_s\left[C_{t,s}^{\mathrm{imb}}-C_{t,s}^{\mathrm{imb}}(\Delta l_{n,t,s}=0)\right].$$

(46)

For coalition-based settlement, an approximate Shapley value is calculated as

$${\varphi }_{i,t}={\displaystyle \frac{1}{M!}}\sum _{{\mathcal{S}}_i\subseteq \mathcal{M}\setminus \left\{i\right\}}|{\mathcal{S}}_i|!(M-|{\mathcal{S}}_i|-1)!\left[v({\mathcal{S}}_i\cup \left\{i\right\})-v\left({\mathcal{S}}_i\right)\right].$$

(47)

The source-side charge is

$${\Gamma }_{g,t}^{\mathrm{ren}}={\omega }_{g,t}^{\mathrm{ren}}{R}_t^{\mathrm{bal}}, g\in \mathcal{G}.$$

(48)

The load-side charge is

$${\Gamma }_{n,t}^{\mathrm{load}}={\mathrm{\omega }}_{n,t}^{\mathrm{load}}{R}_t^{\mathrm{bal}}, n\in \mathcal{N}.$$

(49)

Flexible resources receive capacity and activation payments:

$${\Pi }_{r,t}^{\mathrm{FR}}={\pi }_{r,t}^{\mathrm{cap}}{k}_{r,t}^{\mathrm{cert}}+{\mathbb{E}}_s\left[{\pi }_{r,t,s}^{\mathrm{act}}(u_{r,t,s}^{\mathrm{up}}+u_{r,t,s}^{\mathrm{dn}})\right]-{\Pi }_{r,t}^{\mathrm{pen}}.$$

(50)

Because the weights in Equation (44) sum exactly to one, revenue adequacy is an identity rather than an empirical check:

$$\sum _g{\Gamma }_{g,t}^{\mathrm{ren}}+\sum _n{\Gamma }_{n,t}^{\mathrm{load}}=R_t^{\mathrm{bal}}=\sum _r{\Pi }_{r,t}^{\mathrm{FR}}+\underset{s}{\mathbb{E}}[{\pi }^{\mathrm{cur}}{p}_{t,s}^{\mathrm{cur}}+{\pi }^{\mathrm{VOLL}}{\mathsf{\ell }}_{t,s}^{\mathrm{sho}}]+\sum _r{\Pi }_{r,t}^{\mathrm{pen}}, u_t^{\mathrm{RA}}=0.$$

(51)

Participation compatibility for flexible resource r is

$$\sum _t{\Pi }_{r,t}^{\mathrm{FR}}-\sum _{t,s}{\widehat{c}}_{r,t}\left(u_{r,t,s}\right)\ge {\mathsf{\Omega }}_r^{\mathrm{out}},\forall r.$$

(52)

Settlement fairness is measured by the deviation between assigned cost and marginal contribution:

$${\mathcal{F}}^{\mathrm{fair}}=\sum _{i,t}{\left({\Gamma }_{i,t}-{\displaystyle \frac{{\chi }_{i,t}}{{\sum }_j{\chi }_{j,t}+{ϵ}^{\mathrm{num}}}}{\mathbb{E}}_s\left[C_{t,s}^{\mathrm{imb}}\right]\right)}^2.$$

(53)

A Benders-type master problem optimizes first-stage reserve capacity:

$$\underset{\mathbf{k},\eta }{\min }\sum _{r,t}{\pi }_{r,t}^{\mathrm{cap}}{k}_{r,t}^{\mathrm{cert}}+\eta \mathrm{s}.\mathrm{t}.\eta \ge a_q^{\top }\mathbf{k}+b_q,q\in {\mathcal{Q}}^{\mathrm{cut}}.$$

(54)

The online solution stops when the primal-dual residual is below tolerance:

$$\max \left\{\parallel {\mathbf{z}}^{k+1}-{\mathbf{z}}^k{\parallel }_2,\parallel {\mathsf{\mu }}^{k+1}-{\mathsf{\mu }}^k{\parallel }_2,|{\mathcal{L}}_{\mathrm{DFL}}^{k+1}-{\mathcal{L}}_{\mathrm{DFL}}^k|\right\}\le {ϵ}^{\mathrm{alg}}.$$

(55)

The settlement layer is designed to avoid three distortions common in imbalance-pricing practice: The first distortion is socialization without causality, in which all users share balancing costs even if only a subset of participants caused the reserve activation. The second distortion is mechanical deviation pricing, in which charges are proportional to energy deviation but ignore whether the deviation occurred during scarcity, congestion, or surplus. The third distortion is undercompensation of flexible resources, in which availability is undervalued and resources have little incentive to maintain deliverability. The proposed settlement mechanism mitigates these distortions by linking participant charges to counterfactual marginal cost and linking flexible-resource payments to both certified capacity and real-time activation.

The coalition value in Equation (47) is implemented as $v_t\left(\mathcal{S}\right)={\mathbb{E}}_s\left[C_{t,s}^{\mathrm{imb}}\left({\mathbf{z}}_{t,s}^{★}\left(\mathcal{S}\right)\right)\right]$. Members of $\mathcal{S}$ retain their realized deviations; participants outside $\mathcal{S}$ are set to their contracted schedules. The day-ahead certified capacities and network topology are held fixed, while storage, flexible-load activation, curtailment, line flows, and shortage are re-optimized in every scenario. This construction isolates the cost caused by the coalition without crediting it for a different first-stage procurement decision. Source-side and load-side participants may enter the same coalition; their signs are retained, so a load deviation that offsets renewable underproduction can reduce the coalition value.

Exact enumeration requires $2^M$ redispatches. The reported settlement uses 500 antithetic random permutations, evaluates every participant’s entry along each permutation, and averages the marginal increments. The same scenario draws are used for adjacent coalitions to reduce Monte Carlo noise. Independent permutation batches are run until the maximum change in any participant share is below 1%; the 500-permutation cap is binding only for the 250- and 500-participant cases. Redispatches are parallelized across 32 CPU workers. Table 3 reports the resulting cost and allocation error against a 2000-permutation reference.

Table 3. Computational cost of sampled Shapley settlement.

Participants M	Permutations	Wall Time (s)	Maximum Absolute Difference from 2000-Permutation Allocation (Percentage Points)
20	200	4.7	0.6
50	300	12.9	0.8
100	400	31.8	0.9
250	500	78.6	1.0
500	500	162.4	1.2

Equation (51) guarantees revenue adequacy for every interval and every participant set because the non-negative weights sum to one and are applied to the complete balancing-revenue requirement. The result is not limited to the numerical examples. The equal-weight branch prevents an undefined denominator in the exceptional case where no positive contribution exists. Numerical rounding is performed only after aggregation to currency units; the unrounded settlement residual remained below ${10}^{-8}$ GBP in all experiments.

6. Data, Experimental Design, and Parameters

The empirical design has two distinct evidence layers. The first is a network-constrained regional engineering case used to test the complete learning, reserve, dispatch, and participant-level settlement mechanism. It contains 33 nodes, 32 branches, four conventional units, ten wind plants, six photovoltaic plants, and 200 flexible-resource aggregations in the baseline configuration. The chronology contains 35,040 15 min intervals. Public market trajectories are regionalized by per-unit scaling, while the network, flexible-resource ratings, and response constraints are fixed engineering inputs; day-ahead, intraday, and real-time market outputs are generated endogenously by the optimization model. The second layer is an external replay of the actual Great Britain balancing market. It uses Elexon/BMRS day-ahead wind and solar forecasts (DGWS/B1440), measured or estimated wind and solar output (AGWS/B1630), actual total load (ATL/B0610), system imbalance prices (DISEBSP), and procured upward-reserve prices (PPBR/B1730) [48]. This replay tests the learned reserve policy against recorded market-level uncertainty and prices rather than simulated market-clearing outputs. Keeping the two layers separate prevents the historical replay from being used to support participant-level or nodal claims: the regional case evaluates mechanism completeness, whereas the replay evaluates reserve-policy transfer under recorded market conditions. Table 4 summarizes the sources, resolutions, system sizes, and chronological splits for both evidence layers.

Table 4. Datasets, temporal resolution, system size, and chronological splits.

Experiment	Source and Purpose	Resolution and Size	Split	Samples
Regional engineering case	Market-calibrated renewable/load chronology; complete 33-node dispatch and settlement test	15 min; 33 nodes, 32 branches, 20 generators, 200 flexible assets	1 January–31 August/1 September–31 October/1 November–31 December	23,328 train/5856 validation/5856 test
Actual-market replay	Elexon/BMRS DGWS, AGWS, ATL, DISEBSP, and PPBR; external market-level validation	30 min; system-level trajectories regionalized to a 1000 MW renewable reference and 700 MW mean load	9 October–15 November/16–30 November/1–31 December 2025	1821 train/719 validation/1487 test; 4027 complete intervals

Figure 2 contains 2050 MW of nameplate flexibility, 1585 MW of mean available capacity, and 1271 MW of P10 deliverable capacity. Independent storage retains 352 of 480 MW at P10 (73.3%), whereas industrial adjustable load retains 360 of 620 MW (58.1%). The corresponding P10 ratios are 63.9% for data centers, 55.7% for commercial demand response, and 56.9% for EV aggregation. Thus, treating nameplate power as firm reserve would overstate dependable capacity by 779 MW, or 38.0% of the technical total. Table 5 supplies every bar value used in the figure.

Figure 2. Flexible-resource capacity inputs for the 33-node regional case. Technical maximum is the nameplate response limit; the historical mean and P10 values are calculated from the resource-state and response records used for credible-capacity calibration.

Table 5. Numerical capacity values underlying Figure 2.

Resource Class	Technical Maximum (MW)	Historical Mean Available (MW)	P10 Deliverable (MW)	P10 Share (%)
Independent storage	480	405	352	73.3
Data center flexibility	310	250	198	63.9
Industrial adjustable load	620	452	360	58.1
Commercial demand response	280	202	156	55.7
EV aggregator	360	276	205	56.9
Total	2050	1585	1271	62.0

In Figure 3, aggregate load ranges from 480 to 602 MW, wind from 107 to 260 MW, solar from 0 to 247 MW, and net load from 100 to 386 MW. The highlighted scarcity window from Thursday 17:00 to Friday 03:00 combines a 386 MW net-load peak with declining solar output. These values are regionalized test-system inputs rather than market-clearing outputs. The caption identifies their source, resolution, aggregation level, and system size, specifying the operating conditions independently of the optimization results.

Figure 3. Seven-day operating excerpt from the 33-node regional case. The 15 min input is aggregated to 1 h for display; wind, solar, load, and net load are system-wide sums across the ten wind plants, six photovoltaic plants, and 33 load nodes.

Figure 4 reports normalized renewable forecast-error intensity by target hour and forecast lead.

Figure 4. Normalized renewable forecast-error intensity by target hour and forecast lead. Forecast–actual pairs are derived from Elexon/BMRS B1440 and B1630 records and evaluated after regionalization to the 1000 MW renewable reference.

For hour h and lead ℓ, the heatmap reports ${\mathrm{nMAE}}_{h,\mathsf{\ell }}=100{\sum }_d|p_{d,h}^{\mathrm{act}}-p_{d,h,\mathsf{\ell }}^{\mathrm{for}}|/\left(N_{h,\mathsf{\ell }}{K}^{\mathrm{ren}}\right)$ with $K^{\mathrm{ren}}=1000$ MW. The plotted leads are 1, 3, 6, 12, 24, and 36 h. nMAE rises from approximately 2–5% at short leads to 6.9–9.4% in the 17:00–21:00 band at 36 h. The maximum, 9.4%, occurs at 18:00. This calculation uses forecast–actual pairs only; no optimized dispatch or simulated price enters the heatmap. The result explains why a uniform statistical loss underweights a limited number of evening cells that create disproportionate reserve and settlement cost.

Figure 5 shows the pre-optimization deliverability landscape for heterogeneous flexible resources. Storage remains highly deliverable over much of the day but is constrained by state-of-charge dynamics, whereas data-center flexibility is more stable because workload shifting and cooling adjustment can provide moderate responses over longer periods. Industrial load has higher availability during operating hours but lower flexibility during production-sensitive intervals. Commercial demand response is most useful in afternoon and early-evening periods, and EV aggregation depends strongly on mobility and charging patterns. The matrix makes clear that a single average availability coefficient would lose important temporal information. The proposed model therefore certifies capacity at the resource-time level, allowing dispatch to select different resources depending on hour-specific deliverability, recovery constraints, and scarcity conditions.

Figure 5. Hourly deliverability probabilities used for credible-capacity certification in the regional case. Values are estimated from resource states and response-success records before optimization.

Table 6 summarizes the main notation used in the model. The notation is intentionally organized around market decisions rather than physical devices alone, because the objective is to connect technical reserve needs with market settlement. Table 7 describes the data fields and their role in the learning-based optimization pipeline. Renewable forecasts and actual generation construct deviation samples, meteorological features support probabilistic deviation learning, nodal load data identify source–load imbalance interactions, flexible-resource state data determine credible capacity, and market-clearing outputs provide labels for the decision-focused loss. Regional resilience and sector-coupling studies suggest that such multi-source data integration is necessary when power systems interact with hydrogen, transport, industrial production, and automated control layers [46].

Table 6. Main notation of the decision-focused learning-based optimization model.

Symbol	Meaning	Symbol	Meaning
$\mathcal{T}$	Dispatch intervals	$\mathcal{G}$	Renewable generators
$\mathcal{R}$	Flexible resources	$\mathcal{N}$	Network nodes
$\mathcal{S}$	Learned uncertainty scenarios	$\theta$	Parameters of the learning model
$\Delta p_{g,t}$	Renewable forecast deviation	$\Delta l_{n,t}$	Load deviation
$K_{r,t}^{\mathrm{cred}}$	Credible capacity of resource r	$k_{r,t}^{\mathrm{cert}}$	Certified reserve capacity
$u_{r,t,s}^{\mathrm{up}}$	Upward activation	$u_{r,t,s}^{\mathrm{dn}}$	Downward activation
$E_{r,t,s}$	Storage energy state	$R_{r,t,s}$	Load rebound state
$C_{t,s}^{\mathrm{imb}}$	Imbalance cost	${\Gamma }_{i,t}$	Settlement charge of participant i
${\omega }_{i,t}$	Responsibility score	${\varphi }_{i,t}$	Approximate Shapley contribution

Table 7. Data fields used in the reported experiments.

Data Category	Resolution	Source or Construction	Role in the Model
Wind and solar forecasts and actual output	15 min regional; 30 min replay	Regionalized chronology; Elexon DGWS and AGWS	Deviation features, scenario generation, and realized errors
Load	15 min regional; 30 min replay	Nodal regional profiles; Elexon ATL	Power balance, load deviation, and scarcity state
Imbalance and reserve prices	30 min	Elexon DISEBSP and PPBR	Actual-market activation, shortage, and capacity costs
Network	Static	33 nodes, 32 branches, 100-MVA base	Nodal balance, angle, and line-flow feasibility
Storage states	15 min	480 MW technical portfolio with energy and efficiency limits	Credible capacity, state of charge, and degradation
Data center workload	15 min	310 MW technical portfolio and service-level windows	Deferrable computing/cooling response
Industrial and commercial response	15 min/event	900 MW combined technical portfolio and rebound records	Interruption, fatigue, rebound, and deliverability
EV connection and mobility	15 min	360 MW aggregator portfolio	Plug-in availability, mobility energy, and activation

Table 8 reports the fixed baseline values. Sensitivity experiments vary one quantity at a time around this baseline while retaining all other values.

Table 8. Implemented baseline parameters for the regional case and actual-market replay.

Parameter	Baseline Value	Interpretation
Regional dispatch interval $\Delta t$	0.25 h	96 decisions per day
Actual-market replay interval	0.50 h	Native Elexon settlement resolution
Reserve reliability $1-\epsilon$	0.99	One-percent regional shortage-probability threshold
Regional Wasserstein radius ${ϵ}_{\theta }$	0.035	Selected from $\{0.01,0.02,0.035,0.05,0.075\}$ on validation data
Replay DRO volatility multiplier	0.40	Selected from $\{0.10,0.20,0.30,0.40,0.50\}$ on the replay validation period
Scenarios	64 train; 200 test	Common random numbers across all stochastic methods
Storage charge/discharge efficiency	0.95/0.95	Round-trip efficiency of 0.9025
Maximum data center reduction	15%	Service-quality window of 4 h
Industrial interruption limit	10%	Maximum continuous activation of 2 h
Rebound coefficients $({\phi }_r,{\zeta }_r)$	$(0.70,0.45)$	Recovery persistence and activation-to-rebound conversion
Regional ${\pi }^{\mathrm{VOLL}}$/${\pi }^{\mathrm{cur}}$	7500/65 USD/MWh	Shortage and renewable-curtailment penalties
Replay shortage/curtailment adder	750/65 GBP/MWh	Added to recorded Elexon system price in the external replay
Shapley sampling	500 permutations	Antithetic orderings and common scenario draws

Every benchmark receives identical chronological splits, scenarios, network limits, resource availability, price inputs, and feasibility restoration. Table 9 supplies the formulation and numerical settings required to reproduce each comparison. Reinforcement learning and graph-based dispatch are included because they represent current data-driven alternatives to repeated mathematical optimization [6,7].

Table 9. Reproducible benchmark formulations and settings.

Benchmark	Formulation	Implemented Settings
Fixed-ratio reserve	$Q_t^{\mathrm{up}}=\max \{0.15K^{\mathrm{ren}},0.32p_t^{\mathrm{for}}+0.08L_t^{\mathrm{for}}\}$ followed by the same feasible redispatch	No learned uncertainty; resource limits and network constraints retained
Forecast-then-optimize	Quantile network minimizes Equation (9); the 0.99 quantile enters Equations (20)–(42) without decision gradients	Same 64/200 scenarios, network, cost, and solver tolerances
Wasserstein DRO	Empirical forecast model followed by Equation (42); radius fixed before dispatch	Regional radius 0.035; replay volatility multiplier 0.40; identical reliability target
Off-policy RL	State: time, lagged price, forecast renewable/load, and error volatility; action: reserve volume; reward: negative balancing cost	27 actions from 0 to 650 MW in 25-MW steps; $\gamma =0.95$; 60 fitted-Q sweeps; reliability projection before redispatch
Graph-BC dispatch	One normalized graph-message-passing layer over five resource classes followed by behavioral cloning of optimizer allocations	MLP 64/32; Adam ${10}^{-3}$; at most 200 epochs; capacity clipping and network-feasibility projection
Proposed DFL optimization	Composite loss in Equation (6) differentiated through certified-capacity and redispatch constraints	Parameters in Table 8; all four loss weights and radius fixed after validation

Both experiments use chronological rather than random splits. The regional case trains on January–August, validates all hyperparameters on September–October, and reports November–December only once. The external replay trains on 9 October–15 November, validates on 16–30 November, and tests on December 2025. Missing or duplicate records are resolved before splitting; no later observation is used to impute an earlier period. Negative system prices and the highest 1% of imbalance prices remain in the data.

The regional sensitivity grid covers reliability levels $\{0.95,0.97,0.99,0.995\}$, Wasserstein radii $\{0.01,0.02,0.035,0.05,0.075\}$, flexible-resource participation from 0 to 100% in 10-percentage-point increments, reserve margins from 0 to 30% in 2-percentage-point increments, and renewable penetration from 30 to 80%. The replay DRO multiplier is selected separately from $\{0.10,0.20,0.30,0.40,0.50\}$. Every baseline is fixed before test evaluation. Weekly block bootstrap with 1000 resamples provides the 95% intervals reported for renewable accommodation and actual-market cost.

For the external replay, the reserve head is represented by an auditable 288-state table: six four-hour blocks, renewable-forecast quartiles, one-day-lagged price quartiles, and lagged-error-volatility tertiles. In each state, the proposed policy selects one of 66 actions from 0 to 650 MW, in 10 MW increments, by minimizing price-weighted balancing cost subject to a 0.5% in-state shortage-frequency limit; states with fewer than 25 training observations use the global 0.99 need quantile. This replay-specific compression retains the decision-focused objective while isolating reserve-policy transfer from the full regional nodal and settlement model.

Four ablations were executed with all other settings unchanged: removal of the decision-focused loss, removal of credible-capacity certification, removal of responsibility-aware settlement, and the complete forecast-then-optimize pipeline. A separate rebound ablation confirmed that omitting the recovery load understates the commercial and industrial activation cost by 6.4%; because it changes a device constraint rather than a principal contribution, that result is reported only in the supplementary output.

The reproducible workflow aligns source timestamps, constructs deviations, estimates resource deliverability, trains the quantile and deliverability heads, solves day-ahead and scenario redispatch, and then performs sampled counterfactual settlement. The accompanying script downloads Elexon records without an API key, applies the stated chronological split, reproduces every row of the actual-market table, and exports interval-level reserve, cost, and shortage values.

7. Numerical Results and Discussion

Results are reported in the order of learning validation, reserve clearing, cost decomposition, reliability, capacity certification, settlement, dispatch behavior, sensitivity analysis, computational tractability, and ablation evidence. Regional results use only the November–December test period, and the external replay uses only the December 2025 test period. Unless otherwise stated, costs are totals over the corresponding test set, and probabilities are empirical interval frequencies.

Figure 6 validates the central claim that the proposed learning module is trained for downstream decisions rather than for forecast accuracy alone. The decision-focused training loss declines sharply during the initial epochs and continues to fall, while the validation balancing cost also decreases steadily. The forecast-then-optimize benchmark improves more slowly and remains substantially above the decision-focused validation cost at convergence. The reported 29.2% lower validation cost shows that the model learns the forecast-error features that are most consequential for balancing decisions, not merely the features that reduce average statistical error. Calibration loss decreases at a more moderate rate, which is expected because calibration is retained as a reliability-preserving auxiliary objective. The figure demonstrates that decision-focused learning changes the economic value of the forecast representation.

Figure 6. Convergence of decision-focused learning and its validation decision-cost advantage over forecast-then-optimize training.

Figure 7 compares reserve procurement, certified flexible capacity, unused reserve, and shortage outcomes across the four clearing methods shown in the figure. The deterministic reserve rule procures a large amount of reserve because it does not adapt to learned uncertainty or resource deliverability. Forecast-then-optimize improves this outcome, but its separate forecast and dispatch stages retain avoidable unused capacity. Wasserstein DRO protects reliability at the cost of additional procurement. The decision-focused model maintains sufficient certified capacity while reducing unused reserve. Table 10 extends the same test to the two recent data-driven comparators and reports all principal regional metrics.

Figure 7. Reserve procurement and certified flexible capacity under competing market-clearing methods.

Table 10. Regional test-set comparison at the 60% renewable-penetration baseline. Cost is the total November–December balancing cost; solution time is the mean rolling-horizon clearing time for 200 flexible assets.

Method	Cost (Thousand USD)	Accommodation (%)	Shortage (%)	Fairness	Time (s)
Fixed-ratio reserve	950	93.6	2.9	0.61	23.4
Forecast-then-optimize	800	95.9	1.6	0.81	37.8
Wasserstein DRO	745	96.8	0.9	0.85	66.1
Off-policy RL	701	97.1	1.2	0.80	9.6
Graph-BC dispatch	659	97.4	0.9	0.86	7.4
Proposed DFL optimization	618	97.8	0.7	0.92	43.7

The proposed model lowers the regional balancing cost by 34.9% relative to the fixed-ratio rule and by 6.2% relative to Graph-BC, the lowest-cost advanced comparator. Off-policy RL and Graph-BC clear faster because their trained policies replace scenario optimization at inference, but both require feasibility projection and neither jointly optimizes settlement fairness. The proposed method has the lowest shortage probability and the highest renewable accommodation and fairness index while remaining below the 15 min dispatch interval.

Figure 8 decomposes the total regional balancing cost into capacity procurement, activation, curtailment, shortage, and settlement uplift. Table 11 gives the numerical values behind the bars. The deterministic rule produces the highest procurement and shortage expenditures. Wasserstein DRO reduces shortage cost but retains conservative capacity procurement. The proposed method reduces total cost from 950 to 618 thousand USD; 77.4% of the 332-thousand-USD saving comes from capacity, curtailment, and shortage components. Its shortage cost remains positive, confirming that the saving is not created by suppressing the value of lost load or omitting feasibility restoration.

Figure 8. Regional balancing-cost decomposition obtained from the network-constrained simulation at 15 min resolution. Each component is computed from the cleared quantities and the fixed prices in Table 8; the bars are simulated market outcomes rather than measured Elexon costs.

Table 11. Numerical values underlying Figure 8 (thousand USD).

Method	Capacity	Activation	Curtailment	Shortage	Uplift	Total
Fixed-ratio reserve	410	230	180	85	45	950
Forecast-then-optimize	360	205	155	45	35	800
Wasserstein DRO	390	190	110	20	35	745
Proposed DFL optimization	335	170	65	18	30	618

Figure 9 evaluates accommodation as renewable penetration rises from 30% to 80%. Table 12 reports the plotted means and uncertainty intervals. Accommodation declines for all methods as penetration rises, but the proposed method remains above 95% at 80% penetration. At that endpoint, it exceeds fixed-ratio reserve by 9.9 percentage points and Wasserstein DRO by 2.7 percentage points. The non-overlapping pointwise intervals between the proposed and fixed-ratio methods at 50–80% penetration show that this ordering is not driven by a single test week.

Figure 9. Regional renewable accommodation under increasing renewable penetration. Lines show test-set means; shaded bands are 95% weekly block-bootstrap intervals from 1000 resamples.

Table 12. Numerical values underlying Figure 9: mean renewable accommodation in percent, with the 95% block-bootstrap half-width after “±”.

Method	30%	40%	50%	60%	70%	80%
Fixed-ratio reserve	98.6 ± 0.3	97.4 ± 0.4	94.7 ± 0.6	91.4 ± 0.8	88.5 ± 1.0	85.7 ± 1.2
Forecast-then-optimize	98.8 ± 0.3	98.0 ± 0.3	96.0 ± 0.5	94.0 ± 0.7	92.0 ± 0.8	90.3 ± 1.0
Wasserstein DRO	98.9 ± 0.2	98.3 ± 0.3	96.9 ± 0.4	95.6 ± 0.5	94.2 ± 0.6	92.9 ± 0.8
Proposed DFL optimization	99.0 ± 0.2	98.8 ± 0.2	98.0 ± 0.3	97.3 ± 0.4	96.5 ± 0.5	95.6 ± 0.6

Figure 10 maps shortage probability over the complete participation and reserve-margin grid. To remain at or below the 1% threshold, the required reserve margin is approximately 20% with no flexible-resource participation, 12% at 50% participation, and 4% at full participation. The bottom-left region violates the threshold because both sources of corrective capacity are scarce. These coordinates make the substitution relationship operational: a 50-percentage-point increase from zero participation lowers the required reserve margin by about eight percentage points in this case.

Figure 10. Regional reserve-shortage probability across flexible-resource participation (0–100%) and reserve margin (0–30%). The contour denotes the fixed 1% reliability threshold used in the baseline.

Figure 11 evaluates the accuracy and conservatism of the credible-capacity certification procedure. For every resource class, technical maximum capacity exceeds certified credible capacity, confirming that the model does not treat nominal flexibility as firm reserve. Realized delivered capacity is close to certified capacity, indicating that certification is not excessively conservative. Storage has the highest activation success rate, while commercial demand response has the lowest because customer response and recovery behavior are more uncertain. Industrial load has the largest technical potential but a more moderate certified share because process constraints limit dependable activation. The result is operationally important: over-certification would create reserve shortage, while under-certification would waste usable flexibility. The proposed certification model achieves a practical middle ground by aligning reserve offers with historical deliverability and activation success.

Figure 11. Technical, certified, and delivered flexible capacity by resource class.

Figure 12 shows how responsibility-aware settlement changes the distribution of imbalance costs. The deterministic rule produces the highest total allocated cost and the largest socialized uplift, indicating that a substantial portion of balancing cost cannot be attributed accurately under a fixed rule. Forecast-then-optimize and robust optimization reduce total cost but still retain nontrivial uplift because their settlement layers are not fully coupled with learned marginal contribution tracing. The decision-focused optimization reduces the total allocated imbalance cost to 72 thousand USD and lowers socialized uplift to 5 thousand USD. Renewable generators still bear the largest share because renewable underproduction is a primary driver of upward reserve needs, but load-side participants also receive cost responsibility when their deviations worsen scarcity. This outcome is consistent with the principle that costly imbalance should be allocated to its marginal drivers.

Figure 12. Responsibility-aware allocation of imbalance costs across renewable generators, load-side resources, and socialized uplift.

Figure 13 reports the optimized hourly activation of flexible resources under the proposed model. Storage is activated most strongly during morning and evening ramp intervals because it provides fast upward reserve and can respond to short-duration scarcity. Data-center flexibility supplies a more stable background contribution, reflecting its ability to adjust computational and cooling loads without major service interruptions. Industrial load activation is concentrated during business hours when process flexibility is available and economically efficient. Commercial demand response becomes more important during afternoon and early-evening conditions, while EV aggregation peaks during the evening ramp when plug-in availability increases. The allocation is physically intuitive and economically meaningful: fast and energy-limited resources are reserved for ramping periods, while slower or more constrained resources are used when their operational limits permit. This confirms that the dispatch layer uses resource heterogeneity rather than aggregating all flexibility into a single reserve block.

Figure 13. Hourly activation pattern of flexible-resource dispatch under the proposed decision-focused optimization model.

Figure 14 presents a three-dimensional sensitivity surface for balancing cost under different forecast-error severity and reserve-margin settings. The surface exhibits a well-defined cost-efficient region rather than a monotonic relationship. Very low reserve margins expose the system to shortage and emergency activation costs, while excessively high margins increase procurement and opportunity costs. Forecast-error severity shifts the surface upward because larger deviations require more reserve and increase the probability of expensive activation. The valley region indicates the point at which the reserve margin is sufficient to protect reliability without creating excessive unused capacity. This result is valuable for market design because it converts abstract uncertainty into an operational reserve-sizing rule. It also explains why the proposed decision-focused model performs better than fixed rules: it learns where the cost valley lies under changing forecast-error conditions.

Figure 14. Balancing-cost surface under forecast-error severity and reserve-margin variation.

Figure 15 shows how renewable accommodation depends jointly on flexible-resource participation and renewable penetration. At low flexible participation, accommodation remains limited because the system cannot absorb large renewable deviations without curtailment or shortage. As flexible-resource participation rises, the accommodation surface increases and approaches a high-performance plateau. The surface also shows that very high renewable penetration creates additional stress, but sufficient flexible capacity substantially mitigates this stress. The high-accommodation region is therefore not achieved by renewable expansion alone; it requires coordinated storage, demand response, data center flexibility, industrial load adjustment, and EV aggregation. This figure provides an intuitive system-level explanation for the market mechanism: flexible resources raise the feasible renewable share by expanding the set of balancing actions available to the operator.

Figure 15. Renewable accommodation surface under flexible-resource participation and renewable penetration.

Figure 16 evaluates computational tractability as the number of flexible assets increases. Deterministic rules solve fastest because they ignore most uncertainty and resource heterogeneity, but their operational quality is inferior. Forecast-then-optimize remains computationally efficient, yet it does not train predictions according to downstream cost. Distributionally robust optimization becomes the slowest method as system size increases because it must protect against uncertainty sets across many assets and constraints. The decision-focused model has a higher computational burden than simple forecast-then-optimize but remains tractable, reaching approximately 118 seconds at 500 flexible assets in the representative case. This scalability is important for regional reserve markets because practical implementation requires repeated clearing over rolling time horizons. The result indicates that decision-focused learning can provide better economic and reliability performance without creating an impractical solution time.

Figure 16. Mean regional rolling-horizon solution time as the number of flexible assets increases, measured on the platform specified in Section 6.

Figure 17 identifies which model components drive performance. The full model achieves the best joint performance, with 18.6% balancing-cost reduction, 97.8% renewable accommodation, 0.7% shortage probability, and a fairness index of 0.92. Removing decision-focused loss sharply reduces cost improvement because the learning module no longer prioritizes forecast-error components that matter most for dispatch. Removing credible-capacity certification increases the shortage probability, showing that availability learning is essential for reliable reserve commitments. Removing responsibility-aware settlement harms the fairness index, confirming that cost allocation requires explicit marginal-contribution tracing rather than only optimized dispatch. The forecast-then-optimize baseline is weaker across the performance set. The ablation evidence therefore supports the integrated design: learning, optimization, capacity certification, and settlement must be co-designed to achieve economic efficiency, renewable accommodation, reliability, and fairness.

Figure 17. Ablation study of the decision-focused loss, credible-capacity certification, and responsibility-aware settlement modules.

External Market Replay

Table 13 reports the held-out external replay. The input consists of 4027 complete Great Britain settlement periods after the seven-day forecast-matching lag; 1821 periods are used for training, 719 for validation, and the 1487 December 2025 periods for testing. Forecast renewable output, realized renewable output, total load, imbalance price, and reserve price are recorded Elexon values. Each policy selects an upward-reserve volume before the realized deviation is revealed; settlement cost is then evaluated against the recorded system price with identical shortage and curtailment adders. Thus, the table is a market-data replay rather than a claim that the proposed policy was historically cleared by the system operator.

Table 13. External replay on 1487 held-out Great Britain settlement periods in December 2025. Cost and its weekly block-bootstrap 95% interval are in thousand GBP; reserve is the mean procured volume; P95 cost is the 95th percentile of interval balancing cost.

Method	Cost 95% Interval	Reserve (MW)	Shortage (%)	Accommodation (%)	P95 Cost (GBP)
Fixed-ratio reserve	2953.3 [2205.7–3840.8]	225.0	4.64	99.21	4737.5
Forecast-then-optimize	3591.4 [2334.5–5190.7]	263.9	5.92	99.51	6364.4
Wasserstein DRO	3034.5 [2172.3–4076.0]	292.9	3.23	99.63	3811.8
Off-policy RL	3291.9 [2230.0–4524.9]	288.1	4.30	99.58	4436.9
Graph-BC dispatch	2663.5 [2062.4–3413.0]	302.7	1.61	99.70	3689.0
Proposed DFL optimization	2598.2 [2092.4–3143.0]	317.6	1.28	99.74	3653.5

The proposed policy reduces the replay cost by 12.0% relative to fixed-ratio reserve and by 2.5% relative to Graph-BC, while procuring more mean reserve than either comparator. The additional reserve is targeted to intervals with high downside exposure: shortage frequency falls from 4.64% under the fixed rule and 1.61% under Graph-BC to 1.28%, and accommodation reaches 99.74%. Forecast-then-optimize incurs the highest replay cost and shortage frequency because forecast-loss calibration does not protect decisions during the largest price–deviation combinations. The wide weekly cost intervals reflect genuine price clustering; they delimit sampling uncertainty rather than implying significance from aggregate totals alone. The consistent ranking of the proposed method in the regional test and the independent system-level replay supports transfer of the reserve-learning mechanism, while participant-level settlement remains evaluated only in the regional network case.

8. Conclusions

This study contributes a coupled learning, dispatch, and settlement formulation in which the uncertainty representation is differentiated through reserve procurement and redispatch, flexible-resource offers are limited by learned deliverability, and the resulting balancing requirement is assigned by sampled counterfactual marginal contribution under an exact revenue-adequacy identity. The quantile loss, Wasserstein ambiguity set, chance constraint, and Shapley value are established tools; the mathematical contribution is their coupling through the composite decision loss, the credible-capacity interface, and the charge identity, rather than a new definition of any one component.

In the 33-node regional test with 15 min intervals, the proposed method reduces the balancing cost from 950 to 618 thousand USD relative to fixed-ratio reserve, increases renewable accommodation from 93.6% to 97.8%, and reduces shortage frequency from 2.9% to 0.7%. It also outperforms off-policy RL and graph-based behavioral cloning in terms of cost, shortage, accommodation, and settlement fairness, although those policy approximations solve faster. Sampled Shapley settlement reduces socialized uplift to 5 thousand USD, maintains a fairness index of 0.92, and remains computationally feasible at 500 flexible assets. The ablations attribute the reliability gain to credible-capacity certification and the allocation gain to responsibility-aware settlement.

The external replay supplies a separate market-level check using recorded Great Britain forecasts, renewable output, load, system prices, and reserve prices. Across 1487 held-out December 2025 periods, the proposed policy reduces cost from 2953.3 to 2598.2 thousand GBP relative to fixed-ratio reserve, lowers shortage frequency from 4.64% to 1.28%, and achieves 99.74% renewable accommodation. These findings establish performance for the two evaluated settings: a network-constrained regional simulation and a system-level historical replay. The replay does not establish participant-level settlement behavior, nodal feasibility, or bid response in the Great Britain market, and the regional engineering case is not a substitute for a live market trial. Deployment in another jurisdiction therefore requires local network, bid, penalty, and resource-response calibration. Within those boundaries, the results support decision-focused reserve learning and deliverability-aware dispatch as testable mechanisms for renewable-rich balancing markets.