Improved Reinforcement Learning for Multi-Objective Optimization Operation of Cascade Reservoir System Based on Monotonic Property

Abstract

In this paper, improved reinforcement learning (IRL) is designed for the multi-objective optimization operation of a cascade reservoir system. The primary improvement of IRL is searching within limited solution space, based on the derived monotonic property: the first-order derivative relationship between individual reservoir water release decisions for mainstream use (i.e., hydropower generation) as well as tributary use (i.e., regional water supply) and the cascade system’s or a particular reservoir’s water availability, along with the synchronicity and substitutability assumption of storage distribution in the cascade system. The improved algorithm is then applied to a real-world cascade reservoir system in the Yangtze River of China. The results demonstrate the high computational efficiency and reasonable interpretability of IRL.

1. Introduction

The operation of multi-reservoir systems is essential for regulating the spatial and temporal distribution of water resources to satisfy multiple water use needs, such as hydropower generation and regional water demand [1,2]. However, the optimization of the coordinated operation of a complex multi-reservoir system, particular for reservoirs in series, is always challenged by the high-dimensional decision-making process, nonlinear system dynamics, and stochasticity of runoff [1,2,3]. Various optimization techniques are proposed to address the above challenges, as can be found from the state-of-the-art reviews of Yeh [1], Wurbs [4], Labadie [5], Rani and Moreira [6], Dobson [7] and Giuliani et al [8].

According to Dobson et al. [7], optimization methods for multi-reservoir system operation can be categorized into release sequences, operating policy, and real-time optimization argument. Among these three categories, only the value function estimation based on Markov Decision Processes (MDP) can explicitly capture the dynamic nature of the optimization problem as well as the stochasticity of inflow by decomposing the complex optimization issues into a sequence of sub-problems. As one of the most famous optimization algorithms based on MDP, stochastic dynamic programming (SDP) has been widely used to solve reservoir optimization operation problems, since it can handle the nonlinear characteristics of the objective and constraints [1,5,9]. However, the applicability of SDP is restricted to only a limited number of individual reservoirs, due to the so-called curse of dimensionality, i.e., the exponential increase in computing time and memory requirements as the number of individual reservoirs increases. Various improvements to SDP have been proposed to achieve controlled computational efficiency [10,11]. For example, Tilmant and Kelman [12] introduced Successive Approximation Dynamic Programming (DPSA), which decomposes multi-reservoir operation problems into a series of single reservoir operation problems; Howson and Sancho [13] developed Progressive Optimization Algorithm (POA), transforming the multi-stage operation problem into multiple one-stage operation problems; Turgeon [14] and Turgeon and Charbonneau [15] proposed aggregation–disaggregation methods to reduce the high dimensionality of state variables. Although the computational complexity of SDP can be significantly reduced, a severe loss of solution accuracy might result from those improvements [1,16,17,18,19].

As the other optimization method based on MDP, reinforcement learning (RL) integrates the trial-and-error learning mechanisms with Bellman’s principle of optimality [20,21,22,23] to approximate the optimal solutions. As one of the most popular RL methods for solving Markov decision problems, the Q-learning method [11] employs a depth-first search strategy and incrementally optimizes the visited state variables [24], which is different from enumeration over all discrete combinations of state variables to develop a value function and derive the optimal decisions in SDP. Based on this high efficiency computational advantage, Q-learning is widely adopted to solve single-reservoir and multi-reservoir optimization operation problems [24,25,26]. For example, Xu et al. [26] suggested a stochastic optimization method with the integration of Deep Q-Networks (DQN) with aggregation–disaggregation methods for cascade reservoir system optimization operation. However, the limitation of converging into a local optimal solution, as well as a relatively low interpretability of the decision-making process, needs to be further addressed for Q-learning applications in complex reservoir systems [27,28].

Recently, the use of mathematic characteristics of a reservoir system, e.g., the relationships between the decision variables and state variables [17,29,30], has been proposed to improve the MDP by achieving a high level of computational efficiency and effectiveness. For example, Zhao et al. [29,30,31] proposed improved SDP with a monotonic relationship between the decision variable and state variable for a single reservoir water supply, hydropower generation and flood control operation. Using the spatially distributed characteristics of individual reservoir storage, Zeng et al. [17] proposed an improved SDP algorithm based on a derived monotonic relationship between individual reservoir carryover storage and parallel reservoir system water availability. The high computational efficiency and effectiveness with controlled solution accuracy have been proven in a real case. However, the application of the monotonic decision property to overcome the dimensionality problem of cascade reservoir optimization operation has received little attention.

Following the work of Zeng et al. [17], the objective of this paper is to reduce the computational complexity of RL with a derived monotonic relationship between individual reservoir water release decisions and the water availability of the cascade system or a specific individual reservoir, which provides the theoretical basis for the improved RL algorithm design. According to our knowledge, this study can be the first to apply the monotonic property to address the curse of dimensionality for cascade reservoir optimization operation.

The rest of this paper is organized with a step-by-step structure as follows: Section 2 presents the formulation of the stochastic dynamic optimization model for the operation of reservoirs in series and specifies the optimality condition for the model without binding inequality constraints. On the basis of the specified optimality condition, Section 3 provides the first-order derivative relationship between individual reservoir water release decisions and the water availability of the cascade system or a specific individual reservoir. The monotonic property with reduced searching region of the improved RL method is described in Section 4. A cascade reservoir system in the Yangtze River of China is illustrated as a case study in Section 5. Finally, the conclusions of this paper are given in Section 6.

2. Optimality Condition for Multi-Objective Operation of Cascade Reservoir System

This section first describes a long-term optimization problem for the multi-objective operation of the cascade reservoir system, then develops a two-stage optimization operation model based on the stochastic dynamic programming and reinforcement learning methods. Following that, the optimality condition for the multi-objective operation of the cascade reservoir system is derived.

2.1. Stochastic Optimization Operation Model for Cascade Reservoir System

Considering the stochasticity of hydrologic inflow, the long-term operation of the cascade reservoir system illustrated in Figure 1 requires balancing water release for hydropower generation in the mainstream and water release for regional use in tributaries at each period. The objective is to maximize the expected value of multi-regional basin-wide benefits, integrating diverse water use purposes, such as ecology and hydropower production. In addition, the decision of water release at each period needs to obey the physical and engineering constraints of the cascade reservoir system, which includes the mass balance equation, release, and carryover storage constraints. The mathematical expression of the long-term multi-objective optimization operation for the cascade reservoir system can be written as follows:

$$\underset{\mathit{IR},\mathit{OR}}{\max } \underset{I}{E}〈{\sum }_{t=1}^T\left\{{\sum }_{i=1}^n\left[B^i\left({IR}_{i,t}\right)+B_i\left({OR}_{i,t}\right)\right]\right\}〉$$

(1a)

$$s.t. \left\{\begin{array}{c}{WA}_{1,t}=S_{1,t-1}+I_{1,t}-E_{1,t}=S_{1,t}+I{R}_{1,t}+{OR}_{1,t} \forall t\\ \begin{array}{c}{ WA}_{i,t}=S_{i,t-1}+I_{i,t}-E_{i,t}=S_{i,t}+I{R}_{i,t}+{OR}_{i,t}-{IR}_{i-1,t} \forall i\in \left[2,n\right],t\\ \begin{array}{c} 0\le {IR}_{i,t}\le {ID}_{i,t} \forall i\in \left[1,n\right],t\\ 0\le {OR}_{i,t}\le {OD}_{i,t} \forall i\in \left[1,n\right],t\end{array}\end{array}\\ S_{i,t}^{min}\le S_{i,t}\le S_{i,t}^{max} \forall i\in \left[1,n\right],t\end{array}\right.$$

(1b)

where $E\left( \right)$ is the expectation function; $B^i\left( \right)$ is the benefit function of in-stream water release from i reservoir to meet the hydropower generation or ecological water use [32,33,34,35]; $B_i\left( \right)$ is the benefit function of out-stream water release from i reservoir to meet the regional consumptive water demand; particularly, both $B^i\left( \right)$ and $B_i\left( \right)$ are assumed to be concave [16,32,33,34,36]; $T$ is the planning horizon of the cascade reservoir system; n is the number of reservoirs in the cascade system; $W{A}_{i,t}$ is the water availability of i reservoir at time t, which is defined as the sum of initial water storage plus current inflow and minus evaporation loss during this time period for i reservoir; $S_{i,t-1}$ is the initial storage of the reservoir i at the beginning of period t; $I_{i,t}$ is the inflow of i reservoir at period t; $E_{i,t}$ is the evaporation loss of i reservoir at period t; ${IR}_{i,t}$ is the water release for hydropower generation from i reservoir into mainstream at period t; ${ID}_{i,t}$ is the water demand from mainstream at period t; ${OR}_{i,t}$ is the water supply for tributary water user at period t; ${OD}_{i,t}$ is the tributary water demand of i reservoir at period t; $S_{i,t}^{min}$ and $S_{i,t}^{max}$ are the minimum and maximum storage of i reservoir, respectively.

2.2. Stochastic Dynamic Programming for System Optimization Operation

The aforementioned long-term optimization model encompasses a substantial number of water release decisions for each reservoir, and deriving optimal solutions for such an extensive nonlinear optimization model poses significant challenges [1,2,5]. Based on the optimality principle of dynamic programming, the long-term optimization problem can be decomposed into a series of two-stage stochastic optimization problems. If the runoff of each reservoir can be described as a first-order Markov process, then the randomness of the inflow at the subsequent time t + 1 can be characterized using a prior inflow transition probability $P\left(I_{t+1}\left|I_t\right.\right)$ given any specific $I_{i,t}$. Under this assumption, the long-term stochastic optimization model can be expressed using the following backward recursive equation:

$${CV}_{t-1}\left(S_{t-1},I_t\right)=\underset{{\mathit{IR}}_t,{\mathit{OR}}_t}{\max } \left\{B_t\left({IR}_t,{OR}_t\right)+{\sum }_{I_{t+1}}\left[P\left(I_{t+1}\left|I_t\right.\right)\times {CV}_t\left(S_t,I_{t+1}\right)\right]\right\}$$

(2a)

$$B_t\left({IR}_t,{OR}_t\right)={\sum }_{i=1}^n\left[B^i\left({IR}_{i,t}\right)+B_i\left({OR}_{i,t}\right)\right]$$

(2b)

$$S_{t-1}=\left[\begin{array}{c}{S}_{1, t-1}\\ ⋮\\ S_{n, t-1}\end{array}\right]; I_t=\left[\begin{array}{c}{I}_{1,t}\\ ⋮\\ I_{n,t}\end{array}\right] ; {IR}_t=\left[\begin{array}{c}{IR}_{1,t}\\ ⋮\\ {IR}_{n,t}\end{array}\right]; {OR}_t=\left[\begin{array}{c}{OR}_{1,t}\\ ⋮\\ {OR}_{n,t}\end{array}\right]$$

(2c)

where $S_{t-1}$, $I_t$, ${IR}_t$ and ${OR}_t$ represent the vector of the initial storage, inflow, and water release in the mainstream and tributary of the cascade reservoir system at period t, respectively; $CV\left( \right)$ is the value function of the cascade reservoir system at period t, representing the maximum benefit that can be achieved from period t to the end of the planning horizon, conditioned on the inflow and storage state combination. Previous studies [35,37] have shown that because the value function represents the cumulative utilities from period t to T, it is considered to share similar mathematical characteristics with the benefit function [29,31]. In other words, the value function $CV\left( \right)$ is concave.

To maintain consistency with the mathematical expression of the benefit function used in the reinforcement learning method, Equation (2a) is rewritten with the Q-value function as follows:

$$Q_{t-1}\left(S_{t-1},I_t,{IR}_t,{OR}_t\right)=B_t\left({IR}_t,{OR}_t\right)+{\sum }_{I_{t+1}}\left[P\left(I_{t+1}\left|I_t\right.\right)\times \underset{{\mathit{IR}}_{t+1},{\mathit{OR}}_{t+1}}{\max }\left[Q_t\left(S_t,I_{t+1},{IR}_{t+1},{OR}_{t+1}\right)\right]\right]$$

(3a)

$${CV}_{t-1}\left(S_{t-1},I_t\right)=\underset{{IR}_t,{OR}_t}{\mathrm{m}\mathrm{a}\mathrm{x}}\left[Q_{t-1}\left(S_{t-1},I_t,{IR}_t,{OR}_t\right)\right]$$

(3b)

where $Q\left( \right)$ is the approximate function of $CV\left( \right)$ in the reinforcement learning method.

2.3. Reinforcement Learning for System Optimization Operation

In order to construct the value function of the cascade reservoir system, the dynamic programming method based on the Bellman equation requires perfect information of the underlying stochastic hydrologic processes throughout the entire planning horizon. The well-known “curse of dimensionality” problem is caused by this assumption, even though it can capture the best solutions for long-term operation. To make efficient use of the searching process and reduce computational complexity, the Q-learning method in reinforcement learning adopts the concept of random sampling to gradually approximate the value function through incremental temporal difference methods. The multi-objective optimization model of the cascade reservoir system in the Q-learning approach can be written as follows:

$$\begin{gathered} Q_{t-1}\left(S_{t-1},I_t,{IR}_t,{OR}_t\right)\leftarrow Q_{t-1}\left(S_{t-1},I_t,{IR}_t,{OR}_t\right)+\alpha \left\{B_t\left({IR}_t,{OR}_t\right)+ \\ \underset{{\mathit{IR}}_{t+1},{\mathit{OR}}_{t+1}}{\max }\left[Q_t\left(S_t,I_{t+1},{\mathit{IR}}_{t+1},{\mathit{OR}}_{t+1}\right)\right]-Q_{t-1}\left(S_{t-1},I_t,{IR}_t,{OR}_t\right)\right\} \end{gathered}$$

(4)

where $\alpha$ is the learning rate parameter controlling the balance given to the immediate water use benefit and future value return.

Notably, the Q-learning approach only carries out incremental optimization for the sample state combination, as opposed to exhaustively listing every discrete combination of carryover storage at each period of SDP. This paper specifically assumes that the immediate water use benefit ($B^i\left( \right)$ and $B_i\left( \right)$) is provided at each period. Therefore, for any state combination, the convergence condition of the Q-value function is:

$$Q_{t-1}\left(S_{t-1},I_t,{IR}_t,{OR}_t\right)\leftarrow B_t\left({IR}_t,{OR}_t\right)+\underset{{\mathit{IR}}_{t+1},{\mathit{OR}}_{t+1}}{\max }\left[Q_t\left(S_t,I_{t+1},{IR}_{t+1},{OR}_{t+1}\right)\right]$$

(5)

Combining the above equation with Equation (3b), we have:

$${CV}_{t-1}\left(S_{t-1},I_t\right)=\underset{{\mathit{IR}}_t,{\mathit{OR}}_t}{\max } \left\{B_t\left({IR}_t,{OR}_t\right)+\underset{{\mathit{IR}}_{t+1},{\mathit{OR}}_{t+1}}{\max }\left[Q_t\left(S_t,I_{t+1},{IR}_{t+1},{OR}_{t+1}\right)\right]\right\}$$

(6)

According to the equation above, “Q-Learning converges to the optimum action values with probability 1 so long as all actions are repeatedly sampled in all states and the action values are represented discretely” [21] (p. 279). Following this, the optimality condition of Q-learning and dynamic programming methods is equivalent, and the computational complexity of Q-learning can be significantly lower.

2.4. The Optimality Principle of Equal Marginal Utility

If boundary constraints on release and carryover storage (i.e., Equation (1b)) are non-binding, the Lagrange multiplier approach [36] can be used to specify the optimality condition of the two-stage optimization model with the mass balance equation mentioned above, which leads to:

$$\left\{\begin{array}{c}{\displaystyle \frac{d{B}^i}{d{IR}_{i,t}^{*}}}+{\displaystyle \frac{\partial Q_t}{\partial S_{i,t}^{*}}}\times {\displaystyle \frac{\partial S_{i,t}^{*}}{\partial {IR}_{i,t}^{*}}}+{\displaystyle \frac{\partial Q_t}{\partial S_{i+1,t}^{*}}}\times {\displaystyle \frac{\partial S_{i+1,t}^{*}}{\partial {IR}_{i,t}^{*}}}=0\\ {\displaystyle \frac{d{B}_i}{d{OR}_{i,t}^{*}}}+{\displaystyle \frac{\partial Q_t}{\partial S_{i,t}^{*}}}\times {\displaystyle \frac{\partial S_{i,t}^{*}}{\partial {OR}_{i,t}^{*}}}=0\end{array}\right.\to \left\{\begin{array}{c}{\displaystyle \frac{d{B}^i}{d{IR}_{i,t}^{*}}}={\displaystyle \frac{\partial Q_t}{\partial S_{i,t}^{*}}}-{\displaystyle \frac{\partial Q_t}{\partial S_{i+1,t}^{*}}}\\ {\displaystyle \frac{d{B}_i}{d{OR}_{i,t}^{*}}}={\displaystyle \frac{\partial Q_t}{\partial S_{i,t}^{*}}}\end{array}\right. \forall i\ne n$$

(7a)

$$\left\{\begin{array}{c}{\displaystyle \frac{d{B}^n}{d{IR}_{n,t}^{*}}}+{\displaystyle \frac{\partial Q_t}{\partial S_{n,t}^{*}}}\times {\displaystyle \frac{\partial S_{n,t}^{*}}{\partial {IR}_{n,t}^{*}}}=0\\ {\displaystyle \frac{d{B}_n}{d{OR}_{n,t}^{*}}}+{\displaystyle \frac{\partial Q_t}{\partial S_{n,t}^{*}}}\times {\displaystyle \frac{\partial S_{n,t}^{*}}{\partial {OR}_{n,t}^{*}}}=0\end{array}\right. \to \left\{\begin{array}{c}{\displaystyle \frac{d{B}^n}{d{IR}_{n,t}^{*}}}={\displaystyle \frac{\partial Q_t}{\partial S_{n,t}^{*}}}\\ {\displaystyle \frac{d{B}_n}{d{OR}_{n,t}^{*}}}={\displaystyle \frac{\partial Q_t}{\partial S_{n,t}^{*}}}\end{array}\right.\forall i=n$$

(7b)

The aforementioned formulas illustrate the equal marginal utility optimality principle for the multi-objective optimization operation for the cascade reservoir system. That is, (1) for the individual reservoir not at the lowest elevation $\left(i\ne n\right)$, the marginal benefit of releasing water for the tributary use should be equal to the marginal benefit of carryover storage, and the marginal benefit of releasing water in the mainstream for hydropower generation should be equal to the deviation between this reservoir and the downstream reservoir’s marginal benefits of carryover storage; (2) for the individual reservoir at the lowest elevation $\left(i=n\right)$, the marginal benefits of releasing water for both mainstream and tributary use should be equal to the marginal benefit of carryover storage.

3. Monotonic Property for Multi-Objective Optimization Operation of Cascade Reservoir Systems

This section derives the monotonic relationship between the optimal water release decision of an individual reservoir and the (total) water availability of the cascade system or a specific individual reservoir.

3.1. Monotonic Property of Water Release for the Tributary Use

For further mathematical derivation, two assumptions regarding the synchronicity and substitutability are used to characterize the spatial distribution of individual reservoir storage.

Assumption 1.

The carryover storage of each reservoir is a non-decreasing function of the cascade system’s total storage, i.e., synchronic property. The mathematical expression of this assumption is

$${\displaystyle \frac{\partial S_{i,t}^{*}}{\partial S_t^{Sum}}}\ge 0$$

(8a)

$$S_t^{Sum}={\sum }_{i=1}^n{S}_{i,t}^{*}$$

(8b)

In the equations,$S_t^{Sum}$is the total carryover storage of the cascade system at time t. The assumption of storage state synchronicity (Equation (8a)) indicates that the reservoir system should allocate the increased storage according to the prior optimal storage distribution. Theoretical research and empirical findings [14,15,16,17] have demonstrated that reservoir operation rules based on this assumption can successfully lower the system’s invalid water spillage and improve the coordination of decisions regarding water supply and storage across individual reservoirs [16,38,39]. Therefore, this property is widely adopted for the operation of a complex reservoir system, such as the storage target curves [40,41] and the parametric rule [42].

Assumption 2.

The stored water in each reservoir is substitutable among each reservoir. The mathematical expression of resource substitutability is defined as follows [17]:

$${\displaystyle \frac{{\partial }^2{Q}_t}{\partial S_{i,t}^{*}\partial \left(S_t^{Sum}-S_{i,t}^{*}\right)}}\le 0$$

(9)

The above equation indicates that if the total storage of other reservoirs increases, the marginal value of carryover storage for each reservoir either decreases or stays the same because there is more water available in the system for both mainstream and tributary demand, particularly for the successive drought year [17].

Based on the above two assumptions, the monotonic relationship between the optimal water release for tributary use and the total system water availability at period t can be derived.

For any two optimal cascade systems, carryover storage $S_t^{Sum,1}$ and $S_t^{Sum,2}$ at period t, if $S_t^{Sum,1}\le S_t^{Sum,2}$, then according to the Assumption 1, we have $S_{i,t}^1\le S_{i,t}^2$. Using the contradiction method, it can be deduced that $\left(S_t^{Sum,1}-S_{i,t}^1\right)\le \left(S_t^{Sum,2}-S_{i,t}^2\right)$, otherwise, it would exist the carryover storage of an individual reservoir: $S_{i,t}^1\ge S_{i,t}^2$, which is in violation of Assumption 1.

Subsequently, according to Assumption 2, we have

$${\left.{\displaystyle \frac{\partial Q_t}{\partial S_{i,t}^1}}\right|}_{\left(S_{1,t}^1,\cdots {,S}_{m,t}^1\right)}\ge {\left.{\displaystyle \frac{\partial Q_t}{\partial S_{i,t}^1}}\right|}_{\left(S_{1,t}^2,\cdots {,S}_{m,t}^2\right)}$$

(10)

According to the concavity of the value function (decreasing marginal benefit), i.e., conditioned on the unchanged carryover storage of other reservoirs in the system, a higher storage will lead to a lower marginal benefit:

$${\left.{\displaystyle \frac{\partial Q_t}{\partial S_{i,t}^1}}\right|}_{\left(S_{1,t}^2,\cdots {,S}_{m,t}^2\right)}\ge {\left.{\displaystyle \frac{\partial Q_t}{\partial S_{i,t}^2}}\right|}_{\left(S_{1,t}^2,\cdots {,S}_{m,t}^2\right)}$$

(11)

Combining Equations (10) and (11), we can obtain:

$${\left.{\displaystyle \frac{\partial Q_t}{\partial S_{i,t}^1}}\right|}_{\left(S_{1,t}^1,\cdots {,S}_{m,t}^1\right)}\ge {\left.{\displaystyle \frac{\partial Q_t}{\partial S_{i,t}^2}}\right|}_{\left(S_{1,t}^2,\cdots {,S}_{m,t}^2\right)}$$

(12)

Based on the marginal benefit relationship in Equations (7a) and (7b), it can be derived that ${\displaystyle \frac{d{B}_i}{d{OR}_{i,t}^1}}\ge {\displaystyle \frac{d{B}_i}{d{OR}_{i,t}^2}}$. According to the decreasing marginal benefit property of water release, the optimal water release for tributary use must satisfy the following relationship: ${OR}_{i,t}^1\le {OR}_{i,t}^2$. Similarly, it can be deduced that the optimal water release for hydropower generation from the reservoir at the lowest elevation $\left(i=n\right)$ must satisfy ${IR}_{n,t}^1\le {IR}_{n,t}^2$. The cascade system’s total water balance equation can be used to obtain:

$${WA}_t^{Sum,1}=S_t^{Sum,1}+{\sum }_{i=1}^n{OR}_{i,t}^1+{IR}_{n,t}^1\le S_t^{Sum,2}+{\sum }_{i=1}^n{OR}_{i,t}^2+{IR}_{n,t}^2={WA}_t^{Sum,2}$$

(13a)

$${WA}_t^{Sum}={\sum }_{i=1}^n{WA}_{i,t}$$

(13b)

That is, if the total carryover storage of the cascade reservoir system increases, the carryover storage of each reservoir, water release for the tributary use, and the total available water of the cascade system should increase or remain unchanged, and vice versa. As a result, the non-decreasing monotonicity between a cascade reservoir system’s total water availability, water release for tributary use, and optimal carryover storage is determined.

3.2. Monotonic Property of Water Release for the Mainstream Use

Assume that only the water availability of reservoir k increases and the water availability of other reservoirs remains the same, i.e., for any two available water combinations, $W{A}_{k,t}^1\le W{A}_{k,t}^2$ and $W{A}_{i,t}^1=W{A}_{i,t}^2(i\ne k)$. According to the mass balance equation in Equation (1b), the optimal water release from the upstream and downstream reservoirs can be expressed as:

$$I{R}_{k-1,t}={\sum }_{i=1}^{k-1}\left({{WA}_{i,t}-S}_{i,t}-{OR}_{i,t}\right)$$

(14a)

$$I{R}_{k,t}={\sum }_{i=k+1}^n\left(S_{i,t}+{OR}_{i,t}-{WA}_{i,t}\right)+I{R}_{n,t}$$

(14b)

Water release for each reservoir’s tributary satisfies the monotonically non-decreasing property due to carryover storage. Therefore, according to Equation (14a), the water release of the upstream reservoir has the following relationship: ${IR}_{k-1,t}^1\ge {IR}_{k-1,t}^2$, provided that the amount of water available in the reservoir remains constant.

Similarly, from Equation (14b), it can be deduced that the water release of reservoir k and its downstream reservoirs should satisfy ${IR}_{k,t}^1\le {IR}_{k,t}^2$.

The above derivation implies that as the available water of a specific individual reservoir increases, the water release for the mainstream demand from this reservoir and the corresponding downstream reservoirs should not decrease, while the upstream reservoir should not increase. That is, the non-decreasing monotonicity between the water release for the mainstream use from a specific particular reservoir or any downstream reservoir and the water availability of such individual reservoir is derived, whereas the non-increasing monotonicity between the upstream water release and the water availability is specified.

3.3. Quantitative Analysis Based on the Monotonic Property

Based on the monotonicity of the water storage and release decisions of the cascade reservoir system, the following quantitative analysis can be further conducted:

If the cascade system’s total available water increases with an equal interval length (i.e., IL), the sum increase in the total water storage and the water release for tributaries would not exceed IL, according to the system’s total water balance equation (Equation (13a)). Following the non-decreasing monotonical relationship between the optimal decision of individual reservoir and the total available water of the system, it can deduce with the contradiction method that the carryover storage and the water release of individual reservoir will not exceed IL.

Similarly, if reservoir k’s water availability increases with an interval length (i.e., IL) while other reservoirs’ available water stays constant, then both the decrease amount of water release from upstream reservoir and the increase amount of release from downstream will not exceed IL, since the total increment or decrement does not exceed IL. Following the chain rule of derivation, the monotonic property would be maintained in the subsequent operation periods.

In summary, the monotonic relationship between different water release and the water availability in the cascade system is shown in Figure 2.

4. Improved Reinforcement Learning Method for Optimization Operation of Cascade Reservoir System

Reinforcement learning for solving the joint optimization operation of the cascade reservoir system would not utilize the prior knowledge of water mass balance of each reservoir, as well as the hydrologic connection equation between the upper and lower reservoirs to specify the monotonic decision-making property. Based on the monotonic relationship between the optimal release decisions and the water availability, the efficiency of the reinforcement learning method can be effectively enhanced. The main principle of the improved optimization method can be summarized as two steps: initial state optimization, and searching space reduction, as shown in Figure 3.

(1) Initial state optimization: First, the storage volume of each reservoir is discretized with an equal interval length (i.e., IL) for each reservoir at each period, i.e., the discretization interval of each reservoir is set as IL, and the number of discretization of reservoir i is NS_I. After determining the feasible space of the water release decision for each initial state combination based on the constraints of water supply and storage, Newton’s method is applied to specify the optimal release for the mainstream and tributary from the reservoir’s water balance equation (Equation (1b)) and the marginal benefit equalization equation (Equations (7a) and (7b)). The feasibility of the decision is checked according to the water release and carryover storage inequality constraints, where, if the decision variables that do not meet these constraints, according the suggested procedure [43], the boundary of the inequality constraint is set as the optimal value, and the corresponding water release is adjusted according to the mass balance equation. In this process, the reinforcement learning method employs incremental sampling to identify the set of solutions for each reservoir’s optimal water release over the course of the scheduling period.

(2) Searching space reduction: Assume that the optimal water release and carryover storage for the entire periods corresponding to the j sample are $\left({IR}_1^{\#},\cdots ,{IR}_T^{\#}\right)$; $\left({OR}_1^{\#},\cdots ,{OR}_T^{\#}\right)$ and $\left(S_1^{\#},\cdots ,S_T^{\#}\right)$, respectively. If the initial water storage state of j sample increases with IL, then the water release for the tributary can be defined as the decision variable in Newton’s method, and the release for the mainstream can be specified using Equation (14a). According to the monotonicity of the water release decision, the optimal release would be identified within the range between $\left({OR}_1^{\#},\cdots ,{OR}_T^{\#}\right)$ and $\left({OR}_1^{\#}+IL,\cdots ,{OR}_T^{\#}+IL\right)$. As a result, the improved method only searches in the feasible interval with a variation of IL, which leads to a higher computational efficiency than searching the entire feasible region in the traditional reinforcement learning method.

5. Case Study

5.1. System Description

As illustrated in Figure 4, the WB-Xll-Xjb-Sx reservoir system is a giant cascade reservoir system on the mainstream of the upper reaches of the Yangtze River in China. Among them, the storage capacity of WB reservoir is 13.46 billion m³, and the annual average runoff is 241.34 billion m³, with the annual demand of 120 billion m³ in the mainstream and 3.20 billion m³ in the tributary; the storage capacity of Xll reservoir is 6.46 billion m³, and the annual average runoff is 20.64 billion m³ with the annual demand of 65 billion m³ in the mainstream and 0.5 billion m³ in the tributary; the storage capacity of Xjb is 0.90 billion m³, and the annual average runoff is 7.28 billion m³ with the annual demand of 15 billion m³ in the mainstream and 1.8 billion m³ in the tributary; the storage capacity of Sx is 22.15 billion m³, and the annual average runoff is 280.18 billion m³ with the annual demand of 200 billion m³ in the mainstream and 4.5 billion m³ in the tributary.

In this paper, the commonly used benefit function [44] is set as the objective function to evaluate the system operation performance, as below:

$$B^i\left({IR}_{i,t}\right)=\left\{ \begin{array}{c} {\left({\displaystyle \frac{E_{i,t}}{{IE}_{i,t}}}\right)}^{\alpha }={\left({\displaystyle \frac{{\gamma \times IR}_{i,t}}{{\gamma \times ID}_{i,t}}}\right)}^{\alpha }={\left({\displaystyle \frac{{IR}_{i,t}}{{ID}_{i,t}}}\right)}^{\alpha }; {IR}_{i,t}\le {ID}_{i,t} \\ 1 ; {IR}_{i,t}>{ID}_{i,t} \end{array}\right.$$

(15a)

$$B_i\left({OR}_{i,t}\right)=\left\{ \begin{array}{c} {\left({\displaystyle \frac{{OR}_{i,t}}{{OD}_{i,t}}}\right)}^{\alpha }; {OR}_{i,t}\le {OD}_{i,t} \\ 1 ; {OR}_{i,t}>{OD}_{i,t} \end{array}\right.$$

(15b)

where $E_{i,t}$ and ${IE}_{i,t}$ are the hydropower generation of the release and maximum release from reservoir i to the mainstream at period t, respectively; $\gamma$ is the transition coefficient from water into hydropower, which is given from an experimental test; $\alpha$ is the exponential parameter of the benefit function generated by the water release. $\alpha <1$ is defined for a concave function. According to the literature [35] and sensitivity parameter analysis, the value of $\alpha$ is set to 0.8 in this paper. It is assumed that the weighting factors for various water utilities are equal to 1 for the sake of the subsequent comparison analysis.

5.2. Comparison Analysis of Optimization Results

The WB-Xll-Xjb-Sx cascade reservoir system’s comparison analysis of multi-objective optimization techniques divides a hydrological year into 12 periods, with each period’s operation time step of a month. Historical inflow runoff from 1959 to 2023 is collected from each reservoir to generate the corresponding runoff state transition probability function. The inflow is discretized as four for each reservoir, and six storage discretization levels (i.e., interval lengths), ranging from 0.2 to 0.4 billion m³, are employed for the algorithm implementation. Stochastic dynamic programming (SDP), reinforcement learning (RL), and improved reinforcement learning (IRL) methods with a learning rate of 0.9 are executed with GUN Fortran to solve the optimization operation of the cascade reservoir system on a ThinkPad T15g workstation (i7-10750H/16 GB), provided by Lenovo, Beijing, China. The optimization results under different storage discretization levels are shown in Figure 5, Figure 6 and Figure 7.

To compare the results from different methods, the optimization results of SDP are set as a benchmark, and the differences between SDP and RL or SDP and IRL are measured by relative error in Figure 5. As can be seen, there is no significant difference in the optimization results obtained by the three optimization algorithms under six different discretization intervals, with relative errors all kept within 10%. Only when the reservoir capacity discretization interval is set to 0.24 billion m³, the relative errors between SDP and RL, as well as the error between SDP and IRL, both reach the maximum of 5.73% and 9.87%, respectively. As further illustrated in Figure 5b,c, the maximum relative error mainly results from the optimization operation of the Sx reservoir, which has a relatively large storage capacity and water demand.

The computing time of SDP, RL, and IRL with different discretization levels is illustrated in Figure 6a,b. The computation time of IRL is consistently less than 0.10% of that from SDP and 5% of that from RL for all storage discretization levels. For example, the time of SDP is 4109.47 s and RL of 52.83 s, while that of RIDP is 2.10 s under the discretization level of 0.32 billion m³ for running the models over a one-year time horizon. This conclusion indicates the high computational efficiency of IRL.

The carryover storage distribution of the cascade reservoir system in September is given in Figure 7. As can be seen from Figure 7a,b, the carryover storage of the Xll reservoir obtained from SDP or RL would not always increase with the total storage of the cascade reservoir system increases, particularly for a high system storage level. In contrast, the carryover storage of all individual reservoirs derived from IRL would increase with the total storage of the cascade system in Figure 7c, which follows the synchronic storage distribution property. Since September is often defined as the refill period in the Yangtze River, a relatively high storage of the system would seldom result in a relatively low storage of the individual reservoir, which is not consistent with the previous numerical study [45] or the commonly used operating rules [32,33,34]. These distribution results imply the relatively high interpretability of the solutions from IRL.

6. Conclusions

Based on the synchronicity and substitutability property of storage distribution in the cascade reservoir system, the first-order derivative relationship between individual reservoir water release decisions and the water availability of the cascade system or a specific individual reservoir is derived. Theoretical analysis results show that: (1) the optimal carryover storage and water release for the tributary use from each individual reservoir is a non-decreasing monotonic function of the cascade reservoir system’s overall water availability; (2) the water release for mainstream use from a specific reservoir, or from any downstream reservoir, is a non-decreasing monotonic function of the water availability of that individual reservoir. Conversely, the relationship between upstream water release and water availability is characterized by non-increasing monotonicity.

Considering the derived monotonic properties among carryover storage, water release decisions from each individual reservoir, and the water availability of the cascade reservoir system, an improved reinforcement learning method with a reduced searching region has been developed. To demonstrate the effectiveness and efficiency of the proposed algorithm, the WB-Xll-Xjb-Sx reservoir system in the upper reaches of the Yangtze River in China is employed as a case study. Numerical results from this case study indicate that, compared to standard dynamic programming (SDP) and traditional reinforcement learning (RL), the improved method significantly reduces computation time without substantial loss of solution accuracy, particularly in preserving highly interpretable optimization results.