Two Dimension Reduction Methods for Multi-Dimensional Dynamic Programming and Its Application in Cascade Reservoirs Operation Optimization

: An efﬁcient reservoir operation technique plays a very important role in improving the water resources and energy efﬁciency of reservoirs. In order to effectively avoid or alleviate the “curse of dimensionality” of Multi-dimensional Dynamic Programming (MDP) in the application of cascade reservoirs operation optimization (CROO) and keep a global convergence at the same time, two dimension reduction methods are proposed in this paper. One is a hybrid algorithm of MDP and a Progressive Optimality Algorithm (POA), named MDP-POA, which combines the global convergence of MDP and the strong local search ability of POA. MDP-POA ﬁrst takes the global optimal trajectory of MDP in a low discrete degree as the initial trajectory of the POA, and then implements further optimization to the obtained initial trajectory by the POA with a high discrete degree, so as to avoid the “curse of dimensionality” of MDP in high discrete degree and the dependency of the POA for the initial trajectory. The other is an improved MDP (IMDP), which ﬁrst constructs a corridor by the optimal trajectory of MDP in a lower discrete degree, and then implements further optimization in the corridor by MDP with a relatively high discrete degree, so as to avoid a large number of unnecessary calculations, and shorten the run-time effectively. In a case study, the results of MDP-POA, IMDP, and MDP are compared and analyzed from the aspects of power generation and run-time. The analysis indicates that the proposed MDP-POA and IMDP both have a good application effect and are worthy of further promotion.


Introduction
As one of the most stable forms of renewable energy, hydropower energy can be commercially developed and utilized on a large scale [1][2][3][4].With the rapid development of cascade reservoirs, cascade reservoirs operation optimization (CROO) is attracting the attention of more and more scholars all over the world [5,6].The global optimal solution of CROO has very important practical significance in developing long-term optimal operation schemes and rules for cascade reservoirs [7,8].So, it is very necessary to solve the CROO problem correctly and reasonably.However, CROO is a multivariate coupled and complicated nonlinear programming problem [9], which needs to consider not only the hydraulic connection between upstream and downstream reservoirs, but also a lot of constraints.It has the characteristics of high dimensionality, strong coupling, and uncertainty [10][11][12], and its solution is full of difficulties and has always been the focus of many scholars' research [13].Aiming at the solving of the CROO problem, a wide range of methods have been proposed over the past decades, which mainly involve conventional optimization algorithms and various heuristic random search algorithms [14,15].The conventional methods include Linear Programming (LP) [16], Nonlinear Programming (NLP) [17], Lagrangian Relaxation (LR) [18], Quadratic Programming (QP) [19], and Multi-dimensional Dynamic Programming (MDP) [20,21].They are all elitist algorithms, and have already received different degrees of success in solving CROO problems.The modern heuristic random search algorithms include the Genetic Algorithm (GA) [22], Particle Swarm Optimization (PSO) [23], Ant Colony Optimization (ACO) [24], Fuzzy Neural Network (FNN) [25], and the Differential Evolution algorithm (DE) [26][27][28].These have been extensively used to solve the CROO problem, and have also received a good application effect.
Among the above algorithms, MDP is a powerful optimization technique for CROO problems.The most significant characteristic of MDP is that it is able to obtain a global optimal solution and have no requirement for the initial trajectories.Moreover, MDP imposes no restrictions on the unsmooth and nonconvex nature of CROO problems, which make it boast high popularity among the conventional optimization techniques.Many evolutionary algorithms have been proved to possess a global convergence, while as these algorithms are affected by a stochastic feature, they cannot guarantee a global optimum with finite iterations [29].However, although the MDP can solve CROO problems with a global convergence, the high dimensionality, called the "curse of dimensionality", poses difficulties and limits its application in CROO problems, especially for large-scale hydropower systems [30].
On the whole, there are three ways to avoid or alleviate the "curse of dimensionality" and guarantee the global convergence of MDP.The first one is to improve MDP effectively on the premise of guaranteeing a global convergence, so as to shorten the run-time.The second one is to implement parallel computing by using multi-core processing technology.The third one is to combine MDP with other algorithms which have a high computational efficiency, so as to make up for their deficiencies of each other.Some scholars have done related research work in these aspects, as described below.
In terms of algorithm improvements, a variety of improved Dynamic Programming (DP) algorithms have emerged and been used to avoid or alleviate the "curse of dimensionality", such as the Progressive Optimality Algorithm (POA) [31], Dynamic Programming with Successive Approximation (DPSA) [32], Discrete Differential Dynamic Programming (DDDP) [33], and Incremental Dynamic Programming (IDP) [34].These improved algorithms can effectively avoid the "curse of dimensionality" to a certain degree, but defects still exist.For example, POA is sensitive to the initial trajectories, and may converge to a local optimum in some situations.DPSA and DDDP are difficult to use to solve problems with non-convexity, and may also lead to a local optimum.In addition, other related research has been done by some scholars.For example, Mousavi et al. [35] reduced the run-time of an MDP model for a multi-reservoir system by diagnosing infeasible storage combinations and removing them from further computations, which has a good effect within a certain amount of hydropower stations, but the time consumption is still enormous and intolerable when the scale of a hydropower system reaches a certain large degree.By taking advantage of the monotonic relationship between reservoir storage volume and the optimal release decision.Zhao et al. [36] proposed an improved DP model for reservoir operation optimization (ROO); however, the model can only be applied to a reservoir operation with a concave objective function.Ji et al. [37] proposed a novel multi-layer nested multi-dimensional dynamic programming (MNDP) based on a multi-layer nested structure, but it was mainly used to deal with the problem of computer memory space and computational complexity for MDP in CROO, and the computing time had not been reduced.
In recent years, parallel computing has been widely applied in the field of water resources [38,39].Especially, in the field of CROO and MDP applications, there are several successful examples.Dias et al. [40] improved the performance of stochastic dynamic programming by using parallel processing techniques, and successfully applied it to the long-term operation planning of an electrical power system.Li et al. [41] developed a parallel MDP algorithm to optimize the joint operation of Water 2017, 9, 634 3 of 19 a multi-reservoir system based on a distributed memory architecture and the Message Passing Interface (MPI) protocol.Ji et al. [37] implemented parallel computing to the proposed MNDP algorithm, and achieved a good application effect.In order to evaluate the parallel performance of different parallel modes, Zhang et al. [11] proposed three kinds of parallel MDP algorithms.On the whole, these previous studies demonstrate that run-time can be reduced a lot in optimization or simulation by using parallel computing associated with proper parallelization strategies.However, with the increase of the number of cores used in parallel computing, the parallel efficiency will reduce gradually; specifically, when the number of cores is large, the parallel efficiency is generally very low.
In terms of hybrid applications, i.e., the combination usage of MDP with other algorithms, much related research has been achieved, for example, a combination of the GA and DDDP approaches (GA-DDDP) was proposed and developed to optimize a multiple reservoir system's operation by Tospornsampan et al. [42], and the significant advantage obtained from using GA-DDDP is economizing on computational resources, as GA-DDDP does not require optimizing parameters and the derivation of feasible initial trial trajectories.Lantoine and Russell [43] presented a hybrid variant of the differential dynamic programming (HDDP) algorithm to solve constrained nonlinear optimal control problems, and the hybrid method incorporates nonlinear mathematical programming techniques to increase its efficiency.Zhang et al. [23] joined parallel deterministic dynamic programming and a hierarchical adaptive genetic algorithm to solve an ROO problem.However, in view of the above description and analysis, the effective existing hybrid applications have mainly focused on improved dynamic programming algorithms (such as DDDP and IDP) or intelligent optimization algorithms, and there are very few hybrid applications about the baseline MDP which can converge to the optimal solution without an additional requirement of unsmooth, non-convexity, and initial trajectories.Therefore, it has an important practical significance to carry out research of dimension reduction methods for MDP.
In order to effectively avoid the "curse of dimensionality" of MDP and guarantee a global convergence at the same time, this paper proposes two new dimension reduction methods for MDP based on a structural and characteristics analysis, i.e., a hybrid algorithm of MDP and POA (named MDP-POA), and an improved MDP (named IMDP).A detailed case study was provided in this paper by taking the Qingjiang cascade reservoirs in China as an instance, and in order to evaluate the performance of the proposed MDP-POA and IMDP, the results of MDP-POA, IMDP, and MDP were compared and analyzed from the aspects of power generation and run-time.In addition, the authors have analyzed the varying characteristics of the optimal solution for the proposed MDP-POA and IMDP under different discretization levels, and provided the recommended discretization levels or computing schemes for different conditions.The following part of this paper is organized as follows.Section 2 presents the formulation of CROO problems.Section 3 introduces the principle of MDP and the POA, and presents the optimization principle of MDP-POA and IMDP.Section 4 shows the application of MDP-POA and IMDP in the cascade reservoirs of Qingjiang River in China, and the results are analyzed and discussed in this section.Section 5 presents the conclusions.

Formulation of CROO Problems
Power generation is a significant benefit derived from a cascade reservoirs system.CROO aims at maximizing the power generation by developing an optimal plan over the entire planning horizon, while satisfying all kinds of physical and operational constraints.Generally, CROO is related to a given operation of the hydropower stations for T stages as follows.
Water 2017, 9, 634 where E is the total power generation over the entire planning horizon, unit: kWh; T is the number of stages over the entire planning horizon; N i t is the output of the ith hydropower station in the tth stage, unit: kW, and the reservoir indexes from upstream to downstream are 1, 2, ... , n in this paper; K i is the efficiency coefficient of the ith hydropower station; H i t is the average water level of the ith hydropower station in the tth stage, unit: m, and it is determined by the beginning water level h t of the tth stage of the reservoir, the end water level h t+1 , and the downstream tail water level h down , i.e., H t = (h t + h t+1 )/2 − h down , while h down is determined by the total outflow Q t of the reservoir in the tth stage, and Q t happens to be determined by h t and h t+1 .In addition, the reservoir water level h has a one-to-one mapping relationship with the reservoir volume V, namely the reservoir curve of water level-volume; ∆t is the duration of a stage, unit: h.CROO is subject to the following equality and inequality constraints.
(1) Water volume balance: where q i t is the outflow through the turbines of the ith reservoir in the tth stage, unit: m 3 /s; W i t is the abandoned water outflow through the flood outflow gate of the ith reservoir in the tth stage, unit: m 3 /s; the total outflow Q i t of the ith reservoir in the tth stage contains q i t and W i t , unit: m 3 /s; I i t is the inflow of the ith reservoir in the tth stage, unit: m 3 /s; Ev i t is the evaporation capacity of the ith reservoir in the tth stage, unit: m 3 /s; and V i t is the storage volume of the ith reservoir in the tth stage, unit: m 3 .Because we study the mid-and long-term operation of cascade reservoirs in this paper, the delay of water flow between two reservoirs is not considered.
(2) Reservoir volume limits: where V i t,min is the lower limit of V i t , which usually corresponds to the dead level, unit: m 3 ; V i t,max is the upper limit of V i t , which usually corresponds to the flood control level in flood season and normal level in dry season, unit: m 3 .
(3) Comprehensive utilization of water resources required at downstream reservoir limits: where Q i t,min the lower limit of Q i t , which is usually determined by the ecological flow of the downstream river, unit: m 3 /s; and Q i t,max is the upper limit of Q i t , which is usually determined by the channel capacity of the downstream river, unit: m 3 /s.
(4) Power generation limits: where N i t,min is usually determined by the allowed minimum output, unit: kW; and N i t,max is usually determined by the installed capacity and expected output of the hydropower station, unit: kW.
(5) Boundary conditions limits: where V i 0 is the storage volume of the ith reservoir at the beginning of the first stage, unit: m 3 ; V i b is the storage volume of the ith reservoir at the beginning of the entire planning horizon, unit: m 3 ; V i T is the storage volume of the ith reservoir at the end of the Tth stage, unit: m 3 ; and V i e is the storage volume of the ith reservoir at the end of the entire planning horizon, unit: m 3 .

MDP and POA
DP can be effectively used to solve multi-stage decision-making problems recursively, and the ROO problem can be regarded as a multi-stage decision-making problem by dividing the reservoir operation into sub-operations on the basis of operation intervals [44,45].A reverse recursion procedure and a chronological order recursion procedure are involved in the application of DP to ROO problems.In the reverse recursion procedure, starting from the last stage, the output or power generation is calculated up to the first stage, and the optimal storage water level variations can be obtained at last by the chronological order recursion procedure.The recursive equation for the tth stage of computation is as follows [11].
where is the optimal cumulative output of the beginning state Se at the (t + 1)th stage, unit: kW.The optimal cumulative output mentioned above means the sum of the output from present stage t to last stage T in the optimal output process.
Many variables and constraints are integrated into the procedure of solving a CROO problem, because the number of reservoirs is often two or more in a CROO problem.If the number of discrete points of storage volume for each reservoir is M for a cascade system consisting of n reservoirs, then M n combinations of these discrete points can be obtained.With reference to the reverse recursion procedure and chronological order recursion procedure of DP in an ROO problem, we can get the optimal combination of storage volume for each stage.Taking a hydropower station system consisting of n reservoirs as an example, the combination principle of discrete storage volume is shown in Figure 1, and the recursive equation of MDP can be formulated as follows. where ) is the decision variable vector; and ) is the state variable vector.In formula (9), because of the discretization, ), and is the optimal cumulative output of a storage volume combination at the tth stage, unit: kW; and F * t (V t ) is the optimal cumulative output of a storage volume combination at the (t + 1)th stage, unit: kW.
The detailed exposition on the calculation principles and steps of MDP has been recorded in the literature by Ji [37] and Zhang [11].
The POA is a powerful improved dynamic programming in cascade reservoirs operation optimization, which was first proposed by the Canadian researchers N.G.F. Sancho and H. R. Harvson for solving multi-stage dynamic programming problems in 1975.It transforms a complex multi-stage decision problem into a series of two-stage decision-making problems, which simplifies the solution of multi-dimensional problems and decreases their computational complexity.The calculation steps of the POA in solving a CROO model are as follows.
Step 1 Obtain the initial operation trajectories of storage volume by other conventional methods, these can be represented as {V 0 (0), V 1 (0), . . ., V Mum (0)}, where Mum is equal to the number of reservoirs multiplied by the number of operation stages in a year, i.e., Mum = nT.
Step 2 Within the permitted scope, discretize the point V 0 (0), into {V 0 0 (0), V 1 0 (0), . . .,V M 0 (0)}, where M is the number of discrete points for an operation point on the initial operation trajectories of storage volume, and the value of the other points on the initial trajectories are fixed.
Step 3 For different discrete points of V 0 (0), which are {V 0 0 (0), V 1 0 (0), . . ., V M 0 (0)}, implement the simulation calculations respectively by using long series runoff data.Find out the point of V * 0 (0) that can maximize the power generation of all operation stages of the cascade reservoirs, then update V 0 (0) with V * 0 (0), and save it as V 0 (1).The power generation in the simulation calculation can be obtained by formula (10), in which k represents the serial number of discrete points Step 4 In the same way, perform the Steps 2 and 3 to the other points on the initial operation trajectories, and end up with new operation trajectories {V 0 (1) Step 5 Compare the power generation between the initial operation trajectories {V 0 (0), V 1 (0), . . ., V Mum (0)} and the new operation trajectories {V 0 (1), V 1 (1), . . ., V Mum (1)}; if the error meets the accuracy requirements, then stop counting, otherwise take {V 0 (1), V 1 (1), . . ., V Mum (1)} as the new initial operation trajectories, and repeat Step 2 to Step 4.
Water 2017, 9, 634 6 of 19 The detailed exposition on the calculation principles and steps of MDP has been recorded in the literature by Ji [37] and Zhang [11].
The POA is a powerful improved dynamic programming in cascade reservoirs operation optimization, which was first proposed by the Canadian researchers N.G.F. Sancho and H. R. Harvson for solving multi-stage dynamic programming problems in 1975.It transforms a complex multi-stage decision problem into a series of two-stage decision-making problems, which simplifies the solution of multi-dimensional problems and decreases their computational complexity.The calculation steps of the POA in solving a CROO model are as follows.
Step 1 Obtain the initial operation trajectories of storage volume by other conventional methods, these can be represented as }, where Mum is equal to the number

Hybrid Application of MDP and POA
As we know, for a large-scale cascade hydropower system, the calculation amount of MDP in a high discrete degree will be very huge, which can lead to a very long run-time, while in a low discrete degree, it cannot guarantee the accuracy of the final solution.As described the introduction, the POA is sensitive to the initial trajectories, but it has a strong local search ability and fast computing speed.
Water 2017, 9, 634 Therefore, if we take the global optimal trajectories of MDP in a low discrete degree as the initial trajectories of the POA, and implement further optimization to the obtained initial trajectories by the POA with a high discrete degree, then we can effectively avoid their defects and play their advantages.That is to say, we first obtain the relatively good initial trajectories for the POA by MDP in a short time, and then optimize the initial trajectories further by the POA, which can give full play to its local search ability and only take a very short time.Described above is the basic principle of the proposed hybrid algorithm in this paper, i.e., MDP-POA.
Taking the optimization calculation of one reservoir as an example, the principle of MDP-POA is shown in Figure 2.For cascade reservoirs, its principle is basically identical to Figure 2. In Figure 2, M1 is the number of discrete points for MDP optimization, and it usually takes a small value, such as 20 or 30.M2 is the number of discrete points for POA optimization, and it usually takes a large value, such as 100 or 200.

Hybrid Application of MDP and POA
As we know, for a large-scale cascade hydropower system, the calculation amount of MDP in a high discrete degree will be very huge, which can lead to a very long run-time, while in a low discrete degree, it cannot guarantee the accuracy of the final solution.As described the introduction, the POA is sensitive to the initial trajectories, but it has a strong local search ability and fast computing speed.Therefore, if we take the global optimal trajectories of MDP in a low discrete degree as the initial trajectories of the POA, and implement further optimization to the obtained initial trajectories by the POA with a high discrete degree, then we can effectively avoid their defects and play their advantages.That is to say, we first obtain the relatively good initial trajectories for the POA by MDP in a short time, and then optimize the initial trajectories further by the POA, which can give full play to its local search ability and only take a very short time.Described above is the basic principle of the proposed hybrid algorithm in this paper, i.e., MDP-POA.
Taking the optimization calculation of one reservoir as an example, the principle of MDP-POA is shown in Figure 2.For cascade reservoirs, its principle is basically identical to Figure 2. In Figure 2, M1 is the number of discrete points for MDP optimization, and it usually takes a small value, such as 20 or 30.M2 is the number of discrete points for POA optimization, and it usually takes a large value, such as 100 or 200.The calculation steps of MDP-POA in solving a CROO model are similar to that of the POA.The difference is in the first step, where the POA obtains the initial operation trajectories of the storage volume by conventional methods, but we obtain the initial optimal operation trajectories of the storage volume by MDP with a low discrete degree (such as 10, 20, or 30) in MDP-POA.

Improved MDP
There is a "curse of dimensionality" problem for MDP, mainly because it is a traversal optimization process, which means that all combinations of storage volume in the feasible region are calculated.The calculation amount of MDP is usually very huge when the number of discrete points is large.However, what we ultimately need is just an optimal storage volume process line; not all the The calculation steps of MDP-POA in solving a CROO model are similar to that of the POA.The difference is in the first step, where the POA obtains the initial operation trajectories of the storage volume by conventional methods, but we obtain the initial optimal operation trajectories of the storage volume by MDP with a low discrete degree (such as 10, 20, or 30) in MDP-POA.

Improved MDP
There is a "curse of dimensionality" problem for MDP, mainly because it is a traversal optimization process, which means that all combinations of storage volume in the feasible region are calculated.The calculation amount of MDP is usually very huge when the number of discrete points is large.However, what we ultimately need is just an optimal storage volume process line; not all the calculations in the traversal optimization are required for us in the end.Therefore, the question of how to reduce the unnecessary calculations is the key to shorten the run-time of MDP.The essence of shortening the run-time for MDP is to reduce the amount of discrete combinations of storage volume in the calculation.
On the whole, there are two ways to decrease the discrete combinations.The first way is to implement the calculation with a low discrete degree, which will reduce the number of total discrete combinations, but it will affect the precision of the final solution.The second way is to implement the calculation with a high discrete degree, but remove the unlikely optimal solution region first.However, the optimal result of MDP in a current discrete degree is related to all of the discrete combination calculations.So, before the traversal calculation, we cannot eliminate any discrete combinations that Water 2017, 9, 634 8 of 19 we think are unlikely optimal solution regions in the calculation, and we can eliminate them only after we make sure of it.Therefore, we can implement the calculation with a low discrete degree first, which is used to determine the generally unlikely optimal solution region in a short time, and remove it, and then implement further optimization for the rest of the region with a relatively high discrete degree, so as to ensure the accuracy of algorithm.The rest of the feasible region mentioned above is actually a corridor constructed through the optimal trajectories of MDP with a low discrete degree, and the further optimization also described above is actually implemented in the corridor by MDP with a relatively high discrete degree.Described above is the basic principle of the proposed improved MDP in this paper, i.e., IMDP.
Taking the optimization calculation of one reservoir as an example, the principle of IMDP is shown in Figure 3.For cascade reservoirs, its principle is basically identical to Figure 3.In Figure 3, the corridor size can be two or more discrete units around the initial optimal trajectories, and the times of corridor optimization can be implemented once or several times.
In contrast, there are two differences between IMDP and Incremental Dynamic Programming (IDP).The first is that IDP requires an initial operation trajectory; however, IMDP directly uses the optimal solution of MDP in a low discrete degree as the initial operation trajectory of MDP in further optimization with a high discrete degree, and it does not need to find the initial solution by other methods.The second is that IMDP generally requires only one-time corridor optimization to achieve a good result, while IDP may require multiple iterations, and so its computation time is generally longer.
volume in the calculation.
On the whole, there are two ways to decrease the discrete combinations.The first way is to implement the calculation with a low discrete degree, which will reduce the number of total discrete combinations, but it will affect the precision of the final solution.The second way is to implement the calculation with a high discrete degree, but remove the unlikely optimal solution region first.However, the optimal result of MDP in a current discrete degree is related to all of the discrete combination calculations.So, before the traversal calculation, we cannot eliminate any discrete combinations that we think are unlikely optimal solution regions in the calculation, and we can eliminate them only after we make sure of it.Therefore, we can implement the calculation with a low discrete degree first, which is used to determine the generally unlikely optimal solution region in a short time, and remove it, and then implement further optimization for the rest of the region with a relatively high discrete degree, so as to ensure the accuracy of algorithm.The rest of the feasible region mentioned above is actually a corridor constructed through the optimal trajectories of MDP with a low discrete degree, and the further optimization also described above is actually implemented in the corridor by MDP with a relatively high discrete degree.Described above is the basic principle of the proposed improved MDP in this paper, i.e., IMDP.
Taking the optimization calculation of one reservoir as an example, the principle of IMDP is shown in Figure 3.For cascade reservoirs, its principle is basically identical to Figure 3.In Figure 3, the corridor size can be two or more discrete units around the initial optimal trajectories, and the times of corridor optimization can be implemented once or several times.
In contrast, there are two differences between IMDP and Incremental Dynamic Programming (IDP).The first is that IDP requires an initial operation trajectory; however, IMDP directly uses the optimal solution of MDP in a low discrete degree as the initial operation trajectory of MDP in further optimization with a high discrete degree, and it does not need to find the initial solution by other methods.The second is that IMDP generally requires only one-time corridor optimization to achieve a good result, while IDP may require multiple iterations, and so its computation time is generally longer.According to the above principle of IMDP, we can construct the corridor by the optimal result of MDP in a low discrete degree (such as 10, 20, or 30), and set up some IMDP schemes, for example, schemes "IMDP: 10 × (20/2)", "IMDP: 10 × (40/4)", "IMDP: 20 × (10/2)", and "IMDP: 20 × (20/4)" as shown in Figure 4, in which the discrete degree of the schemes is equivalent to the discrete degree of MDP with 100 discrete points.Taking the IMDP scheme "IMDP: 10 × (20/2)" for example, it has 10 discrete points in the first-stage calculation and 20 discrete points in the second-stage calculation, and the 20 discrete points are distributed in two discrete units uniformly, which can be demonstrated by According to the above principle of IMDP, we can construct the corridor by the optimal result of MDP in a low discrete degree (such as 10, 20, or 30), and set up some IMDP schemes, for example, schemes "IMDP: 10 × (20/2)", "IMDP: 10 × (40/4)", "IMDP: 20 × (10/2)", and "IMDP: 20 × (20/4)" as shown in Figure 4, in which the discrete degree of the schemes is equivalent to the discrete degree of MDP with 100 discrete points.Taking the IMDP scheme "IMDP: 10 × (20/2)" for example, it has 10 discrete points in the first-stage calculation and 20 discrete points in the second-stage calculation, and the 20 discrete points are distributed in two discrete units uniformly, which can be demonstrated by the first figure in Figure 4. So, the discrete degree of scheme "MDP: 100" and "IMDP: 10 × (20/2)" are the same, or at least the discrete degree of the IMDP scheme is not lower than that of MDP.The meaning of other IMDP schemes in Figure 4 are similar to scheme "IMDP: 10 × (20/2)".the first figure in Figure 4. So, the discrete degree of scheme "MDP: 100" and "IMDP: 10 × (20/2)" are the same, or at least the discrete degree of the IMDP scheme is not lower than that of MDP.The meaning of other IMDP schemes in Figure 4 are similar to scheme "IMDP: 10 × (20/2)".The calculation steps of IMDP in solving a CROO model are similar to that of MDP, which can be briefly summarized as follows.
Step 1 Obtain the initial optimal operation trajectories of the storage volume by MDP in a low discrete degree (such as 10, 20, or 30), which can be represented as Step 2 Construct a corridor through the initial optimal operation trajectories of MDP obtained in Step 1.
Step 3 Within the constructed corridor scope, discretize every point in

} by another discrete degree.
Step 4 In the constructed corridor, obtain the optimal storage volume combination for each stage by MDP with a reverse recursion calculation and a chronological order recursion calculation.

Basic Data
The Qingjiang River is one of the main tributaries of Yangtze River below the Three Gorges Dam [46], and its basin area is 17,600 km 2 .Its mean annual rainfall, runoff depth, and annual average discharge are approximately 1460 mm, 876 mm, and 423 m 3 /s, respectively.The total length of the main stream is 423 km with a hydraulic drop of 1430 m [47].There are three hydropower stations on the main stream, i.e., Shuibuya, Geheyan, and Gaobazhou from upstream to downstream.Two The calculation steps of IMDP in solving a CROO model are similar to that of MDP, which can be briefly summarized as follows.
Step 2 Construct a corridor through the initial optimal operation trajectories of MDP obtained in Step 1.
Step 4 In the constructed corridor, obtain the optimal storage volume combination for each stage by MDP with a reverse recursion calculation and a chronological order recursion calculation.

Basic Data
The Qingjiang River is one of the main tributaries of Yangtze River below the Three Gorges Dam [46], and its basin area is 17,600 km 2 .Its mean annual rainfall, runoff depth, and annual average discharge are approximately 1460 mm, 876 mm, and 423 m 3 /s, respectively.The total length of the main stream is 423 km with a hydraulic drop of 1430 m [47].There are three hydropower stations on the main stream, i.e., Shuibuya, Geheyan, and Gaobazhou from upstream to downstream.Two hydropower stations are under regular operation (i.e., Shuibuya and Geheyan), and thus they are selected as the focus of the research in this paper.The basic parameters of the two hydropower stations are shown in Table 1, and the plane sketch map of the Qingjiang River cascade hydropower stations is shown in Figure 5 [48].The long series runoff data of this cascade system is available from 1959 to 2000, a total of 41 years.For this study, the years of 1967, 1999, and 1975 are selected as the typical wet, normal, and dry years, respectively.The runoff data for each typical year is shown in Table 2.The long series runoff data of this cascade system is available from 1959 to 2000, a total of 41 years.For this study, the years of 1967, 1999, and 1975 are selected as the typical wet, normal, and dry years, respectively.The runoff data for each typical year is shown in Table 2.

Results and Analysis
According to the data provided above, we solve the CROO problem described in Section 2 by the proposed MDP-POA and IMDP.In the optimization calculation, the length of the operation stage is set to 10 days, the planning horizon is set to one year, the storage water level at the beginning of the entire planning horizon is set to normal level for each reservoir, and no restrictions are set for the end.The upper limit of the water level for each reservoir is flood-limited water level in flood season and normal level in dry season, and the lower limit for each reservoir is the dead level.
In order to comparatively analyze the dimension reduction effect of MDP-POA and IMDP with the baseline MDP, we established five schemes for MDP, and the number of discrete points for these five schemes is 10, 20, 30, 60, and 100, respectively.Ten, 20, and are the numbers of discrete points for the low discrete degree schemes, which are mainly used to construct the corridor for IMDP or derive the initial trajectories for the POA in MDP-POA.Sixty and 100 are the numbers of discrete points for the high discrete degree schemes, which are mainly used for comparative analysis.
In the optimization calculation of MDP-POA, we took the optimal solution of MDP in a low discrete degree as the initial operation trajectories of the POA, and established 13 schemes in the POA optimization stage.The number of discrete points for each scheme is 10, 20, 30, 40, 60, 80, 100, 120, 140, 160, 200, 300, and 400, respectively.In addition, it is important to note that the number of discrete points used in the POA optimization stage should be greater than that in the MDP optimization stage.
In the optimization calculation of IMDP, we constructed the corridor by the optimal results of MDP in a low discrete degree (i.e., 10, 20, or 30), and set up four IMDP schemes, as shown in Figure 4.The discrete degree of the four schemes is equivalent to the discrete degree of MDP with 100 discrete points.

MDP-POA
The detailed calculation results of MDP and MDP-POA in the dry year, the normal year, and the wet year are shown in Tables 3-5, respectively, where the term "MDP with 10" in the tables means that the optimization result of MDP with 10 discrete points is used as the initial trajectories of POA in MDP-POA optimization.The meaning of "MDP with 20" and "MDP with 30" are similar to "MDP with 10".The places with bold font or bold italics font in Tables 3-5 are the key points to extract the data for Table 6.Through the detailed calculation results of MDP and MDP-POA in Tables 3-5, we can extract the relevant data that can indicate that MDP-POA is superior to MDP (with 60 or 100 discrete points) on power generation, including the initial trajectories, the time consumption for initial trajectories, the number of discrete points for the POA, the total computation time, and the power generation, as shown in Table 6, which is mainly used for contrastive analysis between MDP and MDP-POA.In Table 6, the "Time1" of MDP-POA column represents the time used to obtain the initial trajectories for POA optimization by MDP, and "Time2" represents the time used in POA optimization."Total time" is the sum of "Time1" and "Time2", i.e., the run-time of MDP-POA.Several conclusions can be derived from Table 6, as shown below.less than MDP.In addition, from Table 6, it can be seen that MDP-POA can obtain a relatively better result when the initial operation trajectories are obtained by MDP with 30 discrete points, which means that when the number of discrete points of MDP is 30, on the one hand, it can be guaranteed that the final result of MD-POA is better than that of MDP, and on the other hand, the calculation time is also stable and within an acceptable range.For the discrete points of POA optimization in MDP-POA, the general value is about 125 when the number of discrete points of MDP is 30, which is the average of 80, 120, 140, and 160 in Table 6.

IMDP
Corresponding to the three typical years, the calculation results of each IMDP scheme are shown in Table 7 below.The bold font in Table 7 means the power generation of scheme "IMDP: 10 × (20/2)" and "IMDP: 20 × (10/2)" is slightly insufficient compared with scheme "MDP:100".
Therefore, on the whole, the proposed IMDP in this paper can not only greatly shorten the run-time of MDP, but also can effectively guarantee the global convergence of the algorithm.In addition, from the comparison of the four IMDP schemes, it can be seen that the schemes "IMDP: 10 × (40/4)" and "IMDP: 20 × (20/4)" are better than schemes "IMDP: 10 × (20/2)" and "IMDP: 20 × (10/2)".That is to say, in general, the wider the constructed corridor is in the optimization calculation, the better the final results are at the end.Although the run-time will generally increase with the enlargement of the corridor size, it is still much less than that of MDP.In addition, from the perspective of computing time, comparing the scheme "IMDP: 10 × (40/4)" with the scheme "IMDP: 20 × (20/4)", it can be found that the computing time of the "IMDP: 10 × (40/4)" scheme is much bigger than that of the "IMDP: 20 × (20/4)" scheme, so the scheme "IMDP: 20 × (20/4)" is superior to "IMDP: 10 × (40/4)", which indicates that it is best to set the same discrete degree in the two-stage calculation of IMDP when using it to solve a CROO problem, such as the 20 to 20 in scheme "IMDP: 20 × (20/4)".Such a parameter setting can not only achieve the final expected discrete degree, but also can be very good to avoid the unevenness of the discrete degree in the two-stage calculation of IMDP, which often results in a huge amount of total computation.
Therefore, on the whole, the proposed IMDP in this paper can not only greatly shorten the runtime of MDP, but also can effectively guarantee the global convergence of the algorithm.In addition, from the comparison of the four IMDP schemes, it can be seen that the schemes "IMDP: 10 × (40/4)" and "IMDP: 20 × (20/4)" are better than schemes "IMDP: 10 × (20/2)" and "IMDP: 20 × (10/2)".That is to say, in general, the wider the constructed corridor is in the optimization calculation, the better the final results are at the end.Although the run-time will generally increase with the enlargement of the corridor size, it is still much less than that of MDP.In addition, from the perspective of computing time, comparing the scheme "IMDP: 10 × (40/4)" with the scheme "IMDP: 20 × (20/4)", it can be found that the computing time of the "IMDP: 10 × (40/4)" scheme is much bigger than that of the "IMDP: 20 × (20/4)" scheme, so the scheme "IMDP: 20 × (20/4)" is superior to "IMDP: 10 × (40/4)", which indicates that it is best to set the same discrete degree in the two-stage calculation of IMDP when using it to solve a CROO problem, such as the 20 to 20 in scheme "IMDP: 20 × (20/4)".Such a parameter setting can not only achieve the final expected discrete degree, but also can be very good to avoid the unevenness of the discrete degree in the two-stage calculation of IMDP, which often results in a huge amount of total computation.
Taking the schemes "IMDP: 10 × (40/4)" and "IMDP: 20 × (20/4)" as examples, the storage water level variations of Shuibuya and Geheyan reservoir in the three typical years are shown in Figures 6  and 7, respectively.In the two figures, the "Up reservoir" represents the Shuibuya reservoir, and the "Down reservoir" represents the Geheyan reservoir.The green line (MDP schemes) in Figures 6 and  7 cannot be seen sometimes, because it is covered by the red line (IMDP schemes).From Table 7, it can be seen that the power generation of IMDP is worse than MDP in some cases.The reason is that the constructed corridor by the calculation result of MDP does not cover the optimal solution of MDP with 100 discrete points.In addition, as we know, MDP has a global convergence which was proved [49], and its calculation results must be the optimal value under a  From Table 7, it can be seen that the power generation of IMDP is worse than MDP in some cases.The reason is that the constructed corridor by the calculation result of MDP does not cover the optimal solution of MDP with 100 discrete points.In addition, as we know, MDP has a global convergence which was proved [49], and its calculation results must be the optimal value under a From Table 7, it can be seen that the power generation of IMDP is worse than MDP in some cases.The reason is that the constructed corridor by the calculation result of MDP does not cover the optimal solution of MDP with 100 discrete points.In addition, as we know, MDP has a global convergence which was proved [49], and its calculation results must be the optimal value under a certain discrete degree.However, from Table 7, we can see that the power generation of IMDP is often greater than MDP.The reason for this phenomenon is that there are boundary constraints which make the discrete degree in some parts of the corridor higher than the discrete degree of MDP with 100 discrete points.In fact, for all of the three typical years, this situation often happens in the dry season.
In addition, when comparing the results from this study with the research results in other literature, it can be found that the proposed MDP-POA and IMDP in this paper have a great advantage in computing time.For example, in the literature by Ji et al. [37], the computing time of MNDP is significantly greater than that of MDP, while from the Tables 6 and 7, it can be found that the computing time of MDP-POA and IMDP are much smaller than that of MDP on the premise of ensuring the quality of solution, which means that the MDP-POA and IMDP have great advantage in computing time compared to the MNDP method.For the parallel MDP provided in the literature by Zhang et al. [11], in solving the joint optimization of two reservoirs with a discrete degree of 48, its computing time of the three kinds of parallel modes is between 2 and 4 min, however, for the IMDP algorithm proposed in this paper, besides the scheme "IMDP: 10 × (40/4)", the computing time of the rest of the schemes is less than 1 min.Addtionally, it is important to note that the discrete degree of IMDP in this paper is equivalent to 100, whose number of combinations of storage volume is much bigger than that of 48 in the parallel MDP; moreover, IMDP is a serial calculation mode and only one central processing unit (CPU) core is needed in the calculation, while the calculation of the parallel MDP calls over a dozen CPU cores.Therefore, in general, the IMDP proposed in this paper is better than the parallel MDP provided in the literature by Zhang et al. [11].

Conclusions
Aiming at the "curse of dimensionality" problem of MDP, this paper proposed two dimension reduction methods after a characteristic analysis of the MDP algorithm, i.e., MDP-POA and IMDP.The Qingjiang cascade reservoirs were selected as a study case, with conclusions summarized as follows.
(1) The two methods both can greatly shorten the run-time of baseline MDP and effectively hold its optimality to a certain extent.Although the incremental power generation is not significant, the run-time reduction is especially significant; MDP-POA can shorten the run-time to around 1/10 to 1/100 of the baseline MDP, and IMDP can shorten the run-time to around 1/30 to 1/500 of the baseline MDP.
(2) For MDP-POA, the higher the discrete degree is in the MDP optimization stage, the better the results are by the POA optimization.For IMDP, the wider the constructed corridor is in the optimization calculation, the better the final results are at the end.Although the run-time will generally increase with the augment of the discrete degree or the enlargement of the corridor size, the superiorities of MDP-POA and IMDP are still significant compared with MDP.(3) There is little difference on power generation between MDP-POA and IMDP, but on run-time, IMDP is better than MDP-POA.
In most cases, the power generation of MDP-POA and IMDP is better than MDP, but is insufficient in some cases, which indicates that the proposed algorithms still have a certain deficiency.So, there is still a lot of work that needs to be done in the future.For practical application, more advanced improvements and optimization designs for the two algorithms need to be further studied.For the validity of the proposed methods for other reservoirs, more case studies need to be carried out.

Figure 1 .
Figure 1.The discretization procedure of cascade reservoirs' storage volume and their combination rule in Multi-dimensional Dynamic Programming (MDP).

Figure 1 .
Figure 1.The discretization procedure of cascade reservoirs' storage volume and their combination rule in Multi-dimensional Dynamic Programming (MDP).

Figure 2 .
Figure 2. Initial operation trajectories derivation and its further optimization by the Progressive Optimality Algorithm (POA) in MDP-POA.

Figure 2 .
Figure 2. Initial operation trajectories derivation and its further optimization by the Progressive Optimality Algorithm (POA) in MDP-POA.

Figure 3 .
Figure 3. Potential optimal solution region derivation and corridor construction of Improved Multidimensional Dynamic Programming (IMDP).

Figure 3 .
Figure 3. Potential optimal solution region derivation and corridor construction of Improved Multi-dimensional Dynamic Programming (IMDP).

Figure 4 .
Figure 4.The construction of four schemes of IMDP with an equivalent discrete degree of 100 points in MDP.

Figure 4 .
Figure 4.The construction of four schemes of IMDP with an equivalent discrete degree of 100 points in MDP.

Figure 5 .
Figure 5. Plane sketch map of the Qingjiang cascade hydropower stations.

Figure 5 .
Figure 5. Plane sketch map of the Qingjiang cascade hydropower stations.
t is the state variable; Q t is the decision variable, which is determined by beginning state V Sb

Table 1 .
[48]meters of the Shuibuya and Geheyan hydropower stations in Qingjiang River.stations are under regular operation (i.e., Shuibuya and Geheyan), and thus they are selected as the focus of the research in this paper.The basic parameters of the two hydropower stations are shown in Table1, and the plane sketch map of the Qingjiang River cascade hydropower stations is shown in Figure5[48]. hydropower

Table 1 .
Parameters of the Shuibuya and Geheyan hydropower stations in Qingjiang River.

Table 2 .
Typical runoff data for each month in the wet, normal, and dry years (m 3 /s).

Table 2 .
Typical runoff data for each month in the wet, normal, and dry years (m 3 /s).

Table 3 .
Detailed calculation results of MDP and MDP-POA in the dry year.

Table 4 .
Detailed calculation results of MDP and MDP-POA in the normal year.

Table 5 .
Detailed calculation results of MDP and MDP-POA in the wet year.

Table 7 .
Results of MDP and IMDP.