To evaluate the proposals of this paper, in this section we carry on several experiments in order to confirm the contribution of the model proposed, as well as the metaheuristics used to optimize the inputs to the model.
There is a constraint on the execution time of the algorithm, since every 15 min. the optimization of a plant status must be completed to start a new optimization on another status.
6.4. Heuristic Results
Once the model has been designed and trained, the optimization phase is devoted to find a good combination of input values to optimize the objective functions reported in
Section 3 (maximize the H
${}_{2}$ production; and, maximize the profit of the plant).
From the total dataset of samples, we have selected 40 diverse instances (20 for each objective function) in order to evaluate the optimization algorithms. In general, the larger the test dataset, the better. However, 20 samples represent a reasonable rule-of-thumb in this context (as it is pointed out in [
15]). For each objective we sorted all the samples in a descending order with respect to the quality of the solution. Let us remember that each sample corresponds to a real scenario of the SR plant, i.e., the set of input values to the plant and the produced outputs. Once the samples were sorted, then we selected a random instance in each of the following percentiles: 8, 10, 14, 20, 26, 30, 32, 38, 44, 49, 50, 56, 62, 68, 69, 74, 80, 86, 89, and 92, obtaining a total of 20 different instances per objective function. All of the algorithms were provided with the same initial point (the real values of the plant) for their optimization process.
In
Table 6, we report the average improvement (in kg/h) obtained with the proposed algorithms with respect to the original production of H
${}_{2}$: the Genetic Algorithm (GA), the Memetic Algorithm based on First Improvement (MA-FI), and the Memetic Algorithm based on Best Improvement (MA-BI). We refer the reader to
Section 5 for the description of each procedure. For each method we have tried different population sizes. Notice that the sizes of the population tried in the GA are much larger than the ones used for the MA, because the GA is faster than the MA and, therefore, it can handle more solutions within the same execution time. For each algorithm and population size, we have tested two different probability distributions of the ranking used in the selection operator. Each of them is parameterized by the value of the constant
c in the table). Finally, we have also tested three different values for the probability that a particular gene suffers a mutation (denoted by
${p}_{m}$ in the table). In
Table 6, we also report the production of H
${}_{2}$ of the best solution that is produced by the general black-box optimization algorithm MADS [
31]. In this case, there is not population size and the rest of the parameters provided for the GA, MA-FI, and MA-BI reported in the head of the columns of the table apply. However, we provide the result value in this table in order to ease the comparison among the methods.
The values that are reported in
Table 6 are calculated as the result of the average of the improvement for each of the 20 instances considered for
${f}_{1}$. Notice that each algorithm was running up to 15 min. (i.e., the time limit indicated by the engineers in the real SR plant), and all of them were provided with the same initial point (the actual status of the SR plant). As it is possible to observe, the maximum average improvement for the instances considered was 732.64 kg/h of H
${}_{2}$ (highlighted in bold-type font in
Table 6). This value was reached by the state-of-the-art algorithm MADS. However, the differences with the rest of the algorithms are neglectable, especially with those that were obtained by the GA with different parameter configurations.
Further than the average of the values of the objective function for each instance, it is common to use additional statistical measures in order to compare the results that were obtained by the tested algorithms, such as the traditional deviation to the best solution found. In the black-box optimization context, it is also common to report the performance of the algorithms, when compared for the same number of objective function evaluations, through the use of convergence plots, or performance profiles, among others (see Appendix A in [
15] for a wider description). However, in this case, the use of metrics that are related to the number of evaluations of the objective function lacks of sense, since we are tackling a real optimization problem bounded by the time, but not by the number of evaluations. Furthermore, the evaluation is made by an ANN model that does not fall into any real-cost scenario.
In order to complete our comparisons, in
Table 7, we report the average accuracy of the algorithms for the considered instances (the best results are highlighted in bold-type font), which compares the quality of the solutions provided by an algorithm with respect to the best solution found [
15]. In particular, the accuracy is calculated, as follows:
where
f represents the evaluation function,
${x}_{a}$ is the best solution found by the algorithm being evaluated
a (in this case a={GA, MA-FI, MA-BI, MADS}),
${x}_{0}$ is the initial solution, and
${x}^{\star}$ is the best-known solution for the instance. Notice that, in this case, we report the best value found by the algorithm within the time limit.
As we can observe in
Table 7, and despite the fact of the small differences in the average of the objective function, it is not possible to find differences in terms of accuracy between MADS and most of the configurations of GA, even when reporting three decimal points. Additionally, the differences with respect to the MA configurations are very small.
The results that were found for the
${f}_{1}$ reported in
Table 6 and
Table 7 suggest that the optimization problem in this context does not suppose a hard task for any of the algorithms compared. In fact, a general framework for black-box optimization (MADS) is able to find the best solutions that were found in the experiment.
Similarly, in
Table 8, we report the average improvement (in €/h) that was obtained with the proposed algorithms for the
${f}_{2}$, with respect to the original profit of the SR plant. Again, we also report in this table the best solution produced by the general black-box optimization algorithm MADS [
31]. Notice that, in this context, MADS was unable to find a feasible solution in four out of the 20 instances considered. Despite of the fact that we provided to all the algorithms the current status of the plant (which is considered a feasible starting solution), it might happen that the hydrogen production predicted by the model results in a deviation larger than
$\pm \u03f5$ with respect to the real one and, therefore,
${c}_{14}$ is violated. Subsequently, the solution is considered infeasible in our model. Therefore, we did not include those instances in the average reported for MADS. In this case, we could find larger differences among the methods. The best method was the MA-BI that was configured with a population of eight individuals,
${p}_{m}=0.20$,
$c=0.95$ which obtained an average improvement of 782.76 €/h (highlighted in bold-type font in
Table 8). Notice that, in this case, all of the combinations of parameters within the MA outperformed either the MADS or the different GAs proposed.
Again, in
Table 9, we report the accuracy for the evaluated methods. In this table, we can find significant differences between the best algorithm (highlighted in bold-type font) and the rest of the compared methods. Observing the results, we can conclude that the
${f}_{2}$ is considerably complicated for the evaluated methods than
${f}_{1}$. In this case, all of the proposed MAs outperformed either the rest of the GA variants and also the results that were obtained by MADS.
Notice that the prices per ton necessary to calculate the profit of SR plant are very volatile. In our experiments, we have used the following values to calculate the objective function ${f}_{2}$: natural gas = 16.05 €/ton; low-pressure steam = 13.00 €/ton; high-pressure steam = 19.01 €/ton; and, H${}_{2}$ = 1000.00 €/ton.
As a final experiment, in
Figure 2, we graphically illustrate the behavior of the model (denoted as Modeled in the figure) and the optimized model (denoted as Optimized in the figure) with respect to the real behavior of the SR plant (denoted as Original in the figure) for a six months period. The ANN model predicts the behavior of the plant with high correlation, as it is possible to observe in the figure. On the other hand, the optimized model clearly increases the benefit per hour obtained.