Deep Q-Value Neural Network (DQN) Reinforcement Learning for the Techno-Economic Optimization of a Solar-Driven Nanofluid-Assisted Desalination Technology

Abstract

A solar-driven desalination system, featuring a single-slope solar still is studied here. For this design, Al₂O₃ nanofluid is utilized, and the condition achieving the highest efficiency and cost-effectiveness is found using a reinforcement learning called a deep Q-value neural network (DQN). The results of optimization are implemented for the built experimental setup. Experimental data obtained under the climatic conditions of Tehran, Iran, are employed to compare the enhancement potential of the optimized solar still system with nanofluid (OSTSWNF) with the solar still system with water (STSWWA). The hourly fluid temperatures in the basin as well as the hourly and cumulative freshwater production (HFWP and CFWP) are discussed. A number of other parameters, including daily water production and efficiency in addition to the cost per liter (CPL) of the resulting desalinated water, are also taken into account. The results reveal that annual water production increases from 1326.8 L to 1652.4 L, representing ~25% growth. Moreover, the annual average efficiency improves by ~32%, rising from 41.6% to 54.7%. A great economic enhancement is seen as well, with the CPL decreasing by ~8%, i.e., from USD 0.0258/L to USD 0.0237/L.

1. Introduction

Humanity is dealing with increasing survival challenges; in addition to energy and environmental issues, drinking water shortages are becoming acute and more widespread [1]. Nonetheless, despite the severe shortages of potable water seen in some areas, saline water is abundant on Earth, and therefore, making it available for drinking should be pursued as a practical option [2]. Doing so will require the further development of desalination technologies.

Conventional desalination technologies require huge amounts of energy and impose considerable cost [3]. As a result, there has been a trend toward powering them with renewable energy resources, especially solar energy [4,5,6]. One of the most promising and economical solar-driven desalination technologies is solar still systems [7]. Their cost is much lower than that of rivals, and they can be installed with small or large capacities in both remote and urban areas [8]. Due to their working principle, they are also highly reliable [9].

In a solar still, a basin stores saline water, some of which evaporates by absorbing the energy from sun, leaving behind the salt [10]. On contacting a surface and the heat dissipation to the atmosphere, the evaporated water returns to the liquid phase in the form of desalinated freshwater and is collected in a tank. As is typical with technologies, efforts have been made to boost the performance of solar stills and promote advantages like high thermal conductivity [11,12,13,14], enhance their heat absorption capacity significantly [15,16,17], and make them more widely available [18,19,20,21].

Some studies have investigated the impact of using nanofluids on solar sill performance. Elango et al. [22] examined four nanofluids for a single-slope solar still: SnO₂, Al₂O₃, Fe₂O₃, and ZnO. Al₂O₃ was found to work best, enhancing freshwater production by around 30%. With a pyramid-type solar still, in an experimental study, Sharshir et al. [23] evacuated tube pipes and employed nanofluids for performance improvement. Two materials, namely, CuO and carbon black, were compared. Daily efficiency increased from 48% for the conventional system to, respectively, 64.5% and 61% when carbon black and CuO were utilized.

Rashidi et al. [24], taking advantage of computational fluid dynamics (CFD) and analyzing the performance of a stepwise solar still, demonstrated that using nanomaterials within the range of 0 to 5% increased water production by 22%. Another work, also by Rashidi et al. [25], employed numerical modeling to analyze the performance of a nanofluid-based (Al₂O₃) single-slope solar still.

A number of investigations have also been carried out by Parsa et al. [26,27,28]. In [26], several alternatives, including a double-slope silver-based nanofluid solar still heated by a thermoelectric device, were studied. The findings demonstrated a daily efficiency 50.8% greater than that of the conventional system, in which neither of the two enhancements was employed. In [27], experiments were carried out to investigate the performance of a solar still filled with nanofluids at a place in Tehran, Iran, with a high elevation above sea level. Like [26], a silver-based nanomaterial was utilized. Using nanofluids in this case led to a maximum energy improvement of 55.98%. In [28], the performances of three nanomaterials, gold, silver, and titanium, were analyzed in a pyramid solar still. The economic assessment showed costs per liter (CPL) of USD 0.0289 and USD 0.0065 per liter per square meter for solar stills with the oxides of gold and silver as nanomaterials, respectively.

A new design for a single-slope solar still was proposed and experimentally tested by Sohani et al. [1]. In it, side mirrors and solar tracking strategies were adopted, resulting in a huge rise in system performance: a 22.3% growth in daily efficiency. The performance of the design proposed in [1] was also simulated using machine learning approaches in [29]. In addition, that design was optimized using energy and environmental, economic, and exergy (4E) criteria in [30], in which only dimensions and not operating parameters are the decision variables. Moreover, the priority for using that design was chosen by Jafari et al. [31] among different cities in Iran, each of which was representative of one of the diverse climatic conditions of the country.

Nazari et al. [32] employed thermoelectric modules in a solar still differently than in [26]; this time they were employed to prepare cool air to pass over the still’s glass. The investigated system was of the single-slope type in which Cu₂O was used as the nanomaterial. A CPL of USD 0.0218 per liter per square meter was found for the system. Combining two nanomaterials, copper and aluminum, to enhance solar sill performance was suggested and assessed by El-Gazar et al. [33]. Their results indicate that using this combination led to 23.21% and 49.54% growth in freshwater production in winter and summer, respectively.

With reference to the conducted literature review, to the best of the researchers’ knowledge, there were three conditions for nanoparticles in the studies. One considered a constant concentration for them. Another changed the concentration discretely or continuously without finding the optimal concentration through the systematic approach. Even if the concentration was optimized, the study gave a constant value for the whole year as the optimum value. Therefore, the current study was undertaken. Here, DQN, as one of the cutting-edge reinforcement learning methods for optimization, is employed for the dynamic optimization of the system. Employing DQN, an hourly profile of nanoparticle concentrations during a year is obtained. The system is optimized to maximize the freshwater production and efficiency of the system.

The use of DQN optimization is the novelty of the present manuscript. In contrast with the conventional methods that have been employed for the optimization of solar stills so far, DQN is able to perform dynamic multi-objective optimization. That is, if the decision variable of optimization is nanoparticle concentration as in this study, using conventional multi-objective optimization will introduce a constant concentration for all hours of the day, whereas with DQN, an hourly profile of nanoparticle concentration is obtained, reflecting the improvement give by multi-objective optimization.

Based on the results obtained in [22], Al₂O₃ is used as the nanomaterial. For the solar still system with water (STSWWA) and the solar still system with nanofluid, experimental information is utilized, and the fluid temperature in the basin (T_fluid), hourly freshwater production (HFWP), and cumulative freshwater production (CFWP) are drawn and discussed in detail for a sample day. Moreover, the changes in the freshwater production and daily average efficiency during a year are also plotted and discussed. In addition, and to bring a wider perspective, the daily efficiency and CPL for the two systems are compared.

2. The Studied Solar-Driven Desalination System

Figure 1 introduces the investigated solar still in this study, a single-slope type made of steel. The side walls are painted black to absorb the highest amount of solar radiation from the sun. In addition, a flat-plate solar collector is used in the system, which turns into an active type. The width and height of the basin are both 1 m, which makes a basin with the area of 1 m². In addition, the flat-plate collector has an area of 3 m². The solar still experiments with water were conducted in 2019 on days 9, 18, and 27 of each month. The experimental data for the nanofluid-assisted solar still were collected in 2020 on the same days as in 2019. The measuring instruments are introduced in

Table 1. The specifications of the measuring instruments.

Performance Indicator	Device	Uncertainty	Working Range	Unit
Irradiance	Solar power meter	10.0	0.0–2000.0	W·m−2
Wind velocity	Wind meter	0.2	0.0–10.0	m·s−1
Temperature of water in basin	Thermocouple (K-type)	0.6	0.0–1000.0	°C
Ambient temperature	Ambient thermometer	0.1	0.0–80.0	°C
Fresh water production	Graduated cylinder	5.0	0.0–2000.0	mL

. The annual changes in ambient temperature and solar radiation are depicted in Figure 2.

In addition, it should be noted that the depth of water inside the still is kept at 3 cm, which is within the range of other published experimental investigations in the field [34,35,36]. As one more point about the experiments, the condensate was analyzed to check if the concentrations of nanoparticles were within the allowable limit. The specifications for the nanofluid used and the preparation process employed in this present paper are exactly the same as those in [22], in which further information can be found. The nanoparticle concentration changes during different hours of a day as a decision variable of multi-objective optimization, varying from 0.1 to 0.3%. The best concentration for each hour is determined using DQN.

3. Materials and Methods

This part covers the materials and methods of this study in three subsections: how to calculate the studied performance indicators, the details of the uncertainty analysis, and the introduction of DQN.

3.1. Studied Performance Indicators

Separate from T_fluid and the amount of freshwater production, which were determined from the experiments, two other performance indicators were investigated in this study:

• Efficiency.
• Cost per liter (CPL).

The next paragraphs explain how the two indicators are calculated. It should be noted that the experimental data for STSWWA were recorded on 18 September 2019, whereas the data recording for OSTSWNF occurred on 9 September 2020. This one-year gap exists because the idea of using nanofluid to enhance performance came to the authors long after they had published the previous article [1]. As seen in

Table 2. The hourly values of the significant data measured during the experiments.

Hour	Ambient Air Temperature (°C)		Solar Radiation (W·m−2)		Wind Velocity (m·s−1)
	18 September 2019	9 September 2020	18 September 2019	9 September 2020	18 September 2019	9 September 2020
8	18	19	266.8	268.7	1.1	1.2
9	20	20	441.8	441.6	2.2	2.3
10	23	22	610.4	612.6	1.5	1.4
11	24	24	739.4	741.2	1.4	1.3
12	25	25	809.6	810.3	1.6	1.6
13	26	27	812.0	814.1	1.3	1.0
14	27	28	742.2	744.0	1.1	1.2
15	28	29	612.7	612.9	0.9	0.8
16	29	30	449.2	451.3	1.0	0.9
17	28	28	271.7	273.3	2.0	1.8
18	28	27	263.4	261.9	2.6	2.5

, the conditions on the two days of interest are close enough to ensure a fair comparison.

3.1.1. Efficiency

The ratio of a given input to a gained outcome is called efficiency. In a solar still, the given input is the received solar radiation, and the gained outcome is the heat absorbed to lead to water evaporation. Therefore [37]:

$$eff=\frac{m_{FW}{h}_{evap}}{A_{rec}G}$$

(1)

In Equation (1), $eff$, $m$, $h$, $A$, and $G$ stand for efficiency, enthalpy, area, and solar radiation, respectively. The subscripts FW, evap, and rec represent freshwater, evaporation, and receiving radiation, respectively.

3.1.2. Cost Per Liter (CPL)

As the name shows, CPL reveals how much producing every liter of freshwater costs. It is computed based on Equation (2):

$$CPL=\frac{C_{ST}}{V_{FWP}}$$

(2)

in which $C_{ST}$ is the system cost and is determined using Equation (3):

$$C_{ST}=C_{PEP}+C_{OPM}-C_{SAL}$$

(3)

where $C_{PEP}$, $C_{OPM}$, and $C_{SAL}$ are respectively the costs for the initial purchase, for operating and maintaining the system, and for salvage (selling a part of useable materials and components at the end of the system’s lifetime). These three items are obtained from Equations (4) to (6), respectively [1,38,39]:

$$C_{PEP}=PP\times CREF=PP\times \frac{i}{1-{\left(\frac{1}{1+i}\right)}^N}$$

(4)

$$C_{OPM}=f_{OPM}\times C_{PEP}$$

(5)

$$C_{SAL}=f_{SAL}\times C_{PEP}\times SFUF=f_{SAL}\times C_{PEP}\times \frac{i}{{\left(i+1\right)}^N-1}$$

(6)

In Equations (4) to (6), $PP$, $CREF$, $i$, $N$, and $SFUF$ denote the initial purchase price, capital recovery factor, inflation, year of system operation (lifetime), and sinking fund factor, respectively. The operating and maintenance and salvage costs are considered as a fraction of PP and shown respectively by $f_{OPM}$ and $f_{SAL}$. The parameters that are needed to determine $C_{ST}$ are introduced in

Table 3. The required parameters for the calculation of CPL [1].

Parameter	Unit	Value
$i$	percent	5
$N$	years	15
$f_{OPM}$	percent	15
$f_{SAL}$	percent	20

.

$V_{FWP}$ in Equation (2) is the annual volumetric freshwater production of the solar still. An approach described in the literature [1,40] is utilized to determine $V_{FWP}$ based on the information provided for a day in a mild month like September. Following this approach, $V_{FWP}$ is considered to be 300 times greater than the value of freshwater production on a mild day. The value of 300 comes from the number of average sunny days in Middle Eastern countries like Iran.

3.2. Uncertainty Analysis

For the parameters obtained directly from the experiments, the values provided by the manufacturer are utilized. For the other parameters, i.e., the ones that were computed based on the experimental data, the propagation of uncertainty is employed, according to which the uncertainty of a parameter like Z that is determined based on the parameters X and Y is:

$${\delta }_Z=\sqrt{{\left(\frac{\partial Z}{\partial X}\right)}^2{\partial }_X^2+{\left(\frac{\partial Z}{\partial Y}\right)}^2{\partial }_Y^2}$$

(7)

where $\delta$ is the uncertainty and $\partial$ denotes the partial differentiation.

3.3. Optimization Using DQN

As indicated, in this study, DQN is employed as the method for the dynamic optimization of the nanofluid-enhanced solar still. The working principle of DQN is schematically depicted in Figure 3. In DQN, there is an agent and an environment, in addition to state and action signals, and a reward. The agent receives the state from the environment, and it decides to take an action to maximize the reward at the given time. Based on the action taken in the environment, the state changes, and the new state as well as the reward are observed by the agent. The process goes on until one of the stopping criteria, which is usually when the number of time intervals, known as an epoch, finishes.

In this study, three objectives are defined as the rewards for DQN. They are the hourly yield, efficiency, and CPL. The results of DQN are obtained using the widely used simulation approach for modeling the nanofluid-enhanced solar still available in [41,42,43]. Moreover, the outcome considered is the concentration of nanoparticles in the nanofluid. As discussed, the results obtained are implemented for the built experimental setup to evaluate the enhancement potential of DQN. The process flow diagram of DQN optimization is given in Figure 4.

4. Results and Discussion

Following is a detail discussion of the results of this study including the daily and annual values, as well as the results of uncertainty.

4.1. The Daily Values

When a nanofluid is utilized, the heat absorption capacity is enhanced. Therefore, if other parameters remain constant, the optimized solar still system with nanofluid (OSTSWNF) enjoys a higher T_fluid than the solar still system with water (STSWWA), as Figure 5 shows. For example, T_fluid for STSWWA at 8 am, 10 am, and 12 pm is 34.2 °C, 40.0 °C, and 58.2 °C, respectively. The corresponding values for OSTSWNF increase considerably, reaching 40.2 °C, 48.3 °C, and 68.3 °C, respectively. Moreover, as noon (12 pm) approaches, the difference between T_fluid and the other hours becomes greater. At the three indicated hours, the differences between OSTSWNF and STSWWA were 6.0 °C, 8.3 °C, and 10.1 °C, respectively.

The maximum values of both T_fluid and the temperature difference between OSTSWNF and STSWWA are found for solar noon, i.e., 1 pm. For that hour, T_fluid for OSTSWNF and STSWWA are, respectively, 75.3 °C and 64.4 °C, a difference of 10.9 °C. After solar noon, T_fluid goes down. At 3 pm and 5 pm, T_fluid for STSWWA are 57.5 °C and 52.6 °C, whereas at these two hours, OSTSWNF has T_fluid of 67.7 °C and 60.1 °C, respectively.

A higher T_fluid is accompanied by more water evaporation and, consequently, greater daily efficiency and HFWP, as Figure 6 and Figure 7 demonstrate, respectively. Figure 6 shows that the daily efficiency jumps from 43.1% to 56.5%, a spike of 31.09%, which is remarkable.

The same behavior seen in T_fluid is also seen in HFWP, that is, an increase from 8 a.m. to 1 p.m. and a decrease afterwards. Respectively, OSTSWNF produces 169.7 mL and 571.3 mL freshwater at 9 a.m. and 11 a.m., whereas STSWWA is able to provide 195.1 mL and 691.2 mL at the same times. These results represent differences of 25.5 mL and 120.0 mL, respectively. The greatest difference between HFWP and the two systems is observed at 1 pm, with OSTSWNF producing 1137.8 mL and STSWWA 882.0 mL. The difference between the values is equal to 255.8 mL, which is 10.05 and 2.13 times greater than those for 9 a.m. and 11 a.m., respectively.

At 2 and 4 h later than the peak time, i.e., at 3 p.m. and 5 p.m., OSTSWNF gives 816.6 mL and 297.6 mL freshwater, whereas HFWP for STSWWA is 643.0 mL and 261.1 mL. This result indicates enhancements of 173.6 mL and 36.6 mL when OSTSWNF is utilized instead of STSWWA.

A better insight into the water production improvement gained by using OSTSWNF instead of STSWWA is gained by comparing the hourly profiles of CFWP, as illustrated in Figure 5. At 9 a.m., the CFWP of OSTSWNF is 0.20 L. One hour later, i.e., at 10 a.m., it increases 2.74 times and reaches 0.55 L. At 11 a.m. and 12 p.m., CFWP is 6.17 and 11.04 times better than those at 9 am, with CFWP becoming 1.24 L and 2.22 L, respectively. The observed values at 9 a.m., 10 a.m., 11 a.m., and 12 p.m. indicate 14.91%, 16.85%, 19.12%, and 22.06% growth in the CFWP of the system compared with STSWWA, respectively.

At solar noon, which is 1 pm, CFWP for STSWWA is 3.36 L. It experiences 24.33% growth and becomes 3.36 L for OSTSWNF. One, two, three, and four h after solar noon, i.e., at 2 p.m., 3 p.m., 4 p.m., and 5 p.m., OSTSWNF offers 26.37%, 26.47%, 26.04%, and 25.39% better CFWP than STSWWA, with 4.47 L, 5.29 L, 5.83 L, and 6.13 L freshwater obtained from the beginning of the day, respectively.

The daily freshwater production of each system is CFWP at the end of the investigated time period, i.e., 6 p.m. Based on Figure 8, the daily freshwater production of STSWWA is 5.01 L, whereas 6.26 L freshwater is obtained using OSTSWNF. Therefore, the daily water production has increased by 24.96%, a huge achievement.

4.2. The Values throughout the Year

In addition to the values during the sample days, the monthly changes in the efficiency and freshwater production of STSWWA and OSTSWNF are also obtained from the experiments and compared. Figure 9 and Figure 10 report the annual profiles of the two indicated indicators. Based on Figure 10, STSWWA is able to produce 64.1, 115.7, and 166.8 L in February, April, and June, while the values of yield at the same months for OSTSWNF are 79.7, 143.2, and 208.7 L, respectively. The highest amount of freshwater production is seen for July, where STSWWA is able to produce 171.4 L and OSTSWNF has the capability of 213.0 L, resulting in 24.3, 23.8, 25.1, and 24.3% in the four indicated months, respectively. During a year, 1326.8 and 1652.4 L freshwater are generated by STSWWA and OSTSWNF, respectively. Therefore, around 25% growth in the annual yield of the system is observed when DQN optimization is applied.

As Figure 10 indicates, the DQN optimization also enhances efficiency. Like yield, the maximum efficiency of both systems is found for July. In that month, values of 58.5 and 44.3% for STSWWA and OSTSWNF are reported. On the one hand, in August, October, and December, STSWWA has the efficiencies of 43.7, 42.4, and 38.9%, respectively. On the other hand, the corresponding values for OSTSWNF are 57.6, 55.7, and 51.2%, also respectively. Consequently, improvements of 32.1, 31.8, 31.4, and 31.6% are found for the discussed months, respectively. According to the obtained results, the average annual efficiency is 41.6% in a year but reaches 54.7% with DQN optimization, a 31.6% increase.

As seen, the amounts of daily and consequently annual freshwater production are higher using OSTSWNF compared with STSWWA. However, OSTSWNF imposes a greater cost as well, although the growth in the annual freshwater production of OSTSWNF is much greater than the increase in cost. Therefore, OSTSWNF enjoys a lower CPL than STSWWA. According to the results provided in Figure 11, STSWWA has a CPL of USD 0.0258/L that decreases considerably, reaching USD 0.0237/L, an 8.14% improvement. Consequently, not only does OSTSWNF give much more significant technical performance indicators, it is also much more cost-effective.

4.3. Average Relative Uncertainty Values

In every experimental study, researchers should ensure that the obtained data have been gathered correctly. For this purpose, following similar experimental investigations in the field, uncertainty analysis is employed. The results are provided in

Table 4. The required parameters for the calculation of CPL.

Performance Indicator	ARU
Irradiance	0.041
Wind velocity	0.046
Temperature of water in basin	0.352
Ambient temperature	0.917
Fresh water production	1.408

, where the obtained average relative uncertainty (ARU) for each case is close to the available ones in the literature. Therefore, the accuracy of the experiments has been verified.

5. Conclusions and Remarks

This study has investigated a novel design for a solar still desalination system and assessed the impacts of using Al₂O₃ and DQN optimization on performance enhancement by taking advantage of experimental measurements. The experimental data were obtained under the climatic conditions of Tehran, Iran. Considerable increases in the fluid temperatures in the basin and freshwater production were observed, especially during the two-three hours on either side of solar noon, i.e., 1 pm. Moreover, according to the calculations, OSTSWNF enjoyed daily efficiency, daily water production, and CPL of 56.5%, 6.26 L, and USD 0.0237/L, respectively, which were 31.09%, 24.96%, and 8.14% better than those of STSWWA. Moreover, around 25 and 32% improvements in freshwater production and average efficiency during the year were observed with DQN. The remarkable changes observed suggest that using nanofluids could be an efficient way to enhance the performance of solar desalination technologies.