Parallelization of a 3-Dimensional Hydrodynamics Model Using a Hybrid Method with MPI and OpenMP

Abstract

Process-based numerical models developed to perform hydraulic/hydrologic/water quality analysis of watersheds and rivers have become highly sophisticated, with a corresponding increase in their computation time. However, for incidents such as water pollution, rapid analysis and decision-making are critical. This paper proposes an optimized parallelization scheme to reduce the computation time of the Environmental Fluid Dynamics Code-National Institute of Environmental Research (EFDC-NIER) model, which has been continuously developed for water pollution or algal bloom prediction in rivers. An existing source code and a parallel computational code with open multi-processing (OpenMP) and a message passing interface (MPI) were optimized, and their computation times compared. Subsequently, the simulation results for the existing EFDC model and the model with the parallel computation code were compared. Furthermore, the optimal parallel combination for hybrid parallel computation was evaluated by comparing the simulation time based on the number of cores and threads. When code parallelization was applied, the performance improved by a factor of approximately five compared to the existing source code. Thus, if the parallel computational source code applied in this study is used, urgent decision-making will be easier for events such as water pollution incidents.

1. Introduction

In areas with monsoon climates, including South Korea, changes in hydraulic characteristics throughout the year are significant because weather conditions such as precipitation and air temperature change throughout the year [1,2]. The flow rate is strong during the flood season, but weak during the dry season. During the summer, when the temperature is high, stratification develops and sometimes causes various environmental problems [3]. In particular, the multi-functional weirs built as part of the Four Major River Restoration Project increase the residence time and water depth, which deepens the stratification phenomenon.

Changes in hydraulic characteristics also affect water quality [4]. Therefore, the vertical layers need to be classified to precisely simulate hydraulics and water quality for these conditions. Furthermore, cyanobacteria, which are generated in large quantities during the summer, often have different distribution characteristics along the lateral direction of a river, and the water quality distribution worsens downstream around basic environmental facilities or confluences of major pollutant sources. Vertically integrated 2D or x-z 2D models are limited in their ability to reproduce these 3D variation characteristics. Therefore, the application of a 3D model with an appropriate vertical/horizontal resolution is required, but the computational requirements can increase significantly if a model is constructed using 3D high-resolution grids [5]. In particular, the computational requirements increase sharply if a long-term simulation is performed for an entire river, such as the Nakdong River. To overcome this problem, it is necessary to reduce the time required to conduct the calculations by adjusting the number of calculations or calculation intervals using certain techniques, such as parallel calculation, independent computations of a hydraulic-water quality model, or application of an implicit scheme.

In particular, research on parallelization is very important in the field of modeling. With advancements in computer performance, it is possible to simulate various phenomena on a large scale and over an extended period. However, the number of input data items required for the model has increased. Furthermore, the number of parameters of the model required for simulation has also increased. Most of the models currently in use are executed serially, resulting in the simulation taking a long time to complete [6]. The evolution of the model conversely slows down the process. To overcome this issue, the model must be parallelized. The large-scale hydrological model simulates the water resources, changes in water quality, and water circulation globally. Therefore, much data such as weather, climate, runoff, and topography are required for modeling. For this reason, previous studies have tried to reduce the simulation speed by applying a parallelization technique to a large-scale basin model. Neal et al. [7] applied three parallelization techniques based on OpenMP, message passing, and specialized accelerator cards to improve the simulation speed of a 2D flood inundation model that requires various input data. Rouholahnejad et al. [8] significantly reduced the time to calibrate parameters by parallelizing the Soil and Water Assessment Tool (SWAT) model. Liu et al. [9] improved the simulation speed by parallelizing the grid-based distributed hydrological model with OpenMP. Avesani et al. [6] developed HYPERstreamHS, which parallelizes a large-scale river basin model to efficiently consider various hydraulic structures. In particular, the parallel performance was better in the large-scale model than in the small-scale model [9,10].

Parallel calculation is an efficient calculation technique that uses multiple resources simultaneously through job allocation and data distribution via a communication method, whereby processes with separate local memories send or receive data for sharing. A parallel calculation should be divided into separate task fragments that can be solved simultaneously, and should be solved in less time using multiple calculation resources rather than a single one. For example, Barney [11] proposed a conceptual scheme for parallel computing performed by multiple processes by subdividing a problem. Parallel computing methods include message passing interfaces (MPIs) based on a distributed memory system [12], open multi-processing (OpenMP) based on shared memory [13], and a method based on manycore processors such as an Intel Xeon or a graphics processing unit (GPU). Each parallel computing method has advantages and disadvantages. OpenMP is simple to implement, but it has limited scalability, depending on the size of the shared memory system. In contrast, MPI is scalable for high-dimensional problems, but its computing efficiency tends to decrease as the number of CPU cores increases because of the increase in communication between the CPUs. Thus, a high-performance coding technique is required to solve this problem effectively. In the parallelization of manycore processors such as GPUs, the device dependency is high, and the performance is improved only for certain types of calculations. In recent years, studies have been actively conducted on hybrid techniques that combine the OpenMP and MPI methods [14,15,16,17,18,19,20]. Recently, there has been a study that applied a hybrid technique combining OpenMP and MPI to an urban flood model. It was confirmed that the parallelization ability was better than with OpenMP or MPI solely [21].

Among various process-based numerical models, the Environmental Fluid Dynamics Code (EFDC) was developed with the FORTRAN language by the Virginia Institute of Marine Science (VIMS) in the United States, and the Environmental Protection Agency (EPA) released a generalized vertical grid (GVC) version [22,23,24,25]. The numerical scheme employed in EFDC to solve the equations of motion uses a second-order accurate spatial finite difference on a staggered or C grid, and the EFDC model’s internal momentum equation solution, at the same time step as the external, is implicit with respect to vertical diffusion [25]. See reference [25] for more information about the model concept, etc.

Dynamic Solutions International (DSI) released the EFDC_DS (20100328 version), and has been continually updating the source code since then. The EFDC model is used worldwide, which demonstrates its performance and applicability.

Based on the EFDC_DS (20100328 version), the National Institute of Environmental Research (NIER) in South Korea has added more features, such as the operating function of hydraulic structures in the major rivers of South Korea; a simulation function of multi-algae species; a vertical migration mechanism for cyanobacteria; akinete generation and germination mechanisms; and a mechanism for the effect of salt and toxicity on freshwater and sea algae, wind stress, and benthic flux of inorganic nutrients according to changes in the oxidation/reduction conditions [26].

Ahn et al. [26,27] established a method for short-term prediction of algae by using an improved source code with an operating function of hydraulic river structures and of mechanisms for vertical migration of cyanobacteria and akinete creation and germination. They also proposed an optimal method for predicting algae by applying hyperspectral remote-sensing data to the EFDC-NIER model. The model, which the NIER has named EFDC-NIER, has improved the functions to suit the major rivers of South Korea, and it is used to support the policies for algae and water quality control of major rivers and lakes in South Korea. Because the number of calculations required for analysis has increased in various fields such as hydraulics, hydrology, water quality, and aquatic ecosystems, a parallel computational code needs to be applied to increase the utilization of policies through fast decision-making.

In this study, we applied a hybrid parallel computational code constructed using the OpenMP and MPI methods to the EFDC-NIER model, and compared the calculation times required for the existing version and the parallel computational code version. We also compared the simulation results for the existing EFDC model and the parallel computational code model to check if they corresponded. Furthermore, the optimal parallel combination was determined through a comparative evaluation of the simulation time based on the number of cores and threads.

2. Materials and Methods

2.1. Research Trend in Parallel Calculation of EFDC Model

The EFDC is a general-purpose modeling package used for the simulation of 3D flow, material transport, and biochemical processes in systems such as rivers, lakes, estuaries, reservoirs, wetlands, and coasts. The EFDC model is open-source software developed by John Hamrick at the Virginia Institute of Marine Science (VIMS, Gloucester point, VA, USA) [28]. The EFDC is one of the models recommended by the US EPA for the management of the total maximum daily load (TMDL), and the US EPA continues to support its development. The EFDC has been extensively tested and used by many researchers in several modeling studies.

DSI has developed a version of the code that streamlines the modeling process and connects it to the pre- and post-processing tools of DSI [29]. DSI has developed and commercialized the EFDC_DSI OpenMP code by applying the OpenMP library for parallel computing. Figure 1 shows the comparison between the simulation times when the EFDC_DSI OpenMP is applied [30]. The model using an octa-core processor reduced the execution time of all subroutines by approximately 75% compared to the model that used a single-core processor (the dotted line on the left). However, although the calculation time of the subroutine that simulated the water quality response mechanism decreased slightly when a dual-core processor was used, the calculation time was similar to that when using a single-core processor. As the number of processors increased (the dotted line on the right), the calculation time did not significantly decrease.

IBM used the MPI library to parallelize the GVC version of the EFDC model released by the EPA [31] and published it on GitHub [32]. For continuous management during a parallelization operation, the setup process for parallel execution was automated (1) to limit the changes in a large number of source files to avoid computational errors, (2) to ensure that the results of serial calculations and parallel calculations match, and (3) to ensure that the originally configured serial model runs properly in the parallel code. Figure 2 shows the parallel efficiency as a function of the number of processors using IBM’s parallelized code. When six processors were used, the parallel efficiency was 50%, and when 25 processors were used, an efficiency of 40% was achieved. However, the rate of increase in the efficiency fell when more than six processors were used. The parallel efficiency for the water quality response calculation was unknown because it was calculated by parallelizing only the hydraulic module from the tidal current model of Galway Bay.

2.2. Development of Parallel Computational Code

As described above, DSI performed parallelization using the OpenMP method, and IBM performed parallelization using the MPI method. IBM performed parallelization only on the subroutines related to the hydraulic calculations. Therefore, in this study, we aimed to apply the parallel code to all of the EFDC-NIER model based on a hybrid method of applying both OpenMP and MPI. Because both MPI and OpenMP have advantages and disadvantages, the hybrid method can utilize the advantages of both while minimizing their drawbacks. In hybrid parallel programming, an intensive calculation is performed at a single node using the OpenMP method, and a large number of calculations are performed by executing communications between different nodes over the network based on the MPI method. In 2019, a research team at the University of Bialystok in Poland conducted a study on the K-means parallelization algorithm. Statistics such as the vector sum and count were calculated using OpenMP at each MPI node, and after gathering them at the master node, the remaining statistical calculations were completed and delivered to each processor to continue the next calculation. Similarly, four algorithms were parallelized, and the results of the computational experiments showed that all these algorithms were superior to the conventional Lloyd algorithm in terms of computing time [33].

MPI is a library specification containing standardized subroutines and functions for handling communication [34], whereas OpenMP is a shared memory model and an add-on to a compiler. One of the advantages of OpenMP is that there is no communication between nodes, but the disadvantage is that the user’s desired calculation time cannot be secured when large calculations are performed because a single node cannot be extended infinitely. In contrast, in the case of MPI, calculation nodes can be added to secure the calculation speed desired by the user.

Parallel code development was carried out in two stages: Sequential code optimization and parallel code creation. Sequential code optimization refers to the process of maximizing the performance of the sequential code before parallelizing the code (

Table A1. Examples of source code optimization.

Variables		Code Description
1. CALUVW	Improved	IF (ISCFL.GE.1.AND.DEBUG) THEN IF (MYRANK.EQ.0) THEN OPEN (1, FILE = ‘CFL.OUT’, STATUS = ‘UNKNOWN’, POSITION = ‘APPEND’) ENDIF IF (MYRANK.EQ.0) THEN IF (ISCFL.EQ.1) WRITE (1,1212) DTCFL, N, ICFL, JCFL, KCFL IF (ISCFL.GE.2.AND.IVAL.EQ.0) WRITE (1,1213) IDTCFL ENDIF ENDIF
2. EEXPOUT	Conventional	DO NSP = 1, NXSP DO K = 1, KC; DO L = 2, LA WQ = WQVX(L, K, NSP) WRITE (95) WQ ENDDO; ENDDO ENDDO
	Improved	DO NSP = 1, NXSP WRITE (95) WQVX (:,:,NSP) ENDDO

). The optimization was performed by finding unnecessary statements through measuring the precise computation time for each computational statement in the subroutines defined earlier by hotspot analysis. A DEBUG variable was added to the CALUVW subroutine to compose a log fine (CFL.OUT) when DEBUG was required, and the variable I/O was reduced in EEXPOUT by reducing the loop depth to 3 (NSP, K, L) when writing the result.

Parallel codes were developed using a hybrid parallel programming method that included both OpenMP and MPI. They were created by adding statements used by both parallel libraries, and parallelization was performed for approximately 40 source codes. For parallel code development, we chose the hybrid parallel programming approach and developed a technique for dividing the computational area according to the processor used. The MPI parallel code was composed by partitioning the LA variable (the number of calculation grids) corresponding to the 2D index in a Do loop, and the OpenMP parallel code was composed using a thread fork-join approach with the partitioned indices. To aid in the understanding of how the hybrid parallel code is created,

Table A2. Example of hybrid parallel code application.

Category	Description
Sequential code	DO L = 2,LA RCG_R8(L) = CCC(L)*P(L) + CCS(L)*PSOUTH(L) + CCN(L)*PNORTH(L) & + CCW(L)*P(L-1) + CCE(L)*P(L + 1)-FPTMP(L) ENDDO DO L = 2,LA RS8Q = RS8Q + RCG_R8(L)*RCG_R8(L) ENDDO
Parallel code	!$OMP PARALLEL DO DO L = LMPI2,LMPILA RCG_R8(L) = CCC(L)*P(L) + CCS(L)*PSOUTH(L) + CCN(L)*PNORTH(L) & +CCW(L)*P(L-1) + CCE(L)*P(L + 1)-FPTMP(L) ENDDO !$OMP PARALLEL DO REDUCTION(+:RS8Q) DO L = LMPI2,LMPILA RS8Q = RS8Q + RCG_R8(L)*RCG_R8(L) ENDDO CALL MPI_ALLREDUCE(RS8Q,MPI_R8,1,MPI_DOUBLE, & MPI_SUM,MPI_COMM_WORLD,IERR) RS8Q = MPI_R8

shows an example of a parallel code. First, we designed it such that the statements repeated from 2 to LA in the Do loop of the sequential code would be repeated from LMPI2 to LMPILA and partitioned according to the MPI rank in the parallel code. We then parallelized each Do loop using !$OMP PARALLEL DO, OpenMP Directive. The RS8Q variable is configured to calculate the sum of all indices by simultaneously using OpenMP’s REDUCTION and MPI’s ALLREDUCE.

We developed MPI functions for area partitioning and communication according to the number of MPI nodes, and

Table 1. MPI-only functions of the EFDC-NIER parallel code.

Function	Description
MPI_INITIALIZE	Initializes MPI and sets up MPI variables (number of nodes, rank of each node)
MPI_DECOMPOSITION	Partitions the LA index
BROADCAST_BOUNDARY	Performs communication of boundary values between MPI nodes (1D variable)
BROADCAST_BOUNDARY_ARRAY	Performs communication of boundary values between MPI nodes (2D and 3D)
COLLECT_IN_ZERO	Performs collective communication to send the variables to the master node (1D)
COLLECT_IN_ZERO_ARRAY	Performs collective communication to send the variables to the master node (2D and 3D)
MPI_TIC/MPI_TOC	Timestamp for hotspot analysis (MPI Walltime)

presents the specific roles of each function. Among them, the BROADCAST_BOUNDARY function was developed to communicate the updated boundary values between the MPI nodes to and from adjacent nodes. The COLLECT_IN_ZERO function transmits variables from all nodes to the master node in order to output the results and analyze the consistency at the master node. For the data type used in the MPI communication, a user-defined data type (MPI_TYPE_VECTOR) was used to reduce the frequency and amount of communication. For the topology, a single topology without partitioning was used. MPI-only functions that handled the communication between nodes were inserted to send and receive the boundary values for a minimal number of times; otherwise, a communication load would occur.

2.3. Parallel Calculation Test Model Sets

To evaluate the parallelization performance and consistency of the EFDC-NIER according to the number of grids, we used the model sets presented in

Table 2. Method for evaluating the parallelization performance.

Case		Description	Comparison Method
Case 1		Before optimization and parallel code application	Consistency of calculation times and results for simulations from 1 July 2015 to 9 July 2015
Case 2	1	Sequential code optimization
	2	Case 2-1 + Changed method of writing the result file
Case 3		Case 2-2 + OpenMP (6 threads)
Case 4		Case 2-2 + MPI (6 nodes)
Case 5	1	Case 2-2 + Hybrid (1 thread + 5 nodes)
	2	Case 2-2 + Hybrid (2 threads + 4 nodes)
	3	Case 2-2 + Hybrid (3 threads + 3 nodes)
	4	Case 2-2 + Hybrid (4 threads + 2 nodes)
	5	Case 2-2 + Hybrid (5 threads + 1 node)

and

Table 3. Model and computer specifications used for the evaluation of the parallel calculation.

Category		Description
Model	Number of grids	Horizontal: 6998 units, vertical: 11 layers
Windows PC	CPU	Inter Core i7-8700 3.20 GHz, 6 cores
	Memory	32 GB
	OS	Microsoft Windows 10 Pro 64-bit
Compiler		Intel parallel studio XE 2020, 19.1.2.254 20200623

. Measuring the consistency of the parallel code involves evaluating whether the results for the serial and parallel calculations match.

3. Results and Discussion

3.1. Source Code Analysis for the EFDC-NIER Model

In models such as EFDC-NIER, which rely on data-intensive calculations, it is important to identify the time-consuming calculation codes in the program before creating the parallelization code. Such a time-consuming function or section is called a hotspot, and it is crucial to clearly identify hotspots to perform optimization and parallelization for overall performance improvement.

In this study, we identified 20 subroutines and calculation statements as hotspots among approximately 250 subroutines executed in the source code of EFDC-NIER.

Table 4. Hotspot analysis results for the EFDC-NIER model.

Function Name	Description	Proportion of Execution Time (%)
HDMT2T	Main code of numerical simulation	100.00
WQ3D	Water quality component simulation	38.03
CALUVW	Flow rate/direction component simulation	18.45
CALCONC	Concentration (water temperature, sediment) simulation	10.65
CALQQ2T	Turbulence intensity simulation	7.33
CALEXP2T	Explicit momentum equation calculation	6.17
CALHDMF	Simulation of horizontal viscosity and spreading momentum	4.66
CALPUV2C	Simulation of surface P, UHDYE, and VHDXE	4.12
EEXPOUT	Writing a result file	3.15
CALAVB	Simulation of vertical viscosity and dispersion	3.12
ADVANCE	Updating the next timestep value	0.98
CALBUOY	Buoyancy simulation	0.98
CALTBXY	Bottom friction factor calculation	0.60
QQSQR	Updating the turbulence intensity value	0.56
CALCSER	Updating the boundary data	0.35
CALTSXY	Updating the surface wind stress	0.28
SEDIMENT	Sediment simulation	0.25
NLEVEL	Distribution of variables for each timestep	0.11
SALPTH	Writing the salinity result	0.09
DUMP	Recording the model variable dump	0.06

presents the names of the subroutines identified as hotspots, their execution times, and the proportions of the total execution time for each subroutine. The HDMT2T subroutine is the main execution component of the numerical operations, and for the 20 hotspots, the proportions are calculated by comparing their execution time to the execution time of HDMT2T. Twelve subroutines accounted for more than 1% of the total execution time: Water quality component simulation (WQ3D) > flow rate/direction component simulation (CALUVW) > concentration (water temperature, sediment) simulation (CALCONC) > subroutine description (CALQQ2T) > subroutine description (CALEXP2T). The proportions of these subroutines were high because they simulated 3D variables.

3.2. Parallel Performance Evaluation

For the parallel performance evaluation, we compared the execution times of the model for the cases presented in Table 2 and examined the execution times and performance improvement rates of the major subroutines for each parallel combination.

When the simulation times for each case were compared, the source code optimization and parallel code application resulted in a performance improvement by a factor of approximately five compared to Case 1 (

Table 5. Parallelization performance evaluation method results.

Case		Simulation Time (h)
Case 1		3.22
Case 2	1	2.68
	2	1.69
Case 3		0.74
Case 4		0.69
Case 5	1	0.66
	2	0.73
	3	0.68
	4	0.65
	5	0.79

). When the optimization was performed for EEXPOUT (shown in Table A1), the time decreased by approximately 1.5 h, indicating that a significant amount of time was spent in the writing method. A comparison of OpenMP and MPI showed that the MPI method required 0.05 h less in the simulation time. However, in the case of personal computers, it would be sufficient to use OpenMP because the core resource is limited.

For the combination of OpenMP and MPI, we compared the simulation times by increasing the number of OpenMP threads from one to six and decreasing the number of MPI nodes from five to one in each case. When four threads of OpenMP and two nodes of MPI were applied, the fastest simulation was executed in 0.65 h. It is difficult to determine whether a certain combination of OpenMP thread count and MPI node count is sufficient, and an appropriate combination that accounts for the specifications of the user’s computer should be determined.

We conducted this study using a Windows PC, which offers versatility. OpenMP can use only the cores corresponding to one computer, whereas MPI can utilize all cores of multiple computers. Therefore, the performance improvement will increase further if the software is applied to a Linux-based supercomputer. Appendix B describes the evaluation results for the parallel computing performance in a Linux-based cluster, and Appendix C describes the evaluation results for the parallel computing performance of the OpenMP and MPI methods.

3.3. Consistency Evaluation

In the process of parallelizing the sequential code, two categories can be considered for the factors that violate the consistency of the parallelization code. A complete understanding of the code is required before developing the parallelization code. If parallel programming directives are added without sufficient understanding of the algorithm, the variables that need to be communicated may not be communicated, resulting in improper synchronization. Additionally, subroutines that are calculated at a certain period may not be recognized, resulting in the omission of their parallelization. These cases imply that the parallel code has not been written properly, and a sufficient understanding of the code should be gained before proceeding.

In the second case, because a difference occurs in the order of calculations in the process of parallelizing the sequential code, a rounding error may occur owing to the limitations of floating-point arithmetic operations. Rounding errors occurred in the CONGRAD subroutine in the development process of the parallelization code for EFDC-NIER. To resolve this problem, we changed the types of variables (RPCG, PAPCG, RPCGN, ALPHA, and BETA) used for the calculations from the REAL 4-byte type to the REAL 8-byte type. The consistency of the parallel code means that the parallel execution results in the simulation are the same as the results of the sequential model for the simulation domains and the simulation options that the user can use. We considered the errors of the floating-point arithmetic operations when determining whether the simulation produced the same results. The consistency was evaluated by comparing the sum of the absolute values of all matrix values of the variables simulated up to a certain prediction period based on the variables that were written to the results file. As shown in

Table 6. Consistency analysis results for each evaluation subject.

Variable	Case 1	Case 3	Case 4
TSX	6.173959 × 10−4	6.173959 × 10−4	6.173959 × 10−4
TSY	6.763723 × 10−4	6.763723 × 10−4	6.763723 × 10−4
TBX	2.715987 × 10−2	2.715987 × 10−2	2.715987 × 10−2
TBY	3.796839 × 10−1	3.796839 × 10−1	3.796839 × 10−1
AV	6.224335	6.224335	6.224335
AB	7.667015	7.667015	7.667015
AQ	2.551961	2.551961	2.551961
HP	3.046219 × 104	3.046219 × 104	3.046219 × 104
HU	3.080725 × 104	3.080725 × 104	3.080725 × 104
HV	3.018550 × 104	3.018550 × 104	3.018550 × 104
P	1.726112 × 106	1.726112 × 106	1.726112 × 106
U	1.927491 × 102	1.927491 × 102	1.927491 × 102
V	1.632354 × 103	1.632354 × 103	1.632354 × 103
W	8.197821 × 10−1	8.197821 × 10−1	8.197821 × 10−1
TEM	1.677452 × 105	1.677452 × 105	1.677452 × 105
SEDT	4.259870 × 105	4.259870 × 105	4.259870 × 105
QQ	1.507725 × 10	1.507725 × 10	1.507725 × 10
QQL	1.070996	1.070996	1.070996
WQV	2.552020 × 105	2.552020 × 105	2.552020 × 105
WQVX	5.815869 × 104	5.815869 × 104	5.815869 × 104

and

Table 7. Consistency results of major water quality factors.

Date	Case 1			Case 3			Case 4			Case 5-4
	BOD	T-N	T-P	BOD	T-N	T-P	BOD	T-N	T-P	BOD	T-N	T-P
01-07-2015	1.453	3.590	0.080	1.453	3.590	0.080	1.453	3.590	0.080	1.453	3.590	0.080
02-07-2015	1.481	3.609	0.079	1.481	3.609	0.079	1.481	3.609	0.079	1.481	3.609	0.079
03-07-2015	1.521	3.658	0.080	1.521	3.658	0.080	1.521	3.658	0.080	1.521	3.658	0.080
04-07-2015	1.547	3.700	0.081	1.547	3.700	0.081	1.547	3.700	0.081	1.547	3.700	0.081
05-07-2015	1.520	3.712	0.079	1.520	3.712	0.079	1.520	3.712	0.079	1.520	3.712	0.079
06-07-2015	1.596	3.672	0.081	1.596	3.672	0.081	1.596	3.672	0.081	1.596	3.672	0.081
07-07-2015	1.637	3.612	0.080	1.637	3.612	0.080	1.637	3.612	0.080	1.637	3.612	0.080
08-07-2015	1.563	3.568	0.081	1.563	3.568	0.081	1.563	3.568	0.081	1.563	3.568	0.081
09-07-2015	1.634	3.493	0.081	1.634	3.493	0.081	1.634	3.493	0.081	1.634	3.493	0.081

, the consistency evaluation results confirm that the results exactly match the sequential execution of the parallel code, execution of the OpenMP, and execution of the MPI.

4. Conclusions

In this study, we optimized the source code of the EFDC-NIER model, which has been enhanced by the National Institute of Environmental Research (NIER) since 2010, and applied a parallel computational code. Then, we examined the consistency of the results between the existing EFDC model and the parallel computational code simulations and compared their calculation times. Furthermore, we determined an optimal parallel combination through a comparative evaluation of the simulation times between the number of cores and threads. The major findings of this study are as follows.

(1)

The source code optimization and parallel code application resulted in a performance improvement by a factor of approximately five compared to the existing source code (Case 1). In the case of the existing EFDC, a large amount of time was consumed by a subroutine that wrote the results, and when this was improved, it took approximately half as much time for the calculation. As shown in Appendix B, the parallel calculation performance of the OpenMP and MPI methods applied in this study showed a similar level of performance as the results of the version developed and released by DSI. Therefore, the parallel calculation of the EFDC-NIER is better than or on par with that of the EFDC+ MPI, especially considering that its improvement values include the simulation results of the water quality factors.

(2)

For a Windows PC, there is no difference in the reduction of the calculation speed between the OpenMP and MPI methods because the core and thread resources are limited. However, as shown in Appendix C, when using a Linux server, the simulation is performed more ideally when the MPI method is used compared to the OpenMP method. In the case of the hybrid method that uses both the OpenMP and MPI methods, the optimal computing combination should be applied according to the performance and computing resources of the computer on which the simulation will be performed.

(3)

In the case of South Korea, algae prediction information for the water supply source sections of large rivers is sent to the water quality managers at eight-day intervals. When predicting the algae eight days into the future, all prediction work must be completed within 8 h. The fastest simulation’s calculation time (0.65 h) is a very important factor; if the optimization and the parallel computational source code applied in this study are used, quick calculation will be facilitated when urgent decision-making is required for an event such as a water pollution incident.