Parallel Reservoir Simulation with OpenACC and Domain Decomposition
Abstract
:1. Introduction
2. Related Work
3. Materials and Methods
3.1. The Black-Oil Model Reservoir Simulation
- (Initialization) This part inputs and initializes the raw data (i.e., oil reservoir data records) and parameters, it belongs to a preprocessing stage for black-oil reservoir simulation.
- (Computation) This part is the key of reservoir simulation which repeatedly builds a Jacobian matrix and solves the linear equation. One common method is using Newton’s method to convert the nonlinear equations into linear equations, then solve the linear equations. In each time step, the procedure of solving equations is performed first in the Jacobian matrix and solved until the computing result is convergent. Then the parameter is updated timely. The iteration mentioned above is carried out until the loop termination condition is satisfied.
- (Output) This part outputs the final computation results.
3.2. Parallel Nonlinear Equation Solver with OpenACC
- Specification of a parallel region construct: Two kind of directive constructs, kernel and parallel, are defined in OpenACC to specify which part of the code is to be executed in parallel. While the kernels directive entrusts a compiler with responsibility of analyzing dependencies of variables, the parallel directive implies that responsibility to the user. We use the latter in our implementation.
- Memory allocation and data transfer data construct: Data directive is a representative example. In OpenACC2.5, enter and exit directives are added which allocate and free memory space on the device. Data transfer between host and device is executed by update directive. Then before the parallel region, the clause of the data directive is always present.
- Specification of parallelized loop: This is done by loop directive. In parallel regions, it is necessary to specify this. With this instruction, the user can directly determine the gang, worker and vector parameter. And variables are private to loop.
3.3. The GPU-Aided Domain Decomposition
- Choose the approximate solution ,
- Parallel compute boundary value of subdomain,
- Extend to
- If it does not meet the convergence conditions, make , then go to step 2.
4. Results and Discussion
4.1. Experimental Setup
4.2. Evaluating and Discussing the Efficiency of OpenACC Parallel
4.3. Evaluating and Discussing the Amount of Code During Project Modification
4.4. Evaluating and Discussing the GPU-Aided Domain Decomposition Method
5. Conclusions
Author Contributions
Funding
Conflicts of Interest
Abbreviations
CUDA | Compute Unified Device Architecture |
GPU | Graphics processing unit |
FDM | Finite difference method |
AWS | Amazon Web Services |
DDM | Domain decomposition method |
GDDM | GPU-aided domain decomposition |
ASM | Additive Schwarz Method |
References
- Maliassov, S.; Beckner, B.; Dyadechko, V. Parallel reservoir simulation using a specific software framework. In Proceedings of the Society of Petroleum Engineers-SPE Reservoir Simulation Symposium, The Woodlands, TX, USA, 3–5 February 2013; pp. 1062–1070. [Google Scholar]
- Guan, W.; Qiao, C.; Zhang, H.; Zhang, C.S.; Zhi, M.; Zhu, Z.; Zheng, Z.; Ye, W.; Zhang, Y.; Hu, X.; et al. On robust and efficient parallel reservoir simulation on tianhe-2. In Proceedings of the Society of Petroleum Engineers-SPE Reservoir Characterisation and Simulation Conference and Exhibition, Abu Dhabi, UAE, 27 May 2015; pp. 933–945. [Google Scholar]
- Chien, M.; Northrup, E. Vectorization and parallel processing of local grid refinement and adaptive implicit schemes in a general purpose reservoir simulator. In Proceedings of the SPE Symposium on Reservoir Simulation, New Orleans, LA, USA, 28 February–3 March 1993; pp. 279–290. [Google Scholar]
- Chien, M.C.; Tchelepi, H.A.; Yardumian, H.E.; Chen, W.H. A scalable parallel multi-purpose reservoir simulator. In Proceedings of the SPE Symposium on Reservoir Simulation, Dallas, TX, USA, 8–11 June 1997; pp. 17–30. [Google Scholar]
- Li, X.; Lei, Z.; Huang, D.; Khamra, Y.E.; Allen, G.; White, C.D.; Kim, J.G. Queues: Using grid computing for simulation studies. In Proceedings of the Digital Energy Conference and Exhibition, Houston, TX, USA, 11–12 April 2007; pp. 1–13. [Google Scholar]
- Eldred, M.E.; Orangi, A.; Al-Emadi, A.A.; Ahmad, A.; O’Reilly, T.J.; Barghouti, N. Reservoir simulations in a high performance cloud computing environment. In Proceedings of the SPE Intelligent Energy Conference & Exhibition, Utrecht, The Netherlands, 1–3 April 2014; pp. 1–8. [Google Scholar]
- Nickolls, J.; Dally, W.J. The gpu computing era. IEEE Micro 2010, 30, 56–69. [Google Scholar] [CrossRef]
- Klie, H.M.; Sudan, H.H.; Li, R.; Saad, Y. Exploiting capabilities of many core platforms in reservoir simulation. In Proceedings of the SPE Reservoir Simulation Symposium, The Woodlands, TX, USA, USA, 21–23 February 2011; pp. 1–12. [Google Scholar]
- Liu, H.; Yu, S.; Chen, Z.J.; Hsieh, B.; Shao, L. Parallel preconditioners for reservoir simulation on gpu. In Proceedings of the SPE Latin America and Caribbean Petroleum Engineering Conference, Mexico City, Mexico, 16–18 April 2012; pp. 1–5. [Google Scholar]
- Gilman, J.R.; Uland, M.; Angola, O.; Michelena, R.; Meng, H.; Esler, K.; Mukundakrishnan, K.; Natoli, V. Unconventional reservoir model predictions using massively-parallel gpu flow-simulation: Part-1 bakken reservoir characterization choices and parameter testing. In Proceedings of the Unconventional Resources Technology Conference, San Antonio, TX, USA, 20–22 July 2015; pp. 1–21. [Google Scholar]
- Beckner, B.L.; Haugen, K.B.; Maliassov, S.; Dyadechko, V.; Wiegand, K.D. General parallel reservoir simulation. In Proceedings of the International Petroleum Exhibition and Conference, Abu Dhabi, UAE, 9–12 November 2015; pp. 1–10. [Google Scholar]
- NVIDIA. NVIDIA CUDA C Programming Guide Version 6.0; NVIDIA: Santa Clara, CA, USA, 2014. [Google Scholar]
- Sabne, A.; Sakdhnagool, P.; Lee, S.; Vetter, J.S. Evaluating performance portability of openacc. LNCS 2015, 8967, 51–66. [Google Scholar]
- Lopez, I.; Fumero, J.J.; de Sande, F. Directive-based programming for gpus: A comparative study. In Proceedings of the IEEE 14th International Conference on High Performance Computing and Communications, Liverpool, UK, 25–27 June 2012; pp. 410–417. [Google Scholar]
- OpenACC: OpenACC Programming and Best Practices Guide. 2015. Available online: www.openacc.org (accessed on 18 December 2018).
- Pickering, B.P.; Jackson, C.W.; Scogland, T.R.; Feng, W.C.; Roy, C.J. Directive-based gpu programming for computational fluid dynamics. Comput. Fluids 2015, 114, 242–253. [Google Scholar] [CrossRef]
- Cebamanos, L.; Henty, D.; Richardson, H.; Hart, A. Auto-tuning an openacc accelerated version of nek5000. LNCS 2014, 8759, 69–81. [Google Scholar]
- Calore, E.; Kraus, J.; Schifano, S.F.; Tripiccione, R. Accelerating lattice boltzmann applications with openacc. LNCS 2015, 9233, 613–624. [Google Scholar]
- Wang, Y.; Killough, J.E. A new approach to load balance for parallel/compositional simulation based on reservoir-model overdecomposition. SPE J. 2014, 19, 304–315. [Google Scholar] [CrossRef]
- Have, P.; Masson, R.; Nataf, F.; Szydlarski, M.; Xiang, H.; Zhao, T. Algebraic domain decomposition methods for highly heterogeneous problems. SIAM J. Sci. Comput. 2013, 35, 284–302. [Google Scholar] [CrossRef]
- Fischer, P.F. An overlapping schwarz method for spectral element solution of the incompressible navier–stokes equations. J. Comput. Phys. 1997, 133, 84–101. [Google Scholar] [CrossRef]
Specifications of CPU Platforms | Work Bench |
OS | Windows 10 64 bit |
CPU | i7-5820k (3.3 Ghz, 6 cores) |
Memory | 32GB DDR4 |
Specifications of GPU Platforms | GTX TITAN X |
Architecture | Maxwell |
Memory | 12 GB DDR5 |
bandwidth | bi-directional bandwidth of 16 GB/s |
PGI Visual Fortran | Release Version 16.9 |
Section Name | Amount of Original Code | Amount of Increased Code | The Increased Code Ratio (%) |
---|---|---|---|
Jacobian matrix assembly | 1703 | 307 | 18 |
Linear equation solver | 2073 | 520 | 25.1 |
Sum of the two parts | 3776 | 827 | 21.9 |
© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Kang, Z.; Deng, Z.; Han, W.; Zhang, D. Parallel Reservoir Simulation with OpenACC and Domain Decomposition. Algorithms 2018, 11, 213. https://doi.org/10.3390/a11120213
Kang Z, Deng Z, Han W, Zhang D. Parallel Reservoir Simulation with OpenACC and Domain Decomposition. Algorithms. 2018; 11(12):213. https://doi.org/10.3390/a11120213
Chicago/Turabian StyleKang, Zhijiang, Ze Deng, Wei Han, and Dongmei Zhang. 2018. "Parallel Reservoir Simulation with OpenACC and Domain Decomposition" Algorithms 11, no. 12: 213. https://doi.org/10.3390/a11120213
APA StyleKang, Z., Deng, Z., Han, W., & Zhang, D. (2018). Parallel Reservoir Simulation with OpenACC and Domain Decomposition. Algorithms, 11(12), 213. https://doi.org/10.3390/a11120213