Real-Time Massive Vector Field Data Processing in Edge Computing
Abstract
:1. Introduction
- We propose an edge computing empowered MVFD processing framework. To our best knowledge, we are the first to introduce edge computing for MVFD processing in the literature.
- We invent Data Fluidization Schedule (DFS) strategy, aiming at fine-scale data partitioning, dataflow transmission and computing scheduling so as to ensure the real-time MVFD processing.
- We practically developed and implemented our framework with DFS. Real MVFD data-driven experiment results show that it is superior to both Spark and MapReduce.
2. Related Work
2.1. MVFD Processing
2.2. Edge Computing
3. Edge Computing Empowered MVFD Processing Framework
4. Data Fluidization Schedule
4.1. Fluidization of Data
Algorithm 1 Partition and transmission of MVFD in sequence. |
Require: Raw Dataset, D;
|
4.2. Parallel Computing on Dataflow
4.3. Dataflow Gate
Algorithm 2 Dataflow gate. |
Require: Data consumer threads, ;
|
5. Experiment and Analysis
5.1. Implementation and Settings
5.2. Experiment Results and Analysis
6. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- Nouanesengsy, B.; Lee, T.Y.; Shen, H.W. Load-balanced parallel streamline generation on large scale vector fields. IEEE Trans. Vis. Comput. Gr. 2011, 17, 1785. [Google Scholar] [CrossRef] [PubMed]
- Guo, M.; Huang, Y.; Guan, Q.; Xie, Z.; Wu, L. An efficient data organization and scheduling strategy for accelerating large vector data rendering. Trans. GIS 2017, 21, 1217–1236. [Google Scholar] [CrossRef]
- Chen, L.; Ma, Y.; Liu, P.; Wei, J.; Jie, W.; He, J. A review of parallel computing for large-scale remote sensing image mosaicking. Clust. Comput. 2015, 18, 517–529. [Google Scholar] [CrossRef] [Green Version]
- Zhang, X.M.; He, G.J.; Zhang, Z.M.; Peng, Y.; Long, T.F. Spectral-spatial multi-Figure classification of remote sensing big data based on a random forest classifier for land cover mapping. Clust. Comput. 2017, 20, 2311–2321. [Google Scholar] [CrossRef]
- Garth, C.; Gaither, K. Large-Scale Integration-Based Vector Field Visualization. Math. Vis. 2014, 37, 327–338. [Google Scholar]
- Li, Z.; Huang, Q.; Carbone, G.J.; Hu, F. A high performance query analytical framework for supporting data-intensive climate studies. Comput. Environ. Urban Syst. 2017, 62, 210–221. [Google Scholar] [CrossRef]
- Zheng, K.; Gu, D.; Fang, F.; Zhang, M.; Zheng, K.; Li, Q. Data storage optimization strategy in distributed column-oriented database by considering spatial adjacency. Clust. Comput. 2017, 20, 2833–2844. [Google Scholar] [CrossRef]
- Lohrmann, B.; Warneke, D.; Kao, O. Nephele Streaming: Stream Processing under QoS Constraints at Scale; Kluwer Academic Publishers: Dodrecht, The Netherlands, 2014. [Google Scholar]
- Xu, X.; Xie, F.; Zhou, X. Research on spatial and temporal characteristics of drought based on GIS using Remote Sensing Big Data. Clust. Comput. 2016, 19, 757–767. [Google Scholar] [CrossRef]
- Song, X.; He, G.; Zhang, Z.; Long, T.; Peng, Y.; Wang, Z. Visual attention model based mining area recognition on massive high-resolution remote sensing images. Clust. Comput. 2015, 18, 541–548. [Google Scholar] [CrossRef]
- Solaimani, M.; Iftekhar, M.; Khan, L.; Thuraisingham, B.; Ingram, J.B. Spark-based anomaly detection over multi-source VMware performance data in real-time. In Proceedings of the 2014 IEEE Symposium on Computational Intelligence in Cyber Security, Orlando, FL, USA, 19 January 2015; pp. 1–8. [Google Scholar]
- Yuan, L.; Yu, Z.; Luo, W.; Yi, L.; Hu, Y. Pattern forced geophysical vector field segmentation based on Clifford FFT. Comput. Geosci. 2013, 60, 63–69. [Google Scholar] [CrossRef]
- Kratzke, N.; Quint, P.C. Understanding cloud-native applications after 10 years of cloud computing—A systematic mapping study. J. Syst. Softw. 2017, 126, 1–16. [Google Scholar] [CrossRef]
- Satyanarayanan, M. The emergence of edge computing. Computer 2017, 50, 30–39. [Google Scholar] [CrossRef]
- Hyrkas, J.; Clayton, S.; Ribalet, F.; Halperin, D.; Virginia Armbrust, E.; Howe, B. Scalable clustering algorithms for continuous environmental vector cytometry. Bioinformatics 2015, 32, 417–423. [Google Scholar] [CrossRef] [PubMed]
- Domann, J.; Meiners, J.; Helmers, L.; Lommatzsch, A. Real-time news recommendations using apache spark. In Proceedings of the Conference and Labs of the Evaluation Forum, Évora, Portugal, 5–8 September 2016. [Google Scholar]
- Chen, X.; Wang, Y. Relic vector field and CMB large scale anomalies. J. Cosmol. Astropart. Phys. 2014, 2014, 027. [Google Scholar] [CrossRef]
- Čermák, M.; Jirsík, T.; Laštovička, M. Real-time analysis of NetVector data for generating network traffic statistics using Apache Spark. In Proceedings of the Network Operations and Management Symposium, Istanbul, Turkey, 25–29 April 2016; pp. 1019–1020. [Google Scholar]
- Zheng, K.; Zheng, K.; Fang, F.; Zhang, M.; Li, Q.; Wang, Y.; Zhao, W. An extra spatial hierarchical schema in key-value store. Clust. Comput. 2018, 1–15. [Google Scholar] [CrossRef]
- Cho, W.; Choi, E. A GPS Trajectory Map-Matching Mechanism with DTG Big Data on the HBase System. In Proceedings of the International Conference on Big Data Applications and Services, Jeju Island, Korea, 20–26 October 2015. [Google Scholar]
- Huang, F.; Lan, B.; Tao, J.; Chen, Y.; Tan, X.; Feng, J.; Ma, Y. A parallel nonlocal means algorithm for remote sensing image denoising on an Intel Xeon Phi platform. IEEE Access 2017, 5, 8559–8567. [Google Scholar] [CrossRef]
- Sujudi, D.; Haimes, R. Integration of particles and streamlines in a spatially-decomposed computation. In Proceedings of the Parallel Computational Fluid Dynamics, Los Alamitos, CA, USA, 1996; IEEE Computer Society Press: Washington, DC, USA, 1996. [Google Scholar]
- Pugmire, D.; Childs, H.; Garth, C.; Ahern, S.; Weber, G. Scalable computation of streamlines on very large datasets. In Proceedings of the Supercomputing, Yorktown Heights, NY, USA, 8–12 June 2009. [Google Scholar]
- Pugmire, D.; Childs, H.; Garth, C.; Ahern, S.; Weber, G.H. Scalable computation of streamlines on very large datasets. In Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, Denver, CO, USA, 17–22 November 2013; pp. 1–12. [Google Scholar]
- Camp, D.; Garth, C.; Childs, H.; Pugmire, D.; Joy, K. Streamline integration using MPI-hybrid parallelism on a large multicore architecture. IEEE Trans. Vis. Comput. Gr. 2011, 17, 1702–1713. [Google Scholar] [CrossRef]
- Camp, D.; Childs, H.; Chourasia, A.; Garth, C.; Joy, K.I. Evaluating the benefits of an extended memory hierarchy for parallel streamline algorithms. In Proceedings of the IEEE Symposium on Large-Scale Data Analysis and Visualization (LDAV), Providence, RI, USA, 23–24 October 2011; IEEE Press: Piscataway, NJ, USA, 2011. [Google Scholar]
- Weisong, S.; Jie, C. Edge Computing: Vision and Challenges. IEEE Internet Things J. 2016, 3, 637–645. [Google Scholar]
- Wang, T.; Liang, Y.; Jia, W.; Muhammad, A.; Liu, A.; Xie, M. Coupling Resource Management Based on Fog Computing in Smart City Systems. J. Netw. Comput. Appl. 2019, 135, 11–19. [Google Scholar] [CrossRef]
- Wu, Y.; Huang, H.; Wu, Q.; Liu, A.; Wang, T. A Risk Defense Method Based on Microscopic State Prediction with Partial Information Observations in Social Networks. J. Parallel Distrib. Comput. 2019. [Google Scholar] [CrossRef]
- Zhang, G.; Wang, T.; Wang, G.; Liu, A.; Jia, W. Detection of Hidden Data Attacks Combined Fog Computing and Trust Evaluation Method in Sensor-Cloud System. Concurr. Comput. Pract. Exp. 2018. [Google Scholar] [CrossRef]
- Wang, T.; Luo, H.; James, X.; Xie, M. Crowdsourcing Mechanism for Trust Evaluation in CPCS based on Intelligent Mobile Edge Computing. ACM Trans. Intell. Syst. Technol. 2019. [Google Scholar] [CrossRef]
- Wang, T.; Qiu, L.; Arun, K.S.; Xu, G.; Liu, A. Energy-efficient and Trustworthy Data Collection Protocol Based on Mobile Fog Computing in Internet of Things. IEEE Trans. Ind. Inf. 2019. [Google Scholar] [CrossRef]
- Wang, T.; Zhang, G.; Liu, A.; Bhuiyan, M.Z.A.; Jin, Q. A secure IoT service architecture with an efficient balance dynamics based on cloud and edge computing. IEEE Internet Things J. 2018. [Google Scholar] [CrossRef]
- Wang, T.; Zhang, G.; Bhuiyan, M.Z.A.; Liu, A.; Jia, W.; Xie, M. A novel trust mechanism based on fog computing in sensor-cloud system. Future Gener. Comput. Syst. 2018. [Google Scholar] [CrossRef]
- Wang, T.; Zhou, J.; Liu, A.; Bhuiyan, M.Z.A.; Wang, G.; Jia, W. Fog-based computing and storage offloading for data synchronization in IoT. IEEE Internet Things J. 2018,. [Google Scholar] [CrossRef]
- Chun, B.G.; Ihm, S.; Maniatis, P.; Naik, M.; Patti, A. CloneCloud: Elastic execution between mobile device and cloud. In Proceedings of the 6th Conference Computer System, Salzburg, Austria, 10–13 April 2011; pp. 301–314. [Google Scholar]
- Oueida, S.; Kotb, Y.; Aloqaily, M.; Jararweh, Y.; Baker, T. An edge computing based smart healthcare framework for resource management. Sensors 2018, 18, 4307. [Google Scholar] [CrossRef]
- Ganesh, A.; Paramvir, B.; Peter, B.; Krishna, C.; Matthai, P.; Lenin, R.; Sudipta, S. Real-time video analytics: The killer app for edge computing. Computer 2017, 50, 58–67. [Google Scholar]
- Mao, Y.; Zhang, J.; Song, S.H. Power-Delay Tradeoff in Multi-User Mobile-Edge Computing Systems. In Proceedings of the GLOBECOM 2016—2016 IEEE Global Communications Conference, Washington, WA, USA, 4–8 December 2016. [Google Scholar]
- Auroux, D.; Fehrenbach, J. Identification of velocity fields for geophysical fluids from a sequence of images. Exp. Fluids 2011, 50, 313–328. [Google Scholar] [CrossRef]
- Guo, H.; Hong, F.; Shu, Q.; Zhang, J.; Huang, J.; Yuan, X. Scalable lagrangian-based attribute space projection for multivariate unsteady vector fata. In Proceedings of the Visualization Symposium, Paris, France, 9–14 November 2014; pp. 33–40. [Google Scholar]
- Peterka, T.; Ross, R.; Nouanesengsy, B.; Lee, T.K. A Study of Parallel Particle Tracing for Steady-State and Time-Varying Flow Fields. In Proceedings of the Parallel & Distributed Processing Symposium, Anchorage, AK, USA, 16–20 May 2011; pp. 580–591. [Google Scholar]
- Wei, H.; Zuckerman, S.; Li, X.; Gao, G.R. A Dataflow programming language and its compiler for streaming systems. Procedia Comput. Sci. 2014, 29, 1289–1298. [Google Scholar] [CrossRef]
- Shuo, Z.; Ping, W.; Jie, H. An automatic identification algorithm of cyclone and anticyclone based on wind data. J. Tianjin Univ. 2017, 50, 1271–1279. [Google Scholar]
- Wong, K.Y.; Yip, C.L. Identifying centers of circulating and spiraling vector field patterns and its applications. Pattern Recognit. 2009, 42, 1371–1387. [Google Scholar] [CrossRef]
- Pereira, G.A.S.; Choudhury, S.; Scherer, S. A framework for optimal repairing of vector field-based motion plans. In Proceedings of the International Conference on Unmanned Aircraft Systems, Arlington, VA, USA, 7–10 June 2016; pp. 261–266. [Google Scholar]
- Zaharia, M.; Chowdhury, M.; Franklin, M.J.; Shenker, S.; Stoica, I. Spark: Cluster computing with working sets. In Proceedings of the Usenix Conference on Hot Topics in Cloud Computing, Boston, MA, USA, 22–25 June 2010. [Google Scholar]
© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Zheng, K.; Zheng, K.; Fang, F.; Yao, H.; Yi, Y.; Zeng, D. Real-Time Massive Vector Field Data Processing in Edge Computing. Sensors 2019, 19, 2602. https://doi.org/10.3390/s19112602
Zheng K, Zheng K, Fang F, Yao H, Yi Y, Zeng D. Real-Time Massive Vector Field Data Processing in Edge Computing. Sensors. 2019; 19(11):2602. https://doi.org/10.3390/s19112602
Chicago/Turabian StyleZheng, Kun, Kang Zheng, Falin Fang, Hong Yao, Yunlei Yi, and Deze Zeng. 2019. "Real-Time Massive Vector Field Data Processing in Edge Computing" Sensors 19, no. 11: 2602. https://doi.org/10.3390/s19112602
APA StyleZheng, K., Zheng, K., Fang, F., Yao, H., Yi, Y., & Zeng, D. (2019). Real-Time Massive Vector Field Data Processing in Edge Computing. Sensors, 19(11), 2602. https://doi.org/10.3390/s19112602