Near-Real-Time OGC Catalogue Service for Geoscience Big Data
Abstract
:1. Introduction
2. Framework of the Near-Real-Time OGC Catalogue Service
2.1. Fundamental Elements
2.2. Elements for Updating Metadata in Near Real Time
2.3. Metadata Ingestion Elements
3. Methodology
3.1. Near-Real-Time OGC CSW Server and Metadata Repository
3.2. Near-Real-Time Catalogue Update Approaches
3.3. Timing-Based Catalogue Update Implementation in Parallel
3.3.1. Data Source Monitor
3.3.2. Cluster-Based Catalogue Update in Parallel
4. Application and Catalogue Update Efficiency Analysis
4.1. Unidata IDD Data and Catalogue
4.2. Prototype of Near-Real-Time OGC CSW Service for Unidata IDD Data
4.3. Experiment and Result for Catalogue Update Task
5. Discussion and Conclusions
Acknowledgments
Author Contributions
Conflicts of Interest
References
- Nebert, D.; Whiteside, A.; Vretanos, P. OGC 07-006r1: Catalog Service Specification, Version 2.0.2. 2007. Available online: http://portal.opengeospatial.org/files/?artifact_id=20555 (accessed on 18 March 2016).
- Bai, Y.; Di, L. Providing Access to Satellite Imagery through OGC Catalog Service Interfaces in Support of the Global Earth Observation System of Systems. Comput. Geosci. 2011, 37, 435–443. [Google Scholar] [CrossRef]
- Bai, Y.; Di, L.; Chen, A.; Liu, Y.; Wei, Y. Towards a Geospatial Catalogue Federation Service. Photogramm. Eng. Remote Sens. 2007, 73, 699–708. [Google Scholar] [CrossRef]
- Chen, A.; Di, L.; Wei, Y.; Bai, Y.; Liu, Y. An Optimized Grid-Based, OGC Standards-Compliant Collaborative Software System for Serving NASA Geospatial Data. In Proceedings of the 30th Annual IEEE/NASA Software Engineering Workshop, Columbia, MD, USA, 24–28 April 2006; pp. 159–166. [Google Scholar] [CrossRef]
- Chen, Z.; Chen, N. Use of Service Middleware Based on ECHO with CSW for Discovery and Registry of MODIS Data. Geo-Spat. Inf. Sci. 2010, 13, 191–200. [Google Scholar] [CrossRef]
- Di, L.; Yu, G.; Shao, Y.; Bai, Y.; Deng, M.; McDonald, K.R. Persistent WCS and CSW Services of GOES Data for GEOSS. Geosci. Remote Sens. Symp. IEEE. 2010, 38, 1699–1702. [Google Scholar]
- Shao, Y.; Di, L.; Bai, Y.; Wang, H.; Yang, C. Federated Catalogue for Discovering Earth Observation Data. Photogramm. Fernerkund. Geoinform. Jahrg. 2013, 2013, 43–52. [Google Scholar] [CrossRef]
- ECHO. What is ECHO? 2016. Available online: https://wiki.earthdata.nasa.gov/pages/viewpage.action?pageId=26543757 (accessed on 12 June 2016).
- Battrick, B. The Global Earth Observation System of Systems (GEOSS) 10-Year Implementation Plan Reference Document. ESA (European Space Agency) Publications Division, 2005. Available online: http://www.earthobservations.org/documents/10-Year%20Implementation%20Plan.pdf (accessed on 12 June 2016).
- Beyer, M.A.; Lapkin, A.; Gall, N.; Feinberg, D.; Sribar, V.T. ‘Big Data’ Is Only the Beginning of Extreme Information Management. 2011. Available online: http://www.gartner.com/id=1622715 (accessed on 26 July 2016).
- Mauro, A.D.; Greco, M.; Grimaldi, M. A Formal definition of Big Data based on its essential Features. Libr. Rev. 2016, 65, 122–135. [Google Scholar] [CrossRef]
- Laney, D. 3D Data Management: Controlling Data Volume, Velocity and Variety. 2001. Available online: http://blogs.gartner.com/doug-laney/files/2012/01/ad949-3D-Data-Management-Controlling-Data-Volume-Velocity-and-Variety.pdf (accessed on 12 June 2016).
- Unidata. Unidata Community. 2016. Available online: https://www.unidata.ucar.edu/community/index.html (accessed on 8 May 2016).
- Yoksas, T.; Emmerson, S.; Chiswell, S.; Schmidt, M.; Stokes, J. The Unidata Internet Data Distribution (IDD) System: A Decade of Development. In Proceedings of the 22nd International Conference on Interactive Information Processing Systems for Meteorology, Oceanography, and Hydrology, Atlanta, CA, USA, 27 January–3 February 2006. [Google Scholar]
- Martell, R. OGC 07-110r4: CSW-ebRIM Registry Service—Part 1: ebRIM Profile of CSW. 2009a. Available online: http://portal.opengeospatial.org/files/?artifact_id=31137 (accessed on 18 March 2016).
- Martell, R. OGC 07-110r4: CSW-ebRIM Registry Service—Part 2: Basic Extention Package. 2009b. Available online: http://portal.opengeospatial.org/files/?artifact_id=31138 (accessed on 18 March 2016).
- Voges, U.; Senkler, K. OpenGIS Catalogue Services Specification 2.0.2—ISO Metadata Application Profile. 2007. Available online: http://portal.opengeospatial.org/files/?artifact_id=21460 (accessed on 12 June 2016).
- Yoksas, T.; Almeida, W.G.D.; Coelho, D.G.; Leon, V.C. Internet Data Distribution -- Extending Real-Time Data Sharing throughout the Americas. Adv. Geosci. 2006, 8, 91–95. [Google Scholar] [CrossRef]
- Domenico, B.; Caron, J.; Davis, E.; Kambic, R.; Nativi, S. Thematic Real-Time Environmental Distributed Data Services (THREDDS): Incorporating Interactive Analysis Tools into NSDL. J. Digit. Inf. 2006, 2. [Google Scholar]
- Nogueira, R.; Cutrim, E.M. Extending THREDDS Middleware to Serve OGC Community. Adv. Geosci. 2006, 8, 57–62. [Google Scholar]
- Bergamasco, A.; Benetazzo, A.; Carniel, S.; Falcieri, F.M.; Minuzzo, T.; Signell, R.P.; Sclavo, M. Knowledge Discovery in Large Model Datasets in the Marine Environment: The THREDDS Data Server Example. Adv. Oceanogr. Limnol. 2012, 3, 119–133. [Google Scholar] [CrossRef]
Name | Programming Language | Year Started |
---|---|---|
GeoNetwork | Java | 2001 |
GeoServer with CSW plug-in | Java | 2006 |
pycsw | Python | 2010 |
deegree | Java | 2014 |
Dataset Category | Update Frequency | Count |
---|---|---|
Satellite data | ||
NESDIS GOES satellite data | 5 min | 140,000 |
Radar data | ||
NEXRAD Level II radar WSR-88D | 3–10 min | 660,000 |
NEXRAD Level III radar products | 44,900,000 | |
TDWR Level III radar | 3,200,000 | |
NCEP and FNMOC Forecast Model | 3900 | |
Rapid Refresh (RAP) High Resolution Rapid Refresh (HRRR) | 1 h | 1300 |
Short Range Ensemble Forecasting (SREF) Wave Watch III (WW3) | 3 h | 800 |
Global Forecast System (GFS) North American Model (NAM) Global Ensemble Forecasting System (GEFS) | 6 h | 1400 |
Downscaled GFS (DGEX) | 12 h | 60 |
NAVY Global Environmental Model (NAVGEM) Forecast of Aerosol Radiative Optical Properties (FAROP) | 6 h | 120 |
Coupled Ocean/Atmosphere Mesoscale Prediction System Navy Coupled Ocean Data Assimilation (NCODA) Model Wave Watch 3 (WW3) Model | 12 h | 230 |
NCEP analyses data | 6100 | |
Multi-Radar Multi-Sensor (MRMS) analysis Real Time Mesoscale Analysis (RTMA) | 1 h | 5000 |
National Digital Forecast Database (NDFD) | 30 min | 1100 |
Observation data | 270 | |
METAR Station data Surface buoy point data Surface synoptic point data | 24 h | 270 |
Site Code | Begin Time | End time | Elapsed Time (s) | Count | Elapsed Time to Register One Metadata Record (s) |
---|---|---|---|---|---|
PAKC | 12:24:06 | 12:31:15 | 429 | 282 | 1.52 |
KTLX | 14:48:05 | 14:53:37 | 332 | 262 | 1.27 |
KPBZ | 15:59:32 | 16:03:02 | 213 | 160 | 1.33 |
KABR | 17:05:26 | 17:08:07 | 161 | 141 | 1.14 |
FOP1 | 18:31:15 | 18:37:38 | 383 | 234 | 1.64 |
Average: | 1.38 |
Dataset Category | Data Count | Estimated Time |
---|---|---|
Satellite data | 140,000 | 54 h |
NEXRAD Level II radar data | 660,000 | 10.5 d |
NEXRAD Level III radar products | 44,900,000 | 717 d |
TDWR Level III radar products | 3,200,000 | 51 d |
NCEP and FNMOC Forecast Model | 3900 | 1 h, 30 min |
NCEP analyses data | 6100 | 2 h, 20 min |
Observation data | 270 | 6 min |
Dataset Category | Update Frequency | Update Count of New Data | Estimated Time |
---|---|---|---|
Satellite data | 5 min | 78 | 2 min |
NEXRAD Level II radar data | 3–10 min | 166 | 4 min |
NEXRAD Level III radar products | 11232 | 4 h | |
TDWR Level III radar products | 810 | 19 min | |
NCEP and FNMOC Forecast Model | 1, 3, 6, 12 h | 59 | <2 min |
NCEP analyses data | 30, 60 min | 13 | <1 min |
Observation data | 24 h | 3 | 5 s |
Dataset Category | Catalogue Initialization Phase | Catalogue Update Phase | Schedules |
---|---|---|---|
Satellite data | 10 tasks | 2 timing tasks | every 2 min |
NEXRAD Level II radar data | 20 tasks | 4 timing tasks | every 2 min |
NEXRAD Level III radar products | N/A | N/A | N/A |
TDWR Level III radar products | 30 tasks | 20 timing tasks | every 4 min |
NCEP And FNMOC Forecast Model | 1 task | 1 timing task | every 30 min |
NCEP analyses | 1 task | 1 timing task | every 30 min |
Observation data | 1 task | 1 timing task | every 6 h |
Dataset Category | Catalogue Initialization Phase | Catalogue Update Phase |
---|---|---|
Satellite data | 5 h, 42 min | 1 min, 18 s |
NEXRAD Level II radar data | 14 h | 1min, 26 s |
NEXRAD Level III radar products | N/A | N/A |
TDWR Level III radar products | 1 d, 18 h | 2 min, 43 s |
NCEP And FNMOC Forecast Model | 1 h, 12 min | 1 min, 55 s |
NCEP analyses | 2 h, 3 min | 25 s |
Observation data | 5 min, 36 s | 5 s |
© 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Song, J.; Di, L. Near-Real-Time OGC Catalogue Service for Geoscience Big Data. ISPRS Int. J. Geo-Inf. 2017, 6, 337. https://doi.org/10.3390/ijgi6110337
Song J, Di L. Near-Real-Time OGC Catalogue Service for Geoscience Big Data. ISPRS International Journal of Geo-Information. 2017; 6(11):337. https://doi.org/10.3390/ijgi6110337
Chicago/Turabian StyleSong, Jia, and Liping Di. 2017. "Near-Real-Time OGC Catalogue Service for Geoscience Big Data" ISPRS International Journal of Geo-Information 6, no. 11: 337. https://doi.org/10.3390/ijgi6110337