You are currently viewing a new version of our website. To view the old version click .
MDPIMDPI
Search all articles
ISPRS International Journal of Geo-Information
Download PDF
  • Article
  • Open Access

8 June 2016

Global-Scale Resource Survey and Performance Monitoring of Public OGC Web Map Services

,
,
,
and
1
School of Remote Sensing and Information Engineering, Wuhan University, 129 Luoyu Road, Wuhan 430079, China
2
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, 129 Luoyu Road, Wuhan 430079, China
3
Collaborative Innovation Center of Geospatial Technology, Wuhan University, 129 Luoyu Road, Wuhan 430079, China
*
Authors to whom correspondence should be addressed.
ISPRS Int. J. Geo-Inf.2016, 5(6), 88;https://doi.org/10.3390/ijgi5060088
Version Notes
Order Reprints

Abstract

One of the most widely-implemented service standards provided by the Open Geospatial Consortium (OGC) to the user community is the Web Map Service (WMS). WMS is widely employed globally, but there is limited knowledge of the global distribution, adoption status or the service quality of these online WMS resources. To fill this void, we investigated global WMSs resources and performed distributed performance monitoring of these services. This paper explicates a distributed monitoring framework that was used to monitor 46,296 WMSs continuously for over one year and a crawling method to discover these WMSs. We analyzed server locations, provider types, themes, the spatiotemporal coverage of map layers and the service versions for 41,703 valid WMSs. Furthermore, we appraised the stability and performance of basic operations for 1210 selected WMSs (i.e., GetCapabilities and GetMap). We discuss the major reasons for request errors and performance issues, as well as the relationship between service response times and the spatiotemporal distribution of client monitoring sites. This paper will help service providers, end users and developers of standards to grasp the status of global WMS resources, as well as to understand the adoption status of OGC standards. The conclusions drawn in this paper can benefit geospatial resource discovery, service performance evaluation and guide service performance improvements.

1. Introduction

Web technologies, new standards and commercial applications are rapidly changing the nature and extent of available online geospatial resources. Thus, it is imperative to investigate whether Web Map Service (WMS) remains the best solution to share and interoperate maps over the Internet or if a new standard and direction is needed. To understand the WMS adoption situation and appraise the WMS standard, a global-scale survey to investigate the distribution, usage and quality of public WMSs (i.e., the web map services that are available over the Internet for public use) is highly desirable. WMS is an Open Geospatial Consortium (OGC) standard protocol initially proposed in 2000 for serving geo-referenced maps over the Internet [1]. The latest version WMS 1.3.0 was published in 2006 [2] and is also available as ISO 19128 (i.e., ISO 19128:2005 Geographic information-Web map server interface) [3] issued by the International Organization for Standardization (ISO). WMS has become the most widely-used OGC data portrayal service, recognized globally and supported by both mainstream commercial geospatial tools and open-source software. Currently, a huge number of WMSs are available over the Internet for public use providing abundant map resources, but varying in map content and quality. In this context, finding an appropriate WMS becomes a challenging problem [4].
Therefore, for more effective and efficient utilization of these invaluable online geospatial resources, the following questions need to be asked: What is the adoption situation of WMS; who are the primary contributors; and where are the providers located? Are there any patterns regarding the different attributes of WMS resources? For example, what are the most popular themes, specification versions and map projections, spatial and time coverages of the map layers? How is the quality of global WMSs, in terms of accessibility, successability and performance? Are there any patterns that can provide hints for service selection and server-side improvement?
In this paper, we carried out a global-scale resource investigation and executed distributed performance monitoring on a set of crawled public WMSs. The potential contributions of this research can be drawn from the following perspectives:
(1)
Geospatial resource discovery: the investigation of WMS resources and their metadata help us to grasp the server locations, provider types and content distribution of the global geospatial resources. Consequently, this knowledge will benefit service discovery by bringing to light on-demand WMSs that have those properties expected by service consumers.
(2)
Service performance evaluation: by developing a distributed monitoring framework, spatiotemporal heterogeneity and individual service-level performance differences can be analyzed over space and time. This will guide performance-aware service selection for time-critical applications, as well as steer performance improvements from the service provider perspective.
(3)
Evolution of service standard: this investigation can help researchers and standard makers from both industry and academia review the development and adoption of open service standards for a geospatial data portrayal. By linking cutting-edge web visualization technologies in relation to the prevailing interoperation modes (e.g., crowdsourcing [5,6] and collaboration [7,8]), we can rethink the appropriateness of the WMS standard, thus inspiring us to conceive new directions for advancement.
This article is organized as follows. Section 2 reviews relevant research. Section 3 introduces our monitoring method and data collection. Section 4 details the discoveries. Section 5 concludes with results and discusses future research.

3. Data Collection and Methodology

The data collection and analysis workflow is illustrated in Figure 1. First, WMSs are discovered using a developed topic crawler. After importing the crawled WMSs into a database, the WMS resource survey and QoS analysis were conducted. The WMS resource survey is based on service metadata (i.e., capabilities documents retrieved from GetCapabilities operation). QoS analysis is based on our monitoring result on the two mandatory operations, GetCapabilities and GetMap. Routine monitoring permits acquisition or updating of long-term QoS behaviors (e.g., stability, performance) and the availability status (whether the URL is not valid any more) of all WMSs, while intensive monitoring captures the spatiotemporal features of QoS for selected services.
Figure 1. Data collection and analysis workflow.

3.1. Online Web Map Service Discovery

The WMSs investigated in this research were all collected from the World Wide Web by our active topic crawler [56]. The discovery workflow is illustrated in Figure 2. To collect as many WMSs as possible and to ensure the global coverage of collected WMSs, the crawler adopted a hybrid search strategy that integrates search engine-based search and directed search. Common search engines (e.g., Google and Bing) have powerful crawling and indexing capabilities that capture global web pages. Therefore, a search engine-based search guarantees the breadth of search and, therefore, ensures that our search can reach the regions of the web indexed by common search engines. More to the point, WMS is a type of domain-specific web resource, usually published through geospatial web portals and catalogues, such as the OGC Catalogue Service for the Web (CSW). We used directed search to make a dedicated search of these SDIs using standard APIs and web page crawling. Reputable SDIs (e.g., data.gov [31], GEOSS clearinghouse [30], EuroGEOSS broker [57]) and other geospatial web portals found through search engines were set as seed pages.
Figure 2. Search strategies and discovery workflow of web map services.
In terms of search engine-based search, we developed two methods. The first method searches WMS directly. Keyword-based search (e.g., “WMS”, “Web Map Services” or “OGC”) or advanced search functions (e.g., in Google Search, we use “service = WMS” as a query constraint for an “inurl” statement) provided by the search engine were utilized to retrieve candidate web pages (i.e., seed pages) that may contain WMS URLs. Then, our crawler crawled these web pages recursively to discover WMSs. The second method searches WMS by locating ArcGIS REST Service Directories using keyword-based search (i.e., “ArcGIS REST Service Directory”) because many online service directories [58,59] generated by ArcGIS servers [20] provide OGC-compliant geospatial web services. To discover WMSs from such a service directory, the crawler firstly identified real service directory folder pages from retrieved search results by analyzing the HTML structure and content. Then, a dedicated and recursive crawl was conducted on the qualified web pages.
The procedure for WMS discovery from a specified web page was treated as a matching and validation process on the WMS capabilities document URLs hidden among all of the HTTP URLs included on the web pages. The HTTP request URL of a WMS capabilities document or its URL prefix is usually posted as a hyperlink or text on web pages explicitly. A canonical HTTP request for the WMS GetCapabilities operation contains multiple Key-Value Pairs (KVPs) as essential request parameters, such as “request = GetCapabilities” and “service = WMS”. Based on the WMS URL prefix, the capabilities document URL can be easily formed by appending a necessary query parameter. Thus, we used the KVP feature as a criterion to search or form URLs. To avoid unnecessary URL formation processes, only the hyperlinks whose anchor texts and URL syntax that followed specified rules [56] were selected as candidate URL prefixes. Nevertheless, a URL with such a syntax structure cannot guarantee a valid WMS, and a further HTTP request is needed for validation. Our crawler votes for a URL as a valid one only if the payload of the HTTP response is a valid XML document that contains mandatory element tags (e.g., <WMS_Capabilities>). A valid WMS URL was added into the database if it did not replicate a URL already present in the database.

3.2. Distributed Monitoring Framework and Strategy

To collect QoS monitoring data from geographically-dispersed monitoring sites, we developed a distributed monitoring framework that consists of three components (Figure 3); (1) The data collector collects service metadata and real-time performance data of WMSs using a group of monitoring sites. Each monitoring site monitors the assigned WMSs in parallel using multithreading technologies. A monitoring manager configures and coordinates the monitoring tasks of all monitoring sites and is in charge of data management; (2) The data access service provides both the RESTful and Simple Object Access Protocol (SOAP)-based web service APIs to retrieve monitoring data (e.g., monitoring sites, service metadata and historical performance in a given time period); (3) A web portal, developed based on our previous research [60], visualizes the layers, service performance, as well as the spatial distribution of the monitored WMSs and monitoring sites using maps and charts.
Figure 3. The architecture of the monitoring framework.
Both routine monitoring and intensive monitoring were based on our distributed monitoring framework, but the monitoring strategies were different. Routine monitoring utilized several fixed monitoring sites to guarantee that the available status of the WMSs was not impacted by single monitoring site connectivity. The two mandatory operations of each WMS were both tested on a weekly basis, one request for each operation from each routine monitoring site per week. In contrast, to capture performance differences and investigate spatiotemporal patterns, intensive monitoring used more monitoring sites from different locations, and the monitoring time intervals are more intensive and adjustable. In practice, to provide stable accessibility, some WMS providers may setup license policies [61] to prohibit excessively frequent accesses from a single IP address. We respected these policies and conducted distributed round-the-clock monitoring. Multiple monitoring sites were deployed on dispersed locations around the world. They worked collaboratively to capture the performance difference caused by the diversity of spatial positions where users are accessing the services. In the temporal dimension, WMSs were divided in groups and monitored periodically, according to a configurable schedule. The monitoring schedule helped us to investigate the daily pattern in performance and to avoid being blocked by the WMS servers due to excessively frequent requests. Meanwhile, by dividing WMSs into groups, the accumulated response delay within a group could be controlled and the monitoring time interval for a single WMS could be guaranteed. After iterative monitoring and merging multiple day monitoring data into one day, we obtained intensive daily monitoring records with almost equal time intervals for each mandatory operation of a selected WMS, i.e., a record around every five minutes.
For a consistent and comparable test, we created testing rules for the two operations. During GetMap operation monitoring, the first named layer [2] of the WMS was requested for testing. We restricted the output image to be a 200-pixel height, 400-pixel width and in PNG format. If this format was not supported, JPEG or other formats were specified accordingly. Although output maps may be distorted as the geographic extent (i.e., bounding box) varies, the data volume can be roughly controlled, as well as the data transfer cost. Meanwhile, to eliminate the impact of differences in server-side processing procedures, we limited each request to a single layer, and requests that combined multiple layers simultaneously were excluded. However, a request upon cascading parent layers that contain child layers was allowed, since a WMS treats a parent layer as a single layer in terms of GetMap requests. For GetCapabilities operation monitoring, as WMS may support multiple service versions, testing requests that do not specify the optional parameter “version” were used as the measurement to obtain QoS for the default version.

3.3. Survey Data Collection

From September 2014 to November 2015, 27 public cloud-based (Windows Azure) monitoring sites dispersed at 13 locations were deployed on four continents and in seven countries (Figure 4) in total. All monitoring sites were employed with the same virtual machine configuration, including the network, computing resources and operating system to avoid any impact on monitoring results as much as possible. Four monitoring sites located in U.S. (Bristow, VA; Redmond, WA), Ireland (Dublin) and China (Hong Kong) were constantly employed for routine monitoring from September 2014 to November 2015. The remaining 23 monitoring sites from 12 locations were set up for intensive monitoring from 23 August 2015 to 3 October 2015. At any time point during intensive monitoring, 12 sites from 12 different locations were selected to conduct monitoring simultaneously; other sites from replicated locations were used as alternatives.
Figure 4. The geo-locations of monitoring sites.
Over more than one year of routine monitoring, 46,296 WMSs (from 72 countries and six continents) collected by our web crawler were constantly monitored. Among these services, 41,703 WMSs in total were valid and contained 318,102 layers. Since the total number of WMSs was too large to conduct a comprehensive test, we selected 1210 WMSs to be intensively monitored for 42 days. To avoid the impact of access overload, the number of WMSs from a single provider, the same domain name or IP address was strictly limited to at most five during WMS selection. The global distribution of the selected WMSs was considered, as well. The GetCapabilities test was based on these 1210 WMSs. Among these WMSs, 876 WMSs were actually valid for access. We selected them to conduct further GetMap tests. Table 1 lists the amount of WMSs from each continent in testing. In keeping with the round-the-clock monitoring strategy as described in Section 3.2, we finished a monitoring cycle every six days. From each monitoring site location, we collected 2016 GetCapabilities QoS records (i.e., 48 records per day) for each WMS selected for GetCapabilities test and 2016 GetMap QoS records for each WMSs selected for GetMap test.
Table 1. The WMSs selected in intensive monitoring.
To conduct a comprehensive investigation, we collected more metadata fields (e.g., contact information, CRS, spatial coverages of layers) and fine-grained quality information than the research reported in the literature [42,43,44,45,49,50]. Response time, error type, size of response message, download speed and relevant monitoring site were recorded from each monitoring request. Response time is affected by the latency on the client, network and server side. To analyze the key impact factors of WMS response times in the future, we obtained fine-grained response time costs for different phases in the interaction. The total response time in a complete HTTP request-and-response operation was decomposed into different time spans using the cURL command line tool [62]. The attributes of these time spans include Domain Name System (DNS) parsing time; connecting time; total time for sending a request and server processing; and the data transfer time. In addition, the average response time and the successability for each WMS were updated periodically as quality metrics.

4. Data Analysis

4.1. Global WMS Resource Survey

In the resources survey, primary metadata properties, including the title, abstract, keywords, coordinate reference system, version and provider, were extracted from the WMS capability documents to investigate service usage and resource distribution.

4.1.1. Server Location and Provider Type

The geo-location of monitored public WMSs suggests that North America (especially the U.S.) and Europe have most abundant WMS resources (Figure 5 and Figure 6a). In the U.S. and Europe, Open Government Data initiatives promote the accessibility and re-use of official data and information by citizens, communities and developers via open development repositories. Spatial data infrastructures (SDIs) developed in these regions are widely used by cross-domain users all over the world for geospatial resources discovery and sharing. For examples, data.gov [31] integrates the U.S. government’s open data to build a one-stop data catalog, and the INSPIRE geoportal [63,64] is a European Union (EU) SDI to facilitate the sharing of environmental spatial information for public use. As a result, the global accessibility of geospatial resources published in the U.S. and Europe is significantly enhanced, and online service exploration becomes much easier. The geo-location distribution of WMSs does not necessarily mean there are more map resources in the U.S. and Europe. This distribution indicates that OGC service standards are more widely adopted for publishing open data in those places. Consequently, these resources are more easily reached by global public users through geoportals and search engines.
Figure 5. The geo-location of monitored public WMSs.
Figure 6. The regional distribution and types of monitored public WMS providers; (a) while most services are in North America and Europe, among the remaining 0.6%, Asia contains 0.23%; (b) providers by sector; government has the largest proportion (37.69%), followed by academic institutions (34.19%), while industry has the smallest proportion (1.78%).
From an analysis of the 13,352 WMSs from 989 providers that include provider information, such as organization tags in their capability documents, we found that the server locations and provider types of WMSs are imbalanced. The WMS standard draws wider support from governments, academic institutions and intergovernmental organizations to share non-profit (e.g., public welfare and resources) geospatial data (Figure 6b) than industries. Providing public data services is one of the duties of governments and intergovernmental organizations in order to benefit society. Therefore, they actively publish data related to Societal Benefit Areas (SBAs) [65] and other public interest areas using open standards. Academic institutions also play a major role in proposing and using open standards to facilitate scientific data sharing. In contrast, services provided by industry only represent a small proportion (1.78%), because WMS is an open standard for geospatial data access rather than a commercial-level data exchange protocol.
By counting the number of WMSs published by each of the identified 989 providers, we found that the top ten service providers (1%) of analyzed WMSs published 7367 WMSs (55.18%, 13,352 in total), but the number of WMSs they offered varies significantly (Table 2). The Earth Data Analysis Center (EDAC) of the University of New Mexico and National Oceanic and Atmospheric Administration (NOAA) make the largest contribution. EDAC hosted and managed the New Mexico Resource Geographic information System (RGIS) [66] for over 23 years specifically to share statewide public geospatial data. The data are managed by the RGIS data repository and published through OGC-compliant web services. NOAA, as well, provides a huge amount of oceanic- and atmospheric-related data through WMS. The imbalanced service contribution among providers follows a power law [67], as shown in Figure 7, and reflects the differences in geospatial resource possession, data sharing policies and effectiveness.
Table 2. Top ten service providers in the monitored WMSs.
Figure 7. Log-log plot of the Cumulative Density Functions (CDF) p(x) for the number of WMSs contributed by each provider according to a discrete power law with α = 1.792 and x m i n = 1 .
We further summarized the prevalent software used to publish WMSs by analyzing WMS Uniform Resource Locators (URL) and capabilities documents. Among a total of 46,296 WMSs, there were 3484, 1987 and 515 WMSs published by ArcGIS Server, GeoServer and MapServer, respectively; the publishing software for the remainder is unknown. Government agencies published 1721 of the ArcGIS-based WMSs. In contrast, only 229 WMSs from government agencies were published using GeoServer and MapServer. The two open-source servers have academic origins, hence more widely adopted in the academic sector than the public sector. Both commercial and open-source geospatial software contribute to WMS publishing, but open-source tools are more popular in academia. Commercial software, such as ArcGIS, is more likely to be used by government agencies given the governmental contracts with software companies.

4.1.2. Popular Map Subjects

Investigating the map subject matter of global WMSs can benefit global cross-disciplinary users in discovering and selecting map resources, as well as increasing our understanding of open data sharing policies, public issues, and trends in the Earth sciences and other disciplines. Although the languages used to describe WMS metadata vary (e.g., English, German, French, Dutch, Italian, Norwegian, Swedish and Chinese), according to our investigation, around 87.85% of the valid WMSs (36,638) contained English words in their service descriptions, and 82.57% of the layers contained keyword fields that were described in English. Subsequently, we studied the top map layer subjects based purely on English keyword to reduce the complexity of data analysis. We extracted a group of English keywords for each map layer from layer description fields (e.g., title, abstract, keywords) of capability documents. The stop words, replicated words and words that were not nouns were eliminated during extraction. Then, we calculated and sorted the occurrence frequency of keywords among all map layers. Figure 8 shows the top English language keywords that had the highest word frequencies. According to the GEOSS Societal Benefit Areas (SBAs) [65] and INSPIRE directive [63], we analyzed obtained keywords and found that the following subjects related to the natural environment and resources appear most frequently: geology, climate, energy, land cover, water, biodiversity, agriculture and ecosystem. Given the explosive growth in Earth observation technologies over past several decades, governments, academic institutions and non-profit organizations have collected and processed a large amount of geographic data about natural phenomena. Many of the data were published using standard OGC services and form the majority of open map resources. For example, the INSPIRE directive is primarily oriented to share spatial environmental information among public sector providers [63].
Figure 8. Top English language keywords found in service descriptions for map layers.
The big data era has arrived and given the advancement of location-aware technologies; social sensing is becoming easier and less expensive. Accordingly, social and economic activity-related data are dramatically increasing. The Socioeconomic Data and Applications Center of Columbia University (SEDAC) published a WMS [68] to represent the spatiotemporal distribution of population size, population density and education degree in America. The National Geomatics Center of China (NGCC) published thematic maps of Chinese employment and wages in 2013 as a WMS [69]. It expected that online maps with social behaviors and economic activities as primary subjects will be exploding exponentially in the near feature.

4.1.3. Spatial Coverages of Map Layers

By analyzing the geographic extent (i.e., bounding box) of 318,102 map layers (Figure 9), we found that continents are covered more intensively than the oceans, except for Antarctica, and the Northern Hemisphere has more coverage than the Southern Hemisphere. Many of the layers have a global extent (more than 25,000 layers). This phenomenon reveals our global Earth observation behaviors and also reflects human activities in space, to some degree. The differences in the richness in open geospatial data also reflect the differences in data sharing policies at the country and regional levels. More specifically, spatial coverages of map layers are concentrated in North America and Europe, especially over the mainland of U.S. and Europe, and Northern Africa has the second largest number of coverages, after North America and Europe. However, due to information security and data policy issues, geospatial resources are strictly controlled in some countries. Geospatial data are often shared internally and between governmental agencies through hardcopy or secured private networks rather than by publishing them through standardized web services for public use.
Figure 9. The spatial coverages of Map Layers.

4.1.4. Yearly Distribution of Map Layers and WMSs with Current Map Layers

The data collection time is a measure of the timeliness of geospatial data (i.e., data currency). Among all 318,102 layers, 62,925 layers from 4587 WMSs contain data collection times in their capability documents. For example, the SEDAC GeoServer WMS [68] embeds the data collection time directly in the layer name field (e.g., “Population Density 2000”), and the NASA Earth observation WMS [70] describes an extended time dimension in each layer (e.g., “2015-01-01/2015-08-29/P8D”). Using such information, we can estimate approximately the update status of map layers or service metadata for a WMS. As data become more open, the map resources collected each year are increasing in number. At the same time, most of these data published using WMSs are up-to-date data in general, not archived historical data (as shown in Figure 10).
Figure 10. Yearly distribution of map layers and WMSs with current map layers.
Figure 10 illustrates that the significant increase in WMSs and map layers in 2000 and 2006 was strongly associated with the release of the WMS standards Version 1.0.0 and Version 1.3.0, respectively. The release of the ESRI RESTful Service APIs and successive updates, which are compliant with the WMS standard, caused another significant increase, starting in 2010. Moreover, INSPIRE required member states to provide discovery and view services (i.e., WMS and WMTS) in 2011 at the latest, which may also have promoted the deployment and updating of WMSs. These trends reveal that new technologies, standards and software products are adopted relatively rapidly in the geoinformation domain. However, the enthusiasm for exploring new technologies is hard to maintain when it comes to routine service maintenance. Many of the existing WMSs seldom add new layers or update layer metadata after deployment. Thus, the data collection time for the latest layers in 4124 WMSs among the 4587 WMSs studied (89.91%) is earlier than the year 2013. This may affect the quality of map resources found in online WMSs.

4.1.5. Supported Coordinate Reference Systems and Service Versions

Among the 318,102 layers examined, various ellipsoidal CRSs and projected CRSs were supported. Ellipsoidal CRSs obtained 97.57% of the support. Web Mercator, Universal Transverse Mercator, Antarctic stereographic and Albers are the most used projections (Table 3). Web Mercator obtained 76.39% of the support. Maps in this projection can be conveniently visualized on the web, since it is easy to conduct map splitting and seamless splicing. Meanwhile, Web Mercator guarantees the correctness of direction and relative position on maps. Therefore, it is widely adopted, and many online public map services use this projection. Some layers support the Antarctic stereographic projection or Albers projection due to the geographic extent of the maps or because of their specific application needs. For example, administrative region maps may need equal-area projection.
Table 3. Common map projections of widely-supported projected Coordinate Reference Systems (CRSs).
Among 41,703 valid WMSs, there were 9920, 10,122, 41,325 and 40,861 WMSs supporting Versions 1.0.0, 1.1.0, 1.1.1 and 1.3.0, respectively. Therefore, most of the WMSs were implemented with geospatial software instances compliant with the latest WMS standard Version 1.3.0, released in 2006 and compatible with Version 1.1.0. In Version 1.3.0, the access interfaces were unified, and the data structure of response messages was refined. Therefore, the version is much more mature than previous versions, contributing to the high adoption rate. Meanwhile, after several years of popularization, the recognition of OGC standards was significantly promoted among users. More open-source and commercial GIS software developers began to support the latest OGC standards. It was also noted that most of the WMSs with Version 1.3.0 are downwardly compatible with Version 1.1.1, but only around 1/4 of WMSs are downwardly compatible with the earlier versions (1.0.0 and 1.1.0).

4.2. Stability and Performance Analysis

Stability and performance are two essential service-level measurement quality factors for web services [71] and software entities [72]. Stability measures the reliability and the maintainability of a software entity by investigating the runtime robustness, while performance measures runtime efficiency. Evaluating these two factors is significant in that it may provide guidelines for WMS selection and server-side improvements for service consumers and providers, respectively. In this section, based on the intensive monitoring result of 1210 WMSs (listed in Table 1), we analyze the overall status of the two factors using selected metrics.

4.2.1. Stability Analysis

To analyze the stability, we investigated accessibility, successability and error types based on two mandatory operations. Accessibility represents the probability that a web service operation is accessible (receiving acknowledgement or response message) while the service is available. Successability is the ratio of successful responses to all requests in a time period and measures the ability to correctly respond to user requests. Error type describes briefly the reason why a request failed. We categorized accessibility into three types in this research: always accessible, temporally inaccessible and constantly inaccessible. Table 4 shows that many WMSs are constantly inaccessible (i.e., invalid WMSs), since these are unavailable or they may have changed URLs. Meanwhile, WMS operations identified as temporally inaccessible were a nontrivial problem and due to network and service maintenance issues. Since we cannot detect or be informed of the maintenance times of all WMSs, maintenance time was not excluded from accessibilities analysis. The accessibility of GetMap is worse than GetCapabilities even for the valid WMSs. Only around 1/5 of layer GetMap operations are constantly accessible. The successability histograms for GetCapabilities and GetMap seen in Figure 11 also indicate that the successability of GetCapabilities is higher than GetMap for the valid WMSs.
Table 4. Accessibility of GetCapabilities and GetMap operations.
Figure 11. Successability histograms of the two mandatory operations for valid WMSs: (a) GetCapabilities; (b) GetMap.
To analyze operation errors, we classified various detailed error types into two categories. The server access error represents the errors that occurred when connecting to the servers, e.g., being unable to connect the host, a time-out or no response from the server. Request processing errors happen during server-side processing after successfully connecting to the server, e.g., semantic error of request, server refusing to execute the request or server overload. From Table 5, we can see that more errors were caused by request processing errors for GetMap. On the one hand, the processing of GetMap operations is relatively more complex than GetCapabilities. GetMap needs to load geospatial data and conducts requisite geoprocessing (e.g., subsetting, transformation and rendering) to generate maps, while GetCapabilities only responds to a metadata document that can be generated in advance. On the other hand, incorrect or outdated layer descriptions in the capabilities document are another reason for request processing errors. We can conclude that stability varies significantly for services and operations. Accessibility to the metadata cannot grantee the accessibility to the map layers. Therefore, the stability of WMS servers and metadata timeliness maintenance are big issues and need to be further addressed by service providers.
Table 5. Error types in GetCapabilities and GetMap operations.

4.2.2. The Power Laws in Response Times

Response time, maximum throughput and computing resource occupation are three common measurement of performance. In our research, we selected response time as the performance measurement because testing maximum throughput requires intensive concurrent requests and may result in request rejection by service providers, while computing resource occupation measurements are hard to obtain. We investigated the overall trends in the response times of all tested WMSs by analyzing the minimum, average and maximum response times of the two mandatory operations recorded for each valid WMS among all successful responses from all monitoring sites. Figure 12 indicates that the response time of the two mandatory operations for valid WMSs, in general, must obey power laws. Most WMSs respond rapidly, but a few have very long response times. The numerical differences between the minimum and the maximum response times are illustrated in Figure 13 reflecting instability in WMS performance. The vertical red lines in the bar charts for average response times indicate that a majority of valid WMSs (more than 80%) can respond to user requests within three seconds in most cases. In contrast, our previous investigations show that less than 40% of WMS GetCapabilities and GetMap operations responded within eight seconds [48,49]. Thus, the overall response time of WMSs was significantly reduced, reflecting improvement in WMSs software and hardware environments, as well as upgrades in the global network, to some extent.
Figure 12. Log-log plot of the CDFs P(x) for minimum, average and maximum response times of (a) GetCapabilities operations and (b) GetMap operations for selected WMSs reflecting continuous power laws; a Kolmogorov–Smirnov test (p > 0.05) indicated that the original data were likely to be drawn from the fitted power-law distribution. The plots show a sharp change in the upper boundaries of the response time dropping significantly at about 60 s, especially in the maximum response time, because the maximum timeout was set to 60 s during monitoring.
Figure 13. Unit area histograms for the minimum, average and maximum response times of (a) GetCapabilities operations and (b) GetMap operations for selected WMSs; the vertical red line in each chart divides the WMSs into 80% and 20% proportions, respectively. The density, calculated as frequency/(total_frequency*bin_width), shows the proportions of WMSs for per unit of response time.

4.2.3. Spatiotemporal Characteristics of Response Times

The response time of a web service is affected by various factors. Among these factors, the network connectivity between service providers and users, instantaneous network condition, as well as the concurrency pattern of global users can be generalized as spatiotemporal access factors. In this section, we explore the relation between spatiotemporal access factors and the response time.
(1) Spatial characteristics
Response time is significantly impacted by network connectivity, and the connectivity of a network in cyber space relies on the establishment of network equipment and their linkages in geographic space. Therefore, the association between spatial distance and response time is inherent. We found that 60.27% of valid WMSs (528 of 876 in total) obtained the shortest average response times from their closest monitoring site locations globally. At the continental level, this trend was more apparent (Table 6). Most of the WMSs tend to get the shortest response time from the monitoring sites in the same regions as the server locations, except in the case of South America. We also calculated the linear regression coefficient of determination (R2) of the response time for individual WMSs using the average response times at each monitoring location for multiple locations including both the public cloud-based sites and local sites. The average R2 was 74.08% for all 876 WMSs. Figure 14 reveals a positive correlation between average response time and the spatial distance from monitoring site to server. The scatter plot in Figure 14a is for all 876 valid WMSs. We can see a positive correlation, but the graph also shows data heterogeneity, since response time is impacted by many other factors, such as the response data volume, server performance, network bandwidth and the distribution of global optical cables. The impact from these factors was partially reduced by selecting 393 WMSs located in the U.S. whose capabilities documents were less than 1 MB and had an average response time of less than 2 s. The positive correlation becomes more visible in Figure 14b. Furthermore, the shorter the distance, the smaller the variance in response times, as response time becomes more stable when the uncertainty of a network is reduced. Although the response time is impacted by many factors and hard to predict precisely, we make the following suggestions. From the perspective of service selection, a WMS that has a closer geographic distance to users may have a higher priority among services with comparable functionalities and map resources. From the perspective of performance optimization, a map server should be deployed as close as possible to potential users. Cloud computing can be utilized to achieve dynamic spatiotemporal deployment of servers, and the state-of-the-art site selection algorithms could be developed to improve performance.
Table 6. The percentage of WMSs from each continent that yielded the shortest average response times from the monitoring sites on each continent.
Figure 14. Correlation between response times and spatial distance from monitoring sites to WMS servers. (a) The 876 valid WMSs and (b) the selected 393 WMSs located in the U.S. whose capabilities documents were less than 1 MB with average response times less than two seconds.
(2) Time series characteristics
The monthly response time series for a WMS from a single monitoring site is steady in general but synthesizes trends with few prominent random fluctuations and many small local variations. Within the 24 h of a day, a time series shows local fluctuations. As shown in Figure 15, for a WMS [73] provided by Arizona Geological Survey, there is a set of intensive peaks between 8:00 A.M. to 11:00 A.M. in the local time of the service, while the response time fluctuates slightly during the night. This phenomenon reveals the local network status, as well as concurrent accesses to a WMS during a certain time period, to some extent. The local network traffic and user concurrency from the same or neighboring time zones to the WMS server are relatively small in the nighttime, but increase sharply during the daytime. Concurrent access to the WMS generates server-side load pressure. The fluctuation is more violent for the WMSs with a longer average response time, since the network condition has a huge impact on the stability of the response time [50].
Figure 15. Time series characteristics of WMS response times.

5. Conclusions and Future Work

5.1. Conclusions

We conducted a comprehensive WMS resource survey and quality analysis for global WMSs based on a proposed distributed monitoring framework. Based on a WMS resource survey of 41,703 valid WMSs, we found that the WMS standard is widely adopted with an imbalanced distribution. More specifically, (1) the providers and server locations are extremely imbalanced. A few providers provided a large amount of public WMSs. WMSs are readily adopted by governments, academic institutions and intergovernmental organizations. In contrast, the contribution from companies is relatively small, since WMS is an open standard for geospatial data access rather than a commercial-level data exchange protocol. Public WMSs also have a skewed spatial distribution due to data policy issues and the imbalanced development of SDIs. Specifically, North America (especially the U.S.) and Europe contributed most of the public WMSs (around 99%); (2) Map resources are abundant, but also disproportionate. The natural environment and resources are the dominant map subjects. North America and Europe have the most concentrated layer coverages. Most of the map data were collected since 2000 when the first version of WMS 1.0.0 was released; (3) The ellipsoidal coordinate system is supported by most of the WMSs, and the Web Mercator projection is widely supported. Most WMSs are published based on the latest Version 1.3.0 and compatible with Version 1.1.1, but the downward compatibility with the old versions (i.e., 1.0.0 and 1.1.0) is deficient.
From the quality analysis, we found that the quality monitoring, evaluation and optimization are imperative and of critical importance for WMS. (1) The quality of the WMSs varies on services, operations and request parameters. Plenty of WMSs are inaccessible due to invalid URLs. GetMap has weak stability and accessibility when compared to GetCapabilities due to the relatively complicated processing in the GetMap operation and incorrect layer descriptions. Request processing errors are the major factor that causes request failures; (2) The response times of all valid WMSs obey power laws. The majority of WMSs can respond rapidly (within three seconds) generally, while a small number of them have long response times. However, when compared to contemporary commercial online map services, the large interval between the average and the maximum response times reveals ubiquitous performance issues in public WMSs; (3) The response time shows spatiotemporal patterns. Our experiment results indicate a positive correlation between WMS response time and the spatial distance from users to servers. The closest monitoring site tends to have the smallest average response time. Furthermore, the shorter the distance, the slighter the fluctuation and the more stable the response time. The trend in the response time series fluctuates significantly with local network traffic and synthesizes the minor random variances. These findings are important to understand the factors affecting service performance. Our investigation provides a valuable guideline for selecting WMS resources and optimizing WMS performance.

5.2. Suggestion and Future Work

To improve WMS discovery, selection and application, suggestions for standards developers and our potential future research directions include:
(1)
Redesigning or redefining the WMS standard and improving both client-side and server-side functionality. The current WMS standard provides very simple and easy-to-use operations to retrieve maps rendered with widely-used industrial image formats on the server side. As web technologies develop, the computing and interactive capability of web browsers are becoming more powerful. Under such circumstances, the WMS standard should be refined, leaving more fine-grained control authority free on the client side for enhanced interactivity and visual analytics functions. For example, new operations can be added to provide access and interaction capability to manipulate individual features and layers. Rendering and animation can be customized on the client side, e.g., the style of map symbols. Meanwhile, the server side could improve performance, concurrent access capacity and validation functions for metadata.
(2)
Building sophisticated WMS quality models. Response time prediction can facilitate service selection for time-critical applications. By analyzing the key impact factors and utilizing the spatiotemporal patterns of response times, prediction models could be built to achieve precise prediction. To support quality-driven WMS resource discovery, a comprehensive evaluation quality model could consider more quality metrics, e.g., data quality of maps, user feedback and preferences.
(3)
Developing a state-of-the-art web portal for better service discovery. Interactive query and visual analytics functions must be enhanced for the next generation of geospatial web portals. Firstly, quality (e.g., performance) and user scoring should be integrated and supported as search criteria. Secondly, service comparisons and the visual analytics function should be enabled. For example, users could be permitted to compare the response time, user feedback and successability of selected services visually, in an interactive way.
(4)
Optimizing the proposed monitoring framework. The scalability and flexibility of our distributed framework could be improved with a larger number of monitoring sites and services. More types of geospatial web services (e.g., ESRI RESTful services, OGC CSW, OGC Sensor Observation Service, OGC Web Processing Service) and operations should be supported.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No. 41501434 and No. 41371372), the Research Foundation of Wuhan University (No. 2042014kf0026) and the Natural Science Foundation of Hubei Province (No. 2015CFA053). Thanks to Mr. Steve McClure for language assistance.

Author Contributions

Zhipeng Gui designed the methods and wrote the paper. Jun Cao implemented the monitoring system and performed the experiments. Xiaojing Liu analyzed the data. Xiaoqiang Cheng and Huayi Wu contributed to the conceptualization and methods.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Doyle, A. OpenGIS Web Map Server Interface Implementation Specification Revision 1.0.0; Open Geospatial Consortium: Wayland, MA, USA, 2000. [Google Scholar]
  2. De la Beaujardiere, J. OpenGIS Web Map Service (WMS) Implementation Specification Version 1.3.0; Open Geospatial Consortium: Wayland, MA, USA, 2006. [Google Scholar]
  3. International Organization for Standardization. Geographic Information—Web Map Server Interface; ISO/TC 211, ISO 19128:2005; International Organization for Standardization: Geneva, Switzerland, 2005. [Google Scholar]
  4. Shen, S.; Liu, W.; Wu, H.; Chen, Y. A multi-level comprehensive evaluation method for quality of WMS based on fuzzy mathematics. In Proceedings of the 17th International Conference on Geoinformatics, Fairfax, VA, USA, 12–14 August 2009.
  5. Sui, D.; Elwood, S.; Goodchild, M. Crowdsourcing Geographic Knowledge: Volunteered Geographic Information (VGI) in Theory and Practice; Springer: Berlin, Germany, 2012. [Google Scholar]
  6. Neis, P.; Zipf, A. Analyzing the contributor activity of a volunteered geographic information project—The case of OpenStreetMap. ISPRS Int. J. Geo-Inform. 2012, 1, 146–165. [Google Scholar] [CrossRef]
  7. Gong, J.; Wu, H.; Zhang, T.; Gui, Z.; Li, Z.; You, L. Geospatial Service Web: Towards integrated cyberinfrastructure for GIScience. GSIS 2012, 15, 73–84. [Google Scholar]
  8. Wu, H.; You, L.; Gui, Z.; Hu, K.; Shen, P. GeoSquare: Collaborative geoprocessing models’ building, execution and sharing on Azure Cloud. Ann. GIS 2015, 21, 109–121. [Google Scholar] [CrossRef]
  9. Geller, T. Imaging the world: The state of online mapping. IEEE Comput. Graph. 2007, 27, 8–13. [Google Scholar] [CrossRef]
  10. Zavlavsky, I. A new technology for interactive online mapping with vector markup and XML. Cartogr. Perspect. 2000, 37, 65–77. [Google Scholar] [CrossRef]
  11. Lins, L.; Klosowski, J.T.; Scheidegger, C. Nanocubes for real-time exploration of spatiotemporal datasets. IEEE Trans. Vis. Comput. Graph. 2013, 19, 2456–2465. [Google Scholar] [CrossRef] [PubMed]
  12. Boulos, M.N.K.; Warren, J.; Gong, J.; Yue, P. Web GIS in practice VIII: HTML5 and the canvas element for interactive online mapping. Int. J. Health Geogr. 2010, 9. [Google Scholar] [CrossRef] [PubMed]
  13. Neumann, A.; Winter, A.M. Time for SVG—Towards high quality interactive web-maps. In Proceedings of the 20th International Cartographic Conference, Beijing, China, 6–10 August 2001.
  14. Jenny, B.; Jenny, H.; Räber, S. Map design for the Internet. In International Perspectives on Maps and the Internet; Peterson, M.P., Ed.; Springer: Berlin, Germany, 2008; pp. 31–48. [Google Scholar]
  15. Lienert, C.; Jenny, B.; Schnabel, O.; Hurni, L. Current trends in vector-based Internet mapping: A technical review. In Online Maps with APIs and WebServices; Peterson, M.P., Ed.; Springer: Berlin, Germany, 2012; pp. 23–36. [Google Scholar]
  16. Open Source Geospatial Foundation. Tile Map Service Specification. Available online: http://wiki.osgeo.org/wiki/Tile_Map_Service_Specification (accessed on 18 March 2016).
  17. Masó, J.; Pomakis, K.; Julià, N. OpenGIS Web Map Tile Service (WMTS) Implementation Standard Version 1.0.0; Open Geospatial Consortium: Wayland, MA, USA, 2010. [Google Scholar]
  18. Jiang, L.; Yue, P.; Lu, X. Restful implementation of catalogue service for geospatial data provenance. ISPRS Arch. 2013, 1, 121–125. [Google Scholar] [CrossRef]
  19. Gui, Z.; Yang, C.; Xia, J.; Li, J.; Rezgui, A.; Sun, M.; Xu, Y.; Fay, D. A visualization-enhanced graphical user interface for geospatial resource discovery. Ann. GIS 2013, 19, 109–121. [Google Scholar] [CrossRef]
  20. ArcGIS Server Website. Available online: http://www.esri.com/software/arcgis/arcgisserver/ (accessed on 13 April 2016).
  21. AUTODESK AUTOCAD MAP 3D. To Add an Image from WMS (Web Map Service). Available online: https://knowledge.autodesk.com/support/autocad-map-3d/learn-explore/caas/CloudHelp/cloudhelp/2015/ENU/MAP3D-Use/files/GUID-A9F620AD-6B9A-487D-9B33-7D365307D571-htm.html (accessed on 13 April 2016).
  22. AUTODESK AUTOCAD CIVIL 3D. To Add an Image from WMS (Web Map Service). Available online: https://knowledge.autodesk.com/support/autocad-civil-3d/learn-explore/caas/CloudHelp/cloudhelp/2017/ENU/MAP3D-Use/files/GUID-A9F620AD-6B9A-487D-9B33-7D365307D571-htm.html (accessed on 13 April 2016).
  23. OpenLayers. Web Map Service Layers. Available online: http://openlayers.org/workshop/layers/wms.html (accessed on 13 April 2016).
  24. GRASS GIS. GRASS GIS Manuals. Available online: https://grass.osgeo.org/index.php?mact=News,cntnt01,detail,0&cntnt01articleid=42&cntnt01returnid=58 (accessed on 13 April 2016).
  25. QGIS. QGIS as OGC Data Client. Available online: http://docs.qgis.org/2.8/en/docs/user_manual/working_with_ogc/ogc_client_support.html#wms-wmts-client (accessed on 13 April 2016).
  26. Neumann, A. Web mapping and web cartography. In Encyclopedia of GIS; Shekhar, S., Xiong, H., Eds.; Springer: Berlin, Germany, 2008; pp. 1261–1269. [Google Scholar]
  27. Fan, J.; Kambhampati, S. A snapshot of public web services. Sigmod. Rec. 2005, 34, 24–32. [Google Scholar] [CrossRef]
  28. Li, Y.; Liu, Y.; Zhang, L.; Li, G.; Xie, B.; Sun, J. An exploratory study of web services on the internet. In Proceedings of the 2007 IEEE International Conference on Web Services, Salt Lake City, UT, USA, 9–13 July 2007.
  29. Zhang, T.; Tsou, M.H. Developing a grid-enabled spatial Web portal for Internet GIServices and geospatial cyberinfrastructure. Int. J. Geogr. Inf. Sci. 2009, 23, 605–630. [Google Scholar] [CrossRef]
  30. GEOSS Clearinghouse Website. Available online: http://clearinghouse.cisc.gmu.edu/geonetwork/srv/en/main.home (accessed on 13 April 2016).
  31. Data.gov Website. Available online: http://catalog.data.gov/dataset (accessed on 13 April 2016).
  32. Lopez-Pellicer, F.J.; Béjar, R.; Florczyk, A.J.; Muro-Medrano, P.R.; Zarazaga-Soria, F.J. A review of the implementation of OGC Web Services across Europe. IJSDIR 2011, 6, 168–186. [Google Scholar]
  33. Lopez-Pellicer, F.J.; Rentería-Agualimpia, W.; Nogueras-Iso, J.; Zarazaga-Soria, F.J.; Muro-Medrano, P.R. Towards an active directory of geospatial web services. In Bridging the Geographic Information Sciences; Gensel, J., Josselin, D., Vandenbroucke, D., Eds.; Springer: Berlin, Germany, 2012; pp. 63–79. [Google Scholar]
  34. Refractions Research. OGC Services Survey. Available online: http://www.refractions.net/expertise/whitepapers/ogcsurvey/ogcsurvey/ (accessed on 18 March 2016).
  35. Skylab Mobilesystems Ltd. OGC WMS Server List. Available online: http://www.skylab-mobilesystems.com/en/wms_serverlist.html (accessed on 18 March 2016).
  36. Bartley, J.D. MAPDEX: A global index of distributed web map services. In Proceedings of the FGDC Coordination Meeting Summary, Washington, DC, USA, 7 May 2005.
  37. Li, W.; Yang, C.; Yang, C. An active crawler for discovering geospatial web services and their distribution pattern—A case study of OGC web map service. Int. J. Geogr. Inf. Sci. 2010, 24, 1127–1147. [Google Scholar] [CrossRef]
  38. Florczyk, A.J.; Nogueras-Iso, J.; Zarazaga-Soria, F.J.; Béjar, R. Identifying orthoimages in web map services. Comput. Geosci. 2012, 47, 130–142. [Google Scholar] [CrossRef]
  39. Lee, K.; Jeon, J.; Lee, W.; Jeong, S.H.; Park, S.W. QoS for web services: Requirements and possible approaches. W3C Work. Group Note 2003, 25, 1–9. [Google Scholar]
  40. INSPIRE. Implementing Directive 2007/2/EC of the European Parliament and of the Council as Regards the Network Services. Available online: http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32009R0976&from=EN (accessed on 13 April 2016).
  41. INSPIRE. Technical Guidance for the Implementation of INSPIRE View Services. Available online: http://inspire.ec.europa.eu/documents/Network_Services/TechnicalGuidance_ViewServices_v3.11.pdf (accessed on 13 April 2016).
  42. Seip, C.; Bill, R. Evaluation and monitoring of service quality: Discussing ways to meet INSPIRE requirements. Trans. GIS 2015. [Google Scholar] [CrossRef]
  43. Anderson, B.; Deoliveira, J. WMS performance tests! Mapserver & Geoserver. In Proceedings of the Free and Open Source Software for Geospatial Conference, Victoria, BC, Canada, 24–27 September 2007.
  44. Horák, J.; Ardielli, J.; Horáková, B. Testing of web map services. In Proceedings of the Global Spatial Data Infrastructure Association World Conference, Rotterdam, Netherland, 15–19 June 2009.
  45. Giuliani, G.; Dubois, A.; Lacroix, P.M.A. Testing OGC Web Feature and Coverage Service performance: Towards efficient delivery of geospatial data. J. Spat. Inf. Sci. 2013, 7, 1–23. [Google Scholar] [CrossRef]
  46. Anthony, M.; Nebert, D. Monitoring the performance and reliability of geospatial web services: Service Status Checker (SSC) system overview. In Proceedings of the Global Spatial Data Infrastructure World Conference, Québec, QC, Canada, 14–17 May 2012.
  47. Li, Z.; Yang, C.; Wu, H.; Li, W.; Miao, L. An optimized framework for seamlessly integrating OGC Web Services to support geospatial sciences. Int. J. Geogr. Inf. Sci. 2011, 25, 595–613. [Google Scholar] [CrossRef]
  48. Gui, Z.; Yang, C.; Xia, J.; Liu, K.; Xu, C.; Li, J.; Lostritto, P. A performance, semantic and service quality-enhanced distributed search engine for improving geospatial resource discovery. Int. J. Geogr. Inf. Sci. 2013, 27, 1109–1132. [Google Scholar] [CrossRef]
  49. Wu, H.; Li, Z.; Zhang, H.; Yang, C.; Shen, S. Monitoring and evaluating the quality of Web Map Service resources for optimizing map composition over the internet to support decision making. Comput. Geosci. 2011, 37, 485–494. [Google Scholar] [CrossRef]
  50. Xia, J.; Yang, C.; Liu, K.; Li, Z.; Sun, M.; Yu, M. Forming a global monitoring mechanism and a spatiotemporal performance model for geospatial services. Int. J. Geogr. Inf. Sci. 2015, 29, 375–396. [Google Scholar] [CrossRef]
  51. Wu, H.; Zhang, H. QoGIS: Concept and research framework. Geomat. Inform. Sci. Wuhan Univ. 2007, 32, 385–388. (In Chinese) [Google Scholar]
  52. SETI@home Website. Available online: http://seti.ssl.berkeley.edu/ (accessed on 13 April 2016).
  53. Climate@Home Website. Available online: http://www.nasa.gov/offices/ocio/ittalk/08–2010_climate.html#.VxR-__mF6Ul (accessed on 13 April 2016).
  54. Zhang, H.; Gong, J.; Wu, H. Research on conception and methods of geospatial information services quality evaluation. Sci. Surv. Map. 2012, 37, 161–164. (In Chinese) [Google Scholar]
  55. Yang, C.; Cao, Y.; Evans, J. Web map server performance and client design principles. Gisci. Remote Sens. 2007, 44, 320–333. [Google Scholar] [CrossRef]
  56. Shen, P.; Gui, Z.; You, L.; Hu, K.; Wu, H. A topic crawler for discovering geospatial web services. J. Geo-Inform. Sci. 2015, 17, 185–190. (In Chinese) [Google Scholar]
  57. EuroGEOSS Broker Website. Available online: http://www.eurogeoss-broker.eu/ (accessed on 13 April 2016).
  58. An ArcGIS REST Service Directory from NOAA Office for Coastal Management. Available online: https://coast.noaa.gov/arcgis/rest/services (accessed on 13 April 2016).
  59. An ArcGIS REST Service Directory from IndianaMAP. Available online: http://maps.indiana.edu/arcgis/rest/services (accessed on 13 April 2016).
  60. Wu, S.; Zhang, M.; Huang, Q.; Zhang, Y.; Wan, C.; Zhang, K.; Cao, J.; Gui, Z.; Qin, K. Design a web portal for visualizing and exploring service quality of global OGC web map services. In Proceedings of the Geoinformatics 2015, Wuhan, China, 19–21 June 2015.
  61. USGS Water Services. Frequently Asked Questions. Available online: http://waterservices.usgs.gov/docs/faq.html (accessed on 13 April 2016).
  62. cURL. Man Page. Available online: https://curl.haxx.se/docs/manpage.html (accessed on 13 April 2016).
  63. Directive, I.N.S.P.I.R.E. Directive 2007/2/EC of the European Parliament and of the Council of 14 March 2007 Establishing An Infrastructure for Spatial Information in the European Community (INSPIRE). Available online: http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32007L0002&from=EN (accessed on 13 April 2016).
  64. INSPIRE Geoportal Website. Available online: http://inspire-geoportal.ec.europa.eu/ (accessed on 13 April 2016).
  65. Group on Earth Observations. GEO Societal Benefit Areas. Available online: https://www.aprsaf.org/data/feature/f_086_3.pdf (accessed on 13 April 2016).
  66. RGIS Website. Available online: http://rgis.unm.edu/ (accessed on 13 April 2016).
  67. Newman, M.E. Power laws, Pareto distributions and Zipf’s law. Contemp. Phys. 2005, 46, 323–351. [Google Scholar] [CrossRef]
  68. A GeoServer WMS from the Socioeconomic Data and Applications Center (SEDAC). Available online: http://sedac.ciesin.columbia.edu/geoserver/wms?SERVICE=WMS&REQUEST=GetCapabilities&VERSIO N=1.3.0 (accessed on 18 March 2016).
  69. China Employment and Wage Maps from the National Geomatics Center of China (NGCC). Available online: http://gisserver.tianditu.com/TDTService/ew/2014/wms?SERVICE=WMS&REQUEST=GetCapabilities&VERSION=1.3.0 (accessed on 18 March 2016).
  70. NASA Earth Observations (NEO) WMS. Available online: http://neowms.sci.gsfc.nasa.gov/wms/wms?SERVICE=WMS&REQUEST=GetCapabilities&VERSION=1.3.0 (accessed on 18 March 2016).
  71. Kim, E.; Lee, Y. Quality Model for Web Services; Organization for the Advancement of Structured Information Standards: Burlington, MA, USA, 2005. [Google Scholar]
  72. Coallier, F. Software Engineering–Product Quality—Part 1: Quality Model; International Organization for Standardization: Geneva, Switzerland, 2001. [Google Scholar]
  73. A Geological Data WMS from Arizona Geological Survey. Available online: http://services.azgs.az.gov/ArcGIS/services/OneGeology/AZGS_Arizona_Geology/MapServer/WMSServer?SERVICE=WMS&REQUEST=GetCapabilities&VERSION=1.3.0 (accessed on 18 March 2016).

Article Metrics

Citations

Article Access Statistics

Journal Statistics
Multiple requests from the same IP address are counted as one view.
Download PDF
Guided Classification System for Conceptual Overlapping Classes in OpenStreetMap
Previous
Implementation of Geographical Conditions Monitoring in Beijing-Tianjin-Hebei, China
Next