4.1. Criteria for Testbed Federation
The federation initially started with five testbeds being integrated with the platform to serve as reference implementations for testbed integration [71
]. These were SmartSantander, SmartICS, SoundCity, Grasse Smart Territory and CABIN. In order to enlarge the value of the offer and also to proof the adequacy of the solutions designed to enable interoperability among heterogeneous IoT platforms, two open calls for testbed integration were conducted. As a result of these Calls, six more testbeds were selected.
The main aim of federating more IoT testbeds and not restricting it to the original four ones is to challenge the platform design. This way tuning of that design can be made by following the lessons learnt and best practices that can only be elicited from actual implementation. Moreover, addition of more application domains also brings further challenges that were not initially considered as they were not present in the initial set of testbeds. This selection was based on the following criteria:
Usefulness: the degree of expected future use of the extension, which takes into account the amplitude (number and variety) of the testbed IoT resources, their nature (i.e., real or virtual resources), the testbed availability and the accessibility to the testbed resources for platform users during the whole project duration and beyond.
Complementarity: the degree at which the testbed will provide new datasets and data streams, whereby it contributes to enlarge the critical mass of the existing experimentation support capacity offered by the 4 integrated testbeds, as well as to probe the interoperability solutions developed within the project, by providing additional datasets and data-streams on the domains of interest of the existing ones. Else, it can offer extra scenarios (smart agriculture, smart factory, crowd-sensing, underwater, etc.) with a high potential impact in terms of the real-world innovation enabled through the offered infrastructure and its associated datasets and data-streams.
Sustainability: The guarantee of availability of the services offered by the extension in absence of future funding. This is linked with the history of the infrastructure and its demonstrable ability to support experimentation.
Technical competence: The testbed provider should exhibit prior testbed management experience and the necessary qualifications to integrate their testbeds within the FIESTA-IoT federation.
Feedback: The potential for providing feedback regarding the platform and the process of integrating new testbeds within the federation. Testbed providers must demonstrate value of the FIESTA-IoT federation procedures and/or motivate added-value extensions. Also, the business impact for joining the federation was considered.
illustrates the assessment of the testbeds based on the criteria set out in the Calls processes. As it has been previously mentioned, the six testbeds whose key features are assessed in Figure 2
are those that were selected during the Open Calls process. The remaining five testbeds were the founding members of the federation.
The overall marks of the different testbeds are remarkable and interestingly high in Complementarity. This provides good insight on the heterogeneity that, in general, is exhibited by the compound offering of the federation of testbeds. This conclusion is also supported by the above average marks that all the testbeds received in the Feedback criterion. On the one hand it shows that the integration of these testbeds can generate valuable lessons learnt, which could not be elicited otherwise, and, on the other hand, it indicates, together with the respective Usefulness marks, the added-value of integrating the offerings from these testbeds.
Another criterion with several excellent marks is the Sustainability. While this is not related to the technical challenges brought forward by the inclusion of this testbed, it certainly demonstrates that the federated testbeds have a long track of previous and future IoT experimentation support.
4.2. Overall Federation Summary and Data Marketplace Offering
For almost all the cases, the IoT devices are actually located around the TP premises. However, since the SoundCity testbed is based on data stemming from smartphones, its actual coverage is not limited to Paris.
provides a summary of the eleven testbeds and highlights roughly the application domain and the IoT devices that are part of each of the deployments. Further, the Table 2
presents categorization of the eleven testbed in different application domain.
Most of these testbeds are internally using proprietary platforms which does not follow any specific standard or largely adopted IoT platform. CABIN testbed is based on oneM2M standard. However, it only implements the functional specifications and not the semantic ontology that has been recently defined by the ETSI oneM2M standard. ADREAM testbed uses a proprietary ontology [76
] which is certainly aligned with the oneM2M base ontology but still is not part of any standard. Finally, SmartSantander testbed is based on FIWARE (https://www.fiware.org/
[Online 3 September 2018]) generic enablers and follows the data catalogues. All in all, the heterogeneity is, significantly, the main feature that can be derived from these testbeds.
Overall, there is a reasonable balance between indoor and outdoor sensors (cf. Figure 3
a) covering five wide application domains (cf. Figure 3
b), namely smart city, smart building, smart energy, smart agriculture and smart sea). The spider graph in Figure 3
b roughly represents the depth in which each of the application domains are covered attending to how many of the federated testbeds cover that particular domain. This creates a quite varied offering able to cope with the needs of a good number of potential IoT researchers and innovators which would like to assess their developments in a scenario excerpted from the combined offering.
In addition to the general details, it is important to highlight the raw figures that the federation of testbeds is continuously providing in terms of active sensors and amount of observations generated. In this sense, Figure 4
shows these two key parameters’ evolution during the second half of December 2017.
More or less constantly, every 6 hours (time scale in X-axis is set to quarter of day for the sake of clarity), the testbeds’ federation is generating around 150,000 observations. These observations are coming, on average, from 2000 active sensors that are continuously capturing events in its environment and reporting them to the platform.
(in Appendix A
) and Figure 5
present the detailed analysis of the offerings from each of the FIESTA-IoT testbeds. In contrast with information available on-line [77
], the figures summarized in Table A1
(in Appendix A
) has been extracted from the actual FIESTA-IoT Platform, thus, presenting the actually available number of active sensors and their observations’ generation frequency and not the textual description of testbeds that indicates the deployed devices but not the ones regularly producing observations, which is the most relevant information for the experimenters willing to carry out their experiments on top of the FIESTA-IoT Platform. The key details presented in Appendix A
, apart from the actual amount of devices producing data from each testbed, are the Phenomenon, which stands for the physical parameter observed, and the Quantity Kind, which stands for the Internationalized Resource Identifier (IRI) assigned in the FIESTA-IoT ontology to the respective physical phenomenon. This latter parameter is particularly important as it is the one that has to be used within the FIESTA-IoT Platform when looking for the corresponding phenomenon.
In terms of the covered application domains, as it is shown in Figure 6
, the Smart Energy domain is the one that has both more active sensors and more generated observations. In this respect, the RealDC, SmartICS and ADREAM testbeds have a large set of IoT devices measuring the power consumption at its Data Centre, offices’ desks and buildings respectively, which produce observations quite often.
Following, the Smart City deployments from the SmartSantander and FINE testbeds also contribute with a significant amount of observations and active sensors to the federation offering. Testbeds, such as CABIN and NITOS, characterized under Smart Building accounts for the sensors that are deployment at indoor environments measuring environmental conditions at the different buildings at which some of the federated testbeds are deployed. Last but not least, Smart Agriculture deployments (Tera4Agri) has a smaller but still relevant share of the available offering. Finally, there are several sensors made available by various testbeds that can be catalogued under other application domains (such as CrowdSensing—SoundCity, Smart Sea—MARINE, Testbed Management or Wireless Network Status—Grasse Territory) as well and still can be discovered by a researcher and/or innovator willing to experiment with them. More yet brief description about the testbeds is provided in Section 4.3
Interestingly, for the Smart City domain, several sub-domains (cf. Figure 7
) are covered by the combined deployments. While SmartSantander deployment accounts for most of these active sensors, there are other testbeds like FINE or SoundCity that contribute with several environment monitoring sensors or CABIN that includes parking availability sensors, thus, enabling experiments that can be transparently applied to different cities.
The other two large domains, in terms of available active sensors, are the Smart Energy and Smart Building. They can be combined into one since they are actually building energy management sensors the ones coming mainly from RealDC and ADREAM testbeds. For this combined domain, also some internal categorization can be made as presented in Figure 8
. The other testbeds related to this category, like SmartICS, NITOS and CABIN contribute to the other big sub-domain, namely, building environmental conditions.
4.3. Federated Testbeds Overview
One of the largest testbeds within FIESTA-IoT federation is the SmartSantander testbed. It is an experimental test facility for the research and experimentation of architectures, key enabling technologies, services and applications for the IoT in the context of a city. The infrastructure made up of 12,000 deployed diverse IoT devices covers a wide area of the city of Santander, located in the north of Spain. This testbed goes beyond the experimental validation of novel IoT technologies. It also aims at supporting the assessment of the socio-economical acceptance of new IoT solutions and the quantification of service usability and performance with end users in the loop.
SmartICS focuses on the domain of smart buildings and is deployed in the Institute of Communication Systems at the University of Surrey. It provides a facility for IoT experimentation using a variety of sensor devices deployed within the building. These sensor devices are mainly installed on office desks in the building and are used to capture a number of quantity kinds relating to air quality, ambient environment, energy consumption and desk occupancy. Observations from the devices are reported to a proprietary testbed server every minute. The proprietary testbed server keeps a register of all reporting sensor devices, and a data repository for each sensor device. Experimenters can interact with the testbed through a proprietary RESTful API, whereby sensors are exposed as dereferenceable web resources.
SoundCity testbed leverages the Ambiciti (http://ambiciti.io
[Online 13th August 2018]) mobile application and the sensors within the smartphone to collect ambient noise measurements. On top of collecting such participatory data on a large scale, the Ambiciti mobile application provides its users with the ability to form groups and contribute to that specific group. The SoundCity testbed leverages such grouping feature and only federates data within the “FIESTA-IoT” group in the FIESTA-IoT platform.
CABIN (Context Aware smart BuildINg) is located on the KETI headquarter premises in Seongnam city, South Korea. This testbed is deployed using OCEAN open source software that implement a M2M IoT platform global standard. The main purpose for the deployed infrastructure is to study building energy optimization considering human behavior. In addition to the indoor sensors, parking sensors are also deployed outside the building. The benefit of having the CABIN testbed is to ensure oneM2M TPS replicability for future oneM2M and FIESTA-IoT enabled testbeds.
NITOS Future Internet Facility is an integrated facility with heterogeneous testbeds that focuses on supporting experimentation-based research in wired networks, wireless networks and IoT in general. NITOS is remotely accessible and open to the research community 24/7 and supports evaluation of protocols and applications under real world settings. The testbed is based on open-source software that allows the design and implementation of new algorithms, enabling new functionalities on the existing hardware. NITOS WSN testbed is part of the overall facility and offers several NITOS Wireless Sensor Motes developed in house by NITlab (https://nitlab.inf.uth.gr/
[Online 13th August 2018]) and deployed in an office environment. NITOS WSN is a smart building testbed, capable of measuring environmental parameters with the purpose of providing the infrastructure upon which an experimenter can build own applications.
The MARINE testbed has been deployed by GRIDNET (http://gridnet.gr/MARINE/
[Online 13th August 2018) for testing the performance of own prototype communication hardware enabling IoT applications in the marine and city environments. Testbed nodes are equipped with a wide variety of heterogeneous communication technologies, ranging from IoT related standards (ZigBee, LoRa) to the widely adopted Wi-Fi and LTE protocols, along with a unique real-time power monitoring framework for monitoring consumption of wireless interfaces. Moreover, the nodes feature a wide set of environmental sensors (17 different sensor types), suitable for application scenarios such as monitoring of water and air quality and detection of potential dangers for inhabitants of the area. The facility currently consists of 8 fixed air quality monitoring stations deployed in the city of Volos, Greece and 4 floating seawater quality monitoring buoys deployed in a bathing and recreation coastal area, close to the city.
RealDC provides live Data Centre environmental information into the FIESTA-IoT ecosystem. This integration comes in the form of power consumption, cooling and ambient weather. The data is captured at five-minute intervals.
Tera4agri Testbed comprises of data collected from the monitoring of environment, soil and trees. Thereby enabling the implementation of innovative experiments in the agriculture domain. The testbed is located in Minervino Murge (Italy) in the Tormaresca - “Bocca di Lupo” (one of Italy’s top wineries) estate. The testbed collects data from the sensors using the Tera s.r.l’s Internet of Everything Gateway.
FINE facility provides an experimental testbed that is able to support innovative IoT applications in the smart city domain. It utilizes RERUM architecture [78
] for enabling the interconnectivity of a large number of heterogeneous IoT devices based on the concept of security, privacy and reliability by design. The testbed comprises several indoor and outdoor deployments in the city of Heraklion in Crete, Greece, operating on 6LoWPAN and LoRaWAN communication technologies to aid applications such as environmental monitoring, comfort quality and energy management and smart parking.
The Grasse Smart Territory Testbed is an experimental testbed for Smart City applications for the urban, suburb and rural areas of the City of Grasse. The main purpose of the testbed is to provide experimental digital facilities and applications to the citizens to make life greener and more efficient using state-of-the-art IoT technologies, and to make public authorities’ managers understand the way IoT technologies can benefit to citizens. It is developed with the collaboration of the local authorities and other local associations and companies. The privileges are given to the use of LoRa technology for the connectivity of devices, which can significantly extend the battery life on the field devices. Several environmental sensors, i.e., CO2, pollen, humidity, are being deployed and tested to be connected to the testbed.
The ADREAM building is a living lab providing a horizontal platform to foster research projects, either focused on one aspect of the building or cross-domain. The building is meant to have as little energy footprint as possible and is thus equipped with a large range of sensors to analyze its energy consumption, as well as its production based on solar panels.