What Is an Open IoT Platform? Insights from a Systematic Mapping Study

Round 1

Reviewer 1 Report

This study investigates the use of the term “open” in Internet of Things literature. This is a very important study especially at the current phase of IoT development. Please see below some feedback on the study and manuscript:

1. In the Introduction and Motivation section, more literature review (at least two additional paragraphs) needs to be added to provide a more comprehensive background about the current research on Internet of Things. For instance, the definition and classification of IoT, the areas of development in IoT, the applications of IoT, etc.
2. Please provide the rationale and inclusion/exclusion criterion regarding “well-known publishers” in more detail. This is a critical part of the methodology.
3. Since only 221 papers were selected, is it correct to say “We conducted a systematic mapping study by retrieving data from 718 papers” (in the Abstract)? It would be beneficial to include the number of papers (n) in each stage in Figure 1 to illustrate this process better. The authors need to use extra caution in describing each step in the methodology.
4. Does the classification process involve multiple raters? The authors should discuss the interrater reliability.
5. Is the use of “demographics data” the most appropriate to describe the content in Section 3.1? I would recommend using something similar to the “distribution of papers” or “descriptive statistics.”
6. What is the purpose of Figure 2? (I understand that it is important to emphasize there are a large number of papers published with no explanation related to Open IoT Platforms, but what is the purpose of showing the trend across years?) Figure 3 seems to be more relevant and essential than Figure 2 in this particular section and should go before Figure 2 if you would like to Figure 2. The authors might consider deleting Figure 2 or better positioning Figure 2.
7. The writing needs to be improved. Please check grammatical errors and typos throughout the manuscript. This manuscript would benefit from professional proofreading by a native English speaker.

Author Response

Thank you very much for your valuable feedback that allowed us to improve the quality of our paper. We have taken into careful consideration and addressed all your suggestions. Below you can find our answers point by point.

1. In the Introduction and Motivation section, more literature review (at least two additional paragraphs) needs to be added to provide a more comprehensive background about the current research on the Internet of Things. For instance, the definition and classification of IoT, the areas of development in IoT, the applications of IoT, etc.

• We have added two paragraphs clarifying the current state of IoT research, its classification and application domains as suggested.
• You can see the details between the lines 19 –34.

2. Please provide the rationale and inclusion/exclusion criterion regarding “well-known publishers” in more detail. This is a critical part of the methodology.

• We made that part clearer as suggested. The reason for choosing these sources and databases based on the recommendations of Brereton et al., [30] and Dyba at el., [31].Also, we now list all sources and databases that were used.
• You can see the updated details between lines 131 –145.

3. Since only 221 papers were selected, is it correct to say “We conducted a systematic mapping study by retrieving data from 718 papers” (in the Abstract)? It would be beneficial to include the number of papers (n) in each stage in Figure 1 to illustrate this process better. The authors need to use extra caution in describing each step in the methodology.

• We have addressed “718 papers” in the abstract by clarifying as follows: “As a result of applying the inclusion and exclusion criteria, 221 papers were selected for review”. See the lines 6-7.
• Moreover, we have edited Figure1 by including the number of papers and improving the overall process description. See the changes in between the lines 161-164 and Figure 1.

4. Does the classification process involve multiple raters? The authors should discuss the interrater reliability.

• We have addressed this part by better reasoning and clarifying the overall classification process. In particular, we have added and clarified the following part “Three researchers in parallel were involved in this process. During the process, the researchers discussed their findings and viewpoints with two additional researchers for validity purposes of the overall study. Moreover, all five researchers met regularly. To minimize the threats of our study, during the classification and mapping process ambiguity was addressed among the team, for,e.g., the challenges a researcher faced when mapping some studies in defined categories. During these meetings, all disagreements were discussed and resolved.”
• See the changes in between the lines 176-182.

5. Is the use of“demographics data” the most appropriate to describe the content in Section 3.1? I would recommend using something similar to the “distribution of papers” or “descriptive statistics.”

• We have changed the Section 3.1 name to “Distribution of Papers” (see the change at line 195).

6. What is the purpose of Figure 2? (I understand that it is important to emphasize there are a large number of papers published with no explanation related to Open IoT Platforms, but what is the purpose of showing the trend across years?) Figure 3 seems to be more relevant and essential than Figure 2 in this particular section and should go before Figure 2 if you would like to Figure 2. The authors might consider deleting Figure 2 or better positioning Figure 2.

• We have addressed your suggestions by merging both figures into one (see the changes in Figure 2).

7. The writing needs to be improved. Please check grammatical errors and typos throughout the manuscript. This manuscript would benefit from professional proofreading by a native English speaker.

• We have revised thoroughly the English, typos and grammatical errors and greatly improved the paper.

Reviewer 2 Report

In this paper, the authors show results from a systematic mapping study about Open IoT Platforms. I liked the content of the paper and how the supplementary material was provided in the GitHub. However, I have the following comments.

Comments/Questions (answers should be used to improve the paper):
1- Systematic review/mapping should be explained to enable its replication. However, some pieces of information are missing in the paper.
a) Consider the text "Three researchers in parallel were involved in this process. During the process, the researchers discussed their findings and viewpoints with two additional researchers for validity purposes of the protocol.". Were there disagreements in this process? If so, how were they resolved?
b) Quality assessment (Section "2.2. Search strategy and sources") was based on "well-known publishers". Was there any objective criterion to choose/limit publishers?
2. I miss the content about "open issues" in the manuscript. It would be interesting for further investigation by other researchers. What is lacking in this research area (i.e., open IoT)? This is an expected "insight" in the manuscript (discussion section).
3. Consider the text "One of the main limitations related to openness is our search string that was focused on open IoT Platforms as such (and not open IoT middleware or framework), from which we might miss other relevant data. However, the publications year of the retrieved studies is wide enough for
capturing relevant data 2012-2019." What is the relation between the first sentence and the second? I mean, the first one is about terms, and the second one is about the range of years. The period could be the same but expanding the search string.
4. The same information "NodeMCU and ThingSpeak hold a share of more than 70%" is presented four times in the manuscript. Too repetitive.
5. The authors could provide more information about future work. This information is not only what they are going to do, but possible investigation gaps to be explored.

Specific points:
- line 8: ...rather than the platforms itself. -> themselves
- line 10: a share of more than 70% -> 70% of what?
- lines 24/25: ...that often are referred to as an IoT platform... -> to as an?
- line 26: ...also referred to as enterprise platforms... -> to as?
- line 48:...applications in between with a reliable solution.. -> in between with?
- line 60: ...One factor of this trend, is... -> no comma
- lines 92/98: researchers point of view -> researchers' point of view
- reference 24 is incomplete
- line 122: footnote 1 is missing
- line 126/131: were applied
- line 149: for e.g. -> no "for"
- line 151: is applied -> are applied
...
There are many grammar errors and typos. Please, revise the whole text carefully.

Author Response

Thank you very much for your valuable feedback that allowed us to improve the quality of our paper. We have taken into careful consideration and addressed all your suggestions. Below you can find our answers point by point.

1. Systematic review/mapping should be explained to enable its replication. However, some pieces of information are missing in the paper.
   
   1. Consider the text "Three researchers in parallel were involved in this process. During the process, the researchers discussed their findings and viewpoints with two additional researchers for validity purposes of the protocol.". Were there disagreements in this process? If so, how were they resolved?
      • We have addressed this part adding and clarifying the following part “Three researchers in parallel were involved in this process. During the process, the researchers discussed their findings and viewpoints with two additional researchers for validity purposes of the overall study. Moreover, all five researchers met regularly. To minimize the threats of our study, during the classification and mapping process ambiguity was addressed among the team, e.g., the challenges a researcher faced when mapping some studies in defined categories. During these meetings, all disagreements were discussed and resolved.”
      • See the changes in between the lines 176-182.
   2. Quality assessment (Section "2.2. Search strategy and sources") was based on "well-known publishers". Was there any objective criterion to choose/limit publishers?
      • The reason for choosing these sources and databases based on the recommendations of Brereton et al., [30] and Dyba at el., [31], we now list all sources and databases that were used.
      • You can view the updated details between lines 131 –145.
2. I miss the content about "open issues" in the manuscript. It would be interesting for further investigation by other researchers. What is lacking in this research area (i.e., open IoT)? This is an expected "insight" in the manuscript (discussion section).
   • Thank you for this feedback. At the end of the discussion section, we have summarized the main findings and highlight several open issues to be investigated.
   • You can view the added parts between lines 470 –482.
3. Consider the text "One of the main limitations related to openness is our search string that was focused on open IoT Platforms as such (and not open IoT middleware or framework), from which we might miss other relevant data. However, the publications year of the retrieved studies is wide enough for capturing relevant data 2012-2019." What is the relation between the first sentence and the second? I mean, the first one is about terms, and the second one is about the range of years. The period could be the same but expanding the search string.
   • We have clarified the first sentence and removed the second one.
   • You can view the updated details between lines 484 –486.
4. The same information "NodeMCU and ThingSpeakhold a share of more than 70%" is presented four times in the manuscript. Too repetitive.
   •  We have modified the repetitions of the same information.
5. The authors could provide more information about future work. This information is not only what they are going to do, but possible investigation gaps to be explored.
   • We have provided additional information about future work.
   • You can view the added parts between lines 526-530.
     1. Specific points: There are many grammar errors and typos. Please, revise the whole text carefully.
        • We edited and fixed all the suggested sentences. Additionally, the entire paper has gone through rigorous editing and we believe all sentences containing errors are now fixed.