Community Vital Signs: Measuring Wikipedia Communities’ Sustainable Growth and Renewal

Round 1

Reviewer 1 Report

The study on this topic is very interesting. The structure is clearly and logical and challenging. The research is timely and worthwhile. The authors provide fresh insight into the field.

The article suggests a current and attractive topic for the academy. The effort made is evident, but it requires some adjustments for better understanding and quality.

Authors should follow the style of a structured abstract, which is based on the IMRAD structure of a paper. The abstract should state briefly the purpose of the research, the principal results, and major conclusions. An abstract is often presented separately from the article, so it must be able to stand alone.

I hope you find the following observations helpful:

Materials and methods: I found this section very important for the readability of the paper. Methods should be described in detail. I think the research procedure could be much more clearly described by means of a diagram also highlighting its potential and limit.

The references, although varied, are up to date. Which says a lot.

Authors should take into account more previous works (e.g. theoretical, conceptual, and empirical reviews) published in the literature. Authors should discuss the results and how they can be interpreted from the perspective of previously published studies. I suggest adding this reference: 

1. Korobiichuk I., Syerov Y., Fedushko S. (2020) The Method of Semantic Structuring of Virtual Community Content. In: Szewczyk R., Krejsa J., Nowicki M., Ostaszewska-Liżewska A. (eds) Mechatronics 2019: Recent Advances Towards Industry 4.0. MECHATRONICS 2019. Advances in Intelligent Systems and Computing, vol 1044. Springer, Cham. pp 11-18. https://doi.org/10.1007/978-3-030-29993-4_2

2. Akif Hatipoglu, Sevinç İlhan Omurca,"A Turkish Wikipedia Text Summarization System for Mobile Devices", International Journal of Information Technology and Computer Science(IJITCS), Vol.8, No.1, pp.1-10, 2016. DOI: 10.5815/ijitcs.2016.01.01

I strongly recommend adding this work to the list of references.

Authors should discuss the results and how they can be interpreted from the perspective of previously published studies.

The diagrammatic presentation of the study research will be the strongest section of this work. I suggest adding a visual presentation of obtained outcomes in section Results.

I also suggest a grammar and spelling review. 

Overall, I find the paper adequate but it can be improved by addressing the aforementioned issues.

The conclusion is thorough.

Author Response

 

• Materials and methods: I found this section very important for the readability of the paper. Methods should be described in detail. I think the research procedure could be much more clearly described by means of a diagram also highlighting its potential and limit.

 

We have generated a diagram including the different phases of the work along with its main method and the objectives we pursue. We believe this diagram allows the reader to get a glimpse of the entire paper and see how the different objectives are related to each other.

 

• Authors should take into account more previous works (e.g. theoretical, conceptual, and empirical reviews) published in the literature. I suggest adding this reference: 
• Korobiichuk I., Syerov Y., Fedushko S. (2020) The Method of Semantic Structuring of Virtual Community Content. In: Szewczyk R., Krejsa J., Nowicki M., Ostaszewska-Liżewska A. (eds) Mechatronics 2019: Recent Advances Towards Industry 4.0. MECHATRONICS 2019. Advances in Intelligent Systems and Computing, vol 1044. Springer, Cham. pp 11-18. https://doi.org/10.1007/978-3-030-29993-4_2
• Akif Hatipoglu, Sevinç İlhan Omurca, "A Turkish Wikipedia Text Summarization System for Mobile Devices", International Journal of Information Technology and Computer Science(IJITCS), Vol.8, No.1, pp.1-10, 2016. DOI: 10.5815/ijitcs.2016.01.01
• I strongly recommend adding this work to the list of references.

We have improved the description of the state of the art, adding references to further previous literature, in particular with respect to other visual tools proposed in previous studies to expose different aspects of social interactions in Wikipedia. 

• Authors should discuss the results and how they can be interpreted from the perspective of previously published studies.

We have revisited the sections 5. Discussion and 6. Conclusions, and added references to previously published studies in order to interpret our results in the context of previous literature. 

• The diagrammatic presentation of the study research will be the strongest section of this work. I suggest adding a visual presentation of obtained outcomes in section Results.

We included a table in the Results section. We believe this table helps the reader get a clear overview of the 6 Vital signs introduced in this work and the main aspects of each of them. This new presentation of the results includes those related to Objective 2, which is the paper's main contribution: it encompasses for each Vital sign the research question it stems from, the corresponding indicators, the main findings, and the suggested targets for the Wikipedia language communities.

• I also suggest a grammar and spelling review. 

We have performed an accurate grammar review on the document and caught minor issues that had remained in the previous version of the manuscript.

Reviewer 2 Report

This paper argues that we need a set of metrics, called Vital Signs, that can give a more nuanced picture of the health of an online community, like Wikipedia. They study the stability of the community, as a concern to forecast upcoming problems. The paper introduces the definition of six “Vital Signs,” each of which is associated with one or more indicators that are proposed for assessing and monitoring a different aspect of community sustainability.

Both abstract and introduction of the paper must be substantially improved, improving readability. Even though the objectives are defined but the scientific contributions are not clear. A list of bulleted points in the introduction is needed.

Despite the comprehensive literature, authors have not indicated the importance of social links and text understanding as two references are indicated below:

Hua, Wen, et al. "Understand short texts by harvesting and analyzing semantic knowledge." IEEE transactions on Knowledge and Data Engineering 29.3 (2016): 499-512.

Najafipour, S., et al. "SoulMate: Short-text author linking through Multi-aspect temporal-textual embedding." IEEE Transactions on Knowledge and Data Engineering 34.1 (2020): 448-461.

The structure of the paper would better be focused on scientific contributions, used as the title of subsections. I propose to add a framework overview that can illustrate the components of this research that propose a novelty. Defining the Vital Signs is not considered as a major contribution.

In general, the main drawback of this paper is the fact that the sections are not aligned with each other. A long literature review that is not fully systematic, a short methodology section that does not include a clear justification for a well-defined problem statement, neither declared with preliminary definitions nor equipped with clear mathematical notations.

The experimental section is more like a visualization dashboard, resembling the empirical results. The results section should include some baselines and clear benchmarks to assure the proposed Vital signs are effective enough or can surpass similar works.

 

 

Author Response

 

• Both abstract and introduction of the paper must be substantially improved, improving readability. Even though the objectives are defined but the scientific contributions are not clear. A list of bulleted points in the introduction is needed.

We have improved the abstract and introduction sections to achieve better readability. We added a list of bullet points to summarize the main contributions of the study in the introduction. We think this improvement helps the reader to easily capture the contributions of the study, we thank the reviewer for this suggestion. 

• Despite the comprehensive literature, authors have not indicated the importance of social links and text understanding as two references are indicated below:
• Hua, Wen, et al. "Understand short texts by harvesting and analyzing semantic knowledge." IEEE transactions on Knowledge and Data Engineering 29.3 (2016): 499-512.
• Najafipour, S., et al. "SoulMate: Short-text author linking through Multi-aspect temporal-textual embedding." IEEE Transactions on Knowledge and Data Engineering 34.1 (2020): 448-461.

We have improved the state-of-the-art description, adding references to further previous literature, particularly concerning other visual tools proposed in previous studies to expose different aspects of social interactions in Wikipedia.

• The structure of the paper would better be focused on scientific contributions, used as the title of subsections. I propose to add a framework overview that can illustrate the components of this research that propose a novelty. Defining the Vital Signs is not considered as a major contribution.

Following the reviewer’s suggestion, we have added a diagram in the Introduction section to provide a framework overview that illustrates the various phases and components of our research.  

• In general, the main drawback of this paper is the fact that the sections are not aligned with each other. A long literature review that is not fully systematic, a short methodology section that does not include a clear justification for a well-defined problem statement, neither declared with preliminary definitions nor equipped with clear mathematical notations.

To address the reviewer’s concerns, we added some paragraphs to articulate the relationship between each section of the background and the problem that needs to be solved. We also added two diagrams to clarify the structure of the manuscript and the approach and main findings of the work. The diagram that we added in the Introduction section illustrates the different objectives of the paper along with the methods we employed to provide results. The three objectives build on each other to understand the risk of unsustainable and declining Wikipedia communities. The second diagram, added in the Results Section, helps the reader visualize how each Vital sign stems from a research question, then it is translated into a set of indicators, which we report for eight selected communities; finally, we report our suggested target values for the communities. We hope that the relationship between the sections is now more straightforward with these improvements, and the manuscript’s discourse is easier to follow. 

• The experimental section is more like a visualization dashboard, resembling the empirical results. The results section should include some baselines and clear benchmarks to assure the proposed Vital signs are effective enough or can surpass similar works.

The Vital Signs are the first set of indicators focused on community renewal. To the best of our knowledge, no other existing tools have been implemented to study this problem in Wikipedia. Therefore, it is not possible to compare them to any existing tools or even indicators. Their effectiveness should stem from the fact that they have been designed according to previously detected needs and address problems already framed in previous literature and validated by Wikimedia affiliates and community members. In the absence of baselines or benchmarks to compare our work with, we have assessed the effectiveness of our solution through the feedback received from the affiliates and communities in the various focus groups we have organized, described in Section 4.3, “Validation and affiliates’ feedback,” and we have discussed this issue in Section 5.3 “Study limitations and future steps.”

 

Reviewer 3 Report

The authors propose the creation of 6 sets of language-independent indicators, which have been called "vital signs". Three out of six sentences focus on the general population of active editors who create content: retention, stability, and balance; The other three relate to specific community functions: Specialists, Administrators, and Global Community Involvement. The authors present an analysis for 8 Wikipedia language editions and show that communities renew their productive power while absolute numbers stagnate. This happens to a lesser extent in locations related to specific functionalities or admin flags. Finally, the authors evaluate a framework by discussing these indicators with Wikimedia partners to assist them in driving the changes necessary for communities to grow. 

The paper is very interesting and easy to read. Unfortunately, various scientific questions arise that were not addressed in the work.

• The state of the art is missing, what was written and developed by other authors or community on this topic at the time?
• And what is special about your solution compared to other solutions that already exist in the literature? There are many indicators that have certainly also been discussed in other areas.
• How will you monitor the indicators? I don't find this discussed anywhere, whether it was addressed in the conclusion. 

Author Response

 

• The state of the art is missing, what was written and developed by other authors or community on this topic at the time?

We have improved the description of state of the art, both in the Introduction section - where we define the context and motivation for our study - and in the Background section - where we describe previous studies and community practices. We have added references to additional previous literature in both sections, particularly regarding other visual tools proposed in previous studies to expose different aspects of social interactions in Wikipedia and different dimensions of community dynamics. We have also enriched the discussion of the results with further context and references from previous literature. 

• And what is special about your solution compared to other solutions that already exist in the literature? There are many indicators that have certainly also been discussed in other areas.

Although there exists literature explaining the decline of specific Wikipedia language communities, to our knowledge, there is no study aimed at explaining the state of community renewal and no solution to help communities monitor their state. Therefore, we referenced the solutions which proposed to measure other important aspects of Wikipedia communities. There exist tools for quantifying the outcomes of an edit-a-thon, tracking the winners of an editing campaign, or the controversiality of articles, and we have added references to them in this revised version of the manuscript. As discussed in Section 5.3 Study limitations and future steps, with the dashboards, we are working on turning the results of this work into a valuable tool for the communities.

 

 

• How will you monitor the indicators? I don't find this discussed anywhere, whether it was addressed in the conclusion. 

In Section 5.3, “Study Limitations and Future Steps,” we present some screenshots of the future website that will include the different metrics we designed and validated with the feedback from Wikimedia affiliates and community members. These metrics will be included in several dashboards to allow Wikimedians to navigate and compare the data of the different language communities and prepare actions to improve on them. Similarly, the results from the different metrics will be presented in the form of tables on the Meta-wiki website. The graphs and tables will be updated monthly when new data are available. All these aspects have been now clarified and expanded in section 5.3.

Round 2

Reviewer 1 Report

Accept in present form