New Developer Metrics for Open Source Software Development Challenges: An Empirical Study of Project Recommendation Systems
Round 1
Reviewer 1 Report
The authors propose the use of developer metrics taken from activities contained within Open Source Software Projects (OSS) as a means of providing a project recommendation system. It seems like the created metrics work well in providing recommendations based on the evaluation performed.
The background does not link the papers together, other than by assuming that each portion of an OSS project that has been examined in the literature can be used as a metric.
I would appreciate additional references into studies on how these metrics contribute or how they may interact, such as the one found at (https://doi.org/10.3390/systems8030028). While this focuses more on user trust, it does examine the metrics associated with what users believe are good OSS projects, which are likely also to be needed for a recommendation system. The background reads more as a justification that the metrics shown in Table 1 have been examined rather than showing that they are actually useful. It is possible the papers referenced do this, but this is not clear in the background.
I really appreciate the use of the GitDataSCP dataset. It was not clear in the background that those studies were not using a sparse dataset, which would have been nice.
Overall, there is not enough detail in most of the descriptions in sections 3 and 4 to fully understand what is being done. I am confused with how the recommendation system works. Section 3.2 “The Recommendation Model” does not seem to actually define the recommendation model, instead focusing on the normalization method for the metric matrix. It looks like, in Sections 3.3 and 3.4, you are using the watching metric to evaluate if a recommendation was good, but also calculating unrated projects, implying that the recommendation system is based on the calculations of the similarity ratings. If this is the case, why would watching not be factored into the recommendations? It seems like watching a project shows interest. It seems like a better method for evaluating recommendations is needed, possibly by checking new projects the developers have worked on after the GitDataSCP information was grabbed.
Author Response
Thank you for your all supportive comments.
The responses are in the attached document.
Please see the attachment.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Very interesting article. Innovative in the research area and with great relevance. Very well written, clear and methodologically well designed. A good contribution.
Author Response
Thank you for your all supportive comments.
The responses are in the attached document.
Please see the attachment.
Author Response File: Author Response.pdf
Reviewer 3 Report
In the paper, the authors demonstrate that was created an item-based project recommendation system to solve the issues which is the GitHub project recommendation and to combine similar properties or activities, as like RQ1~RQ4 author propose to.
Most of important thing is how to compare with between the documents. About these things, Authors proposed to based on tf-idf and cosine similarity.
These method have problem which is only using word counts when they make their matrics.
Also, when calculating the metirc, a value of 0 may appear if a lot of same words are counted, because it is using IDF.
So, the researchers who regarding this issue are trying to overcome these problems.
In this focus, this paper is an interesting paper.
Other researchers sometimes study by changing to algorithms like Word2Vec in this regard.
Since various evaluation items are calculated and provided in this study, the above may be treated less importantly, but comparison between documents is a very difficult problem, and is seen as a key part of this paper in terms of algorithms.
If the research on this is supplemented, I think it will be better.
Author Response
Thank you for your all supportive comments.
The responses are in the attached document.
Please see the attachment.
Author Response File: Author Response.pdf
Reviewer 4 Report
In the paper "New developer metrics for open-source software development challenges: An empirical study of project recommendation system" the authors made an interesting attempt to create new metrics for open-source projects recommendation. Their research base on the data from the most popular open-source code repository GitHub. Firstly they analyzed literature about existing solutions (like stars, forks, etc.). Then the dataset of 100 developers and 41,280 projects have been taken and used to find similarity scores between projects. Finally, two evaluation methods (watch/fork and language-based) have been evaluated. Developer metrics scores were clearly described and presented in tables and figures.
Research presented in the article matches the journal "Applied Sciences".
Author Response
Thank you for your all supportive comments.
The responses are in the attached document.
Please see the attachment.
Author Response File: Author Response.pdf
