Gender Differences in Collaboration Patterns in Computer Science
Round 1
Reviewer 1 Report
The paper is a preliminary study of the so-called "gap gender" in CS, with respect to different subareas of CS and perceived collaboration patterns.
Generally, the paper is well written and the results are interesting or line up mostly with what is known; I think the paper invites more exploration, which makes it something worth publishing.
I think it is well organized, but can use more context and more neutrality in the writing, as it is a phenomenon worth studying and it is easy to impose one's own wishes into work that is of a politicized nature as this topic is. For example, the proposed gender gap may just be something that exists, not something to be problematized or fixed (while supporting people, regardless of one's gender). At some points in the paper, it rears its head, when instead more neutral (focusing only on the measurements) writing could be used. As discussed in the paper, there is a well exhibited pattern in women/men with respect to people/things or systems, given that some parts of CS (especially theory and systems) are very intimately tied to systemized thinking and the purest kinds of thing-oriented study; what if this is just an element of CS that is intrinsic to the kind of research occurring? This work is valuable preliminary work, it should remain as neutral as possible about this phenomenon.
General comments:
=================
-How do we know what a "large" gender gap is? I strongly recommend against imposing a meaning to the size of the gap, the gap may, hypothetically speaking, be not large for what is expected. There's an underlying assumption. In all places where this underlying assumption is made, it is strongly advised to focus on your metrics only, or at least provide more context to better quantify the scope of "large" here and in all other places.
-Section 1, lines 22-23: I strongly suggest removing this sentence (with citations 1 and 2) entirely, one seems to be an opinion piece and the other is about engineering careers and seems to be about undergraduate engineering programs (not academic CS research). It weakens the work to introduce irrelevant citations that are not about the topic the paper is about. Just remove this, it does not take away anything from the paper.
-Section 1, last paragraph: Does this work take into account that different CS communities may use different conventions for author order? This should be acknowledged or at least contextualized more carefully. Are these CS-related papers? What limits are there here?
-Section 2, "Conference data instead of journal data". CS can often be multidisciplinary, the list of conferences provided may not include every major conference a CS researcher publishes in, this is very difficult to of course pin down. This should be acknowledged as well, as most areas of CS have this facet to them (from applications to theory). Either make this more explicit here or in the "Limitations" subsection.
-Page 15, lines 497-498: Don't write this, this is incorrect and a strange remark. Theoretical computer science (TCS) is the foundation of CS, it most certainly is an area of CS (speaking as a researcher in this area). I would, for example, not categorize Theoretical Physics as Mathematics, just as shouldn't conflate a mathematician with a theoretical computer scientist (some of which, are mathematicians, some are computer scientists). Instead I urge you to instead relate this more to what you are focusing on instead of speculating what is or isn't CS, applied areas of CS. Perhaps, say that "... perhaps due to the foundational, mathematical nature of theoretical computer science, in contrast to many applied areas of CS".
-Question: In the paper, has the thought about the maturity of an area come into consideration? Systems (I mean, proper systems research, not the more applied topics) and theory, for example, are far more mature areas of CS (especially theory) than some of the more applied examples cited (e.g. data visualization), this may play a role in how easy/difficult it is to enter an area and have success in conducting research in it. What if it's the combination of accessibility to entry/success/interest to/in a field plus the things vs. people evidence taken together, with some other factors (e.g. marketing [how does one eliminate the effects in 2017 or other years of fads/trends in CS, if more jump into these "sexy" areas (as was with ML and Data Science at this time), you will generally see more people in them], the softness of the area [I invite you to look more into this, but in Canada and the USA, lots of CS students tend to have challenges with Maths, which in some places is a broader trend])? Some areas are more fundamental to the field, hence, they are harder to make significant contributions within them (or as a result, may be far more niche, reducing the visibility of an area to, for example, some women). This is something perhaps to invite more discussion about. There are many hypotheses to jump to about this, based on sex differences research (amplified sex differences in IQ in the most extremes, for example).
-Question: Systems research is not only in the CS bubble, does the study account for this? For example, how does one account for those doing CS systems theory, in what are not seen as traditionally CS conferences? In addition, certain collaborators and groups may not be housed in CS departments, is there anything interesting to say about this? I don't think your paper addresses this, but it is just something interesting to note.
I did not check the formatting of the references.
Specific comments:
==================
-Abstract, sentence one: Define "gender imbalance", as opposed to what?
-Section 1, page 1, line 20: Define what you mean by "gender gap", it is an important part of the paper, it should be presented in a clearly formulated manner. If it is well-studied, than most certainly the paper can briefly describe what it is in a neutral manner.
-Section 1, page 1, line 24: Is there a more neutral word than "severe" than can be used here, that is in more exact terms?
-Section 1, page 1, line 29: Be more neutral, do not introduce activist narratives into this work. Avoid phrases like "challenge". It's a phenonemon in some STEM fields (at least where women are underrepresented) that men are overrepresented than women (and vice versa, if one projects a 50/50-based view of how one should expect things [it may not in reality work like that]). I recommend calling it a phenomenon instead of a "challenge".
-Section 1, page 2, lines 42-43: How does this sentence connect to the previous sentence? Are you focusing on these more applied areas or are you focusing on the main areas of CS? Rephrase.
-Section 2, page 4, line 169. As theoretical CS frequently comes up in this paper, it would be an excellent example to describe here. Theoretical CS is quite broad and foundational to CS, but also very niche and generally difficult to enter (regardless of one's identity).
-Section 2, page 4, lines 189-193: Delete all of these sentences. Just say "For this study, the most critical piece of information on these researchers is their perceived gender. We use the gender terms..." It's specific, and explains the limitation without needing to re-state something stated later that is stated as a non-issue in the study (on Page 5, paragraph 2).
-Section 3, page 5, line 221-222: I advise reminding the reader what FAR is, and show the formula for calculating it as it is most relevant in this section.
-Section 4, page 15, line 458: Instead of using "fields", maybe use a phrase like "reseach activities". Mathematics is a field that heavily overlaps with CS (as CS is primarily a mathematical science), theoretical CS is a subfield of CS.
-Section 4, page 15, line 465: What is meant here by "diversity"? You are studying gender, not a presumed broader label like "diversity". I suggest helping the reader by quantifying what is meant by "diversity" here, as you are not studying anything but prescribed labels of genders to researchers based on bibliometric data. Just keep it simple and narrow to what is being studied, or omit it.
-Section 4, page 15, line 470: "worsens the problem for the next woman". Which problem?
Author Response
We thank the reviewer for supporting the publications of the results of the paper. We are also grateful for taking the time to offer detailed feedback and specific suggestions to improve the paper and its language. We do agree that in some instances, the language of the paper can be reworded to focus more on the data and less on aspirational goals. In other instances where we disagreed with the reviewer recommendation, we tried to include clearer and more elaborate language.
The following is the complete list of the reviewer's specific recommendations, with our responses in bold.
General comments:
=================
-How do we know what a "large" gender gap is? I strongly recommend against imposing a meaning to the size of the gap, the gap may, hypothetically speaking, be not large for what is expected. There's an underlying assumption. In all places where this underlying assumption is made, it is strongly advised to focus on your metrics only, or at least provide more context to better quantify the scope of "large" here and in all other places.
We edited the text throughout to describe the gap in quantitative terms, without ascribing meaning to it with the adjective "large".
-Section 1, lines 22-23: I strongly suggest removing this sentence (with citations 1 and 2) entirely, one seems to be an opinion piece and the other is about engineering careers and seems to be about undergraduate engineering programs (not academic CS research). It weakens the work to introduce irrelevant citations that are not about the topic the paper is about. Just remove this, it does not take away anything from the paper.
Respectfully, we disagree. This sentence merely echoes the many established research findings that the gender gap in CS as a whole is both consequential to society and correctable, which is the point here. It's true that CS/IT is more general than academic research, but this introductory statement sets the tone to why the reader should care about our findings for academic CS research. Reference 2 on engineering is relevant because a significant number of CS papers (at least the ones we investigated in the field of systems) have authors with engineering department affiliations. As for reference 1, the opinion piece, we included it because it surveys about a dozen other studies itself, which reinforce this point. We've also surveyed the literature ourselves and added in this revision two more recent references to support this point, including one specifically for CS research.
-Section 1, last paragraph: Does this work take into account that different CS communities may use different conventions for author order? This should be acknowledged or at least contextualized more carefully. Are these CS-related papers? What limits are there here?
That is an excellent question! In our dataset, a very low percentage of papers (15%) with 3+ authors were presented in alphabetical order. We added this statistic to this section for systems papers in particular (our focal point). We did not break down this statistic by subfield because the high variance in the number of coauthors by subfield (e.g., many single-author papers in theory) would make this analysis unreliable for our sample sizes.
-Section 2, "Conference data instead of journal data". CS can often be multidisciplinary, the list of conferences provided may not include every major conference a CS researcher publishes in, this is very difficult to of course pin down. This should be acknowledged as well, as most areas of CS have this facet to them (from applications to theory). Either make this more explicit here or in the "Limitations" subsection.
We made this point more explicit in the Limitations subsection.
-Page 15, lines 497-498: Don't write this, this is incorrect and a strange remark. Theoretical computer science (TCS) is the foundation of CS, it most certainly is an area of CS (speaking as a researcher in this area). I would, for example, not categorize Theoretical Physics as Mathematics, just as shouldn't conflate a mathematician with a theoretical computer scientist (some of which, are mathematicians, some are computer scientists). Instead I urge you to instead relate this more to what you are focusing on instead of speculating what is or isn't CS, applied areas of CS. Perhaps, say that "... perhaps due to the foundational, mathematical nature of theoretical computer science, in contrast to many applied areas of CS".
We stand corrected and have adopted the suggested wording.
-Question: In the paper, has the thought about the maturity of an area come into consideration? Systems (I mean, proper systems research, not the more applied topics) and theory, for example, are far more mature areas of CS (especially theory) than some of the more applied examples cited (e.g. data visualization), this may play a role in how easy/difficult it is to enter an area and have success in conducting research in it. What if it's the combination of accessibility to entry/success/interest to/in a field plus the things vs. people evidence taken together, with some other factors (e.g. marketing [how does one eliminate the effects in 2017 or other years of fads/trends in CS, if more jump into these "sexy" areas (as was with ML and Data Science at this time), you will generally see more people in them], the softness of the area [I invite you to look more into this, but in Canada and the USA, lots of CS students tend to have challenges with Maths, which in some places is a broader trend])? Some areas are more fundamental to the field, hence, they are harder to make significant contributions within them (or as a result, may be far more niche, reducing the visibility of an area to, for example, some women). This is something perhaps to invite more discussion about. There are many hypotheses to jump to about this, based on sex differences research (amplified sex differences in IQ in the most extremes, for example).
The question of the effect of the maturity of the field on gender ratios is an interesting one, and we have not considered it whthin this paper's scope. One reason is that we do not have any objective metric of maturity. Another is that even mature fields, such as distributed systems and cryptography have fads ("grid computing" and "NFTs" are two respective examples). We would appreciate suggestions on how to measure the maturity of a field as a whole but suspect that it requires a subjective and detailed assessment of each paper, which is obviously nonscalable.
-Question: Systems research is not only in the CS bubble, does the study account for this? For example, how does one account for those doing CS systems theory, in what are not seen as traditionally CS conferences? In addition, certain collaborators and groups may not be housed in CS departments, is there anything interesting to say about this? I don't think your paper addresses this, but it is just something interesting to note.
We are not sure what the reviewer is referring to by "the CS bubble". Our choice of conferences necessarily omits some niches of the CS research ecosystem, as does our choice of a single year and no journal articles. Our paper does not make claims for exhaustive comparisons or datasets, only for relative comparisons among the conferences that were selected. If the reviewer feels that some specific conferences merit inclusion in this analysis, we would be glad to consider such suggestions.
Specific comments:
==================
-Abstract, sentence one: Define "gender imbalance", as opposed to what?
As opposed to gender parity. We replaced "imbalance" with "disparity".
-Section 1, page 1, line 20: Define what you mean by "gender gap", it is an important part of the paper, it should be presented in a clearly formulated manner. If it is well-studied, than most certainly the paper can briefly describe what it is in a neutral manner.
"Gender gap" is a term widely used in the literature. We have attempted to define it as clearly as we can in the following paragraph.
-Section 1, page 1, line 24: Is there a more neutral word than "severe" than can be used here, that is in more exact terms?
We replaced it with "noteworthy".
-Section 1, page 1, line 29: Be more neutral, do not introduce activist narratives into this work. Avoid phrases like "challenge". It's a phenonemon in some STEM fields (at least where women are underrepresented) that men are overrepresented than women (and vice versa, if one projects a 50/50-based view of how one should expect things [it may not in reality work like that]). I recommend calling it a phenomenon instead of a "challenge".
We used the suggested wording.
-Section 1, page 2, lines 42-43: How does this sentence connect to the previous sentence? Are you focusing on these more applied areas or are you focusing on the main areas of CS? Rephrase.
We clarified this sentence.
-Section 2, page 4, line 169. As theoretical CS frequently comes up in this paper, it would be an excellent example to describe here. Theoretical CS is quite broad and foundational to CS, but also very niche and generally difficult to enter (regardless of one's identity).
We think it would be difficult to define theoretical CS in the context of this paragraph, which explicitly states that delineating specific subfields is subjective and nonstandard. For example, the SPAA conference may be considered by some to be in systems and by others to be in theory (and in practice, varies by each paper).
Instead, we added your suggested distinction to the end of the discussion section, where theoretical CS is singled out.
-Section 2, page 4, lines 189-193: Delete all of these sentences. Just say "For this study, the most critical piece of information on these researchers is their perceived gender. We use the gender terms..." It's specific, and explains the limitation without needing to re-state something stated later that is stated as a non-issue in the study (on Page 5, paragraph 2).
We feel that is necessary to justify our simplifying approach to gender in a way that isn't detailed on page 5, so we decided to leave this paragraph as is.
-Section 3, page 5, line 221-222: I advise reminding the reader what FAR is, and show the formula for calculating it as it is most relevant in this section.
Oddly, we are unable to find the reference to FAR in lines 221-222. The first use of the acronym FAR in the paper is in line 227 of the first submitted version, where the acronym is explicitly defined.
-Section 4, page 15, line 458: Instead of using "fields", maybe use a phrase like "reseach activities". Mathematics is a field that heavily overlaps with CS (as CS is primarily a mathematical science), theoretical CS is a subfield of CS.
That is a useful suggestion, but to maintain consistency both within this paper and with our previous related publication, we prefer to leave this term as is.
-Section 4, page 15, line 465: What is meant here by "diversity"? You are studying gender, not a presumed broader label like "diversity". I suggest helping the reader by quantifying what is meant by "diversity" here, as you are not studying anything but prescribed labels of genders to researchers based on bibliometric data. Just keep it simple and narrow to what is being studied, or omit it.
We replaced "lower diversity" with "higher gender disparity".
-Section 4, page 15, line 470: "worsens the problem for the next woman". Which problem?
Thank you for pointing out the confusing wording. We have attempted to clarify it.
Reviewer 2 Report
The article characterizes the relationship between gender and multi-author publications in computer science. Bibliometric data are recalled and analysed. The obtained results do not raise any objections. Some graphs are difficult to read, but adding raw data allows you to analyse the context in more detail.
I recommend accepting the article for publication.
Author Response
The article characterizes the relationship between gender and multi-author publications in computer science. Bibliometric data are recalled and analysed. The obtained results do not raise any objections. Some graphs are difficult to read, but adding raw data allows you to analyse the context in more detail.
I recommend accepting the article for publication.
Thank you for this recommendation. We have revised all the figures and captions for clarity, especially figures 3 and 4.
Reviewer 3 Report
The article is devoted to the important topic of assessing gender inequality in the academy (in this case, in computer science). The article is well structured and well written. I have only one recommendation. It seems to me that Figures 3 and 4 are not very well read. The coding of the gender is not very well chosen, so it is difficult to compare the values for males and females. I would advise thinking about choosing a different encoding method that would improve the readability of the figures.
Author Response
The article is devoted to the important topic of assessing gender inequality in the academy (in this case, in computer science). The article is well structured and well written. I have only one recommendation. It seems to me that Figures 3 and 4 are not very well read. The coding of the gender is not very well chosen, so it is difficult to compare the values for males and females. I would advise thinking about choosing a different encoding method that would improve the readability of the figures.
Thank you for your recommendation. We have redrawn figures 3 and 4 to make them easier to read and more consistent with the other figures.
Reviewer 4 Report
I have reviewed the paper “Gender differences in collaboration patterns in computer science”. I found the paper interesting, and the work is a valuable contribution to the topic of gender gaps in science.
One of my main concerns is the sample that is used. Authors should mention something about how representative is the sample. Gender gap varies importantly among areas of knowledge (as acknowledge by the authors) but also among countries. A recent study of Elsevier about gender gap in science shows that there are countries and fields with smaller gender gap, thus it could be interesting to know something about the nationality of the sample. Are they mostly Americans? This should not imply a lot of work since most proceeding papers report the address of the authors.
The S&T indicators of the National Science Foundation (NSF) also reports the number of graduates (undergraduates) in different fields of knowledge. Authors could compare this kind of indicators with the sample that they are using. This is important because it is well known that there is a gender pipeline in graduate studies, so the high FAR could be explained because only a small proportion of women are in academia in Computer Science, and only the most productive attend international (national) conferences.
A more detailed explanation of the selection of the conferences is suggested.
Hope my comments are useful to improve this interesting paper.
Author Response
> One of my main concerns is the sample that is used. Authors should mention something about how representative is the sample. Gender gap varies importantly among areas of knowledge (as acknowledge by the authors) but also among countries.
We make no claims for the representativeness of the sample, and in fact we mention in the limitations section that this sample may very well not be representative. The overall FAR we computed for CS as a whole may or may not be representative, but it is close to the range of what has been computed in comparable studies. On the other hand, by selecting conferences from different fields, we can directly compare them to each other, without needing the overall sample to be representative.
As described in the paper, we selected conferences based on prestige metrics from Google scholar, personal familiarity with the conferences (at least within systems), and an attempt to include enough papers and authors from every major field in CS. There are other papers (e.g., reference 9 in the original paper) that do attempt to capture a more representative sample of all of CS using more scalable (but less accurate) approaches to infer gender. Our goal in this study was different, as mentioned earlier. That said, we have now tried to clarify this point further in the limitations section.
> A recent study of Elsevier about gender gap in science shows that there are countries and fields with smaller gender gap, thus it could be interesting to know something about the nationality of the sample. Are they mostly Americans? This should not imply a lot of work since most proceeding papers report the address of the authors.
Thank you for bringing this study to our attention. We agree that comparing gender statistics across countries could be very interesting and informative indeed, because we have already done so, using two limited subsets of our dataset (see https://doi.org/10.1145/3476480 and https://arxiv.org/abs/2201.01757)
Unfortunately, our experience with these studies is also how we now know that this analysis does actually require significant effort, and we were only able to accomplish it for the systems subset of the data (2,439 papers).
First, all paper PDFs must be obtained, which is both labor-intensive and complicated by copyright and closed-access issues. Second, CS conference papers do not in fact typically include author addresses. They usually include affiliations for most authors, and often email addresses, but the process of inferring country of residence from these data is not straightforward either. For example, what is the country of residence for someone with a @microsoft.com email?
To try to answer the reviewer's questions for systems only (where we've undergone this significant effort), a majority of authors are US based, and of all the large countries, the US has the highest FAR in systems (at about 11.5%).
> The S&T indicators of the National Science Foundation (NSF) also reports the number of graduates (undergraduates) in different fields of knowledge. Authors could compare this kind of indicators with the sample that they are using. This is important because it is well known that there is a gender pipeline in graduate studies, so the high FAR could be explained because only a small proportion of women are in academia in Computer Science, and only the most productive attend international (national) conferences.
Did you mean the S&E indicators? we looked these up again to make sure we didn't miss anything last time we checked them, here: https://ncses.nsf.gov/pubs/nsf21321/data-tables
The data is broken down by broad discipline and by degree. For example, it shows that about 20% of all postdoctoral fellows in CS last year were women. But it does not break down by CS field and does not look at publication counts, which are the focus of our study.
A more detailed explanation of the selection of the conferences is suggested.
We tried to clarify the criteria in the paper in this revision.
Round 2
Reviewer 1 Report
Thank you for addressing my comments. I have no further comments, and thank you for your responses.
Author Response
Thank you.
