Round 1

Reviewer 1 Report

Comments

The paper asks if a particular gender equity program, the German Women Professors Program, had the intended effect to get more women into professorships and for universities to take up initiatives for organizational and cultural changes to promote gender equity. The study uses an interesting way to measure the quantitative effects of this particular program on gender equity on the increase of women professors in Germany since 2007. 

The argument implied is that governmental programs on gender equity that work with financial incentives to fund professor positions for women and at the same time have the goal organizational structural changes can increase at a minimum awareness but also do positively effect the hiring of women, raising the proportion of women among professors. This is an important contribution to scholarship since it shows the effects of policies on the gendered composition of academic leadership positions.

Using an original, unique dataset, a census of higher education institutions (HEIs) in Germany, the paper makes an important methodological contribution to the literature on gender equity policies and institutional change by using a careful methodological approach to estimate the effects on the proportion of women among professors by comparing universities who did and did not participate in the program. In addition, the longitudinal study of the proportion of women professors in Germany overall is convincing.

Key Issues & Questions:

I suggest to rewrite the abstract to keep in mind that the Journal of Social Science has a broader audience who want to understand how gender equity programs can be effective. And the particular methodological approach is of interest to fewer scholars. So I would shorten the method here and emphasize more the uniqueness of the data & findings and the implications!

In the introduction, the paper should state a clear argument – what does the author suggest is the key take away? Also, the literature review could be reformulate to make clearer which hypothesis will be tested.

I am missing a bit of a discussion of how the author evaluates the relatively size of the effect. Critics have argued that while the program did have the desired effect of increasing the proportion of women professors, the size of the effect seems rather small compared to the efforts and money’s spent on the program. As the author finds a difference of 1.8 percentage at the HEI level and particularly the 2.3 % of increase overall is rather small effect size.  How does the author interpret the results then for short-term, medium-term or longer-term potential for change? And related to this:

Page 12 line 397: I am puzzled as to why the correlation only exists for the regular but not for the other professorships.  How does the author explain that the effect only shows up for the regular but not for the additionally earlier created professorships? What are the implications of this finding, can this finding explain the small effect size? But why?

Page 12 line 379: Mechanisms: When testing the hypothesis that program increased the proportion of women, the author should be more clear about to establish causation, first one has to establish correlation. Line 396,  So, I don’t agree that a high correlation between two variables is an indication of causation.  Instead correlation is a necessary condition but not a sufficient condition for causality. More criteria for causation need to be checked.  And even a weak to medium correlation of r=0.263 can be a causal relationship – but this needs to be argued with time order, non-spuriousness & causal mechanism.  So in line 399, it should say instead of “no effect” should say “no correlation and thus no effect” to clarify this process of establishing causality.

Page 14 line 467: I understand that the study can’t do a systematic comparison between the HEI’s who received and didn’t receive funding from the German Excellence Initiative. But the reader would benefit to know how many of the HEI did get relevant funding through the German Excellence Initiative in the same time frame?

I would like to see the author expand on how the program indeed worked.  The author addresses these questions in how the program might have contributed to the outcomes in the conclusion. But what could explain how hiring changed at the university level? What are the “mechanisms” to establish causality? How did funding increase the proportion in a particular university and hiring process? Why were the universities more likely to appoint a women for the existing position? Did they already have a woman in mind for the position? Did the Women Professor Program than help them to be more competitive with other universities, for example, by using the extra funds to increase their chances of getting the woman (compensation/start up packages etc.). So in other words, how did the universities use the money?  Or why did it not make a difference in so many hiring processes?

One factor that could point to spuriousness is the changes in retirements. I do wonder if the percentage of women among the new hires increased with the program or if perhaps the retirement of male professors during the same time period can explain these outcomes. Since the percentage of men among retiring professors is higher, and more men retried during this period than women, then the program’s success might be entirely due to the change in the gender composition among the senior faculty, rather than actually helped more women to get hired.

Smaller points

Page 1 line 25 what was the European average in 2007? Germany was 12% but what was the EU average?

Page 4 line 156: I’d formulate the 2^nd question more simply as: “What are the mechanisms that contributed to the effects?”

Page 4 line 171, explain the vacancies, give an example “for example due to retirements”

Page 10 line 326 Table: The table now shows 202 total HEIs in East and only 45 total HEIs in the West. This seems to be incorrect. Is Eastern German States & Western German states switched around?

Page 10 line 326 Table: “Time-off” is awkward formulation – do you mean partial/part-time? Or is it voluntary? I assume this would include people who have a full-time job and additional responsibilities as gender equity officer. So perhaps just call it not-full/main time.

Page 13 line 415. So does this mean the effect is estimated more conservatively, since there are more HEI included in the time series analysis?

Page 14 line 476. Why did the applied science universities not consider the other instruments to be meaningful?  Were they eligible at all for the Excellence initiative are they members of the DFG? If not explain here.

Finally, speaking as a non-English speaker myself, I suggest that the author hire a native English speaker to edit the paper thoroughly before publication to improve its readability.

Author Response

Thank you for your helpful comments and your careful reading of the paper of the reviewers. I have taken your comments into account as follows:

Abstract

I have rewritten the abstract as recommended (shortened the methodological part, emphasized the findings and implication).

Introduction

I have added the main argument of the paper at the end of the Introduction.

Literature review

I have rewritten the conclusion of the literature review in order to emphasize better the research questions that I derived from the review. For this paper, I preferred to formulate research questions rather than a hypothesis. According to my theoretical approach, the program theory would have been the starting-point for building a hypothesis. However, I realized that a hypothesis would double the number of research questions, which would reduce the readability of the paper.

Furthermore, I decided not to rewrite major parts of the literature review, as the other reviewer did not suggest any changes to this part of the paper.

Size of the effect

I have added a discussion of the size of the effect in the final part of the paper. I calculated the annual average of both effect calculations (comparison of participating and non-participating HEIs, time series analysis) and related this to the average annual increase in the proportion of women professors from 2008-2015. Furthermore, I argue that the assessment of the program should not be limited to the impact on the increase in women professors, as the program is not limited to this one objective. Finally, the evaluation estimates the efficacy of the program, not its efficiency (the evaluation of the latter is not possible in my view).

Mechanism – correlation and causation

I have added an explanation of the difference between regular and additional professorships. I have also modified the wording regarding correlation and effects. My argument is that, although the correlations suggest the presence of an effect of start-up funding on the increase in the proportion of women professors, they do not establish causation.

German Excellence Initiative

At the first mention of the Excellence Initiative, I have added a footnote (footnote 3) that provides some information about funding under the program and the number of universities that have obtained such funding.

Expand on how the program indeed works

In section 5.2.2 (Mechanisms), I have added an explanation of how the program might work in the HEIs. Ultimately, however, an in-depth analysis of the mechanisms within the HEIs would be beyond the scope of this paper and would have to be addressed in a further study that focuses on qualitative data. The present paper aims to evaluate the impact of the program by means of quantitative data and to discuss the possibilities and limits of this approach.

Retirements

In section 5.3 (Real and Expected Proportion of Women Professors), I have added a paragraph in which I reflect on retirements as an interfering factor and explain that, because the number of retirements was much higher in the years before the program, the time series analysis was not disturbed by this factor.

Smaller points

All suggested changes and explanations have been realized.

Editing of the manuscript by a native English speaker

Before the initial submission, the manuscript was proofread and copy-edited by a native speaker who is a professional copy-editor, qualified translator, and sociologist with many years of experience and a large portfolio of translated and copy-edited journal articles and monographs that have successfully undergone peer review. She also proofread and copy-edited the revised version of the manuscript.

Reviewer 2 Report

The paper addresses a research desideratum in the field of gender equality policies namely the lack of impact evaluations. The paper is well written, interesting to read, refers to the relevant literature on evaluation methods as well as to the rare examples of impact evaluations. As the political discussion about gender equality in academia refers to the German “Women Professors Program” as a good practice example, the paper is of interest for researchers as well as for policy makers. Hence, I highly recommend the paper to be published.

One minor comment: I found it a bit irritating when several references are in a bracket with a “see author X” in the middle (e.g. line 104, 190, 312, 513). I guess the “see” should be placed in the beginning of the list.

Author Response

Thank you for your helpful comments and your careful reading of the paper. The wrong placement of "see" was a mistake of the literature management program.