The DOAJ Spring Cleaning 2016 and What Was Removed—Tragic Loss or Good Riddance?
Round 1
Reviewer 1 Report
This article examines a group of journals that were removed from DOAJ due to their failure to reapply for an inclusion within a given time period. It helps readers understand possible reasons, e.g., not necessarily always because of the low quality of these journals. The article summarizes various characters of the journals and attempts to pinpoint the importance of DOAJ listing. I recommend publishing this article but would like to see some revisions before its publication.
1. The author does not mention, and seems to have not read, another published article on the same subject: Andrea Marchitelli, Paola Galimberti, Andrea Bollini, and Dominic Mitchell (2017). “Improvement of editorial quality of journals indexed in DOAJ: a data analysis,” JLIS.it 8(1). DOI: 10.4403/jlis.it-12052 (It is interesting that one of the latter authors is from DOAJ). Both articles share some similar information and provide overlapped analyses.
2. The author needs to pay more attention to language. There are some grammar issues, such as “DOAJ note” on Page 3, Line 101, where DOAJ is a singular; “These” on Page 13, Line 388, where “They” may be a better word … There are also expression concerns, e.g., “hypothesis” on Page 1, Line 35, is a standard expression that may expect scientific measures – please consider a different term …
3. This following reasoning is questionable: “Among indicators of publishing competence could be the use of APCs, permanent article identifiers, journal licenses, article level metadata deposited with DOAJ, archiving policy/solutions and/or having a policy in SHERPA/RoMEO.” Page 1, Lines 38-40.
Author Response
See enclosed, which contains a general comment and replies to all reviewers.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
See attachment.
Comments for author File: 
Comments.docx
Author Response
See enclosed, which contains a general comment and replies to all reviewers.
Author Response File: Author Response.docx
Reviewer 3 Report
I have 4 comments overall:
The criteria for re-accreditation in DOAJ are not discussed or extensively compared with the previous criteria. This is an important methodological gap which needs to be filled. It detracts from the soundness of the conclusions reached.
The statement is often made in the paper that, in the re-accreditation process, issues of process rather than academic quality governed the removal of various titles. Unless the point in (1) is addressed, this statement cannot be substantiated.
In the wake of Plan S, the DOAJ is seen as a major tool for measuring compliance with Plan S's OA philosophy. What is the relationship between the re-accreditation process described in the article and Plan S?
The overall conclusion seems rather too weak. If lack of rigorous process has led to the de-accreditation of a number of journals, what is the academic impact of this? Would interviews with 1 or 2 academics in certain disciplinary fields add to the conclusions that the paper draws? It is extra work, but it would certainly strengthen the arguments the paper is putting forward.
Author Response
See enclosed, which contains a general comment and replies to all reviewers.
Author Response File: Author Response.docx
Reviewer 4 Report
Overall: I was especially interested in the focus of the article examining the size of publisher of journals removed from DOAJ. I believe this investigation to be of interest to a number of readers. I would, however, like the author to address why this investigation is important now--nearly 3 years after the completion of DOAJ's cleanup. Why does the closer look at publishers of excluded journals matter in 2019? Are there longterm repercussions?
Language: Throughout the article, the author should update the language and eliminate the use of "I" "me" and "we" to make the article sound more objective and authoritative. If the author would like to address personal experiences with DOAJ--I would like to see a new section to address all of this in one section. I would also suggest that some footnotes are eliminated, and possibly folded into the text. (Specifically: #1, #3--become reference, #4 #5 #6--become reference)
Formatting: Author should review all included tables--Tables 1-6, 8 and 10, exception 4--present related information with the same column headers. Could some of these tables be combined or eliminated to text only.
Introduction: Address why this article is being written in 2019 and what readers may hope to learn.
Materials and Methods: Address who and why groups were split into publisher size groups--and further explain use of "publishing competence" metric. Could competence be connected with an existing standard? I would suggest using COPE--or even using the standards set by DOAJ's Seal. This could help bridge 2016 and 2019.
Results: Reduce the number of tables in this section, especially under 3.1. If the results section is to address publishing competences, reorder 3.3, 3.4, 3.6, and 3.9. Group these subsection together--and move 3.5, 3.7, 3.8 into one larger analysis section (or perhaps do this in the same section as discussion of size of publisher.) Need additional explanatory text around Table 16.
Summing up and discussion: This section again needs to address why this article is written in retrospect. What has the author learned in years since the DOAJ cleanup. Additionally, the discussion focuses on a specific regional-context. If this is to remain, the geographical representation should be addressed as a focus in the introduction--and section 3.5 should be expanded (with the elimination of table 10) in order to better address how Denmark Sweden and Norway fit into the geographical representation of DOAJ.
Author Response
See enclosed, which contains a general comment and replies to all reviewers.
Author Response File: Author Response.docx
Round 2
Reviewer 2 Report
See attached report.
Comments for author File: Comments.docx
Author Response
First, a general thanks for all the work you have put into this, and valuable input both to my text and my thinking, for this and (maybe) future articles.
Now, your comments are quite extensive, and I am not sure how best to comment to your comments. But I think a general comment/rebuttal is that we obviously do not agree what should be the focus of this manuscript. That is not to say I necessarily disagree with anything you say, but that I do not feel that this manuscript is the best place for it, and that it distracts from the focus I want. (Many of your comments I would like to discuss further, but that would be quite another article.)
Specifically, I do not want to engage deeply in the history of what led to the DOAJ re-accreditation process, only enough to put it in some kind of context. Mentioning both Beall and Bohannon would be fine, and I do of course need to get what facts are included, right. For the article, it suffices that there was a re-accreditation process that – among other things – led to this large exclusion of journals. Thanks for a very comprehensive documentation!
Another aspect that I don’t want to spend too much space on, is the general discussion of whitelists and blacklists, even if a very interesting subject. Some discussion is needed, not least because this removal can be said – with some certainty – to have resulted in DOAJ being less useful as a whitelist, than before.
I have included my comments (more than the above) in the document you provided your comments in, this is attached.
Author Response File: Author Response.docx
Reviewer 3 Report
Personally I would not agree with the statement:
490 Looking back to the title of this article: We probably have no reason to lament all journals lost, but
491 most journals lost are journals we would be better off having kept.
The article draws conclusion based on administrative processes, not academic content. Here and elsewhere in the article, I would rephrase this as something like:
Looking back to the theme of this article. a significant number of journals has been lost from the DOAJ. Whilst they may not meet the criteria posed by the revision of the DOAJ, the loss is a significant one in terms of promoting OA content and services,
Author Response
It is correct that my analysis is done on the basis of purely administrative data, but that is also the only reason for the removal of the journals in question at that point in time: They had not fulfilled the requirement of re-applying.
Re-applications from 6359 journals were evaluated, 2058 journals were rejected during the re-application process. [21] This means more than 2/3 of all journals re-evaluated were accepted.
As we have no reason to believe the scholarly quality of the journals removed due to not re-applying was different from the quality of journals that re-applied, we may assume that roughly the same percentage of journals would have been re-accepted, meaning we have lost roughly 1900 bona fide OA journals (2/3 of 2859). We know nothing about whether any individual journal of the 2859 met the re-application criteria, and can only assume they were not significantly different from other DOAJ journals in this respect.
So I think one can conclude that the majority of the de-listed journals were journals that actually merit a listing in DOAJ:
Reviewer 4 Report
Introduction: Rewrite this to exclude the "our journals" view. The personal experience does not contribute to the overall research, but rather only sets the stage for how the author came up with the hypothesis. Additionally, the personal reflection is not appropriate in this section. Specifically: "A short sidetrack..." Push the "Why is it important" (final paragraph in the intro), to the next section.
There are a few sections with unsupported statements: line 77, 81, 289, 496, 502, 528
Clarify table descriptions/table names. How are tables 2 & 4 different? Unclear given table descriptions.
Mentions of Plan S: Even journals in DOAJ do not meet the (current) requirements for Plan S. Further explain how these relate to each other.
Conclusion: Conclusion is not reflective of the introduction. Paragraph beginning "The re-application process analyzed here..." and the final sentence of the paper does not seem to line up.
Author Response
The reviewers comments left aligned, my responses indented:
Introduction: Rewrite this to exclude the "our journals" view. The personal experience does not contribute to the overall research, but rather only sets the stage for how the author came up with the hypothesis. Additionally, the personal reflection is not appropriate in this section. Specifically: "A short sidetrack..." Push the "Why is it important" (final paragraph in the intro), to the next section.
A common comment from a number of reviewers was the need for an explanation of why this article comes now, and not a few years ago. I understood that to be information that should be included in the article itself, but that is not necessary for me – so I have cut down that part and rewritten the introduction.
There are a few sections with unsupported statements: line 77, 81, 289, 496, 502, 528
77 Reference to information about criteria added
81 This paragraph will be totally rewritten
289 This is specifically said to be a probability, i.e. an opinion, not a fact. Though I have experienced a lack of understanding and capacity about this in a project, where we tried to make national journals create listings in RoMEO.
496 This is implicit in their losing the free distribution of information to various databases that harvest DOAJ for information.
502 A reference added.
528 This is, in my opinion, well supported by the argument in the article itself. And the word “probably” points to this not being a certainty.
Clarify table descriptions/table names. How are tables 2 & 4 different? Unclear given table descriptions.
The Table captions say that table 2 is about loss of journals, table 4 about loss of publishers.
Mentions of Plan S: Even journals in DOAJ do not meet the (current) requirements for Plan S. Further explain how these relate to each other.
Some text about this added.
Conclusion: Conclusion is not reflective of the introduction. Paragraph beginning "The re-application process analyzed here..." and the final sentence of the paper does not seem to line up.
Re-written.
