What Proportion of Systematic Reviews and Meta-Analyses Published in the Annals of Surgery Provide Definitive Conclusions—A Systematic Review and Bibliometric Analysis
Round 1
Reviewer 1 Report
Title: "Conclusive conclusions" is very awkward. More to the point, I think systematic reviews are supposed to yield a definitive answer, if possible. They are generally attempted in a fairly mature field, where a lot of sources are available, and an effort is made to balance those sources and come up with guidance; that's the whole reason for the effort! In this context, is 75% of conclusions being conclusive a lot or a little? Given the reasoning above, it could be construed as relatively little consensus.
Reference #5 has a much better title than this paper: "Nearly One-Third of Published Systematic Reviews and 285 Meta-analyses Yield Inconclusive Conclusions: A Systematic Review". This begs a question; why is this review needed?
Lines 11-12: The statement that the authors intend “to assess the impact of the levels of evidence (LOE) on concluding statements” is a thread not tied up in the abstract, except for a non-significant p value. Given that this is stated as an objective, more needs to be said.
In general, shouldn't the level of evidence (LOE) of a meta-analysis be higher than the LOE of the component studies?
Line 15: There is an errant “and”.
Line 20: A total of “186 systematic reviews were published [in Annals of Surgery?] between 2011–2020”. Be more explicit.
Lines 25-26: “Meta-analyses were more likely to provide conclusive statements than systematic reviews (P=0.009)”. This should absolutely not be a surprise!
Line 39: The phrase “to replicate realistic, large-volume scenarios” ought to be “to synthesize realistic, large-volume approximations of clinical reality”, or something like that.
Lines 48-49: The phrase, “often leading to the authors being unable to commit to coherent and intuitive conclusive statements” seems to be a non-sequitar.
Line 52: “Epidemic proportions” needs a citation.
Lines 69-73: The text reads, “Our hypothesis was that at least 50% of included systematic reviews and meta-analyses would provide comprehensive conclusive statements and that LOE would correlate with the conclusive statements of studies, due to the high quality of research articles published in the Annals of Surgery.” I don’t understand how a meta-analysis can have a single LOE and the methods section does not clarify this. A meta may contain dozens of separate sources, each of which has a different LOE. How is a single LOE for a meta calculated?
Line 82: How are “studies deemed appropriate”? What were the criteria?
Lines 94-95: Burns et al. (reference 1) contains 4 different schema for levels of evidence; which one was used? Why do you only discuss level I and III, not II? It seems to me that most references incorporated in a meta-analysis are likely to be level II.
Of the 4 schema in Burns (1), Table 2 seems the most relevant, but this makes the point that level II seems the most likely kind of evidence.
Lines 101-110: The section of “Inclusion and exclusion criteria” should precede description of the search strategy, otherwise how the search was done remains a mystery.
Lines 131-132: “The majority of included studies were level III evidence (59.%, 111/186), with 24.2% of included studies providing level I evidence (45/186).” Do you mean that 111 of 186 studies integrated evidence only from level III studies, while 45 used only level I? What about level II?
Line 138: We need a definition of “Conclusive conclusions”.
Figure 1: This correlation seems meaningless; whether a conclusion is conclusive or not is dichotomous, so the r value for this correlation should be very high. This is not worth a figure.
Lines 154-156: I don’t know what this means.
Lines 169-170: The text states, “meta-analyses were significantly more likely to provide conclusive conclusions compared to traditional systematic reviews (P=0.016, †)”. This seems obvious to me.
Line 173: I saw no supplementary tables.
Line 185: Nonsense! There is no such thing as an “irrefutable conclusive statement”.
Lines 208-210: “Members of the academic community may be guilty of having the tendency to utilize IF as a proxy of the quality of a journal compared to competing journals in the same field”. Call me guilty; I think IF is an adequate proxy.
Lines 214-227: This seems too long and chatty.
Lines 240-244: “This is evident from the results of the current study where 80.9% (106/131) of meta-analyses (versus 61.8% of systematic reviews (34/55)) provide conclusive statements to their analyses (P=0.009, †). Thus, this study supports the use of meta-analysis methodology where feasible in order to improve the ability to provide conclusive results of surgical research questions.” I think these statements belong in the Abstract.
Lines 247-249: A statement is made that, “In this study, the authors subjectively adjudicated the conclusiveness of the published systematic reviews and meta-analyses in the journal, without applying a reliable scoring system to fairly judge ‘conclusiveness’.” Why not rectify this and figure out some way to conclude objectively how rigorous a conclusion has been made?
Line 276: There are no “Conflict of interest” statements.
References: Listing all references seems excessive. Most can be put into a supplementary file.
Author Response
Reviewer 1.
Reviewer’s Comment:
Title: "Conclusive conclusions" is very awkward. More to the point, I think systematic reviews are supposed to yield a definitive answer, if possible. They are generally attempted in a fairly mature field, where a lot of sources are available, and an effort is made to balance those sources and come up with guidance; that's the whole reason for the effort! In this context, is 75% of conclusions being conclusive a lot or a little? Given the reasoning above, it could be construed as relatively little consensus.
Author’s Response:
First of all, many thanks for providing an extensive and thorough review of our manuscript. The authors believe the manuscript has benefitted greatly from this review and are grateful for the time taken out of your busy schedule to review this work.
Thank you for this suggestion – the authors agree with the suggestion that the title is somewhat awkward. Both the title and discussion have been revised in accordance with this suggestion; the title now reads: Over 75% of Systematic Reviews and Meta-Analyses Published in the Annals of Surgery Provide Definitive Conclusions – A Systematic Review and Bibliometric Analysis.
We have also amended the discussion to suggest that greater than 75% of studies providing definitive conclusions is a good return based on the work of previous authors who have conducted similar analyses:
‘These results support the journal as one that provides strong definitive conclusions on most synthetic reviews, particularly when compared to similar, previously conducted studies (e.g.: Harris et al. reported 1 in 3 studies failed to provide definitive conclusions in their previous analysis of orthopaedic literature).’
Reviewer’s Comment:
Reference #5 has a much better title than this paper: "Nearly One-Third of Published Systematic Reviews and Meta-analyses Yield Inconclusive Conclusions: A Systematic Review". This begs a question; why is this review needed?
Author’s Response:
Thank you for this comment. Reference 5 reviewed 6 top ranking orthopaedic journals and illustrated that one third of synthetic reviews yielded inconclusive results. This differs from our study as we have just reviewed the Annals of Surgery journal only.
Reviewer’s Comment:
Lines 11-12: The statement that the authors intend “to assess the impact of the levels of evidence (LOE) on concluding statements” is a thread not tied up in the abstract, except for a non-significant p value. Given that this is stated as an objective, more needs to be said.
Author’s Response:
Thank you for this suggestion. The authors agree that this requires revision – the objective of the study has been revised to read:
‘To perform a systematic review and bibliometric analysis of systematic reviews and meta-analyses published in the Annals of Surgery during a 10-year eligibility period and determine the conclusiveness of concluding statements of these reviews published in the journal’
Reviewer’s Comment:
In general, shouldn't the level of evidence (LOE) of a meta-analysis be higher than the LOE of the component studies?
Author’s Response:
Thank you for this comment.
Although we appreciate that a meta-analysis of data compiled from randomized controlled trials is at the summit of the academic level of evidence ladder, this does not suggest that all meta-analyses will be of higher LOE than systemic reviews. A systematic review (with or without meta-analysis) can be performed on all LOEs (meta-analysis methodology is not only reserved for randomized controlled trials). Therefore, it is possible that there may be studies which are systemic reviews (without meta-analyses) performed on data of a higher LOE than some of the included meta-analyses.
Reviewer’s Comment:
Line 15: There is an errant “and”.
Author’s Response:
Thank you for highlighting this error. We have amended the text accordingly.
Reviewer’s Comment:
Line 20: A total of “186 systematic reviews were published [in Annals of Surgery?] between 2011–2020”. Be more explicit.
Author’s Response:
Thank you for this recommendation. We have amended the text to provide more clarity. Thank you.
‘In total, 186 systematic reviews (with or without meta-analyses) were published in the Annals of Surgery between 2011–2020 (131 systematic reviews with meta-analyses (70.4%) and 55 without meta-analyses (29.6%)).’
Reviewer’s Comment:
Lines 25-26: “Meta-analyses were more likely to provide conclusive statements than systematic reviews (P=0.009)”. This should absolutely not be a surprise!
Author’s Response:
As described above, this isn’t a surprise but it is a result worth reporting as a systematic review (with or without meta-analysis) may be performed on all LOEs (meta-analysis methodology is not only reserved for randomized controlled trials). We hope this makes sense and to reiterate, we agree this isn’t a huge surprise although it is a result worth reporting.
Reviewer’s Comment:
Line 39: The phrase “to replicate realistic, large-volume scenarios” ought to be “to synthesize realistic, large-volume approximations of clinical reality”, or something like that.
Author’s Response:
Thank you for this recommendation. The text has been amended in accordance with this suggestion.
Reviewer’s Comment:
Lines 48-49: The phrase, “often leading to the authors being unable to commit to coherent and intuitive conclusive statements” seems to be a non-sequitar.
Author’s Response:
Thank you for this recommendation. The text has been revised to ensure the text flows logically:
‘However, higher risks of bias, lower methodological quality, and heterogeneous results impact the validity and the meaningfulness of the conclusions which may be drawn. Despite this, the publication of such studies continues to increase, which often leads to the authors of such studies being unable to report definitive results of their synthetic review.’
Reviewer’s Comment:
Line 52: “Epidemic proportions” needs a citation.
Author’s Response:
Thank you for this suggestion – the 4th reference from Ioannidis et al. is being quoted in this reference to provides “Epidemic proportions” and thus has been cited directly behind this statement to provide more clarity.
Reviewer’s Comment:
Lines 69-73: The text reads, “Our hypothesis was that at least 50% of included systematic reviews and meta-analyses would provide comprehensive conclusive statements and that LOE would correlate with the conclusive statements of studies, due to the high quality of research articles published in the Annals of Surgery.” I don’t understand how a meta-analysis can have a single LOE and the methods section does not clarify this. A meta may contain dozens of separate sources, each of which has a different LOE. How is a single LOE for a meta calculated?
Author’s Response:
Thank you for this comment. When integrating data of several LOE into meta-analysis, the lowest available LOE included in the study is then used to represent the LOE of the meta-analysis due to the review being subject to the biases associated with such studies. This is common in several academic journals and has been clarified in the methodology section to ensure brevity to the reader:
‘For synthetic reviews included which included studies of varying LOE, the study with the lowest included LOE was used to represent the LOE of the synthetic review.’
Reviewer’s Comment:
Line 82: How are “studies deemed appropriate”? What were the criteria?
Author’s Response:
Thank you for this suggestion. All systemic reviews (with or without meta-analysis) published in the journal during the eligibility period (2011 – 2020) were included.
We have clarified this in the text:
‘All titles published in the journal were initially screened, and all systemic reviews (with or without meta-analysis) published in the journal were included and had their abstracts and full texts reviewed.’
Reviewer’s Comment:
Lines 94-95: Burns et al. (reference 1) contains 4 different schema for levels of evidence; which one was used?
Author’s Response:
Thank you for highlighting this confusing area of the paper – we used the LOE for prognostic studies as outlined by Nguyen et al. from the American Society of Plastic Surgeons (Table 3 from the Burns et al. paper). We have amended the text to reflect this and referenced this in the text also. Thank you.
‘The hierarchical level of evidence-based medicine (LOE) were considered in accordance to the previous work of Nguyen et al. [7]. In brief, level I evidence consisted of high-quality RCTs which were adequately powered and the systematic reviews of such studies. Level II studies consisted of lesser quality RCTs and predominantly consisted of prospective cohort studies, and systematic reviews of those studies. Level III studies consisted of retrospective comparative studies. Level IV studies were typically of the case-series variety, and level V articles were usually case reports or expert opinions.’
Reviewer’s Comment:
Why do you only discuss level I and III, not II? It seems to me that most references incorporated in a meta-analysis are likely to be level II.
Author’s Response:
Thank you for this comment. As outlined in the text, there was large proportion of studies that were systemic reviews (with or without meta-analysis) of RCTs or else large systemic reviews (with or without meta-analysis) comprised of retrospective data. In accordance with the work of Nguyen et al., these are level I and level III evidence respectively.
Reviewer’s Comment:
Of the 4 schema in Burns (1), Table 2 seems the most relevant, but this makes the point that level II seems the most likely kind of evidence.
Author’s Response:
As outlined above, the schema from Table 3 aligns with the work of Nguyen et al.’s description of the LOE most relevant for this study. Thank you.
Reviewer’s Comment:
Lines 101-110: The section of “Inclusion and exclusion criteria” should precede description of the search strategy, otherwise how the search was done remains a mystery.
Author’s Response:
Thank you for highlighting this area of confusion for the manuscript. This has been edited in the text in accordance with the reviewers suggestion – thank you once again for highlighting this area of confusion.
Reviewer’s Comment:
Lines 131-132: “The majority of included studies were level III evidence (59.%, 111/186), with 24.2% of included studies providing level I evidence (45/186).” Do you mean that 111 of 186 studies integrated evidence only from level III studies, while 45 used only level I? What about level II?
Author’s Response:
Thank you for this comment. This has been clarified in the methodology as previously outlined.
When integrating data of several LOE into systemic reviews and meta-analysis, most journals will accept that the lowest available LOE included in the review is to one selected to represent the LOE of the synthetic review. Therefore, this was applied to the studies included in this study (e.g.: if RCTs and both prospective and retrospective studies were included in a study, the synthetic review in question is considered level III evidence).
This has been clarified in the methodology section.
Reviewer’s Comment:
Line 138: We need a definition of “Conclusive conclusions”.
Author’s Response:
Thank you for thid suggestion, and one that is acknowledged as s shortcoming of the current study. As outlined in our limitations section, the authors subjectively reviewed each text as determined whether there was a conclusive or inconclusive takeaway message from the study. In order for a study to be classed as conclusive, there had to be unanimous agreement among all authors as to the conclusivity of the study.
‘The current systematic review of systematic reviews and meta-analyses published Annals of Surgery suffers from several limitations: In this study, the authors subjectively adjudicated the conclusiveness of the published systematic reviews and meta-analyses in the journal, without applying a reliable scoring system to fairly judge ‘conclusiveness’.
We acknowledge that this doesn’t provide a definition for conclusiveness.
Reviewer’s Comment:
Figure 1: This correlation seems meaningless; whether a conclusion is conclusive or not is dichotomous, so the r value for this correlation should be very high. This is not worth a figure.
Author’s Response:
Thank you for this suggestion. We have removed this figure from the study under your recommendation.
Reviewer’s Comment:
Lines 154-156: I don’t know what this means.
Author’s Response:
Thank you for highlighting this issue with our manuscript. We have amended the text to ensure clarity. We hope this helps.
‘We performed a subgroup analysis based on region of publication of the synthetic reviews included in this study. When evaluating each region independently, LOE failed to significantly impact the conclusiveness of studies, irrespective of region (all P>0.050).’
Reviewer’s Comment:
Lines 169-170: The text states, “meta-analyses were significantly more likely to provide conclusive conclusions compared to traditional systematic reviews (P=0.016, †)”. This seems obvious to me.
Author’s Response:
Thank you for this comment. As previously outlined, this may be obvious to the reader however it is a result worth presenting in this manuscript. Thank you once again.
Reviewer’s Comment:
Line 173: I saw no supplementary tables.
Author’s Response:
Thank you for highlighting this issue – out supplementary materials have all been added to the publications submission portal. We will for these to be disseminated with you during re-review. Thank you.
Reviewer’s Comment:
Line 185: Nonsense! There is no such thing as an “irrefutable conclusive statement”.
Author’s Response:
Thank you for this comment - this has been amended in the text:
‘This result highlights the value of synthetic reviews published in the Annals of Surgery and supports the authors’ null hypothesis suggesting that over 50% of such studies published in the journal would provide indecisive conclusive statements.’
Reviewer’s Comment:
Lines 208-210: “Members of the academic community may be guilty of having the tendency to utilize IF as a proxy of the quality of a journal compared to competing journals in the same field”. Call me guilty; I think IF is an adequate proxy.
Author’s Response:
Thank you for this comment – we have not amended the text.
Reviewer’s Comment:
Lines 214-227: This seems too long and chatty.
Author’s Response:
Thank you for highlighting this issue. We have shortened the text to improve the readability. Thank you.
Reviewer’s Comment:
Lines 240-244: “This is evident from the results of the current study where 80.9% (106/131) of meta-analyses (versus 61.8% of systematic reviews (34/55)) provide conclusive statements to their analyses (P=0.009, †). Thus, this study supports the use of meta-analysis methodology where feasible in order to improve the ability to provide conclusive results of surgical research questions.” I think these statements belong in the Abstract.
Author’s Response:
Thank you for highlighting this – we have amended the abstract and main text accordingly:
‘Of note, 80.9% (106/131) of meta-analyses and 61.8% of systematic reviews (34/55)) provided conclusive statements (P=0.009, †).’
Reviewer’s Comment:
Lines 247-249: A statement is made that, “In this study, the authors subjectively adjudicated the conclusiveness of the published systematic reviews and meta-analyses in the journal, without applying a reliable scoring system to fairly judge ‘conclusiveness’.” Why not rectify this and figure out some way to conclude objectively how rigorous a conclusion has been made?
Author’s Response:
Thank you for this suggestion – we have explored several different methodologies including minimal clinical importance difference, statistical conclusion validity, to try to establish how to best quantify conclusiveness. These methods all require the raw data to be available which we do not have. Unfortunately, we are therefore unable to provide a more robust conclusiveness calculation. This explanation is added to the limitations section:
‘The current systematic review of systematic reviews and meta-analyses published Annals of Surgery suffers from several limitations: In this study, the authors subjectively adjudicated the conclusiveness of the published systematic reviews and meta-analyses in the journal, without applying a reliable scoring system to fairly judge ‘conclusiveness’. This is due to the requirement of several methodologies used to score ‘conclusiveness’ requiring the analyst to have the raw data from the study. This is an obvious shortcoming of this study.’
Reviewer’s Comment:
Line 276: There are no “Conflict of interest” statements.
Author’s Response:
This has been amended – thank you.
Reviewer’s Comment:
References: Listing all references seems excessive. Most can be put into a supplementary file.
Author’s Response:
This has been amended.
Thank for providing a thorough review of our manuscript – the study has benefitted considerably from your review, thank you.
Reviewer 2 Report
The authors have performed a systematic review on SR and MA published in a single journal (Annals of Surgery) on the association between level of evidence and conclusiveness of studies.
Was the review protocol registered a priori at e.g. Prospero? If so, please provide the registration number. If not, please state this explicitly in the manuscript together with the reason why this was not done.
How was “conclusiveness of studies” defined?
Was the minimally clinically important difference (MCID) used for the classification of conclusive / inconclusive?
What was the interrater agreement of “conclusiveness of studies” between the reviewers?
The statistical methods described in the methods section are to be used for studies with individual patient data (e.g. cohorts and RCTs). Since the present SR uses study level data appropriate meta-analytic techniques are required such as evidence synthesis and meta-regression.
Author Response
Reviewer 2.
Reviewer’s Comment:
The authors have performed a systematic review on SR and MA published in a single journal (Annals of Surgery) on the association between level of evidence and conclusiveness of studies.
Author’s Response:
Thank you for taking the time out of busy schedule too review our manuscript – it is appreciated by all the authors.
Reviewer’s Comment:
Was the review protocol registered a priori at e.g. Prospero? If so, please provide the registration number. If not, please state this explicitly in the manuscript together with the reason why this was not done.
Author’s Response:
Thank you for this suggestion. We attempted to register this study with PROSPERO however this study was deemed ineligible for registration as it does not have a direct link with human health. We have amended the methods to reflect this:
‘This review was not prospectively registered with the International Prospective Register of Systematic Reviews (PROSPERO) as the results of this review does not have a direct link to human health.’
Reviewer’s Comment:
How was “conclusiveness of studies” defined?
Was the minimally clinically important difference (MCID) used for the classification of conclusive / inconclusive?
Author’s Response:
Thank you for highlighting this shortcoming of the paper – we have explored several different methodologies including minimal clinical importance difference, statistical conclusion validity, to try to establish how to best quantify conclusiveness. These methods all require the raw data to be available which we do not have. Unfortunately, we are therefore unable to provide a more robust conclusiveness calculation. This explanation is added to the limitations section.
‘The current systematic review of systematic reviews and meta-analyses published Annals of Surgery suffers from several limitations: In this study, the authors subjectively adjudicated the conclusiveness of the published systematic reviews and meta-analyses in the journal, without applying a reliable scoring system to fairly judge ‘conclusiveness’. This is due to the requirement of several methodologies used to score ‘conclusiveness’ requiring the analyst to have the raw data from the study. This is an obvious shortcoming of this study.’
Reviewer’s Comment:
What was the interrater agreement of “conclusiveness of studies” between the reviewers?
Author’s Response:
This is an interesting question – of the included 186 studies, there was interrater agreement in 181/186 (97.3%). For those 5 studies, a third independent reviewer was asked to arbitrate. The rationale for omitting data on interrater agreement from the current study is that it was not one of the primary objectives of the study and it is somewhat irrelevant once the third reviewer arbitrated over any discrepancy in opinion.
Reviewer’s Comment:
The statistical methods described in the methods section are to be used for studies with individual patient data (e.g. cohorts and RCTs). Since the present SR uses study level data appropriate meta-analytic techniques are required such as evidence synthesis and meta-regression.
Author’s Response:
Statistical analyses such as independent samples t-test were utilised to comparing means (i.e.: number of citations, impact factor, etc). Furthermore, when comparing frequencies Chi Squared and Fishers Exact test were used as appropriate as described by Kim et al.
We appreciate these tests are used in clinical studies including patient data, however their application to this setting is appropriate also.
Round 2
Reviewer 1 Report
Title: I faulted the title as being awkward and improvement has been made here. However, problems remain. The current title is, “Over 75% of Systematic Reviews and Meta-Analyses Published in the Annals of Surgery Provide Definitive Conclusions”. A reader may—depending on bias—be surprised by how low or how high the 75% figure is. I was originally surprised by how low it is, because systematic reviews are generally attempted in a mature field, where a lot of sources are available, and the whole point of a systematic review is to provide reliable guidance.
The other issue is that all but the most naïve reader must wonder how “definitive” is defined. And it isn’t....
Perhaps try a new title entirely, posed as a question; “What Proportion of Systematic Reviews and Meta-Analyses in the Annals of Surgery Resolve a Clinical Equipoise?”. This highlights the fact that some meta-analyses and reviews may not even address a clinical equipoise.
Abstract: Much improved. But “the conclusiveness of concluding statements” remains a very awkward phrase. Perhaps the easiest change to make would be to do a global search-and-replace of “conclusive” with “unambiguous”, so the quoted phrase would become “the unambiguity of concluding statements”. Later (line 60), the phrase “inconclusive conclusions” would become “ambiguous conclusions”.
General comment: I asked, “Shouldn't the level of evidence (LOE) of a meta-analysis be higher than the LOE of the component studies?” The authors responded that, “this does not suggest that all meta-analyses will be of higher LOE than systemic reviews”. This does not address the concern, as I asked in relation to the component studies. This seems to me to be an important issue that is not covered in the Discussion and probably should be covered.
Line 27-28: The authors state that, “Of note, 80.9% (106/131) of meta-analyses and 61.8% of systematic reviews (34/55)) provided conclusive statements (P=0.009, †).” The difference tested here is between meta-analyses and systematic reviews? Please clarify why this is relevant.
Line 72-76: The text originally read, “Our hypothesis was that at least 50% of included systematic reviews and meta-analyses would provide comprehensive conclusive statements and that LOE would correlate with the conclusive statements of studies, due to the high quality of research articles published in the Annals of Surgery.” I did not understand how a meta-analysis could have a single LOE given that a meta may contain dozens of separate sources, each of which has a different LOE. The authors have now clarified how a single LOE is calculated for a meta.
However, I question whether this makes sense; I am unfamiliar with the idea that, “When integrating data of several LOE[s] into a meta-analysis, the lowest available LOE included in the study is then used to represent the LOE of the meta-analysis, due to the review being subject to the biases associated with such studies.” Conceptually, this is saying that there is no strength in numbers, that an entire review can be contaminated by a single bad study.
Perhaps the best way to deal with this is to do a sensitivity analysis. By dropping out those studies with the lowest LOE, do the conclusions change as the LOE increases? This would be an interesting thing to do here, and a paragraph in the Discussion would be appropriate.
Line 89: The phrase should be “as the results of this review do not”.
Line 108-109: I noted that we need a definition of “Conclusive conclusions” and the definition has been hinted at here. However, I think that “conclusive” should be defined in the next section (“Definitions”), just a few lines later, in a sentence of its own.
Line 117-119: Sentence fragment here.
Line 161: Figure 1 is called out here, but there is no longer a Figure 1.
Line 168-169: “Of note, the LOE trended towards significance for studies published from Australia and New Zealand (P=0.072, χ2).” This is a repeated measures problem; I would delete the sentence. Being a little bit significant is like being a little bit pregnant; it can’t happen.
Line 196-200: Harris et al. found that “nearly one-third of published systematic reviews and meta-analysis provided inconclusive conclusions”. You found that 25% of published systematic reviews and meta-analysis provided inconclusive conclusions. This is not substantially different.
Line 220-235: I would delete this paragraph. I don’t think it’s paradoxical that “both LOE and the two-year IF failed to correlate with the conclusiveness of studies published in the Annals of Surgery. IF likely correlates with how novel a study is and some meta-analyses are not novel at all.
This same paragraph is also rendered problematic by the claim that, “Members of the academic community may be guilty of having the tendency to utilize IF as a proxy of the quality of a journal compared to competing journals in the same field”. As I noted before, I am guilty of this, but I don’t think it a crime.
Lines 224-225: The statement that the “Annals of Surgery [is] considered at the summit of surgical journals internationally” is not true, I believe. The impact factor of Annals of Surgery is 12.97, while the impact factor of JAMA Surgery is 14.77.
Line 260-262: I don’t understand your second point.
Lines 289-741: I would put most of these references in supplementary material and cite here only those references cited in the body of the paper.
Author Response
Reviewer 1
Reviewer’s Comment:
Title: I faulted the title as being awkward and improvement has been made here. However, problems remain. The current title is, “Over 75% of Systematic Reviews and Meta-Analyses Published in the Annals of Surgery Provide Definitive Conclusions”. A reader may—depending on bias—be surprised by how low or how high the 75% figure is. I was originally surprised by how low it is, because systematic reviews are generally attempted in a mature field, where a lot of sources are available, and the whole point of a systematic review is to provide reliable guidance.
The other issue is that all but the most naïve reader must wonder how “definitive” is defined. And it isn’t....
Perhaps try a new title entirely, posed as a question; “What Proportion of Systematic Reviews and Meta-Analyses in the Annals of Surgery Resolve a Clinical Equipoise?”. This highlights the fact that some meta-analyses and reviews may not even address a clinical equipoise.
Author’s Response:
Thank you for taking the time to review our manuscript.
This is a reasonable request by the reviewer as the title is somewhat challenging for the reader to interpret. We have taken this recommendation on board, however we believe adding ‘resolve a clinical equipoise’ to the title of a systemic review is almost as awkward as writing ‘provide definitive conclusions’. In hope of making the title concise and coherent, we have attempted to take the advice of the reviewer on board and have revised the title of the manuscript to read: What Proportion of Systematic Reviews and Meta-Analyses in the Annals of Surgery Provide Definitive Conclusions – A Systematic Review and Bibliometric Analysis.
Although not perfect, we hope this is satisfactory. Thank you for this suggestion.
Reviewer’s Comment:
Abstract: Much improved. But “the conclusiveness of concluding statements” remains a very awkward phrase. Perhaps the easiest change to make would be to do a global search-and-replace of “conclusive” with “unambiguous”, so the quoted phrase would become “the unambiguity of concluding statements”. Later (line 60), the phrase “inconclusive conclusions” would become “ambiguous conclusions”.
Author’s Response:
Thank you for this suggestion.
We agree with the reviewer and have made this revision.
Thank you.
Reviewer’s Comment:
General comment: I asked, “Shouldn't the level of evidence (LOE) of a meta-analysis be higher than the LOE of the component studies?” The authors responded that, “this does not suggest that all meta-analyses will be of higher LOE than systemic reviews”. This does not address the concern, as I asked in relation to the component studies. This seems to me to be an important issue that is not covered in the Discussion and probably should be covered.
Author’s Response:
Thank you for this comment. The relevance of this comment is somewhat lost in the clinical setting – we performed this subgroup analysis to determine whether the conclusiveness of systematic review without meta-analysis differed from versus systematic review with meta-analysis. Our rationale for performing such an analysis is to assess whether the integration of data using met-analysis methodology has the potential to improve the ‘conclusiveness’ of the studies, and interestingly, we found it does improve ‘conclusiveness’.
Therefore, although performing an analysis of component studies is a possible, we are uncertain what clinical benefit it has in the context of this study. Therefore, we do not believe it will shed any additional light on the current research question.
Thank you for this suggestion.
Reviewer’s Comment:
Line 27-28: The authors state that, “Of note, 80.9% (106/131) of meta-analyses and 61.8% of systematic reviews (34/55)) provided conclusive statements (P=0.009, †).” The difference tested here is between meta-analyses and systematic reviews? Please clarify why this is relevant.
Author’s Response:
Thank you for this comment.
Yes, this represents the comparison of the proportion of systematic review without meta-analysis and systematic review with meta-analysis which had conclusive conclusions.
This is extremely relevant to clinicians as meta-analyses integrate data from several studies to provide an overarching result of the combined data. Therefore, it is of relevance to assess the impact of these study types (i.e.: studies which have integrated results in meta-analysis vs. those who simply present results in the form of a systematic review (without meta-analysis) on the conclusiveness of the authors statements when they publish their study. This is an interesting finding, which carries clinical relevance as a consequence as future studies may ensure they use meta-analysis methodology in order to ensure they maximise interpretation of their results to improve patient outcomes.
Thank you.
Reviewer’s Comment:
Line 72-76: The text originally read, “Our hypothesis was that at least 50% of included systematic reviews and meta-analyses would provide comprehensive conclusive statements and that LOE would correlate with the conclusive statements of studies, due to the high quality of research articles published in the Annals of Surgery.” I did not understand how a meta-analysis could have a single LOE given that a meta may contain dozens of separate sources, each of which has a different LOE. The authors have now clarified how a single LOE is calculated for a meta.
Author’s Response:
Thank you.
Reviewer’s Comment:
However, I question whether this makes sense; I am unfamiliar with the idea that, “When integrating data of several LOE[s] into a meta-analysis, the lowest available LOE included in the study is then used to represent the LOE of the meta-analysis, due to the review being subject to the biases associated with such studies.” Conceptually, this is saying that there is no strength in numbers, that an entire review can be contaminated by a single bad study.
Author’s Response:
Thank you for this comment.
This is widely accepted by medical journals when addressed the LOE of a systemic review of studies of varying LOE. Including more studies of lower LOE does not strengthen the evidence, which is what we assessed when evaluating whether LOE impacts the conclusiveness of the published studies.
Please see this link to the John Hopkins Nursing Evidence Based Practice webpage for more information:
https://libguides.ohsu.edu/ebptoolkit/levelsofevidence
We hope this is satisfactory.
Reviewer’s Comment:
Perhaps the best way to deal with this is to do a sensitivity analysis. By dropping out those studies with the lowest LOE, do the conclusions change as the LOE increases? This would be an interesting thing to do here, and a paragraph in the Discussion would be appropriate.
Author’s Response:
Thank you for this interesting suggestion. Although this could potentially shed light on the question asked of the reviewer, it unfortunately was not was not the purpose of this analysis in this study – we wished to simply assess whether the LOE of synthetic reviews published in the journal impacted on ‘conclusiveness’. Therefore, performing a sensitivity analysis of the included studies is something that may be considered in a similar, novel study. Thank you.
Reviewer’s Comment:
Line 89: The phrase should be “as the results of this review do not”.
Author’s Response:
Thank you for highlighting this issue.
We have revised the text accordingly.
Reviewer’s Comment:
Line 108-109: I noted that we need a definition of “Conclusive conclusions” and the definition has been hinted at here. However, I think that “conclusive” should be defined in the next section (“Definitions”), just a few lines later, in a sentence of its own.
Author’s Response:
Thank you for this suggestion.
We have added the definition of conclusive conclusions to the definitions section as suggested:
‘Conclusive conclusions were concluding statements to a synthetic review which provided a clear, concise and informative message based on the results of the synthetic review as adjudicated by the independent reviewers. Studies reporting the requirement for ‘further’ investigation or research were considered to be inconclusive.’
Reviewer’s Comment:
Line 117-119: Sentence fragment here.
Author’s Response:
Thank you for this suggestion. We have made the definitions section bullet points to increase readability.
Reviewer’s Comment:
Line 161: Figure 1 is called out here, but there is no longer a Figure 1.
Author’s Response:
Thank you for highlighting this error. We have amended the text accordingly.
Reviewer’s Comment:
Line 168-169: “Of note, the LOE trended towards significance for studies published from Australia and New Zealand (P=0.072, χ2).” This is a repeated measures problem; I would delete the sentence. Being a little bit significant is like being a little bit pregnant; it can’t happen.
Author’s Response:
Thank you for this suggestion. We have amended the text accordingly.
Reviewer’s Comment:
Line 196-200: Harris et al. found that “nearly one-third of published systematic reviews and meta-analysis provided inconclusive conclusions”. You found that 25% of published systematic reviews and meta-analysis provided inconclusive conclusions. This is not substantially different.
Author’s Response:
Thank you for highlighting this.
The authors agree with the reviewer in that it was not substantially different to that of Harris et al. despite the impact factor of the Annals of Surgery being twice that of the Orthopaedic journals included in their study.
We have not made any revisions to the text based on the comment.
Reviewer’s Comment:
Line 220-235: I would delete this paragraph. I don’t think it’s paradoxical that “both LOE and the two-year IF failed to correlate with the conclusiveness of studies published in the Annals of Surgery. IF likely correlates with how novel a study is and some meta-analyses are not novel at all.
Author’s Response:
Thank you for this suggestion.
We have revised the text such that this sentence is not reported in an unexpected manner:
‘Of note, both LOE and the two-year IF failed to correlate with the conclusiveness of studies published in the Annals of Surgery in this systematic review.’
Reviewer’s Comment:
This same paragraph is also rendered problematic by the claim that, “Members of the academic community may be guilty of having the tendency to utilize IF as a proxy of the quality of a journal compared to competing journals in the same field”. As I noted before, I am guilty of this, but I don’t think it a crime.
Author’s Response:
This is a fair suggestion. We have revised the text accordingly:
‘Members of the academic community have the tendency to rely on IF as a proxy of the quality of a journal compared to competing journals in the same field’
Reviewer’s Comment:
Lines 224-225: The statement that the “Annals of Surgery [is] considered at the summit of surgical journals internationally” is not true, I believe. The impact factor of Annals of Surgery is 12.97, while the impact factor of JAMA Surgery is 14.77.
Author’s Response:
Thank you for this suggestion. We have revised the text as per the reviewer’s suggestion:
‘…with the Annals of Surgery being considered among those at the summit of surgical journals internationally.’
Reviewer’s Comment:
Line 260-262: I don’t understand your second point.
Author’s Response:
This point has been removed from the text in order to improve the readability of the limitations section:
‘Firstly, and most importantly, the authors of this study subjectively adjudicated the conclusiveness of the published systematic reviews and meta-analyses in the journal, without applying a reliable scoring system to fairly judge ‘conclusiveness’. This is due to the requirement of several methodologies used to score ‘conclusiveness’ requiring the analyst to have the raw data from the study. This is an obvious shortcoming of this study. Secondly, although this analysis was performed by two independent reviewers, the study design makes presumptions as to the reliability of this assessment, with no formal appraisal of intra- and inter-observer agreement. Thirdly, the failure for LOE and IF to influence the conclusiveness of studies published may be considered a blunt instrument when determining the actual clinical impact these studies may have on influencing or challenging current practice in the field of surgery. Finally, this study fails to evaluate the overall methodology or the risk of bias / quality assessments of the 186 included studies, which limits the conclusions which may successfully be drawn from the data presented in the current study.’
Reviewer’s Comment:
Lines 289-741: I would put most of these references in supplementary material and cite here only those references cited in the body of the paper.
Author’s Response:
Thank you for this suggestion. We have revised the text in accordance with this suggestion.
Thank you once again for your review of our manuscript
Author Response File: 
Author Response.docx
Reviewer 2 Report
The authors have sufficiently addressed my comments
Author Response
Reviewer 2
Reviewers Comment:
The authors have sufficiently addressed my comments
Authors Response.
Thank you.
Author Response File: Author Response.docx
