Application of Association Rules Analysis in Mining Adverse Drug Reaction Signals
Round 1
Reviewer 1 Report
Review
In this manuscript the authors used a data mining method, association rules analysis (ARA) in order to improve the accuracy in detection of adverse drug reactions signals. This might be an interesting method and the authors imply that it is superior to traditional methods such as the MHRA method, which is used in countries such as the UK.
In general the research might be interesting to the reader involved in pharmacovigilance. However the structure of the manuscript is confusing, which may hamper a just interpretation and understanding of the presented data.
In general: the manuscripts contain some sentences without a verb, i.e. p2, l 82.
Abstract: no remarks, best part of the manuscript
Introduction: far too extensive. Parts concerning ARA could be moved to the discussion. The hypothesis and aim of the study should be stated more clearly.
Methods: MHRA and ARA method is clearly described
Performance evaluation: The concept of Precision, recall and F1 are not new in ARA field and can be found in the literature. Refs should be included.
Table 2: abbreviations should be explained.
Results: first part (l202-208) should be moved to the methods section, just as more lines explaining the method of research.
Text about the indication of levofloxacin and the severity of the adverse drug events could be replaced or left out the manuscript. This information is not important for the understanding of this new method. These examples could be presented more comprehensive which improves the reading of the manuscript.
Discussion: the findings of the presented research should be discussed with regards to the available literature. To my opinion the discussion would be improved if the structure would be made more clear.
Author Response
Please see the attachment.
Author Response File: 
Author Response.docx
Reviewer 2 Report
I found the paper interesting and relevant. I had one concern. I was not able to verify that the percentage improvement in performance in Table 5 was statistically significant. Perhaps I just missed it, but, even so, the authors should comment on how they know that the results are an improvement.
Author Response
Please see the attachment.
Author Response File: Author Response.docx
Reviewer 3 Report
Content
----------
The goal of this paper is to use association rules analysis (ARA) to mine signals, improve the performance of signal detection, and provide more reliable decision support for drug safety. The main contribution of this research is to use some important metrics of the ARA, such as Support, Confidence and Lift; three performance evaluation indicators, such as Precision, Recall and F1; the traditional MHRA method. The experimental results showed that the signal detection performance was greatly improved comparing with the traditional MHRA method. To reduce drug risk and provide decision-making for drug safety, more data mining methods need to be introduced and applied to ADR signal detection.
Major comments
--------------
1. writing style
The author does not follow the conventional format of article. For example, the abstract is lengthy. The author use some keyword "Background:", "Methods:" and "Conclusions" to label the sentences. The author also use many numbers such as "40.51%" to justify the significance of the result. The current writing style makes this paper is difficult to follow.
Please use concise sentences and words to carefully rewrite the abstract. The reader want to know what you have done, what is the innovation and impact in this research.
2. There is no "literature review" section.
3. The "Conclusion" section is very simple.
4. There is no figures to support the opinion.
Evaluation
--------------
Given the above, I'm in a position to reject.
Author Response
Please see the attachment.
Author Response File: Author Response.docx
Reviewer 4 Report
It's always intertesting when new methodology is introduced to any field. Data processing and ADR detection is ceratinly an important issue.
I have one major remark for this paper:
1. What data were used for decision on the optimal Lift and Confidence values (Table 4)? Was it the same dataset like the one for ARA perfomance comparison (Table 5)?
If yes then please comment on the way you would approach this task for another data?
If no, then please describe in details how the data was chosen for the search of the optimal Lift and Confidence values? Was it random, stratified or balanced selection maybe? What was the size of this dataset?
This part is crucial for the final judgement of ARA validity on the priniciple of separation cause from effect, namely optimal governing parameters of ARA method and its performance.
Author Response
Please see the attachment.
Author Response File: Author Response.docx
Round 2
Reviewer 1 Report
no remarks anymore
Author Response
Thank you for your affirmation. We have carefully revised the grammar of the full text to ensure that the description is clearer and more accurate.
Reviewer 3 Report
The author revised as my comments.
Author Response
Thank you for your approval. We have checked the grammar of the full text to ensure that the description is clearer and more accurate.
Reviewer 4 Report
Given the response Authors provided I see two possible ways to save this work:
- Enhance your computational part with proper validation of your thresholds for ARA parameters by performing k-fold cross-validation, i.e. k=10. Only then you will have right to claim that your precision, recall and F1 are superior to the method with a priori set thresholds
- Re-write the paper with emphasis on the fact that to demonstrate best achievable ARA model. In this case you simply cannot compare you results with MHRA method directly since your adaptive threshold was already adapted to the data you used for performance validation
Of these two above mentioned ways no. 1 is prefered. I leave it to the Authors which one they will choose
Author Response
Please see the attachment.
Author Response File: Author Response.docx
Round 3
Reviewer 4 Report
Dear Authors. I believe I wasn't specific enough with my last review so please let me rephrase:
- Please perform 10-fold cross-validation, where you will establish individual thresholds for your parameters based on the TRAINING dataset and then perform external validation on the remaining dataset (TEST). After repeating this procedure 10 times you will be able to calculate overall F1 for your external predictions and only then you may compare such result with classical method
- The final thresholds optimized by you on the whole dataset are simple demonstration of best case scenario and could be left in the paper but then MUST be labeled as such - a demonstration of best capability of ARA but not the final result - the latter will be based on the point no. 1, namely 10-fold cv
I hope that this time it is clear and will not require another round of review as I find this paper very interesting yet it has to follow strict modeling methodology, which sadly you don't follow (yet).
Author Response
Please see the attachment.
Author Response File: Author Response.docx
