Review of Time Domain Electronic Medical Record Taxonomies in the Application of Machine Learning
, Imran Khan Niazi 1,2,3, Brian K. Russell 1,4, Catherine Crofts 1, Samaneh Madanian 1 and David White 1,*
Round 1
Reviewer 1 Report
The paper proposes a review that will focus on time series data needed to construct a complete EMR by identifying paradigms that fall within the scope of application of artificial intelligence (AI) based on the principles of translational medicine. Scopus, Web of Science, and PubMed were searched for relevant records. The authors identified 1020 records from research databases that were filtered based on a PRISMA review process and for review were selected 160 studies.
The paper is well organized, and the length is appropriate. The title is chosen correctly and the abstract provides sufficient information to give a clear idea of what to expect from the paper.
The study methods are appropriate, and the data are valid.
The results are well highlighted, and the conclusions are adequate.
The references are relevant and correctly chosen, and related work is discussed and cited appropriately.
The technical depth of the paper meets the requirements for a scientific article published in a quality journal.
Author Response
Response to Reviewer #1
Reviewer’s Comments in overall submission.
Reviewer #1, Comment #1: The paper proposes a review that will focus on time series data needed to construct a complete EMR by identifying paradigms that fall within the scope of application of artificial intelligence (AI) based on the principles of translational medicine. Scopus, Web of Science, and PubMed were searched for relevant records. The authors identified 1020 records from research databases that were filtered based on a PRISMA review process and for review were selected 160 studies.
The paper is well organized, and the length is appropriate. The title is chosen correctly and the abstract provides sufficient information to give a clear idea of what to expect from the paper.
Author response: The authors thank you for your comments and positive feedback.
Author action: No action taken.
Reviewer’s Comments about study methods.
Reviewer#1, Comment# 2: The study methods are appropriate, and the data are valid.
Author response: The authors thank you for your comments and positive feedback.
Author action: No action taken.
Reviewer’s Comments in Results.
Reviewer #1, Comment #3: The results are well highlighted, and the conclusions are adequate.
Author response: The authors thank you for your comments and positive feedback.
Author action: No action taken.
Reviewer’s Comments in References.
Reviewer #1, Comment #4: The results are well highlighted, and the conclusions are adequate.
Author response: The authors thank you for your comments and positive feedback.
Author action: No action taken.
Reviewer’s Comments about technical depth and quality.
Reviewer #1, Comment #5: The technical depth of the paper meets the requirements for a scientific article published in a quality journal.
Author response: The authors thank you for your comments and positive feedback.
Author action: No action taken.
Reviewer 2 Report
This is a typical interdisciplinary study in which the authors integrate multiple research approaches to provide a highly innovative review of advances in the application of artificial intelligence to the time-domain analysis of electronic medical records.
I personally feel that the researcher had rather ambitious goals when designing this review, but the results achieved were not as good as expected.
The principles related to meta-analysis and systematic evaluation were used in the research methodology of this paper to delineate the main elements and structure of electronic medical record research. However, there is no overall analysis of the limitations of the relevant studies in the results, and it is hoped that the authors will add relevant content.
One of the biggest problems with this manuscript is that it is not logical, and the scattered content in the text makes it less systematic and less inspiring to read. For example, lines 479-486, each paragraph throws out a point of view, and even though the point of view is good, it is not closely enough connected to the whole study. I hope the author can reorganize the narrative so that the ideas in this paper can be expressed systematically and clearly.
Author Response
Response to Reviewer #2
Reviewer’s Comments in overall submission.
Reviewer #2, Comment #1: This is a typical interdisciplinary study in which the authors integrate multiple research approaches to provide a highly innovative review of advances in the application of artificial intelligence to the time-domain analysis of electronic medical records.
I personally feel that the researcher had rather ambitious goals when designing this review, but the results achieved were not as good as expected.
Author response: The authors thank you for your comments and acknowledge the coverage of this review paper is very broad, making this is a challenge to address within the submission word count limit.
Author action: Specific areas of concern raised by reviewers have been individually addressed.
Reviewer’s Comments in Results.
Reviewer #2, Comment #2: The principles related to meta-analysis and systematic evaluation were used in the research methodology of this paper to delineate the main elements and structure of electronic medical record research. However, there is no overall analysis of the limitations of the relevant studies in the results, and it is hoped that the authors will add relevant content.
Author response: Thank you for pointing this out. In response, the authors have added to the Results section an overall analysis of the limitations of each of the relevant studies reviewed.
Author action: (Lines 206 to 208) There are several limitations of time series and graph-based healthcare data; these include data sparsity [47], noise [48], and limited generalizability and lack of context [49].
(Lines 277 to 280) In the healthcare context, structured and semi-structured data are typically easier to work with and analyze because they have some inherent structure. Unstructured data, such as free-text notes in electronic health records, can be more challenging to work with because it requires more effort to extract meaningful information from it.
(Lines 305 to 313) The critical limitations of wearable sensors are the contextualization of data and integration with the existing clinical care pathways. Interoperability is another key aspect that needs to be addressed while deciding different sensing elements. This will help increase the generalizability of models by allowing it access to different kinds of data.
(Lines 381 to 395) There are several key limitations to data harmonization standards for electronic medical records for example:
- Complexity - Data harmonization standards can be complex and may require significant resources to implement and maintain.
- Limited adoption - Not all electronic medical record systems may adopt the same data harmonization standards, which can limit the ability to exchange data between systems.
- Changing standards - Data standards can change over time, which can make it difficult to maintain compatibility with other systems.
- Privacy and security concerns - The exchange of patient data between systems can raise concerns about privacy and security. Careful measures must be taken to ensure that patient data is protected when it is shared between systems.
- Cost - Implementing and maintaining data harmonization standards can be expensive, particularly for smaller healthcare organizations.
(Lines 507 to 509) There is currently a lack of standardized evaluation methods for interpretability techniques, making it difficult to compare and contrast the effectiveness of different approaches.
Reviewer’s Comments about structure and coherency.
Reviewer #2, Comment #3: One of the biggest problems with this manuscript is that it is not logical, and the scattered content in the text makes it less systematic and less inspiring to read. For example, lines 479-486, each paragraph throws out a point of view, and even though the point of view is good, it is not closely enough connected to the whole study. I hope the author can reorganize the narrative so that the ideas in this paper can be expressed systematically and clearly.
Author response: The authors thank you for highlighting this very important requirement. In response we have restructured the narrative throughout the document to better connect the content by explaining the processes undertaken and the discussion around the findings. Additionally, we have added text to the Conclusion section that connects the different points of view draw out from this review.
Author action: (Lines 542 to 562) Another issue that needs to be tackled in data collection and preprocessing is interoperability of various devices and sensors, and this review has elaborated on different interoperability methods.
There are issues where collected data is fed to different decision systems. This part of the pipeline has been discussed by this review, especially where the graph-based solutions such as temporal phenotyping used to help identify risks for various morbidities and help cluster disease presentation into various groups. The most recent works and re-viewed literature focus more on applying these solutions in the real world. This application becomes easier when the AI process making predictions can be explained, hence different explainability and interpretability techniques are compared here while highlighting the lack of standard metrics of evaluation for such methods. The validation of accuracy for each individual patient is an open area of research.
Based on the advances mentioned in this review, any future review may include the identification of ethical dilemmas in healthcare interventions and personalized healthcare: continuous healthcare monitoring and better intervention methods. Clinical use of ambulatory data continues to be a challenge for traditional medical practice. The debate between generalizable AI models with the required precision to achieve individual specific health outcomes will likely continue. This work can influence the current healthcare system in a positive manner, however, a way of combining these issues can first develop individual specific models, then explain them using explainability techniques and then cluster them for general exploratory studies.
Reviewer 3 Report
This review presented a paradigm of the application of AI in times series and graph-based healthcare data driven by translational medicine. It looks at the complete pipeline, starting from data collection, harmonization, and quality dimensions. The decision systems are deliberated over, including various kinds of phenotyping, mortality detection, and other methods. Details related to the components related to the data, explainability, algorithms and levels of automation are studied. This paper has some certain guidance and thinking value. Some issues are listed below:
1. Please add an explanation of the standard used to exclude 320 Records after retrieving and eliminating duplicates in the database.
2. The structure of Table 4 can be optimized to make it more intuitive, preferably on the same page.
3. Please supplement the bibliometric analysis to get the development context and research status of the field of electronic medical records data.
4. Please supplement the academic views and judgments on the future development trend in the conclusion section.
5. There are some typographical errors. For example, the first letter of the Abstract is bold at the beginning, there are more punctuation marks on line 37, and there is no period at the end of line 38, line 263, and line 296.
6. Please supplement relevant reference materials in 2022.
Author Response
Response to Reviewer #3
Reviewer’s Comments in overall submission.
Reviewer #3, Comment #1: This review presented a paradigm of the application of AI in times series and graph-based healthcare data driven by translational medicine. It looks at the complete pipeline, starting from data collection, harmonization, and quality dimensions. The decision systems are deliberated over, including various kinds of phenotyping, mortality detection, and other methods. Details related to the components related to the data, explainability, algorithms and levels of automation are studied. This paper has some certain guidance and thinking value. Some issues are listed below:
Author response: The authors thank you for your comments and positive feedback.
Author action: Specific areas of concern raised by reviewers have been individually addressed.
Reviewer’s Comments about Materials and Methods.
Reviewer #3, Comment #2: Please add an explanation of the standard used to exclude 320 Records after retrieving and eliminating duplicates in the database.
Author response: The authors thank you for your comments and positive feedback which we have addressed by adding more detail to the sentence describing the language exclusion criteria.
Author action: (Lines 100 to 102) Appling an English language filter criteria resulted in 320 articles being removed.
Reviewer’s Comments in Results.
Reviewer #3, Comment #3: The structure of Table 4 can be optimized to make it more intuitive, preferably on the same page.
Author response: Thank you for pointing out this formatting issue which we have addressed by revising the table 4 heading and page formatting.
Author action: Added a column headed ‘Evaluation Metrics’ and repositioning the table so that it now spans a single page.
Reviewer #3, Comment #4: Please supplement the bibliometric analysis to get the development context and research status of the field of electronic medical records data.
Author response: Thank you for raising this important matter which we have addressed by adding additional text throughout the submission to better establish the bibliometric development context and research status, as detailed in Author action below.
Author action: Lines 98 to 102) The search was limited to 2015 to November 2022. Only articles were included, and re-views were not made part of the search criteria. Proceedings of various conferences were excluded from the search. Applying these filters and only selecting English language records resulted in 320 articles being removed, resulting in the number of articles selected for review totaling 164.
(Lines 141 to 144) Figure 2 is a representation of the paradigms found in the literature under the topic at hand. The inclined axes represent the major topics in the field, for example, types of data and levels of automation. These inclined axes are also divided, shown as horizontal arrows, to identify the relevant subtopics.
(Lines 305 to 323) The critical limitations of wearable sensors are the contextualization of data and integration with the existing clinical care pathways; hence a challenge exists to show clinical efficacy. Most historic clinical data is taken from a patient at rest (e.g., resting heart rate) with the assumption that only disease can shift homeostasis, and most wearable data is ambulatory (e.g., heart rate during a workout) with confounders such as physical activity making traditional clinical interpretation challenging. Interoperability is another crucial aspect that needs to be addressed when deciding on different sensing elements. This will help increase the generalizability of models by allowing them access to various kinds of data.
(Lines 381 to 395) There are several key limitations to data harmonization standards for electronic medical records for example:
- Complexity - Data harmonization standards can be complex and may require significant resources to implement and maintain.
- Limited adoption - Not all electronic medical record systems may adopt the same data harmonization standards, which can limit the ability to exchange data between systems.
- Changing standards - Data standards can change over time, which can make it difficult to maintain compatibility with other systems.
- Privacy and security concerns - The exchange of patient data between systems can raise concerns about privacy and security. Careful measures must be taken to ensure that patient data is protected when it is shared between systems.
- Cost - Implementing and maintaining data harmonization standards can be expensive, particularly for smaller healthcare organizations.
Reviewer’s Comments in Conclusions.
Reviewer #3, Comment #5: Please supplement the academic views and judgments on the future development trend in the conclusion section.
Author response: The authors thank you for pointing this out and have responded by adding the following paragraph in the Conclusions section that articulates our views around future development trends.
Author action: (Lines 554 to 562) Based on the advances mentioned in this review, any future review may include the identification of ethical dilemmas in healthcare interventions and personalized healthcare: continuous healthcare monitoring and better intervention methods. Clinical use of ambulatory data continues to be a challenge for traditional medical practice. The debate between generalizable AI models with the required precision to achieve individual specific health outcomes will likely continue. This work can influence the current healthcare system in a positive manner, however, a way of combining these issues can first develop individual specific models, then explain them using explainability techniques and then cluster them for general exploratory studies.
Reviewer’s Comments in overall submission.
Reviewer #3, Comment #6: There are some typographical errors. For example, the first letter of the Abstract is bold at the beginning, there are more punctuation marks on line 37, and there is no period at the end of line 38, line 263, and line 296.
Author response: The Authors thank you for raising this important matter.
Author action: Overall submission has been reviewed and resolved for grammatical and typographical errors.
Reviewer’s Comments about study methods.
Reviewer #3, Comment #7: Please supplement relevant reference materials in 2022.
Author response: The authors thank you for your suggestion and have added four additional references.
Author action: Amended description of total papers reviewed in Materials and Method, and amended References accordingly.
(Lines 98 to 102) The search was limited to 2015 to November 2022. Only articles were included, and re-views were not made part of the search criteria. Proceedings of various conferences were excluded from the search. Applying an English language filter criteria resulted in 320 articles being removed, resulting in the number of articles selected for review totaling 164
(Lines 592 to 593)
- Pavlič, J., T. Tomažič, and I. Kožuh, The impact of emerging technology influences product placement effectiveness: A scoping study from interactive marketing perspective. Journal of Research in Interactive Marketing, 2021.
(Lines 661 to 666)
- Liu, Y., et al., Bidirectional GRU networks‐based next POI category prediction for healthcare. International Journal of Intelligent Systems, 2022. 37(7): p. 4020-4040.
- Rodeheaver, N., et al., Breathable, Wireless, Thin-Film Wearable Biopatch Using Noise-Reduction Mechanisms. ACS Applied Electronic Materials, 2022. 4(1): p. 503-512.
- Yang, J., A.A. Soltan, and D.A. Clifton, Machine learning generalizability across healthcare settings: insights from multi-site COVID-19 screening. npj Digital Medicine, 2022. 5(1): p. 1-8.
Reviewer 4 Report
By defining paradigms that fit within the domain of application of artificial intelligence (AI) based on the principles of translational medicine, authors analysed time series data required to develop a full EMR. The subheadings of the five primary subjects they discovered are examined in this review. The abstract is worded well. The paper is really well written. Having said that The article is suitable for publication. However, there are a few grammatical errors in the paper, so it may use some proofreading. Their review has been executed very well from a technical standpoint and is deserving of publication.
Author Response
Response to Reviewer #4
Reviewer’s Comments in overall submission.
Reviewer #4, Comment #1: The paper proposes a review that will focus on time series data needed to construct a complete EMR by identifying paradigms that fall within the scope of application of artificial intelligence (AI) based on the principles of translational medicine. Scopus, Web of Science, and PubMed were searched for relevant records. The authors identified 1020 records from research databases that were filtered based on a PRISMA review process and for review were selected 160 studies.
The paper is well organized, and the length is appropriate. The title is chosen correctly and the abstract provides sufficient information to give a clear idea of what to expect from the paper.
Author response: The authors thank you for your comments and positive feedback.
Author action: No action taken.
Reviewer’s Comments about study methods.
Reviewer #4, Comment #2: The study methods are appropriate, and the data are valid.
Author response: The authors thank you for your comments and positive feedback.
Author action: No action taken.
Reviewer’s Comments in Results.
Reviewer #4, Comment #3: The results are well highlighted, and the conclusions are adequate.
Author response: The authors thank you for your comments and positive feedback.
Author action: No action taken.
Reviewer’s Comments in References.
Reviewer #4, Comment #4: The results are well highlighted, and the conclusions are adequate.
Author response: The authors thank you for your comments and positive feedback.
Author action: No action taken.
Reviewer’s Comments about technical depth and quality.
Reviewer #4, Comment #5: The technical depth of the paper meets the requirements for a scientific article published in a quality journal.
Author response: The authors thank you for your comments and positive feedback.
Author action: No action taken.
Reviewer 5 Report
The paper contains sufficiently new and suitable information, and it adheres to the journal’s standards. The topic and level of formality are appropriate for the journal`s readership. Its style and readability are suitable. There is a huge amount of information given throughout the article, but I would suggest revising the paper.
The methodological concept is clear, and the selected methodology is scientifically appropriate.
Figure 1 (Fishbone diagram) must be explained better.
I suggest citing also PAVLIČ, Jani, TOMAŽIČ, Tina, KOŽUH, Ines. The impact of emerging technology influences product placement effectiveness: a scoping study from interactive marketing perspective. Journal of research in interactive marketing. 2021. DOI: 10.1108/JRIM-02-2021-0041 to explain the systematic review from the scoping study perspective.
The results are analyzed appropriately, but they should be presented more clearly (for example: comparison of unstructured data and comparison of interoperability techniques).
Further, I recommend rewriting the conclusion. The concluding remarks should be more specific and better explained. I suggest adding more future directions. The limitations of the study must be completed.
In summary, the article is sufficiently interesting to warrant publication, but it needs major revision. Please follow all the comments above.
Author Response
Response to Reviewer #5
Reviewer’s Comments in overall submission.
Reviewer #5, Comment #1: The paper contains sufficiently new and suitable information, and it adheres to the journal’s standards. The topic and level of formality are appropriate for the journal`s readership. Its style and readability are suitable. There is a huge amount of information given throughout the article, but I would suggest revising the paper.
The methodological concept is clear, and the selected methodology is scientifically appropriate.
Author response: The authors thank you for your comments and positive feedback.
Author action: No action taken.
Reviewer’s Comments about Methods and Materials.
Reviewer #5, Comment #2: Figure 1 (Fishbone diagram) must be explained better.
Author response: The authors thank you for identifying this important matter and have responded by adding the explanation text.
Author action: (Lines 141 to 144) Figure 2 is a representation of the paradigms found in the literature under the topic at hand. The inclined axes represent the major topics in the field, for example types of data, and levels of automation. These inclined axes are also divided, shown as horizontal arrows, to identify the relevant subtopics.
Reviewer’s Comments about references.
Reviewer #5, Comment #3: I suggest citing also PAVLIČ, Jani, TOMAŽIČ, Tina, KOŽUH, Ines. The impact of emerging technology influences product placement effectiveness: a scoping study from interactive marketing perspective. Journal of research in interactive marketing. 2021. DOI: 10.1108/JRIM-02-2021-0041 to explain the systematic review from the scoping study perspective.
Author response: The authors thank you for making this suggestion which we have added text to the submission as Reference 11.
Author action: (Line 57) Our review of the relevant studies has found numerous pertinent publications [11].
(Lines 571 to 572) 11 Pavlič, J., T. Tomažič, and I. Kožuh, The impact of emerging technology influences product placement effectiveness: A scoping study from interactive marketing perspective. Journal of Research in Interactive Marketing, 2021.
Reviewer’s Comments in Results.
Reviewer #5, Comment #4: The results are analyzed appropriately, but they should be presented more clearly (for example: comparison of unstructured data and comparison of interoperability techniques).
Author response: The authors thank you for making this suggestion which we have implemented in two parts for each of the areas identified. For the comparison of unstructured data, we have added additional paragraphs to the submission addressing these issues.
Author action: (Lines 268 to 273) The different ML techniques used in conjunction with unstructured data are clustering, classification, boosting, and a combination. Clustering can help with phenotyping and grouping together different clinical pathways. Classification requires labeling the data, which can be taxing for a large volume of clinical notes. Boosting models can leverage the different structures present in unstructured data to make meaningful predictions, especially; risk and mortality.
(Lines 277 to 280) In the healthcare context, structured and semi-structured data are typically easier to work with and analyze because they have some inherent structure. Unstructured data, such as free-text notes in electronic health records, can be more challenging to work with because it requires more effort to extract meaningful information from it.
To clarify the comparison of interoperability techniques, we also have added the following paragraph:
(Lines 342 to 354) The methods used for interoperability include NLP, data mapping and transformation, data quality assessment, predictive analytics, and anomaly detection. They are used to promote one or more of these: Standardization of data, using application programming interface (API), using middleware and frameworks such as the Da Vinci project, and health information exchanges (HIEs). While effective, NLP techniques are very resource intensive. Data mapping and transformations can be very narrow in application. Data quality assessments can be used to compare inconsistencies but require constant updates and maintenance. Predictive analytics can help improve care coordination and resource allocation, but this is also effective in a narrow range of situations. Anomaly detection can identify unusual or unexpected patterns in healthcare data, potentially flagging issues that may need to be addressed. However, it requires certain contextual information to be more effective.
Reviewer’s Comments in Conclusions.
Reviewer #5, Comment #5: Further, I recommend rewriting the conclusion. The concluding remarks should be more specific and better explained. I suggest adding more future directions. The limitations of the study must be completed.
Author response: The authors thank you for raising this matter, and in response have rewritten the Conclusion to include and explain specific findings and future directions.
Author action: (Lines 542 to 562) A key tool used to that end is blockchain technology, which has been used increasingly for the deidentification of data.
Another issue that needs to be tackled in data collection and preprocessing is interoperability of various devices and sensors, and this review has elaborated on different interoperability methods.
There are issues where collected data is fed to different decision systems. This part of the pipeline has been discussed by this review, especially where the graph-based solutions such as temporal phenotyping used to help identify risks for various morbidities and help cluster disease presentation into various groups. The most recent works and reviewed literature focus more on applying these solutions in the real world. This application becomes easier when the AI process making predictions can be explained, hence different explainability and interpretability techniques are compared here while highlighting the lack of standard metrics of evaluation for such methods.
Based on the advances mentioned in this review, any future review may include the identification of ethical dilemmas in healthcare interventions and personalized healthcare: continuous healthcare monitoring and better intervention methods. The debate between generalizable AI models and studies and the precision models that tackle individual specific health outcomes will likely continue soon. Both types of work can influence the current healthcare system in a positive manner, however, a way of combining the two can first develop individual specific models, then explain them using explainability techniques and then cluster them for general exploratory studies.
Reviewer’s Comments in overall submission.
Reviewer #5, Comment #6: In summary, the article is sufficiently interesting to warrant publication, but it needs major revision. Please follow all the comments above.
Author response: The authors thank you for your comments and suggestions which we have implemented in the changes made to our submission.
Author action: Action taken as previously detailed for each response.
Round 2
Reviewer 5 Report
I agree with the revised version of the paper.
