Review Reports - Source-Only Transportability of Engineered ECG Features for Healthy-Versus-Myocardial Infarction Classification

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Greetings to authors, the review report is attached.

Comments for author File: Comments.pdf

Author Response

Major Revision 1: The private hospital data come mainly from a single institution and acquisition system. The implications should be strengthened, and raw feature distributions such as mean ST amplitude and QRS duration should be compared between datasets.

Response to the Reviewer: We agree and strengthened the interpretation of the transfer gap. The revised manuscript separates label/prevalence differences, acquisition/device conditions, feature-distribution shift, data-quality effects, and threshold-transportability effects. We also used the supplementary feature-quality summaries to show that missingness and ST extraction failures are unlikely to be the sole explanation: most feature families had zero missingness; the largest observed missingness fraction was for lead-connectivity features (0.044% in the hospital dataset and 0.0013% in PTB-XL); and ST fallback recovery produced complete final ST outputs. We therefore avoid assigning a fixed numerical fraction of the transfer gap to one component, but now describe the plausible components more explicitly and provide the available aggregate feature-quality summaries in the supplementary material.

Added to Data Sources: "Because the hospital source came mainly from one institution and one ECG acquisition system, the transfer gap was interpreted as the combined effect of at least three nonexclusive components: label/prevalence differences, acquisition and device-related feature differences, and true ECG feature-distribution shift. Aggregate raw-feature summaries such as ST-amplitude distributions, fiducial duration summaries, and QRS-related interval summaries could be reported without releasing private ECG records. In the revised supplementary material, the available aggregate quality summaries document feature-family missingness and ST-extraction fallback behavior, helping separate data-quality effects from broader label, acquisition, and feature-shift explanations."

Added to Feature-Extraction Reliability: "Most feature-family blocks had no missing values; the largest missingness fraction was observed for lead-connectivity features, with 0.044% missingness in the hospital dataset and 0.0013% in PTB-XL. ST-feature extraction also produced complete final outputs after fallback recovery, with fallback required for 440 of 20,988 hospital NeuroKit lead slots (2.10%) and 3393 of 179,784 PTB-XL NeuroKit lead slots (1.89%)."

Major Revision 2: Clarify the inclusion and exclusion criteria in the data-collection section.

Response to the Reviewer: We added explicit inclusion and exclusion criteria for the hospital dataset, including the chest-pain workflow, acquisition year/system, 10-second 500 Hz 12-lead XML/HL7 recordings, interpretable canonical leads, cardiologist binary label, and exclusion of broken data, incomplete waveforms, missing or unrecoverable leads, near-zero amplitudes, severe artifacts, and unusable labels.

Added to Cardiologist-Annotated Hospital Dataset: "Explicit inclusion criteria for the hospital source were: presentation within the chest-pain ECG workflow, acquisition in 2024 on the available 12-lead hospital ECG system, availability of a 10-second 500 Hz XML/HL7 aECG record, interpretable waveforms in the canonical 12 leads, and cardiologist assignment to Healthy or MI for the binary task. Exclusion criteria were: broken-data status, incomplete waveform length, missing or unrecoverable leads, zero or near-zero amplitude, severe artifact preventing interpretation, or absence of a usable binary Healthy/MI label after cardiologist review."

Major Revision 3: Add a section for the classifiers used, explain the architectures in detail, and specify the parameters.

Response to the Reviewer: We renamed and expanded the classifier section. The revised text describes the five classifier families and explains the parameter categories. The exact grids and search budgets are now provided in Supplementary Table S3.

Added to Classifiers Used and Hyperparameter Search: "In architectural terms, logistic regression was used as an L2-regularized linear probabilistic classifier; the SVM used a radial-basis-function kernel; random forest and Extra Trees were bagged decision-tree ensembles differing in the degree of split randomization; and XGBoost was a gradient-boosted decision-tree ensemble."

Added to Classifiers Used and Hyperparameter Search: "Exact candidate grids and search budgets are reported in Supplementary Table S3."

Major Revision 4: The manuscript cites deep-learning studies but uses only machine-learning classifiers. Explain the rationale and consider adding a CNN/ResNet baseline.

Response to the Reviewer: We appreciate this suggestion. We retained the classical machine-learning scope because the study is intentionally focused on fixed engineered ECG feature families under a tabular source-only transportability design. The cited deep-learning studies are used to contextualize target-domain retraining, fine-tuning, architecture transfer, and transfer-learning approaches, whereas the present study intentionally excludes target-domain adaptation. Adding a raw-signal CNN/ResNet would require a separate architecture, signal normalization, tuning, calibration, and fairness protocol, and would answer a broader classical-versus-deep-learning benchmarking question rather than the specific engineered-feature transportability question addressed here. To avoid expanding the manuscript into a separate classical-versus-deep-learning benchmark, we kept the detailed rationale in this response and clarified in the manuscript that the present results are complementary to optimized neural models rather than a claim of superiority over them.

Added to Materials and Methods: "The present study intentionally left these adaptation steps out of the main analysis so that model transportability and adaptation to the target setting are not mixed together."

Clarified in Discussion: "The present results should therefore be read as complementary to deep-learning studies: they do not claim to outperform optimized neural models, but they quantify transportability for transparent feature families under a deliberately stricter transfer rule."

Major Revision 5: Merge the evaluation and metrics sections and add a description of qualitative evaluation such as ROCs.

Response to the Reviewer: We revised the Methods structure so that metrics and statistical analysis are nested under the broader evaluation-protocol heading. We also added text distinguishing threshold-dependent quantitative metrics from curve-based diagnostics such as ROC/PR behavior and score clustering.

Added to Evaluation Protocols, Metrics, and Statistical Analysis: "Evaluation combined quantitative point metrics with curve-based qualitative diagnostics. The quantitative component used operating-point metrics at the selected threshold, threshold-free ranking metrics, probability-error summaries, and bootstrap uncertainty."

Major Revision 6: Provide a structured analysis or table separating candidate explanations for directional asymmetry, and consider PTB-XL down-sampling to match hospital size.

Response to the Reviewer: We added a structured interpretation of directional asymmetry and expanded the threshold-calibration explanation using the supplementary transfer values. The hospital-to-PTB-XL strict transfer row retained moderate ranking (ROC-AUC 0.774) but had very low sensitivity at the source-selected threshold (0.164), whereas target-threshold selection increased sensitivity to 0.405 and balanced accuracy to 0.682. PTB-XL-to-hospital transfer failed differently: strict transfer was more balanced but lower-ranking (balanced accuracy 0.632, ROC-AUC 0.635), and target-threshold selection improved balanced accuracy only slightly to 0.640. PTB-XL down-sampling remains a useful optional future sensitivity analysis if the source pipeline is rerun.

Added to Strict Source-Only Cross-Dataset Transfer: "Sample size and prevalence: PTB-XL provided many more training records and a higher MI prevalence than the hospital source. These differences may affect learned probability scale, threshold selection, and stability of feature-label associations. A size-matched PTB-XL down-sampling sensitivity analysis is therefore useful for separating sample-size effects from other dataset-shift effects."

Added to Secondary Target-Threshold Selection: "Threshold calibration should therefore be viewed as a distinct validation requirement beyond ROC-AUC."

Major Revision 7: Add introductory text after the Results and Discussion section headings.

Response to the Reviewer: We added introductory paragraphs immediately after both section headings to improve flow and orient readers before the subsections.

Added to Results: "The results are presented in four steps. First, internal performance within each dataset is summarized to show the apparent performance ceiling under matched source conditions."

Added to Discussion: "The discussion interprets the transfer gap as a methodological and clinical-scope finding."

Major Revision 8: Revise the Abstract to distinguish ranking transportability from operating-threshold transportability, and add threshold-calibration discussion beyond AUC.

Response to the Reviewer: We revised the Abstract and Discussion to distinguish ROC-AUC/ranking behavior from threshold-dependent sensitivity and specificity. We also added a threshold-calibration discussion to emphasize that AUC alone is insufficient for clinical validation when a source-selected threshold is transported to another dataset.

Added to Abstract: "Ranking transportability and operating-threshold transportability diverged, most notably for hospital-to-PTB-XL transfer, where ROC-AUC was 0.774 but sensitivity at the source-selected threshold was only 0.164."

Added to Secondary Target-Threshold Selection: "Threshold calibration should therefore be viewed as a distinct validation requirement beyond ROC-AUC. A model can preserve partial ranking ability in the target data while its source-selected operating threshold produces unacceptable sensitivity or specificity."

Minor Revision 1: Figure 4 heatmap numeric labels are small; increase font size or provide a companion table.

Response to the Reviewer: We revised the caption to direct readers to the supplementary result-export table, which provides print-readable numeric balanced-accuracy values.

Added to Figure 4 caption: "The full numeric balanced-accuracy values are also provided in the supplementary result table."

Minor Revision 2: Figure 5 caption should define FN/TP and class definitions.

Response to the Reviewer: We added a stand-alone caption note defining the positive and negative classes and clarifying the TP/FN and TN/FP notation.

Added to Figure 5 caption: "MI is the positive class and Healthy is the negative class; therefore TP and FN refer to MI recordings, whereas TN and FP refer to Healthy recordings."

Minor Revision 3: Add a histogram of predicted probabilities for PTB-XL-to-hospital transfer.

Response to the Reviewer: We agree that predicted-score histograms would make score clustering more concrete. The supplied supplementary files contain aggregate metrics rather than per-record predicted probabilities, so we did not cite a new histogram that is not yet available. We revised the transfer-figure caption and response text to acknowledge this point; a supplementary histogram can be added if saved prediction outputs are provided.

Added to Figure 6 caption: "The piecewise-linear appearance of the PTB-XL-to-hospital ROC curve is consistent with clustered or coarsely separated predicted scores, as discussed in the Results."

Minor Revision 4: Report variance for Table 4 because imbalance-policy differences are small.

Response to the Reviewer: We revised the manuscript to avoid overinterpreting the small imbalance-policy differences. Because this policy comparison was based on a fixed patient-aware split rather than repeated resampling, variance was not estimated for that table; therefore, the selected policy is now described as pragmatic rather than robustly superior.

Added to Imbalance Handling and Hospital Policy Selection: "Because the balanced-accuracy differences between imbalance policies were small, the selected mild-oversampling policy should be interpreted as a pragmatic source-training choice rather than as a robustly superior clinical preprocessing rule."

Minor Revision 5: State the exact random-search budget or number of iterations.

Response to the Reviewer: We revised the classifier section to point readers to the exact candidate grids and search budgets in Supplementary Table S3. The supplementary table reports the model-specific grids and the random-search budget, while the main text keeps the methods description concise.

Added to Classifiers Used and Hyperparameter Search: "Exact candidate grids and search budgets are reported in Supplementary Table S3."

Minor Revision 6: Report the proportion of hospital training/testing patients with more than one recording and the number/distribution of PTB-XL records excluded from the binary subset.

Response to the Reviewer: We added the known PTB-XL exclusion count and clarified that excluded PTB-XL records do not have a Healthy/MI class distribution under the present binary rule.

Added to PTB-XL Binary Subset: "From the original 21837 PTB-XL records, 6855 records were excluded by the selected binary mapping because they contained neither the retained Healthy definition nor any selected MI-superclass SCP code; these excluded records do not have a Healthy/MI class distribution under the present binary rule."

Minor Revision 7: Tables S1 to S5 are referenced in the Conclusions but not provided.

Response to the Reviewer: We thank the reviewer for identifying this omission. The supplementary tables were inadvertently omitted during final manuscript file preparation and have now been included with the revised submission. To improve readability, the short threshold-policy table was merged into Supplementary Table S3 as a threshold-selection panel, and full result exports are supplied as machine-readable CSV files.

Added to Supplementary Materials: "Supplementary Tables S1-S5 are provided as machine-readable result exports, with compact summary tables for readability where appropriate."

Added to Supplementary Table S3: "Panel B: threshold selection and reporting."

Minor Revision 8: Table S2 should disclose missingness by feature family and dataset.

Response to the Reviewer: We included feature-family definitions, raw feature counts, hospital missingness, hospital rows with any missing values, PTB-XL missingness, and PTB-XL rows with any missing values in Supplementary Table S2.

Added to Supplementary Table S2: "Feature-family definitions, feature dimensions, and missingness summaries by dataset."

Minor Revision 9: Define 'transportability' or cite a formal framework.

Response to the Reviewer: We added an Introduction paragraph clarifying that transportability is used empirically to mean source-only external validity of the predictive ECG pipeline, not formal causal transportability. We also added a citation to the Pearl and Bareinboim formal transportability framework.

Added to Introduction: "The term transportability is used here in the empirical diagnostic-model sense of whether a source-trained feature extraction, classifier, and threshold-selection pipeline retains useful performance in a different ECG data source without target-domain adaptation."

Added to Introduction: "Therefore, transportability denotes source-only external validity of a predictive pipeline, not a proof of causal transportability."

Minor Revision 10: Verify the AI model designation in the acknowledgments and remove unused abbreviations such as ACS, Grad-CAM, and CWT if they do not appear in the main text.

Response to the Reviewer: We removed the model-specific AI designation and replaced it with a general author-responsibility statement. We also removed the unused ACS abbreviation. CWT and Grad-CAM are retained because they appear in the revised related-study comparison table.

Added to Acknowledgments: "Language-editing and coding-assistance tools were used to improve readability, support manuscript revision, and assist code review."

Removed from Abbreviations: ACS was removed from the abbreviation list.

Reviewer 2 Report

Comments and Suggestions for Authors The study is potentially suitable for publication after revision, but several points should be addressed.

Major comments

The task is binary Healthy-vs-MI classification. This is not equivalent to real-world chest-pain triage, where non-MI abnormalities, ischemia without MI, arrhythmias, conduction disease and other ECG pathologies are common. The authors acknowledge this limitation, but it should be emphasized earlier and more strongly. The manuscript should avoid any wording that may imply readiness for clinical diagnostic use.
The hospital MI labels and PTB-XL SCP MI superclass codes may not represent the same clinical construct. PTB-XL includes old infarction/location statements and injury-related codes, while the hospital dataset appears to come from patients presenting with chest pain. The authors should discuss how differences in label definition, disease stage and clinical context may explain the transfer gap.
Many combinations of 11 feature families and 5 classifiers are compared. The authors correctly state that the best rows should not be interpreted as validation of a locked final model. This point should also be reflected in the Abstract and Conclusions, because the “best observed” results may otherwise be interpreted as optimized final model performance.
The strict transfer results are modest, especially hospital-to-PTB-XL sensitivity of 0.164 at the source-selected threshold. The authors should explicitly state that such performance would not be acceptable for clinical MI screening or decision support without further adaptation and validation.

Minor comments

The title is informative but long. Consider shortening it while preserving the key concept of “source-only transportability”.
Some sections of the Introduction and Discussion repeat the same message about internal validation versus transfer performance. The manuscript would benefit from tightening.
Please ensure consistent terminology: “external validation”, “source-only transfer”, “transportability”, “target-threshold selection” and “transfer learning” should be defined once and then used consistently.
The feature families are well listed, but the clinical meaning of the most successful families could be explained more briefly in the Results or Discussion.

Author Response

Major Revision 1: The binary Healthy-vs-MI task is not real-world chest-pain triage and should be emphasized earlier; avoid wording implying clinical readiness.

Response to the Reviewer: We agree. We added targeted cautionary statements in the Abstract, Introduction, Results/Discussion, and Conclusions. We placed the main clinical-scope statement later in the Introduction, after the transportability motivation, to avoid interrupting the opening clinical background.

Added to Introduction: "Consistent with this methodological focus, the present work is an ECG-methodology benchmark rather than a real-world chest-pain triage study."

Added to Conclusions: "The source-only operating-point results are not acceptable for clinical MI screening or decision support without additional calibration, target-setting validation, and prospective assessment."

Major Revision 2: Hospital MI labels and PTB-XL SCP MI superclass codes may not represent the same clinical construct.

Response to the Reviewer: We added text explaining that the PTB-XL MI construct can include old infarction/location statements and injury-related codes, whereas the hospital dataset came from a chest-pain workflow with cardiologist review. We now explicitly treat label definition, disease stage, and clinical context as plausible contributors to the transfer gap.

Added to PTB-XL Binary Subset: "This PTB-XL label construction is not identical to the hospital clinical label construct."

Added to Directional asymmetry table: "Hospital MI labels came from a chest-pain clinical workflow, whereas PTB-XL MI-superclass SCP codes include infarction-location and injury-related statements."

Major Revision 3: Many feature-family/classifier combinations are compared; best observed rows should not be interpreted as a locked final model, and this should be reflected in the Abstract and Conclusions.

Response to the Reviewer: We revised the Abstract and Conclusions to clarify that the leading rows are highest observed configurations from a prespecified benchmark grid, not locked clinical models ready for deployment.

Added to Abstract: "The reported leading rows are the highest observed configurations in a prespecified benchmark grid, not locked clinical models."

Added to Conclusions: "These results should be read as highest observed benchmark performance across a prespecified feature-family/model grid, not as performance of a locked final model ready for deployment."

Major Revision 4: The strict transfer results are modest, especially hospital-to-PTB-XL sensitivity of 0.164, and this would not be acceptable for clinical screening or decision support.

Response to the Reviewer: We added this point explicitly in the Results and Conclusions.

Added to Strict Source-Only Cross-Dataset Transfer: "This sensitivity would not be acceptable for clinical MI screening or decision support if used without recalibration, target-setting validation, and prospective clinical evaluation."

Added to Conclusions: "The modest source-only transfer results, especially the low hospital-to-PTB-XL sensitivity at the source-selected threshold, reinforce that additional calibration, adaptation, and prospective validation are necessary before any clinical decision-support use."

Minor Revision 1: The title is informative but long; consider shortening while preserving the key concept of source-only transportability.

Response to the Reviewer: We shortened the title while retaining the key concept of source-only transportability and the Healthy-versus-MI engineered-feature scope.

Added to Title: "Source-Only Transportability of Engineered ECG Features for Healthy-versus-Myocardial Infarction Classification"

Minor Revision 2: Some sections of the Introduction and Discussion repeat the same message about internal validation versus transfer performance. Tighten the manuscript.

Response to the Reviewer: We concentrated the internal-versus-transfer distinction in the Introduction, Results, and Discussion signpost paragraphs and avoided large deletion-based restructuring so that the revision remains targeted and transparent.

Added to Introduction: "However, strong performance inside PTB-XL or inside a single hospital source does not by itself establish that the same feature representation and source-selected threshold will remain valid in another setting."

Added to Discussion: "The primary finding was a substantial divergence between internal validation performance and source-only transportability."

Minor Revision 3: Ensure consistent terminology: external validation, source-only transfer, transportability, target-threshold selection and transfer learning should be defined once and used consistently.

Response to the Reviewer: We added a Methods paragraph defining the terms and used the definitions to distinguish strict source-only transfer from target-threshold selection and transfer learning.

Added to Materials and Methods: "Throughout the paper, external validation refers to evaluation in a separate data source or setting. Source-only transfer refers to external testing after all model and threshold choices have been made with source-dataset labels only."

Added to Materials and Methods: "Transfer learning refers to retraining, fine-tuning, architecture transfer, or other target-domain adaptation and is not part of the strict source-only analysis."

Minor Revision 4: The feature families are well listed, but the clinical meaning of the most successful families could be explained more briefly in Results or Discussion.

Response to the Reviewer: We added concise clinical interpretation of the leading feature families in the Results and Discussion.

Added to Within-Dataset Classification Performance: "Clinically, the leading hospital interval family reflects timing and duration information from the representative beat, whereas the leading PTB-XL hybrid + ST family combines morphology, interval, repolarization, spatial, and ST-segment summaries."

Added to Feature-Family Variation Across Validation Settings: "Fiducial intervals summarize timing of depolarization and repolarization-related landmarks, ST descriptors target ischemia-relevant segment behavior, repolarization descriptors capture T-wave morphology and polarity, and lead-connectivity descriptors summarize cross-lead coherence."

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

I have reviewed the revised version of the manuscript and the authors’ response to the previous comments. The requested changes have been implemented satisfactorily. In my opinion, the manuscript can be accepted in its current form.