Review Reports

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

I have finished my review of the manuscript entitled “Leakage-Controlled Machine Learning for Territorial Epidemiological Risk Stratification: A Biomedical Informatics Framework Using Administrative Health Data in the Colombian Orinoquía”.

This manuscript addresses a relevant and timely problem in biomedical informatics and public health surveillance: the development of predictive approaches using administrative health data while explicitly addressing one of the most common threats in machine learning applications—data leakage. The study demonstrates awareness of current methodological expectations by incorporating temporal validation, geographic validation, calibration assessment, uncertainty estimation, and model interpretability.

Among its strengths, I particularly value the authors’ explicit identification and exclusion of leakage-generating variables, the use of temporally separated training and testing periods, and the effort to move beyond conventional random train–test partitioning. The manuscript is generally well structured and demonstrates familiarity with contemporary prediction-model reporting principles.

However, I consider that important methodological and interpretative issues remain before the conclusions can be considered sufficiently supported.

My principal concern relates to the analytical unit and the effective independence of observations. Although mathematical leakage was addressed appropriately, the outcome is generated at an aggregated stratum level while analyses are conducted at the record level, creating the possibility of pseudo-replication and optimistic estimation of predictive performance. I encourage the authors to explicitly quantify the number of unique analytical strata and, ideally, repeat analyses using aggregated observations to determine whether model performance remains stable.

A second major concern is the dominant contribution of geographic identifiers, particularly municipality to prediction performance. The current results raise the possibility that the model is capturing territorial identity or administrative reporting patterns rather than transferable epidemiological risk. Additional analyses removing municipality and department variables, or comparing model performance with and without geographic labels, would considerably strengthen the interpretation.

I also recommend reconsidering the current comparison with Logistic Regression, which appears disadvantaged by the encoding strategy and therefore provides limited interpretative value as a benchmark. Likewise, the threshold used to define high risk appears operational rather than epidemiologically grounded and would benefit from additional justification.

From a reporting perspective, figures should be revised to ensure full language consistency and publication-quality presentation. Several figures contain embedded Spanish text despite submission in English. In addition, while the reference list includes strong methodological citations, the discussion would benefit from broader engagement with recent applied literature in public health machine learning, surveillance systems, geographic validation, and fairness-aware predictive modeling.

Some references are to generic to spcecifically sustain the arguments, methods and data processing. This is a major concern that would make readers doubt of the true validity of the study, providing scientific grounds is important.

eventhough, I do not consider the study unsuitable for publication. The study presents methodological effort, transparent acknowledgment of limitations, and potentially useful contributions for territorial surveillance. Nevertheless, the current version of the manuscript overextends some interpretations relative to the available evidence.

Comments on the Quality of English Language

Some figures and graphical elements still contain Spanish text and residual diacritics, suggesting incomplete language standardization.

All visual materials should be fully translated into English before publication.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

Comments and Suggestions for Authors

The research utilizes label encoding for all models, including the Logistic Regression baseline. As noted in the text, label encoding imposes artificial ordinal relationships on nominal categorical variables (such as diagnostic groups or municipalities), which inherently cripples a linear model's ability to learn valid coefficients. Consequently, the Logistic Regression model exhibits a sensitivity of 0.000 at a 0.5 threshold. Presenting a non-optimized, fundamentally flawed linear baseline weakens the methodological comparison, making the tree-based models look superior by default rather than by fair comparison. I suggest implementing a properly encoded linear baseline to ensure a fair comparison. Also suggest converting categorical features into one-hot encodings or target-based encodings specifically for the Logistic Regression pipeline. Additionally, the data represents individual epidemiological registrations, but target variables are calculated at an aggregated stratum level. This setup introduces extreme intra-stratum correlation , which artificially inflates the sample size to 354,088 records while evaluating identical target outcomes across multiple rows. While the research attempts to address this via GroupKFold validation , the reported record-level performance metrics remain highly susceptible to sample-size distortion. To confirm the framework's true clinical and statistical utility, the research should execute the entire pipeline directly at the aggregated stratum level, collapsing the dataset so that each unique stratum constitutes a single row where the target is the binary risk classification and the features are the stratum attributes. Comparing the performance of this collapsed model against the row-expanded model will definitively establish whether the framework is exploiting row duplication or capturing true epidemiological signals. Finally, the target variable is dichotomized into a binary "high-risk" classification based on the 75th percentile of the morbidity rate distribution (15.48 per 100,000). While the sensitivity analysis demonstrates statistical stability across the 50th, 60th, and 90th percentiles , the manuscript lacks an epidemiologically or clinically justified rationale for why the 75th percentile represents a meaningful threshold for public health intervention or surveillance operations. I suggest expanding the methodology and discussion sections to explicitly ground the 75th percentile selection within public health framework economics. For instance, explain if this threshold aligns with regional administrative capacities to handle high-risk alerts, or if it reflects an established epidemiological standard for determining outbreak or high-burden boundaries within Colombian health registries.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The revision is substantial and responsive. The authors clearly invested considerable effort into addressing to methodological concerns.

Issues still remain in interpretation, presentation, framing, and editorial refinement. I do not believe additional analyses are essential beyond what has already been provided, but I would still request one final revision.

I previously mentioned how "the model may be learning territorial identity"

but in the cover letter authors indicate "...Together, these analyses indicate predominantly transferable epidemiological signal rather than memorized territorial identity." this is an overconfident statement, is still stronger than the evidence. Please consider rephraming something like: "...suggest that a substantial proportion of predictive performance is explained by transferable clinical-epidemiological information, although residual geographic dependence remains."

I do not remember seeing PROBAST discussed meaningfully in the revised manuscript.

Any overconfident statement should be minimized

Comments on the Quality of English Language

Some figures and graphical elements still contain Spanish text and residual diacritics, suggesting incomplete language standardization.

All visual materials should be fully translated into English before publication.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

Comments and Suggestions for Authors Even after several improvements I still have concerns regarding the following: In the Discussion section, the authors discuss the critical equity implications of data-driven models, noting that training models on better-served populations can misrepresent risk in underserved territories and exacerbate inequities. They cite generic placeholders (e.g., "[8,9]", "[10]") but do not ground this specific finding in established algorithmic fairness or spatial epidemiology literature.   Also, the selection of the upper quartile  as the binary high-risk target threshold is described purely as a "pragmatic operational criterion" due to institutional capacity constraints. While operationally defensible, this lacks a sound epidemiological or theoretical rationale.   Connected is then the "leakage-controlled feature set" combines features with vastly different scales of cardinality, specifically "departmento" (4 values) and "municipio" (59 values) alongside broad binary/categorical factors like sexo and nombre_cat_edad. Tree-based ensemble models (Random Forest and Gradient Boosting) are heavily biased toward high-cardinality features (municipio), which artificially alters splitting behavior and inflation metrics.   Finally there is still a conceptual and operational contradiction between the methodology and the results regarding the "Person Model". The authors state in the Methods section that all analyses were conducted strictly at the aggregated stratum level. However, in Section 3.8 and Figure 13, they introduce an interactive individual-level profiling tool ("Person Model"), explicitly stating it "has not been validated with individual-level outcome data". Deploying an individual profiling interface derived exclusively from aggregated spatial data is a classic example of the ecological fallacy.  

Author Response

Please see the attachment.

Author Response File: Author Response.docx