Population Survival Kinetics Derived from Clinical Trials of Potentially Curable Lung Cancers

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This is a continuation of the authors previous work in population parmacokinetics. It is original in that it uses modern AI technology to investigate historic published data without having to go back to the original source material, a novel achivement which raises multi possibilities for research purposes. On the downside, it increases the risks for fradulant publications.

The overall statistical description is competent and shows the authors firm grasp of this developing field of biostatistics. However, most readers are not so "competent" hence the method / criteria description for data selection should be adjusted with this in mind. For example, why curves are excluded and the criteria behind that selection (lines 82-87). In addition, this exclusion could bias the data in one way or another, how does the author access the potential impact of data removal remembering that this is not the original data but an algorith calculated from a published chart.

The digitization of kaplan-meier curves works well and realistic estimates are created, however, how does the author control the experimental error for each of the included data sets, this cannot be estimated from the retrieved data. This can be extended into how does the author estimate the accuracy of the underlying statistics subsequently performed with the extracted data. Has the author performed some form of error analysis on the population groups.

Minor point of issue - the equation inline 95-96, this might be a function within excell but is not the source reference (is it not an adaptation the Taylor function ?) I think this should be more carefully described and referenced even if it is to one of the authors previous publications.

Author Response

Response to reviewer 1: We thank the reviewer for their helpful comments.

Comment #1: “It is original in that it uses modern AI technology to investigate historic published data without having to go back to the original source material, a novel achievement which raises multi possibilities for research purposes. On the downside, it increases the risks for fraudulent publications.”

We appreciate the reviewer’s view of the potential benefits of this approach. With respect to the requirement for caution in interpreting results, we have added a section on “Limitations”. It includes the following: “As with any methodology, there is at least some risk of incorrect conclusions due to misinterpretation of data. As much caution is required in assessment of results using these methods as with any analytical method. One advantage of this approach is that these analyses can be reassessed easily by others since they use accessible published data and readily available analytical tools.”

Comment #2: “For example, why curves are excluded and the criteria behind that selection (lines 82-87). In addition, this exclusion could bias the data in one way or another, how does the author access the potential impact of data removal remembering that this is not the original data but an algorithm calculated from a published chart.”

We have added the following: “We excluded curves derived from <50 patients to reduce potential variability arising solely from low patient numbers. Survival curves based on small sample sizes are subject to greater error, and exclusion of these small trials potentially would help to improve comparability of the remaining studies. The decision to use this particular cut point was arbitrary and could potentially be a source of bias. We have not tested the impact of using a higher or lower cut point.”

To provide more detailed information on trials included and excluded, we also added the following in the “Studies assessed” section under “Results”:

“Publications for chemoradiotherapy were identified in PubMed searches with the filter “Clinical Trial” that included the terms “radiotherapy or radiation or irradiation”, “non-small cell lung OR adenocarcinoma of the lung OR squamous cell carcinoma of the lung”, “locally advanced OR stage III OR stage 3”, “NOT metastatic”, “NOT stereotactic”, and “NOT neoadjuvant”. We identified 379 trials and excluded 143 that had fewer than 50 patients per trial arm, 2 with mixed populations of small cell and non-small cell lung cancer, 11 with no published PFS curves, 4 that did not include both chemotherapy and radiotherapy, and 132 that did not have PFS, relapse-free survival or disease-free survival as an endpoint. From eligible trials, we also excluded 6 curves with longest follow up less than 25 months.

Publications for limited small cell lung cancer were identified in PubMed searches with the filter “Clinical Trial” that included the terms “small cell lung”, “NOT non-small cell lung”, and “limited”. We identified 754 trials and excluded 218 with fewer than 50 patients per trial arm, 10 with no published PFS curves, 55 that included previously treated patients, 125 that included mixed populations of limited and extensive small cell lung cancer or small cell lung cancer plus non-small cell lung cancer, and 236 that did not have PFS as an endpoint.”

Comment #3: “The digitization of Kaplan Meier curves works well and realistic estimates are created, however, how does the author control the experimental error for each of the included data sets, this cannot be estimated from the retrieved data. This can be extended into how does the author estimate the accuracy of the underlying statistics subsequently performed with the extracted data. Has the author performed some form of error analysis on the population groups.”

We have added the following: “The models provide an R² value as an indicator of goodness of fit of the model for a particular curve. They also provide 95% confidence intervals and standard errors for model parameters. However, the method has no specific control for experimental error that might be intrinsic within data for a given curve. To reduce the potential impact of experimental error associated with an individual curve, we calculated medians and ranges for model parameters across curves for a given treatment setting.”

We also added the following in the section on “Limitations”: “As noted previously, the method has no specific control for experimental error that might be intrinsic within data for a given curve. However, in the online supplementary tables, we did include the sample size for each trial to help the reader assess the strength of the evidence.”

Comment #4: “Minor point of issue - the equation inline 95-96, this might be a function within excell but is not the source reference (is it not an adaptation the Taylor function ?) I think this should be more carefully described and referenced even if it is to one of the authors previous publications.”

We have provided a reference and a different way of expressing the equation: x=EXP(-t_n *0.693/t_1/2) is the same as x= 2^(- t_n / t_1/2).

Reviewer 2 Report

Comments and Suggestions for Authors

Dear Authors,

It was a pleasure to review this article.

I have a few comments to make:

- the method of selection of the studies included in the analysis should be explained (keywords used for the pub med search, range of years considered...)

- a graph showing the selection method and the studies selected according to the different stages of the disease and therapies could be useful

- the groups evaluated should be described (e.g. control groups: patients who did not receive chemotherapy...)

- the results obtained regarding the proportion of patients potentially cured and the median PFS half-life for relapsed subpopulations in the different patient groups were reported. The information for these patient groups was extrapolated from different studies involving patients with different clinical characteristics. For this reason, the data obtained are not comparable and their consecutive listing is methodologically incorrect.

- You have published previous articles on the same topic: it would be appropriate to add innovative aspects to the article, also in view of the more clinical target audience of this journal.

Author Response

Response to reviewer 2: We thank the reviewer for their helpful comments.

Comment #1: “- the method of selection of the studies included in the analysis should be explained (keywords used for the pub med search, range of years considered...)”

We had included that we had considered curative-intent chemoradiotherapy trials published 2010 to February 2022 and limited small cell lung cancer trials published 1990-2021.

We have added “Publications for chemoradiotherapy were identified in PubMed searches with the filter “Clinical Trial” that included the terms “radiotherapy or radiation or irradiation”, “non-small cell lung OR adenocarcinoma of the lung OR squamous cell carcinoma of the lung”, “locally advanced OR stage III OR stage 3”, “NOT metastatic”, “NOT stereotactic”, and “NOT neoadjuvant”.” We also added “Publications for limited small cell lung cancer were identified in PubMed searches with the filter “Clinical Trial” that included the terms “small cell lung”, “NOT non-small cell lung”, and “limited”.”

Comment #2: “- a graph showing the selection method and the studies selected according to the different stages of the disease and therapies could be useful.”

We have added for chemoradiotherapy: “Publications for chemoradiotherapy were identified in PubMed searches with the filter “Clinical Trial” that included the terms “radiotherapy or radiation or irradiation”, “non-small cell lung OR adenocarcinoma of the lung OR squamous cell carcinoma of the lung”, “locally advanced OR stage III OR stage 3”, “NOT metastatic”, “NOT stereotactic”, and “NOT neoadjuvant”. We identified 379 trials and excluded 143 that had fewer than 50 patients per trial arm, 2 with mixed populations of small cell and non-small cell lung cancer, 11 with no published PFS curves, 4 that did not include both chemotherapy and radiotherapy, and 132 that did not have PFS, relapse-free survival or disease-free survival as an endpoint. From eligible trials, we also excluded 6 curves with longest follow up less than 25 months.”

We have added for limited small cell lung cancer: “We identified 754 trials and excluded 218 with fewer than 50 patients per trial arm, 10 with no published PFS curves, 55 that included previously treated patients, 125 that included mixed populations of limited and extensive small cell lung cancer or small cell lung cancer plus non-small cell lung cancer, and 236 that did not have PFS as an endpoint.”

Comment #3: “- the groups evaluated should be described (e.g. control groups: patients who did not receive chemotherapy...)”

We have clarified that control groups in the adjuvant chemotherapy trials and in the adjuvant osimertinib trial were patients not receiving adjuvant therapy.

Comment #4: “- the results obtained regarding the proportion of patients potentially cured and the median PFS half-life for relapsed subpopulations in the different patient groups were reported. The information for these patient groups was extrapolated from different studies involving patients with different clinical characteristics. For this reason, the data obtained are not comparable and their consecutive listing is methodologically incorrect.”

We completely agree with the reviewer. We have added the statement “Note that for this outcome and for outcomes below, we combined data from trials that included patients with different characteristics and treatment details. These differences across trials will have impacted outcomes and will have contributed to the heterogeneity seen across trials. Future trials using individual patient data and/or more homogeneous populations would improve reliability of conclusions.”

Comment #5: “- You have published previous articles on the same topic: it would be appropriate to add innovative aspects to the article, also in view of the more clinical target audience of this journal.”

We have revised the first paragraph of the discussion to say, “We have previously published on use of population survival kinetics analyses in incurable solid tumors [2-4,10,11], with limited prior data presented on their use in potentially cured populations [3].  Here we expand on the use of the methodology in potentially cured populations. These analyses permit estimation of the proportion of patients potentially cured, the PFS half-life for the subpopulation destined to eventually relapse, and the probability of eventual progression in patients remaining progression-free at different timepoints after therapy initiation.”

Reviewer 3 Report

Comments and Suggestions for Authors

1.      Too few information on “published PFS” in the section of “Materials and Methods” (Line 77).  Hard for readers to understand the theory and rationale of analysis method. 

What journals?

What years of data?

Could this method be adjusted for important confounders like race-ethnicity, gender, age, cancer staging etc.?

Without the important information of the source data – and adjusted for it – any conclusions generated may be very biased.

2.      Very few citations for each of the reference 2-4 and 10-11. “Population survival kinetics analyses” maybe can be performed easily and rapidly – but must have some fundamental fraud in science to cause it was not well-accept in the real world.

3.      “3.1. Studies assessed” Line 103-110 and 114-115 should be in the “Materials and Methods” – so readers could know data details that authors analyzed.

4.      Authors should have a section mentioned the “Limitation of the Study”

Author Response

Response to reviewer 3: We thank the reviewer for their helpful comments.

Comment #1: “Too few information on “published PFS” in the section of “Materials and Methods” (Line 77).  Hard for readers to understand the theory and rationale of analysis method. 

We have revised the first paragraph of the Methods section to say: “We identified relevant papers through PubMed. For a publication to be included in our analyses, it had to present a Kaplan-Meier PFS curve, relapse-free survival curve or disease-free survival curve. Below, we use “PFS” to also refer to relapse-free and disease-free survival.  As previously described [2,4,10,11], we used the application https://apps.automeris.io/wpd/ to digitize the PFS curves from these publications.”

Comment #2: “What journals?”

We included any journals included in PubMed.

Comment #3: “What years of data?”

The adjuvant chemotherapy data included the trials from the LACE meta-analysis and published 2006-2010. The ADAURA trial was published in 2020. As outlined in the first paragraph of the Results section, we assessed chemoradiotherapy trials published from 2010 to February 2022 and limited SCLC trials published from 1990 to 2021.

Comment #4: “Could this method be adjusted for important confounders like race-ethnicity, gender, age, cancer staging etc.?”

As we have noted in the Limitations section, “Using individual patient data could facilitate assessment of role of dose reductions, therapy interruptions and other factors on curve convexity. It could also facilitate assessment of impact on outcomes of various patient and biological characteristics, such as tumor size, tumor differentiation, PD-L1 expression, tumor mutations, gender, age, race/ethnicity, etc. Methods such as nonlinear mixed effects modeling [91] might prove useful in this. This could facilitate increased personalization of optimal scan frequency, prediction of probability of future relapse, etc.”

Comment #5: “Without the important information of the source data – and adjusted for it – any conclusions generated may be very biased.”

We have added the statement in the Results section, “Note that for this outcome and for outcomes below, we combined data from trials that included patients with different characteristics and treatment details. These differences across trials will have impacted outcomes and will have contributed to the heterogeneity seen across trials. Future trials using individual patient data and/or more homogeneous populations would improve reliability of conclusions.”

Comment #6: “Very few citations for each of the reference 2-4 and 10-11. “Population survival kinetics analyses” maybe can be performed easily and rapidly – but must have some fundamental fraud in science to cause it was not well-accept in the real world.”

Because population survival kinetics analyses have some potential advantages, we anticipate that there will be increased interest in it as people gain familiarity.  We think that our inclusion of a tutorial on the method will make it easier for others to use this approach. In addition, since we have used published data and readily available tools, anyone who wishes to assess the validity of our results is readily able to do so. We have also applied to conduct a workshop on the method at the World Conference on Lung Cancer, San Diego, September 2024, but have not yet heard if our request has been successful.

Comment #7: “ “3.1. Studies assessed” Line 103-110 and 114-115 should be in the “Materials and Methods” – so readers could know data details that authors analyzed.”

We have added the following as the second sentence in the Methods section: “The publications included in our analyses are noted at the beginning of the “Results” section.”

Comment #8: “Authors should have a section mentioned the “Limitation of the Study”.”

We have incorporated some of our discussion on limitations into a specific section, as follows: ”Limitations: Our proposed population survival kinetics methods have some potential limitations. As noted previously, early PFS curve convexities could result in overestimation or underestimation of the proportion of patients progressing at 3 months but would be expected to have less of an impact on estimates of progression at later time points. In addition, based on the long PFS half-lives for the adjuvant trial controls and chemoradiotherapy and SCLC groups, we coded the size of the potentially cured subpopulations as being constant. This somewhat overestimates the size of this subpopulation at later times since, even in the absence of relapse, at least some patients would die from other causes.

As noted previously, the method has no specific control for experimental error that might be intrinsic within data for a given curve. However, in the online supplementary tables, we did include the sample size for each trial to help the reader assess the strength of the evidence.

We also recognize that conclusions from across-study comparisons (eg, with respect to induction or consolidation chemotherapy with radiotherapy) have to be interpreted cautiously.

Furthermore, we have not yet tested this methodology with individual patient data and real-world evidence. Using individual patient data could facilitate assessment of role of dose reductions, therapy interruptions and other factors on curve convexity. It could also facilitate assessment of impact on outcomes of various patient and biological characteristics, such as tumor size, tumor differentiation, PD-L1 expression, tumor mutations, gender, age, race/ethnicity, etc. Methods such as nonlinear mixed effects modeling [91] might prove useful in this. This could facilitate increased personalization of optimal scan frequency, prediction of probability of future relapse, etc.

Another limitation is that short patient follow-up is as limiting for these analyses as for any other approach used in assessing clinical trial data. In our 2-phase decay analyses, 95% confidence intervals were generally narrow for proportion of patients in the relapsing subpopulation, but they were typically very wide or not calculable for PFS half-lives for potentially cured subpopulations. As would be expected, more mature data with longer patient follow-up were associated with a greater probability that 95% confidence intervals could be calculated.

Low patient numbers at the terminal portion of a survival curve means that loss of one patient might lead to a large drop in the curve. Conversely, presence of a single long survivor might impact assessment by producing a very long tail on the curve. In some of our earlier analyses [2,4,10], we truncated curves if there were less than an estimated 10 remaining patients. We have not yet adequately assessed the impact of this truncation, but in preliminary assessments, it generally had minimal impact on the overall curve half-life but did somewhat reduce the probability of a curve fitting 2-phase decay models (D. Stewart, unpublished data). Our requirement that each subpopulation had to be >1% of the entire population for a curve to be designated as fitting a 2-phase decay model also means that the method would miss very small favorable subpopulations.

As with any methodology, there is at least some risk of incorrect conclusions due to misinterpretation of data. As much caution is required in assessment of results using these methods as with any analytical method. One advantage of this approach is that these analyses can be reassessed easily by others since they use accessible published data and readily available analytical tools.”

Reviewer 4 Report

Comments and Suggestions for Authors

I would like to congratulate the authors of the interesting article titled “Population survival kinetics derived from clinical trials of potentially curable lung cancers.” The abstract summarizes the content of the article well. In the introduction, the authors summarize the current state of knowledge, identify knowledge gaps and formulate the objectives of the study. The methodology adopted by the authors is appropriate. The results are interesting and well presented. In the discussion, the authors relate the obtained results to the current literature. The conclusions are interesting and correspond to the results obtained. The paper is written in very good quality English. The content of the article is interesting and adds new information to current medical knowledge. Once again, congratulations on a well-planned study and a well-written article.

Author Response

Reviewer #4.

We would like to thank the reviewer for their very kind comments.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

All issues have been addressed from my persepective, its a very interesting approach and will (i hope) be backed up in the future with direct patient / real world data to support or disprove the conclusions drawn. Look forward to that

Author Response

Thank you for your kind comments.

Reviewer 2 Report

Comments and Suggestions for Authors

Dear Authors, 

The article is well written, you have done an excellent job.

I would just like to suggest moving lines 126 to 140 in Materials and Methods.

Author Response

Thank you for your kind comments.

As recommended, the information that had been in section 3.1 on studies assessed has been moved up into the Materials and Methods section.  The different sections under results have been renumbered.

Reviewer 3 Report

Comments and Suggestions for Authors

Add the limitation section making the manuscript more complete.

Author Response

Thank you for your helpful suggestions. Section 5.0 on limitations has been added.