Review Reports - Multivariate Associations Between R-PAS Variables and Suicidal Ideation and Behaviors in Help-Seeking Adolescents

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The authors have conducted a cross-sectional study to evaluate the contribution of the Rorschach Performance Assessment System (R-PAS) in the assessment of suicidal ideation/behaviour in adolescents who seek help. The authors’ main findings are that elastic net models identified three R-PAS variables as “consistent multivariate correlates of suicidal ideation severity and lifetime suicidal behaviors”; and “R-PAS captures a coherent psychological profile beyond self-reports”.

1. While the title and highlights suggest that the R-PAS makes a positive contribution in the assessment of suicidality, the actual reported results support a different conclusion. As the authors themselves state, “Findings do not support the use of R-PAS variables as suicide risk predictors …” Again, “Model performance indicated modest explanatory capacity …” I also note that the AUC for suicidal behaviour was just 0.66 (for the optimal model), while the r-squared coefficient of determination for suicidal ideation was merely 0.09. An AUC of 0.66 is barely better than chance and would be considered poor for any risk stratification tool. This, together with the very small value of r-squared, are clinically negligible.

2. The risk profiles, presentation and completion rates in adolescents are known to show a marked sex difference. However, the present study has a marked sex bias, with only 14% of subjects being male. A model derived from a sample which is 86% female should not be generalised to male adolescents; this is a major flaw for a tool intended for general (both male and female) adolescent assessment.

3. The integrity of the data is questionable. I am particularly concerned by the fact that over half of the CFC-FC index data are missing (79 missing values out of 153 participants). How did over half the data for a specific R-PAS variable go missing? Was this caused by administrative errors, or scoring difficulties or data recording failures? This raises concerns about the overall quality control of the R-PAS coding in this study.

4. It is not sufficient to correlate the R-PAS variables with C-SSRS scores (concurrent validity). This does not demonstrate that the R-PAS adds predictive value above and beyond standard clinical assessment. To show that R-PAS “complements” C-SSRS scores, it must be demonstrated that adding R-PAS variables to a C-SSRS-containing model (or a model containing clinical interview data) significantly improves prediction. What the authors have attempted is a demonstration of correlation rather than incremental utility. Given the low r-squared coefficient (see above), it is unclear if the R-PAS adds anything that a clinician could not glean from the C-SSRS alone.

5. The cross-sectional design invalidates the core premise of using the tool for assessing suicidal ideation and behaviours in a predictive sense.

6. There is no mention of pre-registration of this study. Given that the authors tested a large number of R-PAS variables and used a penalised regression model to select the “best” predictors, the lack of pre-registration raises the serious issue of analytical flexibility. Without a pre-registered plan, how can one distinguish between confirmatory hypotheses and exploratory data mining?

7. The absence of a power calculation or a clear rationale for the sample size is an important methodological flaw, particularly in light of the authors’ claims.

8. One of the authors is disclosed as a member of the LLC which owns the rights to the R-PAS. This creates an inherent incentive to find positive results for the R-PAS. Given the weak results (see the AUC and r-squared results, above), the pressure to spin these findings as clinically useful is a concern.

Author Response

While the title and highlights suggest that the R-PAS makes a positive contribution in the assessment of suicidality, the actual reported results support a different conclusion. As the authors themselves state, “Findings do not support the use of R-PAS variables as suicide risk predictors …” Again, “Model performance indicated modest explanatory capacity …” I also note that the AUC for suicidal behaviour was just 0.66 (for the optimal model), while the r-squared coefficient of determination for suicidal ideation was merely 0.09. An AUC of 0.66 is barely better than chance and would be considered poor for any risk stratification tool. This, together with the very small value of r-squared, are clinically negligible.

Response: We thank the reviewer for this observation. We agree that the model performance indices would not support the use of R-PAS variables as standalone tools for suicide risk stratification. We would like to clarify that the aim of the study was not to develop or validate a predictive model, but rather to explore multivariate patterns of association between R-PAS variables and clinically assessed suicidal ideation and behaviour. However, we acknowledge that aspects of the original title, highlights, and interpretation may have conveyed an overly strong implication of clinical contribution or predictive utility. In response to the reviewer’s comment, we have revised the manuscript to ensure alignment among the study aims, results, and interpretation.

The risk profiles, presentation and completion rates in adolescents are known to show a marked sex difference. However, the present study has a marked sex bias, with only 14% of subjects being male. A model derived from a sample which is 86% female should not be generalised to male adolescents; this is a major flaw for a tool intended for general (both male and female) adolescent assessment.

Response: We thank the reviewer for this important observation. We agree that the predominance of female participants represents a limitation with respect to generalizability, particularly given known sex differences in adolescent suicidality. To address this concern, we conducted exploratory analyses comparing males and females across sociodemographic, clinical, suicidality, and R-PAS variables. As reported in Supplementary Table S2, no statistically significant differences were observed between genders in suicidal ideation severity, suicidal behavior indicators, or R-PAS variables. These findings suggest that, within the present sample, the observed multivariate associations are unlikely to be driven by gender-related differences. However, we acknowledge that the relatively small number of male participants limits statistical power to detect subtle gender-specific effects. Accordingly, we have revised the manuscript to explicitly state that findings should be interpreted primarily in the context of predominantly female help-seeking adolescents and require replication in more balanced samples before generalization to male populations.

The integrity of the data is questionable. I am particularly concerned by the fact that over half of the CFC-FC index data are missing (79 missing values out of 153 participants). How did over half the data for a specific R-PAS variable go missing? Was this caused by administrative errors, or scoring difficulties or data recording failures? This raises concerns about the overall quality control of the R-PAS coding in this study.

Response: We thank the reviewer for raising this point. We checked the dataset and found an issue with the column names. That allowed us to recover the variables that we thought were missing. We updated the manuscript and the results accordingly. Importantly, the inclusion of the complete CFC–FC variable did not alter the overall pattern of findings.

It is not sufficient to correlate the R-PAS variables with C-SSRS scores (concurrent validity). This does not demonstrate that the R-PAS adds predictive value above and beyond standard clinical assessment. To show that R-PAS “complements” C-SSRS scores, it must be demonstrated that adding R-PAS variables to a C-SSRS-containing model (or a model containing clinical interview data) significantly improves prediction. What the authors have attempted is a demonstration of correlation rather than incremental utility. Given the low r-squared coefficient (see above), it is unclear if the R-PAS adds anything that a clinician could not glean from the C-SSRS alone.

Response: We thank the reviewer for this methodological observation. We agree that the present study does not allow conclusions regarding the incremental or predictive value of R-PAS variables beyond established clinical assessment tools such as the C-SSRS. Our analyses were designed to examine multivariate patterns of association between R-PAS variables and clinically assessed suicidal ideation and behaviour, and therefore reflect concurrent, cross-sectional relationships rather than incremental predictive validity. In response to this comment, we have revised the manuscript to remove any statements suggesting that R-PAS “complements” or adds clinical utility beyond the C-SSRS. We now explicitly clarify that the findings are limited to exploratory associations and do not establish incremental validity or practical advantage in clinical assessment.

The cross-sectional design invalidates the core premise of using the tool for assessing suicidal ideation and behaviours in a predictive sense.

Response: We agreed with the reviewer’s comment. To make it clearer that the study has not intended to serve as predictive, we specified from the beginning that we aimed to investigate whether performance-based indicators capture statistically detectable patterns of association with clinically assessed suicidal thoughts and behaviours.

There is no mention of pre-registration of this study. Given that the authors tested a large number of R-PAS variables and used a penalised regression model to select the “best” predictors, the lack of pre-registration raises the serious issue of analytical flexibility. Without a pre-registered plan, how can one distinguish between confirmatory hypotheses and exploratory data mining?

Response: We thank the reviewer for raising this important methodological point. We acknowledge that the study was not preregistered, and therefore the analyses should be interpreted within an exploratory framework rather than as confirmatory hypothesis testing. To address this concern, we have revised the manuscript to more clearly position the study as exploratory. In particular, we emphasize that the use of elastic net regression was intended to identify multivariate patterns of association in the presence of potentially correlated variables, rather than to test predefined hypotheses or establish predictive models.

The absence of a power calculation or a clear rationale for the sample size is an important methodological flaw, particularly in light of the authors’ claims.

Response: We thank the reviewer for the comment. We acknowledge that the study did not include an a priori power calculation or formal sample size determination. The sample size was determined by the availability of participants within the clinical setting during the study period and therefore reflects a convenience sample rather than a predetermined target based on statistical power considerations. We agree that this represents a limitation, and the use of penalized regression with cross-validation was intended to partially mitigate overfitting in the context of a relatively modest sample size; however, we recognize that this does not substitute for an adequately powered design.

One of the authors is disclosed as a member of the LLC which owns the rights to the R-PAS. This creates an inherent incentive to find positive results for the R-PAS. Given the weak results (see the AUC and r-squared results, above), the pressure to spin these findings as clinically useful is a concern.

Response: We thank the reviewer for the comment regarding potential conflicts of interest. We agree that such an affiliation underscores the importance of maintaining a cautious and transparent interpretation of findings. In response, we have revised the manuscript to ensure that the results are presented in a balanced and non-interpretive manner. In particular, we have clarified that the study does not support the use of R-PAS variables as predictors of suicidal outcomes, emphasized the modest explanatory and classification performance of the models, reframed the findings as exploratory associations rather than evidence of clinical utility, and expanded the Limitations section to explicitly address potential sources of bias, including both methodological constraints and interpretive considerations.

Reviewer 2 Report

Comments and Suggestions for Authors

Thank you for the opportunity to review this manuscript. The study addressed an important and clinically relevant question regarding the potential contribution of performance-based assessment to understanding suicidal ideation and behaviors in adolescents. The topic was timely and aligned with ongoing challenges in suicide risk assessment, particularly the need to move beyond purely predictive approaches toward understanding underlying psychological processes. The manuscript demonstrated several strengths; however, there were also a number of conceptual, methodological, and interpretive issues that limited confidence in the robustness and implications of the findings.

Introduction: Introduction provided a clear overview of the epidemiology and clinical importance of adolescent suicidality. The authors highlighted the limitations of current risk prediction approaches and offered a compelling rationale for exploring performance-based assessment as a complementary method. The presentation of the R-PAS and SC-Comp framework was generally coherent and logically connected to the study aims. However, the manuscript would have been strengthened by the following revisions:

The discussion of R-PAS validity and utility was largely uncritical and did not sufficiently acknowledge ongoing debates or controversies surrounding performance-based measures.
The conceptual link between specific R-PAS variables and suicidality was presented as relatively established, despite a more mixed and context-dependent evidence base.
The distinction between psychological processes associated with suicidality and actual risk prediction was not consistently maintained throughout the section.
The rationale for selecting the specific subset of R-PAS variables (beyond SC-Comp) was not fully theoretically justified.
The introduction did not sufficiently clarify whether the study aimed to test theory, explore associations, or inform clinical practice, leading to some ambiguity in purpose.
The incremental contribution of this study relative to prior Rorschach-based research on suicidality was not clearly delineated.

Methods: Methods section demonstrated several strengths. The use of a clinically relevant help-seeking sample increased ecological validity, and the inclusion of standardized instruments such as the C-SSRS provided a solid clinical anchor. The use of elastic net regression to address multicollinearity and model multivariate patterns was appropriate and represented a thoughtful analytic choice. The reporting of interrater reliability for R-PAS coding was also a notable strength. However, the manuscript would have been strengthened by the following revisions:

The sample was heavily skewed toward females, yet this imbalance was not addressed in sampling strategy, analysis, or interpretation.
The clinical composition of the sample (e.g., diagnoses, severity, comorbidities) was insufficiently described, limiting interpretability and generalizability.
The recruitment process and potential selection biases (e.g., help-seeking population, consent-based participation) were not critically examined.
The cross-sectional design was appropriate for exploratory purposes but was not explicitly framed as limiting causal or temporal interpretations in the design section.
The operationalization of suicidal ideation as a 0–5 ordinal scale treated as continuous may have introduced measurement limitations that were not justified.
The dichotomization of suicidal behavior obscured differences between types, frequency, and severity of behaviors.
The high level of missing data for the CFC–FC variable (n = 79) was substantial and not adequately addressed beyond exclusion from analyses.
The rationale for including additional exploratory variables (M–, WSumCog) alongside SC-Comp variables lacked clear theoretical integration.
Details regarding the stability of the elastic net models (e.g., variability across folds, sensitivity to parameter selection) were not provided.
No external validation or replication strategy was included, increasing the risk of overfitting.
Potential confounders beyond age and sex (e.g., SES, psychiatric diagnoses, medication status) were not included in the models.
The procedures for standardizing variables and handling potential non-normality were not described.
The description of clinical supervision as a guarantee of reliability for C-SSRS administration was insufficiently detailed.
The study did not address potential shared method variance between clinician-administered tools.

Results: Results section presented the findings in a concise and organized manner. The use of penalized regression allowed for the identification of multivariate patterns rather than isolated predictors. The authors reported modest model performance and avoided overstating predictive accuracy. However, the manuscript would have been strengthened by the following revisions:

The reported model performance (R² = .09; AUC ≈ .65) indicated weak explanatory and predictive value, yet the interpretation of a “systematic multivariate signal” may have overstated the findings.
Effect sizes for retained variables were small and not consistently interpreted in terms of clinical relevance.
Confidence intervals for coefficients and odds ratios were not reported, limiting evaluation of estimate precision.
The absence of retained sociodemographic variables was not discussed, despite their potential relevance.
No sensitivity analyses were conducted to assess robustness of findings across model specifications.
The results did not explore potential interactions between R-PAS variables or between predictors and covariates.
The exclusion of variables due to missingness (e.g., CFC–FC) was not examined in terms of potential bias.
The distributional properties of outcome variables (e.g., skewness of ideation scores) were not reported.
The reliance on a single modeling approach limited triangulation of findings.
Calibration of the classification model was only briefly mentioned and not fully interpreted.
The distinction between statistical significance and practical/clinical significance was not clearly articulated.
There was no comparison with simpler models (e.g., baseline or covariate-only models) to contextualize added value.
The results did not examine whether model performance differed across subgroups (e.g., sex, severity levels).
The descriptive tables were informative but not fully integrated into the interpretation of findings.

Discussion: Discussion provided a clinically oriented interpretation of the findings particularly in linking MOR, LSO-Cmplx, and VFD to broader psychological processes such as dysphoria, cognitive overcontrol, and avoidance. The authors emphasized that R-PAS variables should not be used as standalone predictors and framed their findings within a multivariate, process-oriented perspective. The inclusion of clinical implications was a strength. However, the manuscript would have been strengthened by the following revisions:

Several interpretations (e.g., MOR as communicative intent, LSO-Cmplx as perfectionism) were speculative and not directly supported by the data.
The discussion occasionally blurred the distinction between association and mechanism, implying causal interpretations.
The modest model performance was acknowledged but not sufficiently emphasized in limiting conclusions.
The discussion did not adequately address potential shared method variance or measurement overlap.
The broader controversies surrounding R-PAS validity were not revisited in interpreting findings.
The clinical recommendations (e.g., intervention targets) extended beyond what the data could support.
The role of unmeasured confounders (e.g., diagnostic heterogeneity) was not integrated into interpretation.
The absence of longitudinal data limited implications for risk trajectories, but this was not fully emphasized.
The notion of a “coherent psychological profile” may have overstated the strength and stability of the observed associations.
The discussion did not clearly differentiate between exploratory findings and established knowledge.
The integration of findings with existing suicide theories (e.g., interpersonal theory, stress-diathesis models) was limited.
The limitations section, while present, did not fully engage with issues of sampling bias and model generalizability.
The potential clinical utility of R-PAS was suggested without comparison to existing assessment approaches.

Author Response

Comment: Thank you for the opportunity to review this manuscript. The study addressed an important and clinically relevant question regarding the potential contribution of performance-based assessment to understanding suicidal ideation and behaviors in adolescents. The topic was timely and aligned with ongoing challenges in suicide risk assessment, particularly the need to move beyond purely predictive approaches toward understanding underlying psychological processes. The manuscript demonstrated several strengths; however, there were also a number of conceptual, methodological, and interpretive issues that limited confidence in the robustness and implications of the findings.

The discussion of R-PAS validity and utility was largely uncritical and did not sufficiently acknowledge ongoing debates or controversies surrounding performance-based measures.
The conceptual link between specific R-PAS variables and suicidality was presented as relatively established, despite a more mixed and context-dependent evidence base.
The distinction between psychological processes associated with suicidality and actual risk prediction was not consistently maintained throughout the section.
The rationale for selecting the specific subset of R-PAS variables (beyond SC-Comp) was not fully theoretically justified.
The introduction did not sufficiently clarify whether the study aimed to test theory, explore associations, or inform clinical practice, leading to some ambiguity in purpose.

Response: We thank the reviewer for highlighting this lack of information. We provided information about suicidal behaviors and Rorschach variables across different methodologies and integrated the results. Unfortunately, suicidal behaviors were not systematically explored in other performance-based measures besides the Rorschach. Furthermore, we added more specific explanations for selected variables. We hope the information we included will be sufficient to address the issues raised.

Comment:

The incremental contribution of his study relative to prior Rorschach-based research on suicidality was not clearly delineated.

Response: We thank the reviewer for this important suggestion. We have clarified the contribution of the present study relative to prior Rorschach-based suicidality research in the manuscript. While previous studies have primarily examined bivariate associations between individual Rorschach variables (or S-CON–based indicators) and suicidal outcomes, the present study applies a multivariate penalized regression framework (elastic net), allowing the simultaneous evaluation of multiple R-PAS variables while addressing multicollinearity and overfitting. Second, it examines both suicidal ideation severity and suicidal behavior within the same analytical framework in a help-seeking adolescent sample, a population less represented in prior work. Third, the study incorporates bootstrap-based stability analyses to evaluate the robustness of variable selection, providing additional information on the reliability of identified multivariate patterns beyond traditional significance-based approaches.

Comment:

The sample was heavily skewed toward females, yet this imbalance was not addressed in sampling strategy, analysis, or interpretation.

Response: We thank the reviewer for this important observation. We acknowledge that the sample is predominantly female and that this may limit generalizability, particularly given known gender differences in adolescent suicidality. In the revised manuscript, we have addressed this issue at multiple levels. First, we clarified that the sample reflects a help-seeking clinical population, in which female adolescents are typically overrepresented. Second, we conducted exploratory analyses comparing males and females across sociodemographic, clinical, suicidality, and R-PAS variables. As reported in Supplementary Table S2, no statistically significant differences were observed, suggesting that the observed multivariate patterns are unlikely to be driven by sex-related differences within the present sample. Finally, we expanded the Discussion to explicitly acknowledge the limitation of the sample imbalance and to clarify that findings should be interpreted primarily in the context of predominantly female help-seeking adolescents.

Comment:

The clinical composition of the sample (e.g., diagnoses, severity, comorbidities) was insufficiently described, limiting interpretability and generalizability.

Response: We thank the reviewer for this important comment. In response, we have substantially expanded the description of the clinical composition of the sample. We agree that this level of detail is essential for interpretation and generalization. Accordingly, we now explicitly state in the manuscript that this clinical heterogeneity supports ecological validity for real-world adolescent psychiatric assessment contexts, while also limiting generalization to non-clinical populations

Comment:

The recruitment process and potential selection biases (e.g., help-seeking population, consent-based participation) were not critically examined.

Response: We thank the reviewer for this important observation. We have expanded the description of the recruitment process and added a more critical discussion of potential selection effects. Specifically, we clarify that the sample was drawn from a help-seeking clinical population, which may limit generalizability to non-clinical or community adolescents. In addition, participation was based on informed consent, which may introduce self-selection bias, as individuals or families consenting to participation may differ systematically from those who declined (e.g., in terms of symptom severity, treatment engagement, or willingness to undergo psychological assessment).

Comment:

The cross-sectional design was appropriate for exploratory purposes but was not explicitly framed as limiting causal or temporal interpretations in the design section.

Response: We thank the reviewer for this observation. We agree that the limitations inherent to a cross-sectional design should be stated more explicitly. Accordingly, we have revised the Methods section to clarify the inferential scope of the study design. Specifically, we now state that the cross-sectional design allows for the examination of contemporaneous associations but does not permit causal, temporal, or predictive inferences regarding the relationships between R-PAS variables and suicidal outcomes. In addition, we have expanded the Limitations section to further emphasize that the identified associations should not be interpreted as reflecting causal mechanisms or risk trajectories, and that longitudinal research is required to evaluate directionality and predictive utility.

Comment:

The operationalization of suicidal ideation as a 0–5 ordinal scale treated as continuous may have introduced measurement limitations that were not justified.

Response: We appreciated that the reviewer raised that point. Treating suicidal ideation as a continuous variable was intended to preserve information on severity gradients, which are clinically meaningful, and to avoid the loss of variability associated with categorization. This approach prioritizes sensitivity to differences in ideation severity, although it assumes approximate continuity between ordinal levels. However, we have expanded the Limitation section to deepen this point.

Comment:

The dichotomization of suicidal behavior obscured differences between types, frequency, and severity of behaviors.

Response: We thank the reviewer for this insightful comment. We agree that dichotomizing suicidal behavior reduces the granularity of the data and may obscure differences in type, frequency, and severity of behaviors. In the present study, suicidal behavior was operationalized as the presence versus absence of any lifetime suicidal behavior in accordance with the broader behavioral framework of the C-SSRS. This approach was chosen to capture a clinically meaningful threshold of behavioral involvement while maintaining sufficient statistical stability. We acknowledge that this operationalization prioritizes sensitivity to the occurrence of suicidal behavior but comes at the cost of reduced specificity and loss of information regarding heterogeneity. We have expanded the Limitations to explicitly address this issue.

Comment:

The high level of missing data for the CFC–FC variable (n = 79) was substantial and not adequately addressed beyond exclusion from analyses.

Response: We thank the reviewer for the comment. We checked the dataset and found an issue with the column names. That allowed us to recover the variables that we thought were missing. We updated the manuscript and the results accordingly. Importantly, the inclusion of the complete CFC–FC variable did not alter the overall pattern of findings.

Comment:

The rationale for including additional exploratory variables (M–, WSumCog) alongside SC-Comp variables lacked clear theoretical integration.

Response: We thank the reviewer for this request for clarification regarding the inclusion of M– and WSumCog alongside SC-Comp variables. The decision to include these indices was guided by their established relevance to cognitive-perceptual disruption and thought disorganization, which are theoretically and empirically linked to severe psychopathological states, including suicidality. In particular, prior work by Silberg and Armstrong (1992) has highlighted the utility of Rorschach-derived indices in adolescent populations, suggesting that disruptions in thought processes and cognitive integration may be particularly relevant in developmental samples. More broadly, WSumCog and M– capture dimensions of cognitive inefficiency and perceptual-cognitive distortion that are not fully encompassed by SC-Comp variables. We have revised the manuscript to clarify this complementary, exploratory rationale and to explicitly distinguish these indices as theoretically adjacent but not redundant to SC-Comp.

Comment:

Details regarding the stability of the elastic net models (e.g., variability across folds, sensitivity to parameter selection) were not provided.

Response: We thank the reviewer for this methodological suggestion. We agree that reporting model stability and sensitivity to parameter selection is essential in penalized regression frameworks. In the present study, we reported that model selection was based on 10-fold cross-validation within an elastic net framework, with systematic variation of the mixing parameter (α = 0.1–1.0). For each α, both λ.min and λ.1se were evaluated, and model performance was assessed across folds to reduce dependency on a single data split. To further address model stability, we have reported the bootstrap stability selection analyses (500 resamples), estimating the frequency with which each variable was selected across resampled datasets. These analyses provide an estimate of the robustness of variable selection under resampling variability and have been added to the revised manuscript.

Comment:

No external validation or replication strategy was included, increasing the risk of overfitting.

Response: We thank the reviewer for raising the important issue of model generalizability. We agree that the absence of external validation limits the ability to assess out-of-sample generalizability and increases the risk of overfitting. To mitigate this limitation, we implemented repeated 10-fold cross-validation during model training and applied regularization to reduce model complexity and the risk of overfitting. In addition, we conducted bootstrap stability selection analyses to evaluate the robustness of predictor selection across resampled datasets.

Comment:

Potential confounders beyond age and sex (e.g., SES, psychiatric diagnoses, medication status) were not included in the models.

Response: We thank the reviewer for this important suggestion. In the revised analyses, we extended the set of variables to include relevant clinical and sociodemographic variables (SES, psychiatric diagnoses), in addition to age and sex at birth, within the elastic net framework. Although covariates were included in the penalized regression framework, elastic net does not distinguish between “covariates” and “predictors”; all variables were treated symmetrically as candidate features and were subject to shrinkage and selection. Given the relatively small sample size and the risk of overfitting with highly correlated clinical variables, we adopted a penalized regression approach to allow simultaneous inclusion of these variables while controlling model complexity through regularization. This strategy enabled evaluation of whether R-PAS variables retained incremental associations with suicidality above and beyond clinically relevant background factors. Results indicated that the core R-PAS predictors remained stable even after inclusion of these covariates, supporting the robustness of the identified multivariate pattern.

Comment:

The procedures for standardizing variables and handling potential non-normality were not described.

Response: We thank the reviewer for highlighting the need for clarification regarding data preprocessing procedures. All variables were centered and scaled to unit variance. This procedure ensured comparability across variables, particularly important in elastic net regression. With respect to distributional properties, elastic net regression does not require strict multivariate normality assumptions, and the combination of penalization and cross-validation provides robustness to deviations from normality. We have updated the manuscript to address this comment.

Comment:

The description of clinical supervision as a guarantee of reliability for C-SSRS administration was insufficiently detailed.

Response: We thank the reviewer for this comment. We enlarged the Methods section to address this comment.

Comment:

The study did not address potential shared method variance between clinician-administered tools.

Response: We thank the reviewer for this methodological observation. We agree that the potential influence of shared method variance should be considered when interpreting associations between clinician-administered measures. Even if both the C-SSRS and R-PAS involve clinician administration, they differ substantially in their method characteristics. The C-SSRS is a semi-structured clinical interview based on patient self-report, whereas R-PAS is a performance-based assessment with standardized administration and coding procedures. As such, the degree of shared method variance may be more limited than in studies relying exclusively on self-report or interview-based measures.

Comment:

The reported model performance (R² = .09; AUC ≈ .65) indicated weak explanatory and predictive value, yet the interpretation of a “systematic multivariate signal” may have overstated the findings.

Response: We thank the reviewer for this clarification. We agree that the terms “systematic multivariate signal” may have implied a stronger interpretation than warranted by the magnitude of the effects. The model performance must be interpreted as indicating limited explanatory and discriminative ability. We have revised the manuscript to ensure that the results are described in more neutral terms and that no clinical inference beyond the observed effect sizes is implied. The revised text emphasizes the exploratory nature of the findings.

Comment:

Effect sizes for retained variables were small and not consistently interpreted in terms of clinical relevance.

Response: We appreciated the comment. We agreed that, from a clinical perspective, the observed effect sizes do not support the use of individual R-PAS variables for risk stratification but may reflect subtle psychological processes relevant to a broader integrative assessment framework. We stressed that in the manuscript.

Comment:

Confidence intervals for coefficients and odds ratios were not reported, limiting evaluation of estimate precision.

Response: Thank you for the comment. Given the penalized nature of elastic net regression, traditional CIs for coefficients were not computed. Instead, uncertainty was assessed using bootstrap stability analyses and cross-validation–based performance metrics. We updated the proper sections to highlight those analyses.

Comment:

The absence of retained sociodemographic variables was not discussed, despite their potential relevance.

Response: We agree that sociodemographic variables may play a relevant role in adolescent suicidality. In the present analyses, age, sex, and socioeconomic status were included in the elastic net models alongside R-PAS variables. However, none of these sociodemographic variables were retained after penalization in the final models for either suicidal ideation severity or suicidal behavior. We have now clarified this finding in the Results and expanded the Discussion to explicitly note that, within a multivariate penalized framework accounting for shared variance with psychological and cognitive R-PAS indices, sociodemographic variables did not contribute. This does not imply absence of association at the univariate level, but rather suggests limited unique contribution when considered jointly with R-PAS variables in this sample.

Comment:

No sensitivity analyses were conducted to assess robustness of findings across model specifications.

Response: We thank the reviewer for raising that point. In response, we have expanded the analytic strategy to include sensitivity and robustness checks. Specifically, we conducted bootstrap resampling analyses (500 iterations) for both outcomes, re-estimating the elastic net models across resampled datasets and recording variable selection frequencies and coefficient distributions. In addition, we evaluated both λ.min and λ.1se solutions to assess the stability of variable selection under different levels of regularization. Results were consistent in identifying the same core variables, with expected variation in secondary variables under more stringent penalization.

Comment:

The results did not explore potential interactions between R-PAS variables or between predictors and covariates.

Response: We thank the reviewer for this suggestion. We agree that interaction effects may be theoretically relevant in complex psychological models. However, given the relatively modest sample size and the high dimensionality of the variables, explicitly modeling interaction terms would substantially increase model complexity and the risk of overfitting, particularly in the context of penalized regression. Elastic net models partially accommodate collinearity and shared variance among variables, but do not explicitly estimate interaction effects unless pre-specified.

Comment:

The exclusion of variables due to missingness (e.g., CFC–FC) was not examined in terms of potential bias.

Comment:

The distributional properties of outcome variables (e.g., skewness of ideation scores) were not reported.

Response: We thank the reviewer for raising this point. We updated the descriptive table and the statistical analysis section to address this comment. The distribution of suicidal ideation scores showed mild negative skewness, indicating a slight concentration of higher severity values, but no substantial deviation from symmetry.

Comment:

The reliance on a single modeling approach limited triangulation of findings.

Response: We thank the reviewer for this important observation. We agree that the use of a single modeling approach may limit the triangulation and robustness of the findings. The choice of elastic net regression was motivated by the need to model multivariate associations in the presence of potentially correlated variables and a relatively modest sample size. This approach allowed us to identify patterns of association while reducing the risk of overfitting through penalization and cross-validation. At the same time, we acknowledge that reliance on a single analytic framework may limit the assessment of the stability of the results across different modeling approaches and agreed that future research should examine the robustness of these associations using alternative modeling strategies and independent samples.

Comment:

Calibration of the classification model was only briefly mentioned and not fully interpreted.

Response: We agree with the reviewer, and to address this comment, we further discussed the model's calibration and the meaning of the result.

Comment:

The distinction between statistical significance and practical/clinical significance was not clearly articulated.

Response: Thank you for this comment. We have included clinical observations wherever possible while striving to remain consistent with the findings.

Comment:

There was no comparison with simpler models (e.g., baseline or covariate-only models) to contextualize added value.

Response: We thank the reviewer for this methodological suggestion. We agree that comparisons with simpler or baseline models would provide useful context for evaluating the relative contribution of R-PAS variables. The present study was designed as an exploratory analysis aimed at identifying multivariate patterns of association among R-PAS variables rather than formally testing incremental or comparative model performance. Accordingly, baseline model comparisons were not included in the original analytic plan.

Comment:

The results did not examine whether model performance differed across subgroups (e.g., sex, severity levels).

Response: We thank the reviewer for this insightful suggestion. In response, we conducted additional exploratory subgroup analyses. Specifically, we examined the distribution of variables and outcomes across males and females (Supplementary Table S2). These analyses did not reveal significant differences in suicidal ideation severity, suicidal behavior, or R-PAS variables between genders. Given the limited number of male participants, formal sex-stratified elastic net modeling or subgroup-specific performance comparisons were not statistically reliable and risked producing unstable estimates.

Comment:

The descriptive tables were informative but not fully integrated into the interpretation of findings.

Response: We thank the reviewer for this useful comment. We have revised the Results and Discussion sections to improve integration between descriptive statistics and inferential findings. In particular, we now more explicitly link sample characteristics (Table 1) and R-PAS distributions (Table 2) to the interpretation of the elastic net models.

Comment:

Several interpretations (e.g., MOR as communicative intent, LSO-Cmplx as perfectionism) were speculative and not directly supported by the data.
The discussion occasionally blurred the distinction between association and mechanism, implying causal interpretations.
The modest model performance was acknowledged but not sufficiently emphasized in limiting conclusions.
The discussion did not adequately address potential shared method variance or measurement overlap.
The broader controversies surrounding R-PAS validity were not revisited in interpreting findings.

Response: We are aware of the controversies surrounding the validity and reliability of the Rorschach test. However, we recognize that these controversies have primarily concerned the CS. We chose the R-PAS because it is the most well-validated performance-based instrument in the literature. Additionally, we did not further explore this debate because doing so would result in more self-citations.

Comment:

The clinical recommendations (e.g., intervention targets) extended beyond what the data could support.

Response: Thank you for your comment. We have revised our clinical considerations to align them as closely as possible with the results.

Comment:

The role of unmeasured confounders (e.g., diagnostic heterogeneity) was not integrated into interpretation.
The absence of longitudinal data limited implications for risk trajectories, but this was not fully emphasized.

Response: We thank the reviewer for raising that point. We updated the Limitations section accordingly.

Comment:

The notion of a “coherent psychological profile” may have overstated the strength and stability of the observed associations.

Response: We agree with the reviewer and have deeply revised the manuscript to avoid overstatements linked to “psychological profile” considerations.

Comment:

The discussion did not clearly differentiate between exploratory findings and established knowledge.
The integration of findings with existing suicide theories (e.g., interpersonal theory, stress-diathesis models) was limited.

Response: We agreed with the reviewer and enlarged the Discussion accordingly.

Comment:

The limitations section, while present, did not fully engage with issues of sampling bias and model generalizability.

Response: We thank the reviewer for this comment. We enlarged the Limitations section to disclose sampling bias and model generalizability.

Comment:

The potential clinical utility of R-PAS was suggested without comparison to existing assessment approaches.

Response: We expanded the Discussion and Conclusion to address this comment

Reviewer 3 Report

Comments and Suggestions for Authors

This manuscript addresses an important and clinically relevant issue, namely the assessment of suicidal ideation and behaviors in adolescents using performance-based measures such as the Rorschach Performance Assessment System (R-PAS). The study is well-positioned within current debates on the limits of self-report measures and the need for complementary approaches in suicide risk assessment.

The use of elastic net regression to explore multivariate associations is appropriate and represents a methodological strength. The findings are cautiously interpreted, and the authors correctly avoid overclaiming predictive utility, instead emphasizing the potential descriptive and clinical value of R-PAS variables.

However, several aspects require clarification and improvement.

First, the introduction would benefit from a more comprehensive and updated review of the literature. In particular, the authors should better contextualize the role of performance-based assessment within contemporary suicide risk frameworks, including dimensional and transdiagnostic approaches. The theoretical link between R-PAS variables and suicidality (e.g., MOR, LSO-Cmplx) could be more explicitly developed.

Second, the methods section, although generally clear, would benefit from additional detail. Specifically:

More information is needed on sample characteristics (e.g., recruitment setting, diagnostic composition, treatment status).

The rationale for selecting specific R-PAS variables (SC-Comp and developmental indices) should be further justified.

It would be helpful to clarify how missing data (if any) were handled.

Additional details on model tuning and validation procedures (e.g., cross-validation strategy) would strengthen reproducibility.

Third, the results are clearly presented, but the clinical interpretation could be expanded. The manuscript would benefit from a more explicit discussion of how the identified profile (dysphoric ideation, cognitive strain, disengagement) may inform clinical decision-making beyond risk prediction.

Fourth, the discussion is balanced but could be strengthened by:

A deeper integration with existing literature on adolescent suicidality and cognitive-emotional dysregulation.

A clearer distinction between explanatory and predictive models.

A more explicit consideration of the limitations of performance-based measures in clinical settings (e.g., time, training, ecological validity).

Finally, the limitations section should be expanded. In particular, the modest model performance deserves further reflection, including implications for clinical utility and generalizability.

Author Response

Comment: The use of elastic net regression to explore multivariate associations is appropriate and represents a methodological strength. The findings are cautiously interpreted, and the authors correctly avoid overclaiming predictive utility, instead emphasizing the potential descriptive and clinical value of R-PAS variables. However, several aspects require clarification and improvement.

Response: We thank the reviewer for highlighting this lack of information. We provided information from a narrative review that examined suicidal behaviors in Rorschach across different methodologies and integrated the results.

Comment: Second, the methods section, although generally clear, would benefit from additional detail. Specifically:

More information is needed on sample characteristics (e.g., recruitment setting, diagnostic composition, treatment status).

Response: We thank the reviewer for this helpful suggestion. We agree that additional detail on sample characteristics would improve the clarity and interpretability of the study. Accordingly, we have expanded the Methods section to provide a more comprehensive description of the recruitment setting and assessment procedures, and the Results section providing more data concerning diagnoses and medications.

Comment: The rationale for selecting specific R-PAS variables (SC-Comp and developmental indices) should be further justified.

Response: We better explained the rationale behind selecting the variables. We hope the information we included will be sufficient to address the issues raised.

Comment: It would be helpful to clarify how missing data (if any) were handled.

Response: We thank the reviewer for this suggestion. We have revised the Methods section to provide a clearer description of missing data handling procedures. Specifically, participants with missing data on the primary study instruments (i.e., C-SSRS and R-PAS protocols) were excluded from the analytic sample, as illustrated in the participant flow diagram (Figure 1), ensuring that all analyses were conducted on complete cases for the primary measures. Regarding the CFC–FC variable, we checked the dataset and found an issue with the column names. That allowed us to recover the variables that we thought were missing. We updated the manuscript and the results accordingly. Importantly, the inclusion of the complete CFC–FC variable did not alter the overall pattern of findings.

Comment: Additional details on model tuning and validation procedures (e.g., cross-validation strategy) would strengthen reproducibility.

Response: We thank the reviewer for this suggestion. We have now expanded the description of model tuning and validation procedures to improve reproducibility. Specifically, elastic net models were estimated using 10-fold cross-validation, with systematic evaluation of a grid of mixing parameters (α = 0.1–1.0). For each α, model performance was assessed using cross-validated mean squared error (for suicidal ideation) and cross-validated AUC (for suicidal behavior). The final models were selected based on optimal cross-validated performance and refitted using the corresponding λ.min. In addition, λ.1se solutions were inspected to evaluate model robustness under more regularized conditions. To further enhance transparency, bootstrap stability analyses (500 resamples) were reported to assess the consistency of variable selection across resampled datasets.

Comment: Third, the results are clearly presented, but the clinical interpretation could be expanded. The manuscript would benefit from a more explicit discussion of how the identified profile (dysphoric ideation, cognitive strain, disengagement) may inform clinical decision-making beyond risk prediction.

Response: Thank you for your comment. We have revised our clinical considerations to align them as closely as possible with the results.

Comment: Fourth, the discussion is balanced but could be strengthened by:

A deeper integration with existing literature on adolescent suicidality and cognitive-emotional dysregulation.

Response: we thank the reviewer for the comment and updated the manuscript with more literature to address this suggestion.

Comment: A clearer distinction between explanatory and predictive models.

Response: Thank you for your comment. We have clarified the objectives of our study in the hope that this addresses your concern.

Comment: A more explicit consideration of the limitations of performance-based measures in clinical settings (e.g., time, training, ecological validity).

Response: We thank the reviewer for highlighting this interesting and important point. We agree that performance-based measures such as the R-PAS involve practical constraints that may limit their scalability in routine clinical settings, including administration time, scoring complexity, and the need for access to the R-PAS website for scoring and for specific training to ensure coding reliability. We have added an explicit statement in the Discussion acknowledging these limitations, as well as considerations regarding ecological validity in acute and time-constrained clinical contexts. This clarification helps to better situate the findings within real-world implementation constraints.

Comment: Finally, the limitations section should be expanded. In particular, the modest model performance deserves further reflection, including implications for clinical utility and generalizability.

Response: We thank the reviewer for the comment and have addressed the suggestion by expanding the limitations section.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

I thank the authors for making major amendments to their submission. Unfortunately, the "Highlights" and Abstract (particularly the Conclusions) still over-sell the R-PAS. Furthermore, the key concerns from my previous review still apply, for example in respect of the lack of pre-registration, the lack of a power calculation and the cross-sectional design. I am pleased, however, to learn that the authors were able to recover some of the previously missing data.

Author Response

Comment: I thank the authors for making major amendments to their submission. Unfortunately, the "Highlights" and Abstract (particularly the Conclusions) still over-sell the R-PAS. Furthermore, the key concerns from my previous review still apply, for example in respect of the lack of pre-registration, the lack of a power calculation and the cross-sectional design. I am pleased, however, to learn that the authors were able to recover some of the previously missing data.

Response: We thank the reviewer for carefully re-evaluating the manuscript and for highlighting the remaining concerns regarding the Highlights and Abstract. We agree that, given the modest explanatory performance of the models, the cross-sectional design, and the absence of preregistration and an a priori power calculation, the findings must be interpreted strictly within an exploratory framework. In this revision, we have substantially rewritten the Highlights and the Abstract to ensure that the study is explicitly framed as exploratory; any language implying predictive, incremental, or clinical utility has been removed.

Reviewer 2 Report

Comments and Suggestions for Authors

Thank you for the opportunity to review the revised version of this manuscript. The authors have made visible improvements in response to the prior feedback, and the manuscript is now clearer, more transparent, and better aligned with its exploratory aims. In particular, the study purpose is more explicitly framed as cross-sectional and non-predictive, the introduction now acknowledges debates surrounding performance-based assessment, and the sample is much better characterized through the inclusion of diagnostic, sociodemographic, and clinical information. The methodological section has been strengthened with clearer descriptions of standardization procedures, distributional properties, and model estimation, as well as the addition of bootstrap stability analyses and more detailed reporting of elastic net tuning. The authors also made a commendable effort to temper interpretive claims, acknowledge modest model performance, and expand the limitations section to include key constraints such as cross-sectional design, lack of external validation, and generalizability concerns.

Despite these improvements, several issues remain that should be addressed before publication:

The distinction between statistical associations and inferred psychological mechanisms is improved but still not consistently maintained; some interpretations (e.g., links to perfectionism, communicative intent, or specific cognitive styles) extend beyond what the data directly support.
The characterization of findings as reflecting a “coherent psychological profile” remains somewhat overstated given the modest effect sizes, low explanatory power (R² = .09), and only moderate stability of some predictors.
The operationalization of suicidal ideation as a continuous variable and the dichotomization of suicidal behavior are acknowledged but not sufficiently justified or supplemented with sensitivity analyses (e.g., ordinal or alternative modeling approaches).
The handling of missing data for the CFC–FC variable, while described, is not examined in terms of potential bias or impact on results.
Although additional clinical and sociodemographic variables were included in the candidate set, their exclusion through penalization is not fully interpreted, and the role of potential confounding (e.g., diagnosis, comorbidity, medication) remains underdeveloped.
Model evaluation would benefit from additional context, such as comparison with simpler baseline models and reporting of uncertainty estimates (e.g., confidence intervals), to better situate the practical significance of findings.
The discussion of clinical implications, while more cautious, still slightly exceeds the evidentiary base, particularly in suggesting how R-PAS findings might inform intervention targets or clinical decision-making without direct comparative evidence.
The implications of the strongly imbalanced sample (predominantly female) are acknowledged, but conclusions regarding the absence of sex-related effects may be overstated given limited statistical power for subgroup analyses.

Author Response

Comment: Thank you for the opportunity to review the revised version of this manuscript. The authors have made visible improvements in response to the prior feedback, and the manuscript is now clearer, more transparent, and better aligned with its exploratory aims. In particular, the study purpose is more explicitly framed as cross-sectional and non-predictive, the introduction now acknowledges debates surrounding performance-based assessment, and the sample is much better characterized through the inclusion of diagnostic, sociodemographic, and clinical information. The methodological section has been strengthened with clearer descriptions of standardization procedures, distributional properties, and model estimation, as well as the addition of bootstrap stability analyses and more detailed reporting of elastic net tuning. The authors also made a commendable effort to temper interpretive claims, acknowledge modest model performance, and expand the limitations section to include key constraints such as cross-sectional design, lack of external validation, and generalizability concerns.

Response: We thank the reviewer for the careful re-evaluation of the revised manuscript and for acknowledging the improvements in clarity, methodological reporting, and transparency. We appreciate the remaining constructive comments, which have helped us further refine the manuscript and better align interpretation with the exploratory nature of the study.

Comment: Despite these improvements, several issues remain that should be addressed before publication:

The distinction between statistical associations and inferred psychological mechanisms is improved but still not consistently maintained; some interpretations (e.g., links to perfectionism, communicative intent, or specific cognitive styles) extend beyond what the data directly support.

Response: We thank the reviewer for this important observation. We have further revised the Discussion to rephrase any language implying an interpretation in terms of psychological mechanisms. The revised text now consistently frames findings in terms of associational patterns, without attributing causal or explanatory psychological processes beyond the observed data.

Comment: The characterization of findings as reflecting a “coherent psychological profile” remains somewhat overstated given the modest effect sizes, low explanatory power (R² = .09), and only moderate stability of some predictors.

Response: We agree with the reviewer that this formulation may overstate the strength and stability of the findings. We have removed references to a “coherent psychological profile” throughout the manuscript and replaced them with more cautious descriptions, consistent with the observed effect sizes and stability analyses.

Comment: The operationalization of suicidal ideation as a continuous variable and the dichotomization of suicidal behavior are acknowledged but not sufficiently justified or supplemented with sensitivity analyses (e.g., ordinal or alternative modeling approaches).

Response: We thank the reviewer for this comment. Suicidal ideation was treated as a continuous ordinal variable to preserve information on severity gradients and avoid loss of variability associated with categorization. Suicidal behavior was dichotomized due to sample size constraints and the need to ensure sufficient event stability for penalized regression modeling in a relatively small sample. We agree that alternative specifications could provide complementary insights; however, given the exploratory nature of the study and constraints related to model stability, sensitivity analyses using alternative specifications were not conducted but are identified as an important direction for future research.

Comment: The handling of missing data for the CFC–FC variable, while described, is not examined in terms of potential bias or impact on results.

Response: We thank the reviewer for the comment. This point reflects a data management issue that has been fully resolved. The previously missing CFC–FC data were recovered following correction of a variable labeling error. The analyses have been updated accordingly, and the inclusion of the complete variable set did not alter the overall pattern of results.

Comment: Although additional clinical and sociodemographic variables were included in the candidate set, their exclusion through penalization is not fully interpreted, and the role of potential confounding (e.g., diagnosis, comorbidity, medication) remains underdeveloped.

Response: We thank the reviewer for this comment. We would like to clarify that psychiatric diagnoses, SES, and related clinical variables were not excluded a priori from the analysis; rather, they were included in the full candidate set entered into the elastic net models alongside R-PAS variables. In penalized regression, all variables are treated symmetrically and are subject to shrinkage and selection based on their incremental contribution in the multivariate context. Accordingly, the absence of these variables in the final models reflects the penalization process rather than a manual exclusion step.

Comment: Model evaluation would benefit from additional context, such as comparison with simpler baseline models and reporting of uncertainty estimates (e.g., confidence intervals), to better situate the practical significance of findings.

Response: We thank the reviewer for the comment. This study was designed as an exploratory multivariate analysis rather than a formal incremental validity test. Nonetheless, future work should compare penalized models against baseline covariate-only models to better evaluate incremental explanatory value.

Comment: The discussion of clinical implications, while more cautious, still slightly exceeds the evidentiary base, particularly in suggesting how R-PAS findings might inform intervention targets or clinical decision-making without direct comparative evidence.

Response: We thank the reviewer for this important observation. We agree that the evidence provided by the present study does not support direct clinical decision-making or the specification of intervention targets. In response, we have revised the Discussion to further limit clinical implications and removed statements that could be interpreted as suggesting applied or intervention-level utility of the findings. However, since another reviewer listed the clinical considerations as a strength, we did not remove them entirely; we just aligned them with the results.

Comment: The implications of the strongly imbalanced sample (predominantly female) are acknowledged, but conclusions regarding the absence of sex-related effects may be overstated given limited statistical power for subgroup analyses.

Response: We thank the reviewer for this comment. We agree that the strong female predominance in the sample limits generalizability and statistical power for sex-stratified inference. While exploratory comparisons by sex were conducted and reported in Supplementary Table S2 (showing no significant differences), we have revised the manuscript to avoid any language suggesting evidence of equivalence or absence of sex-related effects.

Round 3

Reviewer 1 Report

Comments and Suggestions for Authors

I thank the authors for their second revision.

Author Response

Comment: I thank the authors for their second revision.

Response: We thank the reviewer for taking the time to accurately read the second revision and for the comments that improved the paper