Review Reports

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Dear author,

Thank you for the opportunity to review your work. You approached an interesting Klein’s framework within a military context. However, in its current form, there are significant conceptual and methodological hurdles that must be addressed to provide a robust contribution to the field. I offer the following suggestions in the spirit of helping you refine this research for a future successful submission.

Some points are unclear to a reader unfamiliar with Klein’s idea. For instance, you start with the proposition that Klein claims/structure is well known, but it is not (of course, it depends on the public (lines 35-39). So, I recommend more explicitly defining these for a broader audience.

Why Klein’s decision-making claim could be a single dimension (or a multiple dimension)? I mean, a factor analysis is based on a latent construct. In other words, there appears to be a conceptual leap in treating "decisions" as a latent construct for factor analysis. Since decision-making is typically viewed as a process rather than a static latent trait, the theoretical justification for your factor structure remains unclear.

What is not clear is the relationship between the 11 claims, the dual-capacity, and the 3 dimensions described: “sensemaking frames what options become salient; decisions change the environment practitioners must interpret; and adaptation updates both understanding and action”. You may suggest, if I understood correctly, that those 11 claims could be translated into competencies, which could then be organized into 2 (heuristic vs systematic) or 3 dimensions (sensemaking, adaptation and decision). However, based on Table 1, what you used is how people evaluate whether they agree with a sentence, not how they decide. If you would like to test the dual-process as Klein named, dual-capacity (dual-process has already been suggested and tested since 1980 with Richard Petty and Alice Eagly; although, as you said, Klein may not have been tested, but his idea is not new and there are several pieces of evidence with different methodologies…).

What you describe based on military demands, in fact, is more related to institutional logics that operate behind the military structure.  This military institutional logic is more than 2 (e.g., bureaucratic, legalist, vigilant etc). But I understand it is not the main point.

Method

Your sample is quite small (N=225), which falls below any psychometric sample cutoff/suggestion, and you should defend your point. Furthermore, it would be helpful to clarify why you expected a different factor structure, specifically within a military population (although you don't have any other sample to compare).

Additionally, evidence of validity requires several steps and comparisons. Those steps are not presented in the present work.

Klein’s model often hinges on the distinction between experts and "juniors". Your data showed no significant differences based on age (a common proxy for seniority). This raises a critical question: is this due to a lack of variance in the sample's experience levels, or is the instrument perhaps not sensitive enough to capture these distinctions or the construct/items/factors are not well measured?

The manuscript would benefit from a more detailed description of how the instrument was constructed and administered. Currently, the items seem to lack a unified "target"; some refer to leaders, some to "our plans," or "we can" and others to generic decision-making biases (e.g., "decision biases distort" or "it is bad to draw early conclusions"). This makes it difficult to capture a cohesive and integrative latent construct.

Results:

I noted that the mean score for the item regarding "teaching people procedures helps them make decisions and perform tasks more skillfully" was 1.8 on a 5-point scale. This seems surprisingly low for a military cohort. It might be worth double-checking this data point and/or providing a contextual explanation.

Please ensure you include internal consistency measures (such as Cronbach’s alpha or, ideally, McDonald’s omega) for your factors to support the evidence of validity of your findings (at least an alpha, as you used SPSS, I guess SPSS does not offer omega yet).

Discussion:

Given the many problems with your instrument, sample, and dual-process, I would consider your study a starting point and would suggest a review in the sentences based on the psychometric literature.

I believe the following sources may offer valuable perspectives on scale development as you revise your work (they are not mine):

http://journals.sagepub.com/doi/10.1177/109442819800100106

https://doi.org/10.1016/j.leaqua.2018.07.001

Doi: 10.1590/1982-7849rac2022210085.en

Doi: 10.3389/fpubh.2018.00149

I hope you find these comments constructive, which was my intention. Your work addresses an interesting point.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

The​‍​‌‍​‍‌ paper examines 11 core claims of Gary Klein as part of his natural decision making and Recognition-Based Decision (RBD) theories on a sample of Norwegian Armed Forces personnel (N=225). The writers believe that these claims represent a two-factor structure, i.e. "Planning/Structure" and "Analytical/Evidence-Based Applications," rather than a single-dimension one. The subject matter is very important to the understanding of decision-making mechanisms under VUCA (Volatility, Uncertainty, Complexity, Ambiguity) conditions. Nonetheless, the introduction and literature review are not giving enough attention to the theoretical aspects of why Klein's claims are contradictory or why they should be multidimensional. Besides, the use of factor analysis as an exploratory tool results in a methodological inconsistency with the way hypotheses (H1) are presented.

Areas for improvement:

1. The paper is promoted as an “Exploratory Factor Analysis” (EFA). Generally, EFA seeks to ‘identify’ the number of factors a certain structure is composed of. Nevertheless, by saying in H1 that “a multi-factor model will fit better than a single-factor model,” the authors have basically given the research a semi-confirmatory character. The testing of a particular hypothesis (H1) would require Confirmatory Factor Analysis (CFA) as the analysis method. On the other hand, if the dataset is new and the structure has to be uncovered, the H1 should be an “expectation” or the research question (RQ1) should be put forward.
2. 11 items derived from Klein's (2009) book do not represent a standard “scale,” but rather the authors' theoretical claims. The authors haven't indicated how they took these statements and turned them into a Likert-type questionnaire (the choice of words, the scaling technique). Adding a paragraph of “Item Development” can help describe the transition from the original items to the survey format. If experts were consulted to assess the face validity of the items, this should also be mentioned.
3. After the analysis, the authors concluded that two dimensions, “Planning” and “Analytical,”, were identified (Abstract section). Nevertheless, the manuscript doesn't provide a sufficient theoretical rationale for the differentiation of these two dimensions as evidenced by the literature review. More emphasis should be rather put on the discussion about the opposition of rational/analytical decision-making with intuitive/structural decision-making (for instance, with reference to Dual-Process Theory). Doing so will support your finding of a two-factor structure.
4. 225 participants constitute a fair sample size for EFA according to the 20 participants per item criterion. However, “dropping two items because of low commonality” (Line 12) might have resulted in the weakening of the scale from the content perspective. The Introduction or Methodology sections should mention which items were dropped, and how the changes weakened the content validity of the “decision-making” domain.
5. The description of the VUCA concept is very superficial. What the Norwegian Armed Forces (2024-2025) are specifically facing (e.g., Arctic security, hybrid threats, etc.) should be tightly tied to Klein's claims. Referencing briefly Norwegian military doctrines on decision-making paradigms in the “Military-specific studies” (Line 125) section would definitely add value to the study both locally and ​‍​‌‍​‍‌globally.
6. ​‌‍​‍‌ The selection of “Principal Axis Factoring with Oblimin rotation” by the authors is a methodologically correct choice but a reasoning should be given why the factors are believed to be correlated (oblique). Hence, it would be best to refer to literature when explaining the expected correlation between the factors.

7. It seems that Klein's original statements were not a “scale.” The translation process of these sentences into Norwegian (did you carry out a back-translation?) and the expert review stages (content validity) are not clear. How was the psychometric suitability of the items for a survey ensured?
8. The 5-point Likert scale was characterized as “1=Strongly Disagree, 5=Strongly Agree” (Line 176). Nevertheless, in Table 1, the means (e.g. Q11 = 1.64) reveal that almost all participants have disagreed with the statements. Hence, these items seem to possess very little discriminative power and that “social desirability” bias might also take place. It is necessary to discuss this situation.
9. Convenience sampling was chosen as the sampling method. 225 persons are an almost adequate number for EFA. Demographic variables that have a direct influence on decision-making styles, such as participants' rank and years of experience (seniority), should be included.
10. That a two-factor solution accounts only 29.24% of the total variance is, by academic standards, a very low number (usually one expects from 50 up to 60%). It may be interpreted as a lack of congruence of the model with Klein's theory or, alternatively, the items are insufficiently related. The low rate should not be neglected or regarded as “modest” but rather as a serious limitation.
11. The Cronbach's Alpha for the Factor 1 was .65. In social sciences, a .70 level is considered as a generally accepted lower limit. Thus, a value of .65 is perceived as “weak/borderline” and consequently, using this factor as a “scale” is a risky decision.
12. Some of the questions, namely Q2, Q7, and Q1, have been removed. However, Q1 and Q5 also have very low communality values of .17 and .18 respectively. A stronger rationale should come up technically for decisions on retention or removal of these items.
13. After demonstrating the necessity of greater clarity the authors mention that they have carried out an “exploratory” analysis and “validated” H1 (Line 329). We cannot validate hypotheses with EFA; instead, we can only identify the structures. This error in nomenclature needs to be fixed everywhere in the text. An update of the analysis to Confirmatory Factor Analysis (CFA) or the avoidance of the term “validation” is also a way out.
14. Referring to the authors' statement that these outcomes will be implemented in training programs (AARs, etc.) (Line 421). A prudent military training session would perhaps not consider a “diagnostic tool” whose inherent property appears to explain the 29% of the variance and be reliable up to .65. It is advised that this part be softened. It is a misconception to consider the limitation of Klein items not being written for psychometric purposes (Line 425) simply a “limitation”. This is, in fact, the very essence of the problem of construct validity of the study. The impact of this situation on the results should be candidly ​‍​‌‍​‍‌revealed.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

All revisions I recommended have been addressed, and the manuscript has been accepted.