Review Reports - “Robust” but Tiny: Methodological Influences and Inter-Individual (Un)Stability of the Serial Order Effect in Creativity

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The manuscript addresses an important and timely question regarding the robustness, magnitude, and inter-individual stability of the Serial Order Effect (SOE) in divergent thinking. However, several major issues require attention before publication. First, although the study emphasizes effect sizes, the theoretical implications of consistently small effects are not sufficiently developed; the authors should more clearly articulate what such “robust but tiny” effects mean for creativity theory and assessment practice. Second, the reliance on a secondary dataset limits causal and methodological control; stronger justification is needed for why this dataset is adequate to test inter-individual stability, particularly given short ideation sequences. Third, the manuscript would benefit from a clearer distinction between group-level robustness and individual-level unreliability, including stronger links to the “reliability paradox” literature. Fourth, the influence of scoring methods especially base-rate issues in frequency scoring should be more systematically framed as a methodological bias rather than a substantive SOE difference. Finally, clearer guidance for future research designs (longer tasks, alternative DT measures) is needed.

Author Response

Comment 1:

Response : Thank you for the general comment on our manuscript and your careful evaluation. Regarding the theoretical and practical implications of consistently small, yet replicable effects, we first note that characterizing the observed SOE as “tiny” is, to some extent, context dependent. While the effect explains at most 3.5% of the variance, such magnitudes are not uncommon in psychological research and may be considered substantive when effects are theoretically meaningful and robust across studies. It also emphasizes potential sources of variability unaccounted for, which invites for a close examination of potential moderators of the effect. Correspondingly, the revised manuscript strengthens the rational for the present study by acknowledges that while “tiny” this range of effect sizes is not trivial and may mask important sources of variability to be scrutinized (In the introduction, p. 5, “Together, while modest effect sizes can be meaningful in certain psychological contexts (for instance, when the outcome has high real-world impact or when the effect accumulates over time) they also invite closer examination of moderators that may account for variability in the magnitude of the SOE. That is, even an overall small effect may become considerable in specific conditions or subpopulations, making moderator analysis a valuable next step.”).

Furthermore, we fully agree that the coexistence of replicability with small effect size warrants careful theoretical interpretation rather than being taken as evidence for a stable, trait‑like phenomenon. In line with your comment #3, we have substantially revised and expanded the Discussion to situate the SOE within the broader literature on the reliability paradox and on levels inference. In particular, we now explicitly distinguish between (a) robustness at the aggregate level and (b) reliability as an individual‑differences measure. Drawing on Hedge et al. (2018) and related work, we argue that paradigms yielding consistent mean‑level effects often do so precisely because they minimize between‑person variance, which in turn limits their usefulness for ranking individuals. This framework helps understand why the SOE can be reproducible yet consistently small, and why it fails to show adequate inter‑individual stability.

We have also clarified the theoretical implications of this pattern for creativity research. Rather than interpreting small SOE effects as weak evidence for a general (or “universal”) ideation strategy, we now argue that they are more consistent with a view of ideation as involving flexible, context‑dependent processes (according to the task at hand, experience), in which individuals may shift between different strategies (e.g., memory‑based retrieval versus more effortful associative search) across responses, tasks, or stimuli. This refinement solidifies our initial assumption of heterogenous ideation strategies at the individual level and reframes the SOE as a group‑level regularity emerging from heterogeneous (and possibly adaptive) individual processes.

These clarifications are now explicitly articulated in the revised Discussion (p. 15),

“In short, people may adapt flexibly, with strategy use (e.g., memory retrieval vs. effortful associative search) varying across responses according to task context, stimulus salience, and experience, challenging the notion of a uniform ideation strategy at the individual level. An important direction for examining this hypothesis is therefore to identify typical response sequences (e.g., via mixture or latent‑class modeling) that may reflect qualitative-ly distinct ideation strategies unfolding over ideation sequences.”

Finally, we have expanded the discussion of assessment implications. Specifically, we now emphasize two key points: (1) the SOE should primarily be interpreted as a group‑level phenomenon that does not currently support reliable inferences about enduring individual differences, and (2) progress in assessing individual ideation dynamics will require methods explicitly designed to accommodate individual‑level heterogeneity. To this end, we outline future research directions focused on identifying typical ideation pathways (e.g., via mixture or latent‑class modeling of response sequences) and on developing approaches capable of measuring SOE‑related dynamics at the individual level with adequate reliability. (p 15). “This carries important theoretical and practical implications: while the SOE is informative for characterizing general processes that unfold during idea generation, it does not presently support inferences about enduring individual differences, highlighting the need for future work examining how individual-level ideation dynamics can be measured with adequate reliability (cf. Kievit et al., 2013).

Comment 2:

Second, the reliance on a secondary dataset limits causal and methodological control; stronger justification is needed for why this dataset is adequate to test inter-individual stability, particularly given short ideation sequences.

Response : This paper is considered for the special issue “Analysis of a divergent thinking dataset” https://www.mdpi.com/journal/jintelligence/special_issues/J908U24N0Q which purposely aims to showcase different approaches to analyze a dataset publicly available. Our initial manuscript stated “The data used in the present study stem from a dataset collected in Germany and analyzed in several prior publications by the original team (e.g., Forthmann et al., 2017; Forthmann & Doebler, 2025) and recent secondary analyses conducted as part of the current special issue (e.g., Jendryczko, 2024; Myszkowski & Storme, 2025; Pellegrino et al., 2025; Weiss et al., 2025). »

However, we acknowledge the inherent trade-offs involved in using a secondary dataset. While we agree that this approach offers less causal control, we believe the dataset was well suited to address our research questions, notably the availability of three prompts and three scoring methods, as well as a time on task aligned with the typical “standards” of DT research (see review Saretzki et al., 2024). To address your points, we first added the link to the special issue in which the present work is being considered, so the reader can refer to the call of the SI for better context. Second, we added a sentence in the introductory paragraph of the Limitation and future directions section stating “A primary consideration is the reliance on a secondary dataset, limiting causal and methodological control. While the dataset was suitable for the analyses conducted here (multiple scoring approaches, multiple DT prompts and typical time on task; see Saretzki et al., 2024), some key limitations inherent to the original data collection constrain the generalizability of our findings” (p.16). The paragraph then expands on the limitations of the dataset already outlined (i.e, sample size, only a single task/modality), and we have expanded on the relatively short ideation sequences represented in the dataset, a limitation somewhat mitigated by the use of empirical Bayes shrinkage estimators. “Third, the fixed 2.5-minute time limit likely constrained ideational sequences, reducing the reliability of individual SOE slopes. Although ideation sequences were relatively short, the use of empirical Bayes shrinkage estimators helps mitigate the unreliability typically associated with limited observations. Thus, while longer sequences would be prefer-able, the dataset still provides a reasonable basis for examining inter-individual stability. Yet, studies with longer time-on-task would likely yield more stable estimates.” (p.16).

Comment 3:

Third, the manuscript would benefit from a clearer distinction between group-level robustness and individual-level unreliability, including stronger links to the “reliability paradox” literature.

Response : Thank you for this comment. As outlined above (response to comment #1) we have engaged more thoroughly with the reliability paradox literature in the discussion section. Because there was no study to date examining interindividual stability of the SOE and/or reliability of the effect, the introductory section could not emphasize the reliability paradox, which is why it is only developed in the discussion. Specifically, we now state the following to address your comment (p. 15): “In sum, the robustness of an aggregate effect does not imply reliability as an individual differences measure. As emphasized by Hedge et al. (2018), experimental paradigms tend to become established precisely because they minimize between-person variance while maximizing within-person regularities, yielding reproducible mean-level effects but little stable variance with which to rank individuals. Although the SOE can be consistently demonstrated at the group level, it captures little dependable between-person variance once stimulus and method-specific influences are considered (here reflected in the very low scoring and prompt stability of individual SOE slopes). From a psychometric perspective, the SOE may therefore be better understood as a within-task ideational dynamic elicited by DT tasks, rather than as a marker of a stable, “universal”, trait-like ideation strategy.”

Comment 4:

Fourth, the influence of scoring methods, especially base-rate issues in frequency scoring should be more systematically framed as a methodological bias rather than a substantive SOE difference.

Response : Thank you for this insightful suggestion. We agree that the specific influence of frequency-based scoring can be interpreted as a methodological bias that may artificially inflate the observed serial order effect (SOE). However, we also maintain that these variations reflect substantive differences in the underlying constructs being measured. As noted in our initial discussion, human judgment (creativity), frequency-based scoring (uncommonness), and semantic distance (dissimilarity) are not interchangeable; they tap into distinct dimensions. While an idea may be statistically “rare” (infrequent), it is not necessarily more semantically distant or qualitatively “creative”. Hence, there are theoretical reasons explaining why sequential patterns vary across scoring methods (see paragraph under 4.1 starting with “Second, the variations of the SOE’s effect size according to the scoring method examined in this study…”).. When it comes to frequency-based scoring, we agree that the base-rate issue may partly represent a bias that amplifies the SOE, as also discussed in section 4.1. To address your point more explicitly, we have revised Section 4.1 to clarify that the stronger effect observed in frequency scoring may be partly artifactual. We have added the following: “With that said, the stronger effect observed with the frequency-based scoring method may be partly artifactual. Specifically, the high prevalence of common, fixating responses in early ideation may disproportionately amplify the SOE when scored via frequency-based scoring, a methodological bias further discussed in this section "

The issue is indeed amply developed further in this revision (paragraph p.14 starting with “Along the same lines, our study detected a significant moderation effect of the DT stimulus (i.e., AUT object prompt) when using frequency-based scoring”

Comment 5:

Finally, clearer guidance for future research designs (longer tasks, alternative DT measures) is needed.

Response : Thank you for this suggestion. Within the limitation and future directions section, we now further emphasize the need for future research purposely designed to address our research questions with a new paragraph (p.17) “In sum, future studies purposely designed to address our research questions should be conducted to examine the generalization of our findings (regarding effect sizes and their moderators, inter-individual stability) to different tasks, scoring methods, and samples. Such studies should consider longer time-on-task to increase the length of ideation sequences, implement rigorous experimental controls (e.g., blinded scoring), and the inclusion of diverse task modalities, as well as a broader range of individual difference moderators, to more precisely isolate the mechanisms driving the SOE.”

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

The paper is an interesting in-depth analysis of the serial order effect in creativity. The study demonstrates the replicability of the SOE effect with three different stimuli and three different scoring methods (subjective judgements, frequency-based scoring, semantic distance). Despite the negligible to small magnitude of the effect, some moderators of the SOE effect emerged: 1. The SOE is greater for participants with lower fluency or flexibility; 2. The SOE is stronger when responses are scored with frequency-based scoring; 3. The strength of the SOE may change as a function of the stimulus (perhaps depending on the “solution space” size of the stimulus itself); 4. Participants’ SOE slopes do not reflect uniform, trait-like tendency. That is, whereas the effect is in general stable, inter-individual SOE slope differences are unstable across prompts.

The paper is methodologically sound, easy to follow and well-motivated. I think, however, that some concepts could be better clarified. This is the case for “fixating response” and “solution space”, both introduced in the discussion. They may be introduced in the introduction or at least described more clearly in the discussion.

It’s not clear to me how “response creativity” (y-axis) is measured (figure 1 and graphs in the Supplementary Materials).

12, hybrid DT instructions. I would explain briefly what you mean by “hybrid instruction” in the method section (p.5) and at this point comment on the effects that this specific instruction may have with respect to other possible instructions.

Figure 1, panel c. It seems like creativity decreases as SOE increases but it should be the opposite (see p. 7 of the text).

Minor points:

p.5, 1^st line of section 2.1. It would be appreciated a direct link or a complete reference for the mentioned dataset.

14, 3^rd and 4^th line of section 4.3: something appears to be missing here.

Graphs: Axis labels are really difficult to read in the print version. Please, use bigger fonts.

Author Response

Comment 1:

Response : We thank you for your careful review and the positive evaluation of our initial submission. The clarifications requested were indeed useful to improve clarity and we have revised the text accordingly. Specifically, when describing the classic interpretation of the SOE and in particular the associative theory of Mednick, we added a sentence in which “fixating response” is introduced and clearly defined (p.2). “That is, the SOE in DT reflects the progressive search for increasingly distant semantic concepts (e.g., Beaty & Silvia, 2012; Wang et al., 2017). Starting from salient or even “fixating” responses (i.e., ideas driven by highly accessible or conventional information due to cognitive constraint arising from previously activated knowledge, examples, or dominant representations, even when these are irrelevant or suboptimal for the task; e.g., Chrysikou et al., 2016; Smith et al., 1993), ideation progressively shifts towards more remote concepts.”

Similarly, the concept of solution space is now introduced in the introductory section regarding the moderators of the SOE. Building on Gonthier and Besancon’s findings, which highlighted variations in SOE by task we now state (p.3). “This moderating effect of task may lie in differences in task-specific “solution space”, de-fined as the total set of mental representations and associations that an individual con-siders accessible or relevant for a given task (e.g., Forthmann et al., 2020). From this perspective, tasks with a narrower solution space may constrain ideational sequences by limiting the range of representations that can be explored during idea generation.”

Comment 2:

It’s not clear to me how “response creativity” (y-axis) is measured (figure 1 and graphs in the Supplementary Materials).

Response : Thank you for pointing this out. In figure 1 (and some figures in ESM), we indeed used the generic “response creativity” label to refer to one of the three scoring methods investigated and spelled out the scoring method in the panel titles. For clarity and accuracy, we have now relabeled all figures with the corresponding scoring method examined.

Comment 3:

12, hybrid DT instructions. I would explain briefly what you mean by “hybrid instruction” in the method section (p.5) and at this point comment on the effects that this specific instruction may have with respect to other possible instructions.

Response : We have expanded the definition of “hybrid instruction at first utterance (p6) : “For each prompt, participants had 2.5 minutes to generate alternate uses using “hybrid instructions”, which emphasize both fluency (quantity of ideas) and originality (quality of ideas) as core dimensions of divergent thinking (Reiter-Palmon et al., 2019). The instructions read: "Please try to write down as many uncommon and creative uses for a [object-prompt] as you can think of" (Forthmann et al., 2017).” and when referred to in discussion “the data analyzed in the present study were based on hybrid DT instructions (e.g., Reiter-Palmon et al., 2019) which emphasizes both fluency and originality.” (p.13)

Expanding on the effect this type of instruction may have, we now state (still under the method section “While such hybrid instructions have been shown to increase creative performance (e.g., Butler et al., 2003), they might also emphasize individual “preferences” for either fluency (i.e., quantity) or originality (i.e., quality; e.g., Torrance, 1998).”. As we did not formulate a specific hypothesis on the instruction effect (which can not be tested in the present study), we expanded further on the effect of a hybrid instruction in the discussion section which was already outlined in the initial submission (i.e. p.12 “. In contrast to that study, which used a “be creative” instruction, the data analyzed in the present study were based on hybrid DT instructions (e.g., Reiter-Palmon et al., 2019) which emphasizes both fluency and originality. Although research suggests an attenuated SOE when “be creative/be original” instructions are prompted over “standard” or “be fluent” instructions (Kaya & Acar, 2019; Kleinkorres et al., 2021), hybrid instructions might emphasize individual “preferences” for fluency versus originality, or the other way around (i.e., the quantity-quality trade-off), which has been shown to overshadow the SOE (Gonthier & Besançon, 2024).”

Comment 4:

Figure 1, panel c. It seems like creativity decreases as SOE increases but it should be the opposite (see p. 7 of the text).

Response : Thank you for this careful observation. The issue stems from the operationalization of the semantic score. Our analyses used semantic similarity (e.g., GloVe cosine similarity), not semantic distance. Higher similarity indicates that ideas are semantically closer (typically less creative), whereas lower similarity indicates more remote and creative ideas. As shown in the revised Figure 1c (now relabelled Semantic similarity), semantic similarity decreases with increasing response order. This means that later ideas are more remote and thus, more creative. The interpretation in the text (“later responses were more creative”) remains correct, and we have kept it unchanged. However, to avoid confusion, we have relabelled the y‑axis in Figure 1c (and all related supplementary figures) from “Response creativity” to “Semantic similarity” . Importantly we now used consistent phrasing throughout the text (e.g., semantic similarity is now consistently use “semantic similarity” rather than “semantic distance” throughout the manuscript (when referring to the actual metric used in the study; in the background and discussion section “semantic distance” is preferred as it is a more usual way to refer to this construct). We have also added a clarifying sentence in the main text (Section 2.2, Materials and Procedures) to explicitly state that the semantic scores reflect similarity, with lower values indicating greater creativity: “Finally, semantic similarity scores obtained from the unweighted dewak GloVe full semantic space (Forthmann & Doebler, 2025) were calculated at the response level. Note that, akin to the frequency-based scoring method, higher values reflect responses of lower creativity (i.e., here more semantically similar).”

We believe these changes eliminate confusion and align the visual display with the written description, throughout.

Comment 5:

Minor points:

p.5, 1st line of section 2.1. It would be appreciated a direct link or a complete reference for the mentioned dataset.

Response : Thank you. The link to the dataset is now added in the manuscript’s body as well as in the Data Availability Statement.

Comment 6:

14, 4.3: something appears to be missing here.

Graphs: Axis labels are really difficult to read in the print version. Please, use bigger fonts.

Response : Thank you. As stated, we have revised the figures.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The paper has been revised and is recommended for publication.