Lighting the Dark Side of the Model: Psychometric Probing of Dark Triad Responses in LLMs

Abstract

Background: Large language models (LLMs) are increasingly deployed in socially sensitive contexts, raising concerns about how their outputs reflect or reproduce socially aversive response patterns. Prior research has focused on socially desirable traits (e.g., the Big Five), while less attention has been given to the Dark Triad—Machiavellianism, narcissism, and psychopathy. Methods: This exploratory study applies a psychometric probing approach using the Short Dark Triad (SD3) as a structured questionnaire to analyze response patterns in 12 LLMs. Through controlled persona prompting (gender, age, religion), more than 540,000 responses were generated across model–persona–item observations. Outputs were scored using standard psychometric procedures and compared with pooled human SD3 reference values using Hedges’ g as an interpretive benchmark. Results: Across models, Machiavellianism-related scores tend to be lower than human references, narcissism-related scores slightly higher, and psychopathy-related scores largely within the human range. Persona prompting produces systematic but uneven effects: gender has minimal impact, age yields small shifts, and religious framing leads to the strongest and most consistent decreases in socially aversive response patterns. However, observed differences showed notable heterogeneity across models. Conclusions: The findings should be understood as response patterns shaped by context and specific prompting conditions. Our findings suggest that persona effects reflect normative activation, not identity simulation. Persona prompts seem to trigger patterns in the training data rather than consistent, identity-specific traits. Overall, the study highlights both the potential and limitations of psychometric-style probing for evaluating LLM behavior and underscores the importance of prompt context in shaping model outputs.

1. Introduction

The increasing integration of large language models (LLMs)—such as GPT, Llama, Grok, Mistral, and Gemini—into diverse socio-technical and organizational contexts has intensified the need to systematically analyze their observable behavior under controlled conditions [1]. This need is salient in domains involving affective interaction, decision support, or health-related communication, where subtle differences in generated language can shape user perceptions and downstream decisions. At the same time, the growing prevalence of AI-generated content complicates the distinction between human- and machine-authored text, motivating methods that characterize how models respond to structured stimuli.

One emerging approach is psychometric-style probing, in which human questionnaire formats are used not to diagnose models as if they were persons, but to systematically elicit and compare patterned outputs. we adopt this perspective and use the Short Dark Triad (SD3) as a structured probing instrument. We selected the SD3 because it offers broader construct coverage than very short alternatives such as the Dirty Dozen while remaining substantially more practical for large-scale repeated sampling than administering separate, longer instruments for Machiavellianism, narcissism, and psychopathy. This makes it well-suited for comparing response distributions across many models, items, and persona conditions.

A central interpretive issue in persona-based probing is whether persona prompts induce identity simulation or normative activation. By identity simulation, we mean the emergence of a coherent persona-specific response profile analogous to a stable role or identity. By normative activation, we mean the triggering of learned social and moral associations attached to prompt wording, without implying coherent or stable identity-level behavior. We treat SD3 categories as response dimensions rather than personality traits of the model. Accordingly, expressions such as Machiavellianism, narcissism, and psychopathy refer to the scoring dimensions of the SD3 instrument, and not to stable dispositional properties of LLMs.

Prior work has examined LLM outputs using established personality frameworks, most notably the Big Five [1,2]. These studies suggest that questionnaire-based prompting can reveal systematic and reproducible response patterns, while also highlighting the interpretive limitations of applying human psychometric constructs to generative systems. Extending this line of work, we focus on the Dark Triad—Machiavellianism, narcissism, and psychopathy [3]—as a lens for probing socially aversive response patterns that have received less attention in the LLM literature. In human psychology, these constructs capture tendencies such as manipulativeness, self-enhancement, and reduced empathy. In the present study, however, they are treated as categories for organizing and comparing model outputs under a standardized questionnaire format rather than as intrinsic model properties.

To ensure terminological consistency, we distinguish clearly between human reference, baseline, and norm. The term human reference is used exclusively for comparisons between model-generated response scores and the pooled human SD3 comparison sample. The term baseline is used exclusively for comparisons within the LLM study itself, especially when persona-conditioned responses are evaluated relative to the neutral persona condition. By contrast, we do not use the term norm to denote human comparison data in the analyses, although such usage is common in parts of the psychometric literature. Instead, norm is reserved solely for the interpretive level of the discussion, where it refers to norm activation, that is, the activation of trained normative associations by persona prompts.

The SD3 is therefore used here as a structured probe of questionnaire-conditioned response patterns in LLMs. By presenting its items under controlled prompting conditions and repeated sampling, we analyze how outputs vary across models and persona framings. This design allows us to compare model-generated response distributions with pooled SD3 reference values from prior human studies as an interpretive benchmark, while also examining how contextual cues embedded in prompts shift those distributions. A plausible interpretive lens is that persona prompts activate broader normative associations rather than stable, identity-specific “personality” patterns. In this sense, persona-conditioned SD3 responses may reveal context-dependent norm activation embedded in model behavior.

Against this backdrop, we address two central research questions:

• Research Question 1 (RQ1): How do response distributions generated by large language models under SD3 prompting relate to human SD3 reference distributions as an interpretive benchmark?
• Research Question 2 (RQ2): How does persona-based prompt framing influence SD3-conditioned response distributions in LLM outputs, and to what extent can these effects be understood as context-dependent (e.g., norm-activating) variations?

This paper makes five main contributions. First, it extends psychometric-style LLM evaluation from socially desirable traits to socially aversive response tendencies by introducing the Dark Triad as a structured probing lens. Second, it presents a large-scale empirical study comprising more than 540,000 responses across 12 models, 27 questionnaire items, and 17 different persona conditions in three groups (age, gender, religion). Third, it develops a systematic persona-based evaluation framework that enables controlled comparison of prompt-induced shifts in response distributions. Fourth, it contextualizes model outputs relative to pooled human SD3 reference values using effect sizes, while explicitly treating these as a benchmark by reference rather than as evidence of human-equivalent trait measurement. Fifth, it provides aggregated response statistics that may serve as reference points for future work on psychometric probing and persona-sensitive LLM evaluation.

Overall, we contribute to a more cautious and methodologically grounded understanding of how human-derived psychometric instruments can be used to analyze LLM behavior. Rather than treating questionnaire scores as evidence of stable personality, we use them to examine structured, context-sensitive response patterns and the extent to which these vary across prompting conditions.

2. Foundations and Related Work

The scientific study of the “dark side” of human personality has shifted over the past two decades from clinical pathology to mainstream personality, social, and organizational psychology. Central to this shift is the Dark Triad—narcissism, Machiavellianism, and psychopathy—defined as distinct but overlapping socially aversive traits present in the general, nonclinical population [4,5,6]. Unlike clinical disorders, these traits are understood as continuous dimensions rather than binary diagnoses [5,7]. The Dark Triad describes a constellation of tendencies including low empathy, manipulativeness, and inflated self-view [2,8,9]. Narcissism refers mainly to its grandiose form, involving entitlement and a need for admiration; Machiavellianism reflects strategic manipulation and cynicism; and psychopathy is characterized by impulsivity and reduced remorse [5,10]. These traits are strongly interrelated and are often linked to a broader “Dark Factor” (D-factor), defined as the tendency to maximize personal benefit at others’ expense [10].

Applying such constructs to large language models requires caution. Psychometric instruments like the SD3 are designed as human self-report measures and assume stable dispositions, introspection, and consistent identity—assumptions that do not hold for LLMs. Model outputs are instead generated probabilistically and are highly sensitive to prompt context, sampling, and alignment constraints. Consequently, any “trait-like” measurements obtained from LLMs should be understood as properties of responses under specific questionnaire conditions rather than as indicators of underlying psychological traits. In this sense, psychometric instruments function here as structured probes of model behavior rather than diagnostic tools.

2.1. Psychometric Measurement

While separate psychometric instruments exist to measure each trait of the Dark Triad individually, this results in long and very time-demanding questionnaires [5]. Therefore, more comprehensive measurement tools have been developed ([9] see Table 1). Among them, the 27-item Short Dark Triad (SD3) developed by Jones and Paulhus [8] and the concise 12-item Dirty Dozen (DD) developed by Jonason and Webster [11] are the most widely used, with the latter being criticized for its brevity, which may sacrifice accuracy for efficiency [5]. Furthermore, several measures exist for individual dark traits (see Table 1).

Table 1. Psychometric measures of Dark Traits [5,8,9].

Scale	Items	Target Population	Construct Depth
SD3	27	General/subclinical	High; captures specific nuances of each trait
Dirty Dozen	12	General/subclinical	Moderate; excellent for rapid screening
NPI-40	40	Non-clinical	High for grandiose narcissism; low for vulnerable
Mach-IV	20	Non-clinical	Standardized for cynicism and manipulation
LSRP	26	Non-clinical	Detailed breakdown of psychopathic factors

Establishing a single “prevalence percentage” for the Dark Triad is difficult because these are dimensional traits. However, researchers have identified thresholds for what constitutes “high” or “pathological” levels of the different traits in non-clinical settings.

Psychopathy: Meta-analytic reviews of psychopathy in the general adult population provide the most rigorous prevalence data. Based on a study of 11,497 individuals across 16 samples, the overall prevalence of psychopathy is estimated at 4.5% [12]. However, this figure is highly dependent on the instrument used. When the Hare Psychopathy Checklist-Revised (PCL-R), considered the “gold standard,” is used, the prevalence in the general population is estimated at only 1.2% [12]. This suggests that “true” clinical psychopathy is rare. However, using self-report instruments like the LSRP or SRP quadruples the estimated prevalence to 5.4%, as these scales capture individuals who exhibit significant psychopathic tendencies but may not meet the full forensic criteria [12]. Furthermore, prevalence varied greatly depending on subsamples (e.g., general community vs. student or prison samples; see Table 2).

Table 2. Prevalence of psychopathy in different population segments [12].

Population Segment	Psychopathy Prevalence	Sample Characteristics
General Community	1.9%	Lower risk, stable environments
University Students	8.1%	Higher impulsivity, young adulthood peak
Organizational Samples	12.9%	Concentrated in leadership and management
Prison (Male)	15.7%	High forensic risk, clinical psychopathy
Homicide Offenders	34.4%	Extreme antisocial manifestation

Narcissism: Estimates for narcissism in the general population range from 0.5% to 6.2%, though some studies using larger community surveys suggest it may be as high as 7.7% for men and 4.8% for women [13]. Similarly to psychopathy, prevalence varies according to population segments. For example, a study by Pourramzani and Monajemi found NPI-40 scores of 20 or higher, indicating significant narcissistic tendencies, in 18.1% of their sample of medical personnel [14].

Machiavellianism: Machiavellianism, lacking a clinical diagnosis, is primarily measured through the Mach-IV. While specific prevalence percentages for “high Machs” are less frequently cited than for psychopathy, research indicates that the trait is more common than psychopathy but less visible due to the Machiavellian’s focus on long-term reputation management [8].

2.2. Demographic Variance

Research in human populations shows systematic variation in Dark Triad scores across gender and age [10,14,15]. For example, men generally score higher than women on all three dimensions, with the differences being especially pronounced in psychopathy [12]. Similarly, age-related trends suggest that socially aversive traits decline over the lifespan, consistent with broader personality development patterns.

These findings offer valuable context for understanding the results of persona-based prompting. However, when transferred to LLM settings, such comparisons should be treated cautiously. Persona prompts do not isolate demographic variables in a clean experimental sense but instead introduce broader semantic framings that may activate multiple associations simultaneously.

2.3. Professional and Organizational Distribution

Dark Triad traits in human populations are frequently examined within professional settings, especially in high-power fields such as corporate leadership, healthcare, and politics [16,17,18]. These findings illustrate how socially aversive tendencies can manifest in real-world behavior and decision-making. In the context of LLMs, such literature primarily serves as conceptual background rather than a direct point of comparison, as model outputs do not arise from lived social roles or incentives. For example, narcissism and Machiavellianism strongly predict political ambition. Leaders high in Dark Triad traits can increase public polarization, as followers adopt their confrontational style and hostility toward opponents [19,20].

2.4. Cross-Cultural Comparisons

Cross-cultural research shows that the distribution of Dark Triad traits varies across socioecological environments [21,22]. For example, narcissism is sensitive to cultural and economic conditions, while psychopathy tends to be less variable across regions (see Table 3). These patterns highlight the importance of context in interpreting observed score levels within a given SD3 dimension. For LLMs, however, such variation is mediated indirectly through training data and alignment processes rather than lived cultural experience. Any apparent cross-cultural patterns in model outputs should therefore be understood as learned statistical associations rather than genuine cultural differences.

Table 3. Cultural differences in Dark Traits [21].

Factor	Relationship to Narcissism	Relationship to Mach/Psych
Economic Development	Higher in less developed countries	Generally stable
Gender Equality	Larger sex differences in equal societies	Slightly higher Mach in equal societies
Cultural Value	Higher in hierarchical/embedded cultures	Minimal direct relationship

2.5. Related Work

The rapid deployment of large language models has led to increasing interest in characterizing their behavior using methods adapted from psychology. For an overview, see [23]. Here, we summarize work applying psychometric-style evaluations to LLMs.

Personality Assessment in Language Models (RQ1): Prior studies have adapted instruments such as the Big Five Inventory (BFI) to LLMs [1,23,24]. In these setups, models complete human-oriented questionnaires and are scored using standard procedures. Results often show consistent response patterns under fixed prompting conditions, sometimes resembling human distributions but typically with reduced variance or shifted means [23,24,25,26]. These findings are sometimes described as “personality profiles,” but they are more appropriately understood as artifacts of the measurement setup and prompt design rather than evidence of stable traits. Recent work has begun to extend this approach to socially aversive constructs such as the Dark Triad [27].

Advances and Limitations (RQ2): Methodological extensions include embedding-based scoring, open-ended formats, and cross-cultural adaptations [28,29]. While these approaches improve robustness, many studies still lack systematic validity analysis and rely on single-shot evaluations. Moreover, LLM responses are highly sensitive to prompt framing, making it difficult to disentangle underlying tendencies from contextual effects.

Psychometric Validation (RQ1, RQ2): Reliability measures such as Cronbach’s alpha are increasingly reported, but deeper validation (e.g., factor analysis) and careful comparison to human reference distributions remain limited. Effect size measures are also not consistently used, complicating the interpretation of differences.

Gaps and Rationale: Several gaps remain: (1) limited focus on socially aversive constructs such as the Dark Triad; (2) reliance on small samples or single-response evaluations; (3) limited use of human reference data as a contextual benchmark; and (4) insufficient analysis of persona prompting as a structured source of variation. To address these issues, we adopt a large-scale repeated-sampling design using the SD3 as a probing instrument, combined with persona-based prompting and effect-size-based comparisons to human reference values. This design aims to characterize response distributions under controlled conditions while making the limitations of such comparisons explicit.

3. Methodology

We selected the Dark Triad as a structured framework to examine socially aversive response patterns in language model outputs. The three dimensions—narcissism, Machiavellianism, and psychopathy—serve as analytical lenses rather than measures of intrinsic model personality. To operationalize this approach, we use the Short Dark Triad (SD3; [8]) as a probing instrument adapted to LLM outputs.

Unit of analysis. The unit of analysis is the response distribution produced under a specific model–persona–item condition. Rather than inferring latent personality traits, repeated sampling is used to characterize central response patterns, response variability, and response stability.

Human reference values. Model outputs are contextualized using pooled SD3 summary statistics from [8] (means, standard deviations, and sample sizes), from which pooled reference values were derived (Table 4). These values provide a descriptive human reference for calculating Hedges’ g [30] and interpreting model responses on a familiar scale. Because the underlying human samples differ fundamentally from prompted LLM outputs in population, context, and measurement conditions, these comparisons are intended solely as an interpretive reference and should not be considered evidence of psychological equivalence. We use the term human reference exclusively for these pooled human-comparison values, whereas baseline refers to within-study comparison conditions, such as the neutral persona prompt.

Table 4. Overall descriptive statistics by trait (pooled SD3 reference values calculated from the original SD3 research [8]). These values serve as the human reference for illustrative purposes.

SD3-Trait	Mean (Pooled)	SD (Pooled)	n (Pooled)
Machiavellianism	3.198	0.585	998
Narcissism	2.828	0.526	998
Psychopathy	2.161	0.616	998

Effect sizes were computed using Hedges’ g [30] as a bias-corrected standardized mean difference. Given the very large number of observations, statistical significance (p-values) is expected even for small differences. We therefore emphasize effect sizes as the primary measure of practical relevance (see Table 5). The chosen thresholds follow common conventions but should be seen as heuristic rather than absolute.

Table 5. Definition of effect sizes. Effect sizes are based on Hedges’ g [30]. A medium effect corresponds to $0.5\le \left|g\right|<0.8$, relative to either the human reference (just for illustrative purposes) or the average across all models. Positive g values indicate SD3 response score levels that are more pronounced than the human reference values, while negative values indicate less pronounced response score levels.

Effect Size	Threshold
Negligible	$\left|g\right|<0.2$
Small	$0.2\le \left|g\right|<0.5$
Medium	$0.5\le \left|g\right|<0.8$
Large	$0.8\le \left|g\right|<1.3$
Extreme	$\left|g\right|\ge 1.3$

Data quality and statistical inference. All responses were required to conform to a strict Likert scale format enforced via DSPy [31]. Invalid outputs (e.g., malformed responses, refusals, or parsing errors) were excluded, resulting in minor deviations from the intended number of repetitions. Degenerate cases, such as zero variance across repetitions, were retained for descriptive reporting but interpreted cautiously because variance-dependent statistics can become unstable. Extreme effect sizes were manually inspected and discussed where likely attributable to methodological artifacts.

Each model–persona–item combination was sampled 100 times. Because repeated outputs are not independent, responses were averaged per condition before computing Hedges’ g. Consequently, the aggregated model–persona–item condition constitutes the inferential unit of the analysis. Given the exploratory nature of the study, no formal multiple-comparison corrections were applied. Results are therefore interpreted descriptively, emphasizing effect sizes and consistent response patterns rather than formal hypothesis testing.

From an application perspective, the magnitude of deviation can be understood in terms of likely perceptibility. For example, in chatbot-like scenarios, small effect sizes would often remain difficult for users to notice in isolated interactions, because individual answers may still appear well within a plausible response range. Larger deviations, however, are more likely to become apparent over time, as users encounter a more consistent pattern in the model’s responses. In this sense, effect sizes do not directly measure safety or realism, but they do indicate how noticeable a response bias may become in practice as interactions accumulate.

3.1. Overall Experiment Design

Figure 1 illustrates the overall study design, which follows a structured multi-stage workflow to systematically analyze the response behavior of large language models. The process begins with two preparatory steps identified in the literature review (see Section 2): selecting language models and defining personas. While model selection ensures a representative and diverse set of systems, persona definition operationalizes relevant demographic attributes for bias analysis.

Figure 1. Study design and model evaluation workflow. The study followed a multi-stage procedure to systematically assess large language models’ answer behavior: a screening phase, followed by a more detailed assessment phase, in which the models were asked to impersonate a set of specific personas to elicit bias-prone behavior.

The core experimental procedure is divided into two sequential phases. In the screening phase, models are evaluated using a neutral “human” persona to assess general response characteristics, in particular the effects of reasoning and the stability of repeated answers. The insights gained in this phase inform a methodological decision point: whether to include reasoning in subsequent analyses.

Based on these observations, the study proceeds to the persona assessment phase, in which models are systematically conditioned on predefined personas (e.g., gender, age, religion). In this phase, each model answers all SD3 items multiple (100) times for each persona, enabling analysis of how assumed identities influence the variation in SD3-conditioned response scores. All collected responses are aggregated and subjected to statistical analysis, including comparisons with human reference data. This staged design allows for a clear separation between general model behavior and persona-conditioned effects, ensuring both methodological rigor and interpretability of results.

3.2. Persona Definition

A persona-based approach [32] enables the systematic conditioning of language models on controlled prompt variations that describe different identities. Rather than isolating demographic variables in a strict experimental sense, these personas should be understood as holistic semantic framings that bundle multiple attributes, associations, and contextual cues. This approach therefore provides an approximation of how outputs change under different identity-related prompt contexts, rather than clean estimates of individual demographic effects. The personas in Table 6 are derived from the literature review to probe potential biases and contextual sensitivities in model outputs. We include age and gender as commonly studied dimensions, while religion is incorporated as a more sensitive and less well-understood domain. Other dimensions (e.g., profession or cultural background) were excluded due to their complexity and the difficulty of operationalizing them in a controlled and interpretable manner. Simultaneously, the pronounced effects associated with certain persona descriptions—especially in the religious domain—indicate that more nuanced and elaborate prompt framings may be revealing and deserve further exploration in future research.

Table 6. Definition of personas and data points collected per persona. Each of the 12 models answered all 27 items across all traits 100 times for each persona. Deviations from the expected 32,400 responses arose because slightly more than 100 questions were sometimes asked for technical batching reasons (e.g., OpenAI’s API batch size is 8 instead of 10 like most other model providers, which is not divisible by 100), and some models had problems delivering consistent answers in the correct DSPy format—these were counted as errors. For example, this occurred with Phi 3.5 Mini in the religious personas group. Religious personas are annotated with the approximate share of the world’s population.

Group	Persona	Persona Description	Data Points	Share (%)	Rationale
Neutral	Human	Human	64,367	$11.40$ *	(1) Screening the reasoning effect; (2) neutral persona baseline per model
Gender	Man	Man	33,086	5.86	Identify male-related biases
	Woman	Woman	33,206	5.88	Identify female-related biases
	Non-binary	Non-binary person	33,095	5.86	Identify biases related to non-binary persons
Age	8 years old	8-year-old child	33,144	5.87	Identify age-related biases
	16 years old	16-year-old teenager	33,310	5.90	Identify age-related biases
	24 years old	24-year-old trainee	33,105	5.86	Identify age-related biases
	36 years old	36-year-old employee	30,585	5.42	Identify age-related biases
	54 years old	54-year-old senior expert	30,406	5.38	Remark: senior expert might have triggered narcissism not related to the age
	72 years old	72-year-old pensioner	30,216	5.35	Identify age-related biases
Religion	Atheist (16%)	Person who was raised in an atheist or agnostic family and has not adopted any religion in adulthood	30,041	5.32	Baseline for religious biases
	Buddhist (7%)	Religious person of Buddhist belief being raised in this faith from childhood	30,061	5.32	Identify religion-related biases
	Christian (31%)	Religious person of Christian belief …	30,134	5.34	Identify religion-related biases
	Hindu (15%)	Religious person of Hindu belief …	30,054	5.32	Identify religion-related biases
	Jew (0.2%)	Religious person of Jewish belief …	30,002	5.31	Identify religion-related biases
	Muslim (24%)	Religious person of Muslim belief …	29,973	5.31	Identify religion-related biases
	Sikh (0.3%)	Religious person of Sikh belief …	30,000	5.31	Identify religion-related biases

* The “human” persona was the only persona that was assessed both with and without reasoning. All other personas were assessed without reasoning. This explains why the proportion is twice as high. However, only the proportion without reasoning has been included in our persona analyses.

Persona prompts can introduce unintended semantic associations beyond the intended variable. For example, the “54-year-old senior expert” persona may evoke cues related to status, competence, or authority in addition to age, potentially influencing responses independently of the demographic attribute of interest. Other persona descriptions may produce similar confounding effects, especially those involving culturally or normatively loaded concepts such as religion.

The neutral “human” persona serves as a baseline prompt condition rather than a true representation of unbiased behavior. Gender personas (man, woman, non-binary person), age-based personas, and religious personas are used to explore how different prompt framings influence response patterns. However, the resulting differences should be viewed as responses to the full persona descriptions rather than as direct effects of isolated demographic variables. Overall, persona-based prompting is best understood as a method for systematically varying contextual framing, enabling us to examine how LLM outputs shift across identity-related prompts while acknowledging the inherent entanglement of semantic cues within each persona.

Persona prompts may activate broader normative and cultural schemas beyond the intended demographic attribute. For example, descriptors such as “senior expert” may evoke associations with authority or competence, while religious personas may primarily trigger moral norms (e.g., prosociality, self-restraint) rather than identity-specific characteristics. As a result, observed effects should be understood not as isolated demographic influences but as responses to composite prompt-induced normative framings. Accordingly, persona effects are understood as responses to composite prompt framings, potentially including norm activation, rather than as direct estimates of demographic influences.

3.3. Model Selection

The model selection in Table 7 reflects a deliberate mix of providers, cultural origins (EU, USA, China), and access types, including free, self-hostable, and commercially restricted (“paywalled”) models, ensuring a balanced and representative overview. Rather than focusing solely on top-tier proprietary foundation models, the selection enables a realistic comparison with widely accessible open or self-hosted alternatives.

Data were collected for each model under two conditions: with reasoning (screening phase) and without reasoning (persona assessment phase). The expected observations per model are 2700 (with reasoning) and 45,900 (without reasoning), based on the experimental design. Deviations arise from two factors: unprocessable outputs due to formatting or parsing errors (notably for Phi-3.5 Mini) and differences in maximum batch sizes across providers. For instance, OpenAI models allow smaller batches (e.g., 8), while others permit up to 10, occasionally increasing repetitions. Overall, the final dataset reflects both the intended design and practical API and model constraints.

Table 7. Analyzed language models, including hosting configuration, regional origin, hosting type, and number of data points. Hosting types: institution-hosted (THL), broker-hosted (e.g., together.ai), company APIs (e.g., OpenAI, Google, xAI), and local deployments. “Type” distinguishes free (self-hostable) vs. commercial (non-self-hostable).

				Collected data points
Model	Hosting	Region	Type	With Reasoning	Without Reasoning	Total
Qwen3 VL-8B	Institute hosted (THL)	China	Free	2700	45,900	48,600
Qwen3 VL-32B	Broker hosted (together.ai)	China	Free	2700	45,900	48,600
DeepSeek V3.1	Broker hosted (together.ai)	China	Free	2700	45,890 (10 errors)	48,590
Mistral Medium 2508	Company hosted (Mistral)	Europe	Commercial	2700	45,870 (30 errors)	48,570
EuroLLM 22b	Local (on host)	Europe	Free	2698 (2 errors)	45,796 (104 errors)	48,494
Ministral3 Mini	Local (on host)	Europe	Free	2700	42,690 (3210 errors)	45,390
Gemini-2.5 Flash	Company hosted (Google)	USA	Commercial	2592 (108 errors)	47,520 (1620 repeats)	50,112
Grok-4.1 Fast	Company hosted (xAi)	USA	Commercial	2700	45,900	48,600
GPT-5 Mini	Company hosted (OpenAI)	USA	Commercial	2592 (108 errors)	45,792 (108 errors)	48,384
Phi-3.5 Mini	Local (on host)	USA	Free	2700	14,180 (31,720 errors)	16,880
Llama-4 Maverick 17B-128E	Broker hosted (together.ai)	USA	Free	2700	45,900	48,600
GPT-OSS 120B	Institute hosted (THL)	USA	Free	2700	45,800 (100 errors)	48,500

Remark: Reported errors indicate how many responses could not be generated in the prescribed response schema. Repeats indicate how many excess responses were generated (due to interrupted and resumed data collection runs).

Table 7. Analyzed language models, including hosting configuration, regional origin, hosting type, and number of data points. Hosting types: institution-hosted (THL), broker-hosted (e.g., together.ai), company APIs (e.g., OpenAI, Google, xAI), and local deployments. “Type” distinguishes free (self-hostable) vs. commercial (non-self-hostable).

				Collected data points
Model	Hosting	Region	Type	With Reasoning	Without Reasoning	Total
Qwen3 VL-8B	Institute hosted (THL)	China	Free	2700	45,900	48,600
Qwen3 VL-32B	Broker hosted (together.ai)	China	Free	2700	45,900	48,600
DeepSeek V3.1	Broker hosted (together.ai)	China	Free	2700	45,890 (10 errors)	48,590
Mistral Medium 2508	Company hosted (Mistral)	Europe	Commercial	2700	45,870 (30 errors)	48,570
EuroLLM 22b	Local (on host)	Europe	Free	2698 (2 errors)	45,796 (104 errors)	48,494
Ministral3 Mini	Local (on host)	Europe	Free	2700	42,690 (3210 errors)	45,390
Gemini-2.5 Flash	Company hosted (Google)	USA	Commercial	2592 (108 errors)	47,520 (1620 repeats)	50,112
Grok-4.1 Fast	Company hosted (xAi)	USA	Commercial	2700	45,900	48,600
GPT-5 Mini	Company hosted (OpenAI)	USA	Commercial	2592 (108 errors)	45,792 (108 errors)	48,384
Phi-3.5 Mini	Local (on host)	USA	Free	2700	14,180 (31,720 errors)	16,880
Llama-4 Maverick 17B-128E	Broker hosted (together.ai)	USA	Free	2700	45,900	48,600
GPT-OSS 120B	Institute hosted (THL)	USA	Free	2700	45,800 (100 errors)	48,500

Remark: Reported errors indicate how many responses could not be generated in the prescribed response schema. Repeats indicate how many excess responses were generated (due to interrupted and resumed data collection runs).

3.4. LLM Instrumentation

Readers seeking more detail on our analysis of the language models should consult Appendix A and Listings A1–A5. All code used for instrumenting the language models, the code used for the analysis, and the collected data are openly available in this Zenodo repository: https://zenodo.org/records/20110630 (accessed on 28 July 2026). All details regarding the conduct of the study, the configuration of the LLMs, and the data recording can be traced there in detail.

For most readers, it suffices to note that we used the Stanford-based LLM integration library DSPy (Version 3.1) for data collection. DSPy [31,33] is a declarative framework for building modular AI software with structured code, enabling consistent execution across models. In this work, we used only its structured features to ensure comparability across models and did not use its automatic prompt optimization capabilities, as these could have influenced model behavior.

All 27 SD3 items were answered by each model using a fixed Likert scale, exactly following the human questionnaire responses [8]. Non-conforming responses were discarded under strict DSPy output constraints. Our setup defined a structured input/output schema (see Listing A4 in Appendix A), disabled caching to ensure independent answers, and used both reasoning (ChainOfThought) and non-reasoning (Predict) strategies. Each persona–item combination was queried 100 times; responses were converted into numerical Likert scale values, including reverse coding, and saved with metadata in JSON files for analysis.

3.5. Screening Phase

The aim of the screening phase was twofold. First, we examined whether models produce significantly different results when required to justify their answers. Prior work [1] shows that reasoning can influence responses to multiple-choice questions, such as Likert scale items, but its effects are unpredictable: it may amplify or reduce bias. In some “highly moderated” contexts (violence, harm, hate speech, discrimination, sexual content, illegal activities, etc.), additional tokens generated through reasoning can even trigger stereotypes and lead to more problematic answers. To investigate this for the Dark Triad, we had the “human” persona complete the SD3 for all models, both with and without reasoning. Based on the results, we then selected the less computationally intensive inference method without reasoning (DSPy Predict) for the remaining personas, as it was expected to trigger bias more strongly. Results are reported in Section 4.1; to anticipate the main finding, differences between reasoning and non-reasoning conditions had no substantial effect.

Second, we analyzed response consistency, i.e., whether models answer psychometric questions randomly or consistently. Using a stability definition from prior work [1], we measured how often, across n repeated responses, the same answer was chosen and took the most frequent response as the stability score (e.g., if, at most, the same answer was given 50 times to the same question asked 100 times, this corresponds to an answer stability of 50%). Results (see Section 4.1) show no significant differences between reasoning and non-reasoning conditions, though some model-specific patterns emerge.

3.6. Persona Assessment Phase

Following the screening phase, the main analysis examined variation in SD3-conditioned response scores across selected language models using a persona-based approach. As reasoning showed negligible effects, subsequent experiments omitted it (DSPy Predict) to improve stability and reduce computational cost. Each model was conditioned on predefined personas (see Table 6) and prompted to answer all 27 SD3 items. For each model–persona–item combination, 100 responses were generated and averaged to mitigate repeated sampling effects. Responses were mapped onto a fixed Likert scale, including reverse coding, to enable comparison with human reference data. Persona-defining attributes (e.g., gender, age, religion) were varied, while other factors were held constant to assess potential biases and contextual effects. Results were aggregated by persona, trait, and model, and compared with human and neutral persona baseline data using effect sizes (Hedges’ g) to quantify persona effects.

4. Results

The reader can obtain an overview of the collected data in Table 8. We will now discuss these in detail in the following sections.

Table 8. Overview of the generated Dark Triad LLM dataset.

Category	Description
Objects of analysis	Large language models (LLMs)
Number of models	12
Model providers	OpenAI, Google, xAI, Mistral, Alibaba, DeepSeek, Meta, EuroLLM (EU Horizon, European Research Council)
Model origins	USA (6), EU (3), China (3)
Model types	Proprietary (commercial) and free/open-weight models
Model hostings	LLM as a Service (commercial), model broker, academic/self-hosted deployments, local on data collection host (access via OpenAI-compatible APIs)
Language	English
Psychometric instrument	Short Dark Triad (SD3) [8]
Questionnaire structure	27 items, 3 traits (Machiavellianism, narcissism, psychopathy)
Response scale	5-point Likert scale
Response conditions	With vs. without reasoning (screening phase); persona-conditioned responses without reasoning (assessment phase)
Personas	Neutral baseline (1), gender (3), age (6), religion (7)
Repetitions per item	100 uncached responses per model, persona and item (averaged to mitigate repeated sampling effects)
Responses per persona	Approximately 30,000 (depending on model and batching constraints; see Table 6)
Responses per model	Approximately 48,000 (depending on model and instrumenting constraints; see Table 7)
Total LLM responses	Approximately 540,000+
Human reference sample	SD3 reference data (pooled $n=998$) from [8]
Statistical comparison	Hedges’ g effect sizes
Heuristic reporting threshold	Effect size $\left|g\right|>0.2$
Primary analysis focus	Deviations from human reference, persona-induced variation, and model-specific response distributions due to trait-related prompt framings
Data collection period	March + April 2026
Data collection method	DSPy-based structured prompting with strict output constraints and no caching
Data format	JSON files containing responses and metadata (persona, item, trait, model)
Dataset availability	Zenodo link: https://doi.org/10.5281/zenodo.20110630 (accessed on 28 July 2026)

4.1. Screening Phase Results

During the screening phase, we examined whether the inclusion of explicit reasoning affects SD3-conditioned response patterns in LLM outputs. As shown in Table 9, differences between reasoning and non-reasoning conditions are associated by effect sizes that are negligible (Hedges’ $\left|g\right|<0.2$). In practical terms, this indicates that the two prompting strategies yield highly similar response distributions, with only minor shifts in mean scores.

Across all three SD3 dimensions, responses generated under reasoning prompts tend to yield slightly lower scores (i.e., less pronounced SD3-related response patterns). However, given the very small effect sizes, these differences are unlikely to be meaningful in applied settings. The values reported in Table 9 should therefore be interpreted as descriptive summaries of response distributions under different prompting conditions rather than as substantively distinct behavioral regimes.

A second focus of the screening phase was response stability under repeated sampling. Figure 2 shows stability scores for each model and SD3 item. Overall, reasoning and non-reasoning conditions produce similar stability patterns, with a slight shift toward lower stability under reasoning prompts. This is consistent with prior work suggesting that additional generated tokens can introduce variability [1].

Table 9. Comparison of variation in SD3-conditioned response scores (Machiavellianism, Narcissism, Psychopathy) between reasoning vs. no-reasoning conditions using a neutral “human” persona. For each trait, n, M, and $SD$ are reported; items are averaged, yielding $n=12\times 9=108$ aggregated model–item observations. $C{I}_{L/U}$ denotes the $95\%$ confidence interval.

	With Reasoning			Without Reasoning
Trait	Mean	SD	n	Mean	SD	n	g	Var(g)	${\mathit{CI}}_{\mathit{L}}$	${\mathit{CI}}_{\mathit{U}}$	Effect	Direction
Machiavellianism	2.607	1.138	108	2.806	1.288	108	−0.164	0.019	−0.431	0.103	Negligible	Non-Reasoning > Reasoning
Narcissism	2.988	0.859	108	3.129	0.858	108	−0.163	0.019	−0.430	0.104	Negligible	Non-Reasoning > Reasoning
Psychopathy	2.175	0.799	108	2.259	1.003	108	−0.092	0.019	−0.359	0.175	Negligible	Non-Reasoning > Reasoning

Table 9. Comparison of variation in SD3-conditioned response scores (Machiavellianism, Narcissism, Psychopathy) between reasoning vs. no-reasoning conditions using a neutral “human” persona. For each trait, n, M, and $SD$ are reported; items are averaged, yielding $n=12\times 9=108$ aggregated model–item observations. $C{I}_{L/U}$ denotes the $95\%$ confidence interval.

	With Reasoning			Without Reasoning
Trait	Mean	SD	n	Mean	SD	n	g	Var(g)	${\mathit{CI}}_{\mathit{L}}$	${\mathit{CI}}_{\mathit{U}}$	Effect	Direction
Machiavellianism	2.607	1.138	108	2.806	1.288	108	−0.164	0.019	−0.431	0.103	Negligible	Non-Reasoning > Reasoning
Narcissism	2.988	0.859	108	3.129	0.858	108	−0.163	0.019	−0.430	0.104	Negligible	Non-Reasoning > Reasoning
Psychopathy	2.175	0.799	108	2.259	1.003	108	−0.092	0.019	−0.359	0.175	Negligible	Non-Reasoning > Reasoning

Figure 2. Response stability across models and SD3 items. Radar plots show stability scores for the human persona under reasoning and non-reasoning conditions. Grey areas denote responses without reasoning; red areas denote responses with reasoning. Higher values (closer to 100) indicate greater stability. Dashed rings mark thresholds: 50% (dominant response < half of outputs) and 20% (random guessing on a 5-point Likert scale). Larger areas indicate more stable responses; values near 20% suggest random or inconsistent responding.

Figure 2. Response stability across models and SD3 items. Radar plots show stability scores for the human persona under reasoning and non-reasoning conditions. Grey areas denote responses without reasoning; red areas denote responses with reasoning. Higher values (closer to 100) indicate greater stability. Dashed rings mark thresholds: 50% (dominant response < half of outputs) and 20% (random guessing on a 5-point Likert scale). Larger areas indicate more stable responses; values near 20% suggest random or inconsistent responding.

Most models exhibit relatively high stability means (see Table 10), indicating that repeated responses to the same prompt tend to cluster around a dominant answer. Table 10 shows the relative share of the dominant answer. However, some model-specific deviations are evident. For example, EuroLLM 22B shows substantially lower stability, with values approaching the threshold of random responding for some items. This suggests that not all models produce equally consistent response distributions under repeated prompting. Table 11 presents a detailed breakdown of the Dark Triad values by model. Here, too, we observe that all models, except Ministral 3 Mini, show no descriptively relevant differences between reasoning and Non-reasoning.

All analyses in this paper are based on repeated samples from the same model–persona–item (prompt) configurations. However, to mitigate problematic effects of repeated sampling (e.g., violations of independence, underestimated standard errors, inflated statistical significance, overly narrow confidence intervals, overweighting of heavily repeated model–persona–item configurations) in our statistical analysis, we averaged the item values. Thus, we count each model–persona–item combination as a single observation by treating the 100 repetitions not as 100 separate events, but as one event represented by the mean value across the 100 repetitions.

Table 10. Comparison of model answer stability (each model was tested on $n=17\times 27=459$ aggregated persona–item observations; each item was repeated 100 times and averaged. Models are sorted by decreasing stability. The stability of 1.0 means identical answers every time; a stability of 0.2 (five-point Likert scale) indicates uniformly random responses.

Model	Mean Stability	SD	n
Mistral Medium 2508	0.931	0.132	459
Llama-4 Maverick 17B-128E	0.920	0.150	459
Grok-4.1 Fast	0.903	0.158	459
Qwen3 VL-8B	0.900	0.173	459
Qwen3 VL-32B	0.879	0.175	459
GPT-5 Mini	0.802	0.181	459
GPT-OSS 120B	0.785	0.170	458
Gemini-2.5 Flash	0.760	0.190	459
Ministral3 Mini	0.748	0.219	459
DeepSeek V3.1	0.704	0.206	459
Phi-3.5 Mini	0.668	0.215	458
EuroLLM 22b	0.572	0.160	459

Table 10. Comparison of model answer stability (each model was tested on $n=17\times 27=459$ aggregated persona–item observations; each item was repeated 100 times and averaged. Models are sorted by decreasing stability. The stability of 1.0 means identical answers every time; a stability of 0.2 (five-point Likert scale) indicates uniformly random responses.

Model	Mean Stability	SD	n
Mistral Medium 2508	0.931	0.132	459
Llama-4 Maverick 17B-128E	0.920	0.150	459
Grok-4.1 Fast	0.903	0.158	459
Qwen3 VL-8B	0.900	0.173	459
Qwen3 VL-32B	0.879	0.175	459
GPT-5 Mini	0.802	0.181	459
GPT-OSS 120B	0.785	0.170	458
Gemini-2.5 Flash	0.760	0.190	459
Ministral3 Mini	0.748	0.219	459
DeepSeek V3.1	0.704	0.206	459
Phi-3.5 Mini	0.668	0.215	458
EuroLLM 22b	0.572	0.160	459

Since we were unable to identify any significant differences between reasoning and non-reasoning, either in the cross-model comparison (Table 9) or in the model-specific analysis (Table 11), we decided to conduct the persona analysis in non-reasoning mode. In the model-specific comparison as well, trait values were generally higher in non-reasoning mode than in reasoning mode, suggesting that effects would be a bit more likely to be observed in non-reasoning conditions.

4.2. Persona Results

Figure 3 summarizes SD3-conditioned response scores aggregated across all personas and models. When interpreted relative to SD3 reference values derived from human data, a consistent pattern emerges: Machiavellianism-related response scores tend to be lower, narcissism-related scores slightly higher, and psychopathy-related scores largely within the reference distribution. Figure 4 summarizes mean Dark Triad scores across gender, age, and religion personas relative to the neutral persona and the human reference. Overall, significant deviations from the neutral persona baseline data were most pronounced for religion-based personas, less frequent for age-based personas, and rare for gender-based personas.

Figure 3. Model response scores (Machiavellianism, narcissism, psychopathy; $n=17\times 9=153$ per model, averaged). Blue: model means (±1 SD). Red dashed: human mean; red band: ±1 SD. Models sorted by psychopathy. Response levels of models are lower for Machiavellianism and slightly higher for narcissism vs. humans; psychopathy lies mostly within the human range.

Figure 4. Mean scores (±1 SD) for Machiavellianism, narcissism, and psychopathy across persona groups ($n=12\times 9=108$ per persona; item values averaged). Blue points show persona means with SD error bars. Red dashed lines and shading indicate the human reference (±1 SD); black dashed lines and grey shading indicate the neutral baseline (±1 SD). Variation across panels highlights differences in SD3-conditioned response scores, with notably lower Machiavellianism and psychopathy among religious personas.

Age-based personas: Table A2 (in Appendix C) shows results for age-based personas. Of 18 comparisons, five (27.8%) showed descriptively relevant differences ($\left|g\right|>0.2$), while 13 (72.2%) did not. Differences appeared across all traits (1 Machiavellianism, 2 narcissism, 2 psychopathy) and were more common in older personas (once for an 8-year-old, twice each for 54- and 72-year-olds). Effects were mostly small (four small, one medium), none large, and largely negative (four below baseline). Overall, age-based personas produced mostly small, infrequent deviations, more often associated with older personas.

Gender-based personas: Across nine persona–trait comparisons for gender-based personas (see Table A3 in Appendix C), one case (11.1%) showed a descriptively relevant deviation from the neutral baseline ($\left|g\right|>0.2$), while eight (88.9%) did not. The only effect was a small decrease in Machiavellianism for the non-binary persona; no effects were found for narcissism or psychopathy. Overall, gender-based prompts had minimal impact on model response scores.

Table 11. Comparison of reasoning and non-reasoning conditions (each model impersonated the neutral ‘human’ persona and was tested on n = 27 items; each item was repeated 100 times and averaged to avoid repeated-sampling effects). Relevance threshold: $\left|g\right|>0.2$. $C{I}_{L/U}$ show the 95%-Confidence Interval.

	With Reasoning			Without Reasoning
Model	Mean	SD	n	Mean	SD	n	g	Var(g)	${\mathit{CI}}_{\mathit{L}}$	${\mathit{CI}}_{\mathit{U}}$	Rel.	Effect	Direction
DeepSeek V3.1	2.284	0.796	27	2.522	0.973	27	−0.264	0.075	−0.800	0.271	True	Small	Non-Reasoning > Reasoning
EuroLLM 22b	3.056	0.962	27	3.180	0.767	27	−0.141	0.074	−0.675	0.393	False	Negligible	Non-Reasoning > Reasoning
GPT-5 Mini	2.714	1.014	27	2.855	0.884	27	−0.146	0.074	−0.680	0.388	False	Negligible	Non-Reasoning > Reasoning
GPT-OSS 120B	2.509	1.003	27	2.545	0.968	27	−0.037	0.074	−0.570	0.497	False	Negligible	Non-Reasoning > Reasoning
Gemini-2.5 Flash	2.330	1.239	27	2.315	1.202	27	0.012	0.074	−0.521	0.546	False	Negligible	Reasoning > Non-Reasoning
Grok-4.1 Fast	2.022	1.128	27	2.417	1.479	27	−0.296	0.075	−0.833	0.240	True	Small	Non-Reasoning > Reasoning
Llama-4 Maverick 17B-128E	2.652	0.937	27	2.650	1.301	27	0.002	0.074	−0.532	0.535	False	Negligible	Reasoning > Non-Reasoning
Ministral3 Mini	2.434	0.702	27	2.976	0.609	27	−0.813	0.081	−1.369	−0.256	True	Large	Non-Reasoning > Reasoning
Mistral Medium 2508	2.436	1.167	27	2.926	1.310	27	−0.389	0.076	−0.928	0.150	True	Small	Non-Reasoning > Reasoning
Phi-3.5 Mini	3.233	0.941	27	3.181	0.879	27	0.056	0.074	−0.478	0.589	False	Negligible	Reasoning > Non-Reasoning
Qwen3 VL-32B	2.823	0.808	27	2.743	1.306	27	0.072	0.074	−0.462	0.606	False	Negligible	Reasoning > Non-Reasoning
Qwen3 VL-8B	2.589	0.639	27	2.465	1.219	27	0.126	0.074	−0.408	0.660	False	Negligible	Reasoning > Non-Reasoning

Table 11. Comparison of reasoning and non-reasoning conditions (each model impersonated the neutral ‘human’ persona and was tested on n = 27 items; each item was repeated 100 times and averaged to avoid repeated-sampling effects). Relevance threshold: $\left|g\right|>0.2$. $C{I}_{L/U}$ show the 95%-Confidence Interval.

	With Reasoning			Without Reasoning
Model	Mean	SD	n	Mean	SD	n	g	Var(g)	${\mathit{CI}}_{\mathit{L}}$	${\mathit{CI}}_{\mathit{U}}$	Rel.	Effect	Direction
DeepSeek V3.1	2.284	0.796	27	2.522	0.973	27	−0.264	0.075	−0.800	0.271	True	Small	Non-Reasoning > Reasoning
EuroLLM 22b	3.056	0.962	27	3.180	0.767	27	−0.141	0.074	−0.675	0.393	False	Negligible	Non-Reasoning > Reasoning
GPT-5 Mini	2.714	1.014	27	2.855	0.884	27	−0.146	0.074	−0.680	0.388	False	Negligible	Non-Reasoning > Reasoning
GPT-OSS 120B	2.509	1.003	27	2.545	0.968	27	−0.037	0.074	−0.570	0.497	False	Negligible	Non-Reasoning > Reasoning
Gemini-2.5 Flash	2.330	1.239	27	2.315	1.202	27	0.012	0.074	−0.521	0.546	False	Negligible	Reasoning > Non-Reasoning
Grok-4.1 Fast	2.022	1.128	27	2.417	1.479	27	−0.296	0.075	−0.833	0.240	True	Small	Non-Reasoning > Reasoning
Llama-4 Maverick 17B-128E	2.652	0.937	27	2.650	1.301	27	0.002	0.074	−0.532	0.535	False	Negligible	Reasoning > Non-Reasoning
Ministral3 Mini	2.434	0.702	27	2.976	0.609	27	−0.813	0.081	−1.369	−0.256	True	Large	Non-Reasoning > Reasoning
Mistral Medium 2508	2.436	1.167	27	2.926	1.310	27	−0.389	0.076	−0.928	0.150	True	Small	Non-Reasoning > Reasoning
Phi-3.5 Mini	3.233	0.941	27	3.181	0.879	27	0.056	0.074	−0.478	0.589	False	Negligible	Reasoning > Non-Reasoning
Qwen3 VL-32B	2.823	0.808	27	2.743	1.306	27	0.072	0.074	−0.462	0.606	False	Negligible	Reasoning > Non-Reasoning
Qwen3 VL-8B	2.589	0.639	27	2.465	1.219	27	0.126	0.074	−0.408	0.660	False	Negligible	Reasoning > Non-Reasoning

Religion-based personas: Across 21 persona–trait comparisons for religion-based personas (see Table A4 in Appendix C), 18 cases (85.7%) showed descriptively relevant deviations from the neutral baseline, while three (14.3%) showed none. This represents the highest deviation rate among persona groups. Descriptively relevant differences occurred equally across traits (six each for Machiavellianism, narcissism, and psychopathy) and were evenly distributed across religious personas (Buddhist, Christian, Muslim, Hindu, Jewish, Sikh; three each). Effect sizes ranged from small to large (seven small, nine medium, two large). Compared to age and gender, religion-based personas showed more frequent and stronger deviations. All 16 relevant effects were lower than the neutral baseline, indicating a consistent downward shift. Overall, religion-based prompts had the strongest and most systematic impact on model outputs.

Model-specific trait results: As shown in Table A5 (in Appendix C), 22 of 36 model–trait comparisons (61.1%) showed meaningful deviations ($\left|g\right|>0.2$) from the neutral persona baseline, while 14 (38.9%) did not, indicating that most comparisons diverged from baseline. Deviations were most frequent for psychopathy ($n=9$), followed by Machiavellianism ($n=7$) and narcissism ($n=6$). Effects were small to medium (eight small, 14 medium), with no large effects, suggesting limited practical magnitude. Effects were asymmetric: in 18 of 22 cases, model means were below baseline, indicating a general shift toward lower Dark Triad scores. At the model level, Llama-4 Maverick 17B-128E and Gemini-2.5 Flash showed the most deviations ($n=3$ each), all below baseline. Several models (e.g., DeepSeek V3.1, GPT-OSS 120B) showed two deviations, also below baseline, while EuroLLM 22b was the only model with two above-baseline deviations. Overall, deviations were common but predominantly reflected reduced score-level in a given SD3 dimension, especially for psychopathy, whereas narcissism remained relatively stable.

5. Discussion

The reader in a hurry will find our main findings summarized in Table 12. In line with our terminology, comparisons to pooled human SD3 data are discussed as comparisons to a human reference, whereas comparisons to the neutral persona condition are treated as baseline contrasts. We interpret the findings with respect to the two research questions introduced in Section 1 and discuss their methodological implications. Throughout, SD3 scores are interpreted as properties of SD3-conditioned response patterns under controlled prompting conditions rather than as evidence of stable personality traits. Specifically, we address:

• RQ1: How do response distributions generated by large language models under SD3 prompting relate to human SD3 reference distributions as an interpretive benchmark?
• RQ2: How does persona-based prompt framing influence SD3-conditioned response distributions in LLM outputs, and to what extent can these effects be understood as context-dependent (e.g., norm-activating) variations?

Table 12. Summary of main findings.

Domain	Comparison	Main Finding	Evidence
Screening phase	Reasoning vs. no reasoning	Across all three SD3 dimensions, reasoning produced slightly lower scores than no reasoning, but differences were negligible in magnitude and not practically meaningful.	Hedges’ $\left|g\right|<0.2$
	Response stability	Most models showed consistent repeated responses under fixed prompt conditions, suggesting reproducible output patterns rather than purely random variation. Stability nevertheless differed substantially between models.	Mean stability: $0.574$–$0.931$
RQ1	Human reference: Machiavellianism	Model outputs tended to fall below pooled human SD3 reference values.	Consistent downward shift
	Human reference: Narcissism	Model outputs tended to be slightly above pooled human SD3 reference values.	Small upward shift
	Human reference: psychopathy	Model outputs were generally within the human SD3 reference range, though some model-specific descriptively relevant deviations occurred.	Broad overlap with reference range
RQ2	Gender personas	Gender prompting had minimal impact on SD3-conditioned outputs.	$1/9$ observed descriptively relevant deviations
	Age personas	Age prompting produced occasional, mostly small deviations, with somewhat stronger effects among older personas.	$5/18$ observed descriptively relevant deviations
	Religion personas	Religion prompting showed the strongest and most systematic effect pattern, especially lowering Machiavellianism and psychopathy scores.	$18/22$ observed descriptively relevant deviations
Interpretation	Persona mechanism	Persona effects are best understood as context-dependent norm activation rather than simulation of stable demographic traits.	Pattern consistency across persona groups
Model heterogeneity	Model-specific differences	Models varied substantially in trait levels and persona sensitivity; most deviations were descriptively relevant and below the neutral baseline.	22/36 observed model–trait descriptively relevant deviations; $18/22$ below baseline
Conclusion	Methodological takeaway	SD3-style psychometric probing is useful for characterizing structured patterns in LLM output, but should not be interpreted as measuring human-like personality traits.	Overall study pattern

Table 12. Summary of main findings.

Domain	Comparison	Main Finding	Evidence
Screening phase	Reasoning vs. no reasoning	Across all three SD3 dimensions, reasoning produced slightly lower scores than no reasoning, but differences were negligible in magnitude and not practically meaningful.	Hedges’ $\left|g\right|<0.2$
	Response stability	Most models showed consistent repeated responses under fixed prompt conditions, suggesting reproducible output patterns rather than purely random variation. Stability nevertheless differed substantially between models.	Mean stability: $0.574$–$0.931$
RQ1	Human reference: Machiavellianism	Model outputs tended to fall below pooled human SD3 reference values.	Consistent downward shift
	Human reference: Narcissism	Model outputs tended to be slightly above pooled human SD3 reference values.	Small upward shift
	Human reference: psychopathy	Model outputs were generally within the human SD3 reference range, though some model-specific descriptively relevant deviations occurred.	Broad overlap with reference range
RQ2	Gender personas	Gender prompting had minimal impact on SD3-conditioned outputs.	$1/9$ observed descriptively relevant deviations
	Age personas	Age prompting produced occasional, mostly small deviations, with somewhat stronger effects among older personas.	$5/18$ observed descriptively relevant deviations
	Religion personas	Religion prompting showed the strongest and most systematic effect pattern, especially lowering Machiavellianism and psychopathy scores.	$18/22$ observed descriptively relevant deviations
Interpretation	Persona mechanism	Persona effects are best understood as context-dependent norm activation rather than simulation of stable demographic traits.	Pattern consistency across persona groups
Model heterogeneity	Model-specific differences	Models varied substantially in trait levels and persona sensitivity; most deviations were descriptively relevant and below the neutral baseline.	22/36 observed model–trait descriptively relevant deviations; $18/22$ below baseline
Conclusion	Methodological takeaway	SD3-style psychometric probing is useful for characterizing structured patterns in LLM output, but should not be interpreted as measuring human-like personality traits.	Overall study pattern

5.1. Discussion of Screening Results

The screening phase assesses whether SD3-style probing yields sufficiently stable response distributions to support subsequent analyses. Overall, the results suggest that it does. Reasoning prompts had only negligible effects on SD3-conditioned scores. Across nearly all models, explicit reasoning produced response distributions nearly identical to those obtained without reasoning, indicating that chain-of-thought prompting does not materially affect SD3 outcomes. The slightly lower scores observed for reasoning prompts may reflect a weak alignment-related regularization effect, but the magnitude is too small to materially affect the substantive interpretation.

Repeated sampling further demonstrated that LLM outputs are not purely random but exhibit reproducible response distributions under fixed prompting conditions. Although repeated generations from the same model–prompt configuration are not statistically independent, they provide a practical approximation of the model’s conditional response distribution. Repeated sampling can thus serve as a methodological basis for psychometric-style probing, with the important caveat that the resulting analyses are best understood as descriptive rather than as conventional inferential statistics.

Because very large numbers of observations make statistical significance likely even for trivial differences, effect sizes provide the more informative basis for interpretation. Accordingly, the screening results support the use of SD3 as a structured probing instrument and reinforce that the resulting scores characterize context-dependent response behavior rather than latent model traits.

5.2. Discussion of Human Reference Results (RQ1)

RQ1 examined how SD3-conditioned response distributions generated by large language models relate to pooled human SD3 reference distributions used as an interpretive benchmark. The results suggest that the relationship between model outputs and the human reference is neither random nor uniform across traits. Instead, a differentiated pattern emerges: Machiavellianism-related scores are systematically lower, narcissism-related scores are modestly higher, and psychopathy-related scores remain mostly within the human reference range. This asymmetry suggests that the model’s responses are not merely driven by a global shift toward lower or higher endorsement of socially aversive items. Rather, all models appear to respond differently depending on the trait domain being probed.

One plausible interpretation is that these differences reflect the effects of instruction tuning, safety alignment, and the normative regularities of the training data. Lower Machiavellianism-related responses may indicate a general pattern of avoiding endorsement of manipulative or exploitative statements, whereas slightly elevated narcissism-related scores may reflect the relative acceptability of self-referential or self-affirming language in questionnaire settings. By contrast, psychopathy-related responses overlap broadly with the human reference range, suggesting that not all socially aversive item types are equally suppressed or amplified. Thus, RQ1 offers not only a descriptive benchmark comparison but also evidence that trait-related prompt content shapes model behavior in domain-specific ways.

At the same time, this pattern should not be overstated. Considerable heterogeneity remains across models, and some deviate more strongly from the aggregate trend than others. The comparison with pooled human SD3 values therefore does not identify a single “LLM Dark Triad profile” but rather a recurring aggregate response pattern alongside substantial model-specific variation. The human benchmark is useful for contextualizing deviations, but it does not imply psychological equivalence between human self-reports and model-generated questionnaire responses.

Taken together, our exploratory findings suggest that SD3-style probing could help reveal structured differences between human reference distributions and LLM-generated response distributions. The value of this comparison lies less in assigning human-like trait levels to models than in identifying systematic response biases across construct domains. In this sense, RQ1 provides a reference map of how LLM outputs align with, diverge from, or selectively overlap with human psychometric reference patterns under controlled questionnaire prompting.

5.3. Discussion of Persona Assessment Results (RQ2)

The persona assessment primarily addresses RQ2 while providing additional context for RQ1. Overall, the response pattern relative to the SD3 reference values remained stable across persona conditions: Machiavellianism-related scores tended to be lower than the human reference, narcissism-related scores were slightly higher, and psychopathy-related scores were largely within the reference range. Persona prompting therefore influenced the magnitude of deviations more than the overall structure of the response distributions.

The effects of persona prompting were highly uneven across persona types. Gender prompts produced only minimal changes, while age prompts resulted in occasional, mostly small shifts. In contrast, religious personas consistently produced the strongest effects across models, mainly by reducing Machiavellianism- and psychopathy-related scores. The consistency of these effects across different religious personas and model architectures suggests that they reflect a robust contextual influence rather than model-specific artifacts.

A plausible explanation is that persona prompts function primarily as contextual cues that activate normative associations learned during pretraining and reinforced through alignment. Under this interpretation, persona descriptions do not induce coherent identity-specific behavior but instead trigger context-dependent response patterns. Religious framings appear to activate moral schemas associated with prosociality and self-restraint, resulting in systematically less socially aversive responses. Consequently, the observed differences are better understood as prompt-induced norm activation than as evidence of simulated personality traits.

The results also highlight substantial model-specific heterogeneity. Models differed both in their baseline SD3-conditioned scores and in their sensitivity to persona prompts, indicating that persona effects depend on model-specific training data and alignment strategies. Moreover, persona prompts inevitably introduce composite semantic cues rather than isolated demographic variables. For example, a prompt describing a “54-year-old senior expert” likely conveys experience, authority, and social status in addition to age. Accordingly, observed effects should be interpreted as responses to the complete prompt framing rather than to individual demographic characteristics.

A particularly instructive example is the “54-year-old senior expert” persona. The increase in narcissism-related scores observed in this condition should not be interpreted as a pure age effect, because the prompt does not vary age alone. Rather, the descriptor “senior expert” likely activates additional associations, such as authority, competence, social status, and self-enhancement. Importantly, this is not merely a limitation of the prompt design but also a substantive methodological finding. It illustrates that persona prompts in LLM studies often function as composite semantic cues, simultaneously activating multiple learned associations. In this sense, the result strengthens our broader interpretation that persona effects are better understood as prompt-induced norm activation shaped by bundled semantic signals rather than as clean simulations of isolated demographic variables. Future research should therefore use more neutral age descriptions, such as “54-year-old person,” or factorial prompt designs that separate age from professional or status-related cues.

Taken together, these findings reinforce our central conclusion: SD3-conditioned outputs are systematically shaped by prompting context rather than reflecting stable underlying traits. Persona prompting therefore provides insight into how contextual framing influences model behavior, but it should not be interpreted as evidence that language models possess coherent, human-like personalities.

5.4. Model Heterogenity

The heterogeneity observed across models raises the question of which properties of the models underlie these differences. The present design does not allow clear attribution to any single factor because the systems differ in architecture, scale, training data, instruction tuning, safety alignment, and deployment context. Some characteristics of proprietary models are also not fully transparent. Therefore, the results should not be interpreted as evidence that any one feature directly determines SD3 response patterns. At the same time, the differences are unlikely to reflect random fluctuations alone. Most models produced stable responses under fixed prompt conditions, suggesting that the heterogeneity reflects systematic properties of the model pipelines. A plausible interpretation is that between-model differences arise mainly from post-training calibration—especially instruction tuning and safety alignment—as well as from differences in training data. These possibilities remain exploratory, but they help explain why SD3-conditioned response profiles may diverge even under identical questionnaire prompts.

5.5. Threats to Validity and Limitations of This Study

The following limitations should be considered when interpreting our results. First, regarding construct validity, the Short Dark Triad (SD3) is a self-report instrument for stable human traits, whereas LLMs lack stable identities and generate probabilistic outputs [34]. Thus, SD3 scores reflect response patterns under a questionnaire format rather than underlying traits. Second, human reference values should be understood only as exploratory, descriptive references. The pooled SD3 values from [8] differ from the LLM setting in cultural context, language, demographics, and measurement conditions, limiting direct comparability, because they originate from historically and culturally different human samples measured under conditions (North American college students, 2014) that are not directly comparable to the present LLM evaluation setting (2026 conditions, cross-country, and multilingual models). Third, internal validity may be affected by prompting. Despite controlled formats, model outputs remain sensitive to prompt wording and system instructions [35,36], and persona prompts may introduce unintended semantic cues (e.g., the 54-year-old senior expert triggered narcissism, which is likely not correlated with age; the Buddhist persona lowered narcissism, whereas other religious personas did not; see Figure 4). Fourth, measurement validity is influenced by response variability and data handling. Invalid outputs were excluded, leading to minor sample imbalances, and repeated sampling ($n=100$) reduces but does not eliminate stochastic variation. Fifth, conclusion validity is limited by dependence on repeated sampling and aggregation. Large sample sizes may inflate significance, while variance issues can affect effect sizes. Results are therefore interpreted descriptively, focusing on consistent patterns rather than isolated tests. By averaging repeated outputs within each model–persona–item condition, the analysis reduces pseudoreplication and emphasizes central, observable response pattern trends, but it also becomes less sensitive to rare or extreme responses; the reported findings therefore characterize typical response patterns more than tail risks. Sixth, external validity is constrained by simplified personas and a non-exhaustive set of models, limiting generalizability to broader populations of users and LLMs [37]. Finally, ecological validity is limited, as structured questionnaire responses differ from real-world, multi-turn LLM interactions [38].

6. Conclusions

Our analysis makes a dual contribution: substantively, it extends LLM psychometric evaluation to the Dark Triad; methodologically, it establishes a large-scale framework for probing socially aversive response patterns in language models. Across 12 models and more than 540,000 responses, stable, non-random, and context-sensitive response patterns emerged that differ systematically by trait domain and persona framing. The overall pattern supports a normative activation account: persona prompts do not appear to elicit coherent simulated identities but instead activate learned social and moral associations embedded in training and alignment.

Relative to human SD3 reference values, Machiavellianism-related scores tended to be lower, narcissism-related scores slightly higher, and psychopathy-related scores largely within the human reference range. The relationship between LLM-generated SD3 response distributions and human SD3 reference values thus follows a differentiated cross-trait pattern rather than a uniform shift. This suggests that psychometric probing can detect regularities in model behavior: some socially aversive response patterns are systematically attenuated, others remain near the human benchmark, and still others are modestly elevated. The main value of this comparison, therefore, lies not in trait attribution but in the structured characterization of how LLM outputs diverge from human reference distributions under controlled questionnaire prompting. Persona prompting influenced these distributions unevenly: gender and age prompts produced only modest changes, whereas religious personas consistently shifted responses toward lower socially aversive SD3 scores across models.

The methodological contribution is central. Rather than treating psychometric scores as though they reveal model “personality,” the present approach demonstrates how psychometric instruments can be used more rigorously as behavioral probes. The combination of repeated sampling, controlled persona manipulation, human reference benchmarking, and effect-size-based interpretation offers a transferable design for future LLM evaluation. In that respect, the findings help move psychometric LLM research from illustrative, one-off questionnaires toward a more systematic, comparative, and scalable methodology.

These findings matter beyond the Dark Triad. They show that psychometric-style probing can reveal how context, alignment, and model heterogeneity shape observable outputs in systematic ways. If prompt wording can reliably shift measured response patterns, then prompt sensitivity is not merely a limitation to acknowledge; it should itself be an object of investigation. This is perhaps the most important insight of our analysis. If changing only a few words in a prompt can consistently reshape model behavior, then what we observe may say as much about our prompts as about the models themselves.