Reliability in Judging Basic Emotions: Lessons from Session Transcripts

Abstract

Background/Objectives: Psychotherapists have become increasingly interested in identifying how recurrent emotional themes might add value to the clinical process, especially in the context of recent developments in affective neuroscience. However, much of this literature remains anecdotal, and it is unclear whether therapists can reliably identify these emotions in routine practice, especially in the context of a free-flowing narrative. In a parallel development, large language models (LLMs) have recently been deployed to classify emotion from narrative texts, though this literature has yet to be aligned with the findings of modern affective neuroscience. Methods: The present study addresses this gap directly and, to our knowledge, represents the first attempt to code psychotherapy transcripts using both independent human raters and an AI coding methodology. The data were drawn from two remarkable patients with focal brain lesions, both with profound amnesia, in long-term psychotherapy. In an exploratory investigation, one session from each patient was coded independently by four clinically experienced psychotherapists, and separately by a single large language model operating under two different prompting conditions. Transcripts were coded for the presence of basic emotional systems. Results: The data suggest that overall inter-rater agreement about basic emotions was poor for both human and AI coders. Conclusions: The present study offers proof-of-concept that this approach is of scientific as well as clinical interest. The findings suggest that greater rigour needs to be applied to the way in which coders are prepared for, and implement, the classification of basic emotion systems.

1. Introduction

When it comes to feelings (which are, never forget, conscious things), what you see is what you get. The psychoanalyst’s initial task is not analysis, then, but classification.

Solms [1] (p. 120)

The last few decades have seen enormous progress in our understanding of the biological substrates of basic emotions [2,3,4,5]. There is broad agreement about the existence of basic emotions, their cross-cultural and cross-species consistency [6], and their biological basis in terms of both anatomy and neurochemistry [7]. There is also clear agreement on these topics in relation to some basic emotions, notably sadness, anger, fear, and a positive novelty-seeking emotion [8,9]. Beyond these four, there remains disagreement in the field about whether there are a small number of additional basic emotions [4,8,10,11,12,13]. One of the most widely used of these taxonomies is that of Panksepp [9], whose list contains the usual four core emotions (PANIC/GRIEF, RAGE, FEAR, and SEEKING), with an additional three which are more unique to Panksepp’s scheme (PLAY, CARE, and LUST). Note that Panksepp uses uppercase text to describe these emotions, to reflect not only their subjective basis but also their neurobiology.

1.1. Emotion in Narratives

Since antiquity, we have sought to identify common themes from the narratives told by humans [14,15]. For more than a century, psychotherapists have reported that patients also describe common themes across sessions [16,17], such as anxiety or sadness across a number of life circumstances [18,19], for example in a work setting and in the therapy room [17]. The psychotherapy community is broadly in agreement that identifying common themes of basic emotions is a useful therapeutic tool [20,21], allowing therapist and patient to formulate a coherent, overarching emotional narrative that may lie beneath a range of presenting problems [17,22]. This pattern can then be used to inform the selection of therapeutic interventions and provide a shared language for understanding change [20,21,23,24], enhancing case formulation, and allowing for more accurate monitoring of the therapeutic progress [22,25,26,27].

However, despite this growing interest and the associated clinical benefits of working with basic emotional themes, it remains unclear whether therapists can reliably identify these emotions in routine practice. Much of this literature remains anecdotal. To our knowledge, there has been no empirical work undertaken to examine whether therapists are reliably able to identify these affective neuroscience emotions, especially in the context of a free-flowing narrative. This absence is part of a more general challenge for psychotherapies. The field has tended to rely on case reports, or large-scale trials of therapeutic interventions [28], with less attention being paid to fundamental issues of reliability.

The broader field of emotion research has well-verified methods for measuring basic emotions, for example, the classification of facial expressions [29,30] or evaluation by questionnaire [31,32]. In contrast, psychotherapists appear to assume that they can accurately and reliably evaluate basic emotions from the narrative accounts of their patients in therapy [23,33,34,35,36].

1.2. Measuring Basic Emotions

Various authors have struggled with the question of how to classify and measure emotions, especially the relationship between accounts of emotion derived from behaviour and those derived from language. Notably, Plutchik [37] observed that these traditions do not always arrive at the same categories. Indeed, the vocabulary through which people describe their feelings maps imperfectly onto the emotional systems inferred from behaviour and physiology (see especially Chapter 11 of [37]). An influential lexically based approach is Bucci’s Multiple Code Theory [38]. Here, emotional meaning is carried in several formats: subsymbolic bodily and sensory processing, symbolic imagery, and symbolic verbal representation. These are connected by what she terms the referential process, through which emotion schemas are progressively symbolised before they can be named [38,39,40].

Rather than the top-down approach adopted by Bucci and others, a bottom-up neuroscience-based approach has been pioneered by authors such as Panksepp. As discussed above, this approach is based on neuroanatomical and neurochemical categories which may well represent the ‘natural kinds’ of basic emotions [6,9]. Whilst the bulk of this work was pioneered in non-human animals [9], the approach has been adapted for measurement in human populations [31] through the Affective Neuroscience Personality Scales (ANPS), a questionnaire designed to assess individual differences in the seven primary emotional systems at the trait level. The ANPS has since generated a substantial body of research [41,42,43,44].

However, no study in the ANPS literature, nor in the broader affective neuroscience tradition, appears to have attempted to identify Panksepp’s seven basic emotion systems [9,45] at the level of individual utterances within psychotherapy session transcripts, perhaps because the ANPS questionnaire measures stable personality dispositions [31,46], and does not capture the moment-to-moment affective texture of a therapeutic session. This distinction is important, as a patient may score low on the ANPS RAGE scale as a trait measure, while nevertheless producing discrete expressions of RAGE within the consulting room, particularly under conditions of frustration or perceived abandonment.

In brief, we have a well-established classification of basic emotions [9,45], and a psychotherapy literature which acknowledges the importance of identifying these basic emotions [47,48,49]. However, these two fields have yet to be brought together. The present study addresses this gap directly, and to our knowledge represents the first attempt to code psychotherapy transcripts for the presence of Panksepp’s seven primary emotional systems using independent human raters and a synonym-based methodology.

1.3. Automated Classification

A separate tradition of computational research, using machine learning and neural network methodologies, has addressed the measurement of emotion in psychotherapy through natural language processing (NLP) and sentiment analysis. This pioneering work has involved databases of tens of thousands of utterances from psychotherapy transcripts [50,51], and has developed a series of models (e.g., LIWC [52] and BERT [53]) integrated with computationally derived emotion classifications such as the NRC Emotion Lexicon [54] and GoEmotions [55] to classify the emotional valence of therapeutic speech [51,56]. These have varied in their efficacy, but have shown reasonable levels of agreement with human raters, with κ values typically in the range of 0.25 to 0.47 [51,55,56]. These findings represent the most systematic attempt in the literature to automate the identification of emotion in psychotherapy transcripts.

However, there are some limitations to these otherwise impressive endeavours. Most importantly, the approach lacks integration with the basic emotion systems so well-established from decades of research in affective neuroscience [8,9]. The result has been a diversity of categories. Early machine learning research operated within a framework of a three-point valence classification: positive, negative, and neutral [50,51]. More recent research has broadened the range of emotions substantially, but has operated independently of the well-established affective neuroscience classification, including ‘emotions’ such as trust and anticipation [54], or a 28-emotion classification scheme derived from the GoEmotions dataset [55,56]. It could plainly be argued that a binary classification cannot distinguish between RAGE, PANIC/GRIEF, and FEAR (all of which would be coded as negative), nor between SEEKING and PLAY (both of which would be coded as positive). Furthermore, it would be greatly advantageous to have a classification system consistent with the well-established affective neuroscience literature, of which Panksepp’s system is the most clearly elucidated [7,8,12].

A further gap in the existing literature concerns the rate at which discrete emotional expressions occur within psychotherapy sessions: how many identifiable emotional instances per minute of therapeutic speech appear. Remarkably, this issue does not seem to have been addressed in the published literature [13,41]: the literature reports accuracy and agreement metrics for their classifiers, but not frequency-of-emotion-per-unit-time data [51,56]. The absence of a published benchmark against which to evaluate the frequency of emotional expression in our data is significant, and it raises the question of what the base rate of identifiable affective content in therapeutic speech might be.

This paper seeks to rectify these several limitations by looking at transcripts from psychotherapy sessions, using for the first time both human and machine learning raters.

Such data can be gathered for any of a range of psychiatric disorders. However, there are particular advantages in using material derived from neurological patients, especially when controlled for premorbid variables and localisation of lesion site [57]. Patients with focal brain lesions are, as a population, premorbidly typical examples of humanity, with few of the confounding issues of aberrant development that often arise in psychiatric populations [58,59]. Moreover, advances in structural imaging make it possible to identify the neural basis of the clinically observed phenomena [60,61,62,63]. The present study uses therapy session content from two remarkable amnesic patients, both seen in psychotherapy for more than a year. There is a particular advantage in using therapeutic material from amnesic patients because of the repetitive and reliable nature of their presentation in therapy [64,65].

This exploratory study also attempts to address the gap by looking at how clinically experienced psychotherapists and a large language model (AI) ‘coder’ identify basic emotions in transcripts. The study assesses (a) the degree of agreement between human coders in attributing basic emotions to portions of the transcripts, and (b) the extent to which AI-based coding converges with human coding. In doing so, and as a proof-of-concept, we aim to clarify the foundational question of reliable identification of basic emotions from psychotherapy transcripts, and also the potential role that AI tools might play in supporting this coding.

2. Materials and Methods

2.1. Participants

There is now a considerable literature on psychotherapy with neurological patients, which suggests that this group can be more stable and more consistent in presentation than comparable psychiatric populations, in part because they are premorbidly typical examples of humanity, and lack the developmental and characterological complexity often found in psychiatric samples [57,58,59,66]. In addition, the lesion site in neurological patients can be (as in these cases) known with precision.

The present study reports two such patients with focal lesions within the Papez circuit implicated in episodic memory.

2.1.1. Patient JL

During the treatment period, JL was a 38-year-old single man with master’s level education and premorbid employment in a professional role, who sustained a severe anoxic brain injury three years earlier (see [64] for more detail). The injury occurred in the context of complex diabetes-related medical deterioration and three cardiac arrests, and JL subsequently presented with the neuropsychological profile of profound anterograde amnesia with largely preserved retrograde memory. The rehabilitation intake history also recorded significant premorbid adversity, including a fractious family life and frequent relocation (often international). There was early and escalating alcohol misuse (reported from early adolescence, and later revised to age 10), an acrimonious parental separation when he was 20 that precipitated an abrupt move with his mother and siblings to their country of origin, and, within a few years, the deaths of both parents (father first, then mother three years later). This was followed by a marked spiral into alcohol and cannabis addiction, occupational instability, and considerable neglect of self-care.

JL was diagnosed with type 1 diabetes at 16 and, prior to the acquired brain injury, had experienced recurrent cardiac events contributing to, and culminating in, the anoxic injury. Post-injury, he lived in a residential neurological unit after having spent a year in a rehabilitation hospital. He experienced the unit as socially isolating (being the youngest resident), repetitive in routine, and restrictive due to close nursing monitoring of diabetes, while maintaining limited family contact, and expressing a strong wish for more independent or assisted living.

Clinically, at the initial consultation, he appeared friendly, polite, lucid, and oriented, spoke clearly with a foreign accent, and showed intelligent curiosity. His amnesia became evident within minutes through rapid repetition of information, while his presentation was notable for somewhat neglected grooming and idiosyncratic but stable personal features. For fuller clinical and contextual details regarding JL, readers are referred to Moore et al. [64].

2.1.2. Patient GA

GA was 19 when he underwent surgery to remove a colloid cyst, during which a craniotomy sectioned the anterior portion of the fornix (see [65] for more detail). Consistent with what has been reported in some cases of fornix and related limbic disruption [67,68,69], the operation left him with a dense anterograde amnesia in the context of preserved working memory and broadly intact higher executive functions. Alongside this, GA developed marked hyperphagia, suggestive of hypothalamic involvement (potentially via cyst-related pressure or adjacent disruption), and he appeared to retain a notable degree of insight into his memory impairment, his hyperphagia, and the broader implications of his brain injury. After surgery, he moved to a residential unit and, seven years later, was referred from his brain injury service to the first author (PM) for psychoanalytic psychotherapy, on the view that he might benefit from a sustained opportunity to process the emotional and developmental consequences of a sudden, devastating, and life-altering neurological event.

Immediately prior to the referral, a brain injury service report described a baseline presentation characterised by irritability and occasional aggressiveness, with episodes of tearfulness and depressed affect. The report also recorded clinically striking features relevant to his later therapeutic presentation. First, tragically, GA’s mother was terminally ill around the time of his diagnosis and died shortly after his surgery, with the report noting that although he attended the funeral, he had no memory of it and was described as not being aware of her death. Second, when asked his age, he reportedly and consistently replied 18 or 19. The report further highlighted the severity and centrality of his hyperphagia, persistent food-seeking that was problematic enough to be embedded in behaviour management plans. Interestingly, this did not occur in the therapy sessions with PM, while also noting no premorbid psychiatric history.

GA undertook fifty-six once-weekly psychoanalytic psychotherapy sessions across eighteen months. These were held regularly at the same time and day, with sessions audio-recorded and transcribed. Following his expressed curiosity during initial consultations, he agreed in the first session to use the couch, an element later discussed in relation to the phenomenology of his amnesia. For a fuller account of GA’s history, clinical course, and the psychotherapy process and findings, readers are referred to Moore & Turnbull [65].

2.2. Session Selection

For this proof-of-concept study, two psychotherapy sessions were selected, one each from patient JL and patient GA. In order to minimise the impact of early ‘settling in’ sessions and late termination dynamics, these sessions were defined as those occurring after the initial assessment and early therapeutic alliance formation phases, and before any explicitly planned termination phase, and at approximately the midpoint of the total number of sessions in the treatment of both patients. The aim of this selection strategy was to capture typical representative psychotherapy sessions, as opposed to a session with beginning or end dynamics and effects. On this basis, session 28 of 56 was chosen for patient GA, and session 36 of 72 for patient JL.

2.3. Transcription Process

The sessions were audio-recorded and transcribed using automatic speech recognition software (OpenAI Whisper). The automatically generated transcripts were then manually checked and corrected by the first author (PM), who compared each transcript line by line with the original audio. Corrections included rectifying misheard words, adding punctuation for clarity, and standardising anonymisation of names and identifying details. An example of this for JL was the OpenAI Whisper raw output ‘The sisters keep checking my sugar, and it drives me mad. I want to move into a sister house’ was corrected manually to ‘The sisters keep checking my sugar, and it drives me mad. I want to move to assisted housing’. Possibly because of JL’s accent and the AI’s algorithm, ‘assisted housing’ was rendered as ‘a sister house’. Similar to the JL transcription, OpenAI Whisper’s raw output ‘They keep pointing me to the menu card’ was corrected manually to ‘they keep pointing me to the memory card’. It seemed as if, because of GA’s accent, ‘memory’ was misrendered by Whisper as ‘menu’. Finally, all audio recordings were deleted from the platform after transcription.

2.4. Coding

2.4.1. Human Coders

Four human coders contributed to the manual emotion coding, including the first author (PM). All coders were clinically experienced psychotherapists with more than 20 years of post-qualification practice. Coders had backgrounds in different psychoanalytic and psychodynamic traditions; only one coder (the first author, PM) had formal training and considerable experience in neuropsychoanalysis. In order to ensure consistency, all participants completed training on the Panksepp basic emotion systems, based on definitions and synonyms taken directly from Panksepp (see table of synonyms in Appendix B). This selection was intended to approximate how experienced clinicians, as opposed to specialist experts, might identify and understand basic emotions in ordinary therapeutic contexts.

2.4.2. AI Coding

In addition to human coders, we used a large language model (ChatGPT 4o, OpenAI) as an AI coder. The AI was provided with the same session transcripts in text format and the same instructions given to human coders. Initially, the AI was requested using the same prompt as human coders to read the entire transcript and to identify passages where any of Panksepp’s basic emotional systems were present (the full prompts used to instruct the models are reproduced in Appendix A). In a second approach, ChatGPT was asked to review its own process resulting from the initial ‘prompt’ inputted to it, and to improve its performance based on that review (this process of reviewing its performance is also reported in Appendix A). These configurations are referred to as ‘AI₁’ and ‘AI₂’ in Section 3 and Section 4. Importantly, AI₁ and AI₂ are the same model, working on the same transcripts, under two different prompts. The contrast between them therefore isolates the effect of prompt formulation, holding both model and material constant.

2.4.3. Coder Training

Prior to coding, all human coders were provided with written instructions. This included definitions of each of Panksepp’s basic emotional systems, adapted from his primary texts and lists of synonyms and related affective terms for each emotion, compiled from Panksepp’s writings [9,45], and standard thesaurus entries [70]. The final list of synonyms was roughly 50% Panksepp and 50% thesaurus (See Appendix B for more details). Coders were instructed to use these materials as an aid to recognising the presence of each emotional system in the clinical material. These training instructions are included as Appendix A.

2.4.4. Coding Procedure

After training, each coder received a PDF copy of transcripts of the selected sessions. Coders were asked to read the transcript, highlighting and annotating any passages in which they judged one or more of Panksepp’s [9] basic emotional systems to be present in the material (e.g., SEEKING, FEAR, RAGE, LUST, CARE, PANIC/GRIEF, PLAY).

Coders were free to mark multiple emotions in the same passage and to annotate as many or as few passages as they wished. After completion, the annotated transcripts were returned for data collation. All highlighted excerpts were entered into a spreadsheet for analysis.

It is worth noting that coders were not provided with predefined units of analysis. This was a deliberate design decision, intended to preserve the ecological validity of the task, since no comparable segmentation is imposed upon a clinician listening to a session. It does, however, mean that disagreement between coders may reflect variability in the identification of units, as well as variability in the recognition of emotions. From the perspective of Multiple Code Theory [38,39], this is not an incidental feature of the method but a consequence of the material itself, since emotion schemas are symbolised to differing degrees in spontaneous speech, and the point at which an affective event becomes lexically identifiable is correspondingly variable. We report the consequences of this decision in the Results, and consider its implications in the Discussion.

3. Results

Two speaker transcripts (GA and JL) were annotated for the seven Pankseppian basic emotion systems: SEEKING, CARE, FEAR, LUST, PLAY, RAGE, and PANIC/GRIEF, by four independent human raters (A, B, C, PM). As discussed in the Methods section, counts reflect the number of instances each coder identified for each basic emotion. The Results of the two AI coders (AI₁ and AI₂) are reported separately. The data suggest substantial variability across coders, in a number of ways.

As described in the Methods, coders determined their own units of analysis. The counts reported below therefore reflect variability in the identification of emotionally meaningful units, as well as variability in the attribution of emotion categories to those units.

3.1. Coders Rated the Two Patients at Very Different Rates

Some coders were more likely to identify instances than others. Table 1 presents the total number of emotion instances identified by each of the four human coders across both patients.

Table 1. Total number of coded emotion instances by human coder for patients GA and JL. A, B, and C denote independent human raters; PM denotes the treating clinician.

Patient/Coder	A	B	C	PM	Total
GA	19	18	36	21	94
JL	4	6	12	21	43
Total	23	24	48	42	137

There was considerable variability in the number of instances coded by human raters. For example, for GA, coder C identified 36 instances while coder B identified only 18. For JL, PM identified 21 instances while A identified only four, a five-fold difference. Across both patients, C was the most prolific coder (48 instances) and A the most conservative (23 instances). GA attracted substantially more coded instances (94) than JL (43), a pattern that is consistent with the first author’s subjective clinical experience of the two patients, a theme which we return to in the Discussion.

3.2. Coders Rated Basic Emotions at Very Different Rates

It is worth noting that these totals derive from 50 min sessions and aggregate the identifications of four coders, any of which may refer to the same passage, so they cannot be read as counts of discrete emotional events. Expressed per coder, the mean number of identifications was 23.5 for GA (approximately 0.47 per minute, or about one every two minutes) and 10.75 for JL (approximately 0.22 per minute, or about one every four to five minutes). Whether this rate, at which a single clinical coder identifies affective expression, is typical of psychotherapy sessions is an empirical question to which we will return in the Discussion.

Figure 1 presents the distribution and rate of coded emotion instances across Panksepp’s seven basic emotion systems for both patients, using human coder data only. As can be seen, GA attracted substantially more coded instances than JL across nearly every emotion system. It is of some interest that this difference was not uniform across systems. CARE showed the largest absolute discrepancy (GA = 0.50 instances per minute; JL = 0.14), a nearly four-fold difference, while SEEKING showed a more modest gap (GA = 0.38; JL = 0.24). RAGE was the only system in which the two patients’ rates were approximately equivalent (GA = 0.26; JL = 0.14), and FEAR was the only system in which JL’s rate exceeded GA’s (JL = 0.04; GA = 0.00).

Figure 1. Distribution and rate of coded emotion instances across Panksepp’s seven basic emotion systems for patients GA and JL (human coders A, B, C, PM only; 50 min sessions).

The rate of identification varied considerably across the emotion systems, from ratings at or near zero for FEAR and LUST in both patients, to ratings above 20% for CARE and SEEKING. In the GA transcripts (N = 94), CARE was the most frequently coded system (26.6%), followed by SEEKING (20.2%) and PANIC/GRIEF (18.1%), while FEAR was not coded at all and LUST accounted for only 5.3% of instances. The JL transcripts yielded a smaller total (N = 43) but a broadly similar rank ordering: SEEKING (27.9%) was the most frequently identified system for JL, followed by PANIC/GRIEF (20.9%) and then CARE and RAGE (both 16.3%). It is of some interest that RAGE was proportionally more prominent in JL (16.3%) than in GA (13.8%), and that SEEKING, rather than CARE, predominated in JL.

Across both transcripts (N = 137), CARE was the most frequently coded system (n = 32, 23.4%), followed by SEEKING (n = 31, 22.6%), PANIC/GRIEF (n = 26, 19.0%), and PLAY (n = 21, 15.3%). Together, CARE and SEEKING accounted for 46.0% of all coded instances, and the four most frequent systems (CARE, SEEKING, PANIC/GRIEF, PLAY) accounted for 80.3% of coded instances. LUST and FEAR were rarely identified, together comprising only 5.1% of the total. The virtual absence of FEAR from the human-coded data is a finding of particular note: across both patients, only two instances were identified, both by a single coder (PM) in the JL transcript.

3.3. Patterns of Human Coder Agreement

Agreement among the four human coders emerged in a small number of high-salience segments. Where it did, it most often took the form of three-out-of-four agreement (A, B, C, PM) rather than unanimity. Surprisingly, there was no segment in which all four human coders converged on the same emotion classification. This is a finding of some importance, and we return to it in Section 4.

Figure 2 presents the weighted agreement scores for human coder consensus across each of the seven basic emotion systems. The scores were calculated by assigning a weight of 1 to items identified by a single human coder (1/4 agreement), 2 for two coders, 3 for three, and 4 for all four, and then computing the mean across all items within each system.

Figure 2. Descriptive weighted agreement scores for human coder consensus across Panksepp’s seven basic emotion systems, rank-ordered from highest to lowest. Scores represent the mean agreement level across items within each system, where 1 = single-coder identification (1/4) and 4 = unanimous identification (4/4).

No emotion system achieved a mean score approaching unanimous agreement, and the range was notably compressed, falling between 1.00 (FEAR) and 1.62 (PLAY). PLAY, CARE (1.55), SEEKING (1.50), and LUST (1.50) formed a cluster at the upper end, while PANIC/GRIEF and RAGE (both 1.31) occupied an intermediate position. FEAR, with a score of 1.00, was the only system in which every identified instance was coded by a single rater alone.

These summary data are also interesting to unpack at the individual item level. For SEEKING in JL, coders B, C, and PM converged on a prototypical ‘approach-oriented’ cluster around the patient’s expressed desire to see a film. These were associated with formulations such as a sense that there was “something out there” that he felt compelled to find. However, coder A did not identify SEEKING at all in that transcript.

For PLAY in GA, coders A, C, and PM converged on recurring basketball references, whereas coder B did not identify these as instances of PLAY. For PANIC/GRIEF in JL, coders A, C, and PM converged on clear separation-distress material, including a passage describing the patient’s response to a significant other’s departure. Coder B did not code these as PANIC/GRIEF. For RAGE in GA, coders B, C, and PM converged on overtly aggressive statements, including explicit references to killing, while coder A did not classify these as RAGE. In sum, even the most semantically transparent material in the dataset did not attract unanimous coding, underscoring the degree to which apparently straightforward affective content can be subject to divergent classification.

3.4. Common Factors and Idiosyncrasies Across Emotion Systems

Appendix C works through examples of each basic emotion in order to establish which features appear common to multiple coders and which appear idiosyncratic. Text examples are provided for these instances, so that the reader can gain a better impression of the nature of these emotional events. The basic emotions are reported in order of frequency in Appendix C.

A summary of the data in Appendix C is that human coders varied substantially in the extent to which basic emotions were identified, ranging from frequent instances of SEEKING and CARE to infrequent instances of FEAR and LUST. There was broad agreement across coders and the two patients about these phenomena. However, individual human coders varied in the extent to which they agreed with other coders, including a number of striking omissions.

3.5. AI Coder Results

The same transcripts were independently coded under two AI coding conditions (AI₁ and AI₂) and were generated by a single large language model (OpenAI’s ChatGPT) operating under two different prompts, using the same synonym lists provided to the human raters. The differences in prompting for these two coders were clarified in the Methods section. This section presents the AI coding data and examines the relationship between AI and human coder outputs.

As can be seen in Table 2, the two AI coders differed markedly from one another. AI₁ was the most conservative coder in the entire dataset, identifying only eight instances for GA and six for JL (totaling 14), well below the human coder range of 23 to 48. AI₂, by contrast, was the most prolific, identifying 47 for GA and 40 for JL (totaling 87), nearly doubling the output of even the most prolific human coder (C, at 48). With AI data included, the combined total for instances for humans-plus-AI rises from 137 to 238, with AI₂ alone contributing 36.6% of the entire dataset.

Table 2. Total number of coded emotion instances by all six coders for patients GA and JL. A, B, and C denote independent human raters; PM denotes the treating clinician; AI₁ and AI₂ denote the two AI prompting conditions.

Patient	A	B	C	PM	AI1	AI2	Total
GA	19	18	36	21	8	47	149
JL	4	6	12	21	6	40	89
Total	23	24	48	42	14	87	238

Overall AI Coding Rates

These data, as can be seen in Table 3, can also be analysed by basic emotion (in this case shown with data from both patients combined). AI₂ was the most prolific coder for every emotion system except SEEKING (where B and C produced more) and LUST (where B produced more). AI₁ produced zero instances for PLAY and LUST.

Table 3. Distribution of coded emotion instances across Panksepp’s seven basic emotion systems by all six coders (GA and JL combined). Human coders presented first, then AI coders. Emotion systems ordered by total frequency.

Emotion	A	B	C	PM	AI1	AI2	Total
CARE	10	0	11	11	3	25	60
SEEKING	0	13	14	4	5	14	50
PLAY	2	0	10	9	0	21	42
PANIC/GRIEF	11	0	6	9	3	11	40
RAGE	0	7	7	6	2	11	33
LUST	0	4	0	1	0	2	7
FEAR	0	0	0	2	1	3	6
Total	23	24	48	42	14	87	238

3.6. AI-Specific Observations by Emotion System

Appendix D reports the AI coders’ contributions to the coding of each emotion system, highlighting where AI coders converged with or diverged from the human coder consensus. For each system, we report what the AI coders identified and what the human coders did in relation to the same material.

Across the seven emotion systems, several consistent patterns emerged in the AI coding that are worth drawing together. AI₁ was the most conservative coder in the dataset (14 instances total across both patients), and its items showed a notable tendency to overlap with human-coded segments. In SEEKING, all three of AI₁’s GA items were also identified by at least one human coder. In CARE, both of AI₁’s items were multiply endorsed by human coders. In PANIC/GRIEF, AI₁ converged with three human coders on one of the strongest agreement segments in the entire dataset. This pattern suggests that AI₁, while highly selective, was operating on criteria that were broadly recognisable to the human raters.

AI₂ presented a strikingly different profile. With 87 instances in total, AI₂ was the most prolific coder by a substantial margin, and a recurring feature of its output was the apparent influence of surface-level lexical features on classification decisions. Across the emotion systems, we observed: “I need some garages” coded as RAGE (perhaps misclassifying ‘rage’ within ‘garages’?); “CD players, small coins in the car” coded as PLAY (the word ‘players’ containing ‘play’?); “there’s a lot of players missing” coded as PANIC/GRIEF (the word “missing” driving the classification?); “I don’t have any worries” coded as FEAR (identifying the keyword ‘worries’ despite its negation?); “but their bodies are paralysed” coded as LUST (driven by the word ‘bodies’?); and multiple references to ‘dad’ and ‘daddy’ coded as CARE regardless of context.

AI₂ also appeared to double-classify certain segments across emotion categories: “so it would be dangerous for me” was coded as both FEAR and RAGE, and “I’m getting it done tomorrow no matter what, I don’t care” appeared under both CARE and RAGE. These patterns are consistent with a coding strategy driven substantially by keyword detection rather than contextual interpretation, and they stand in clear contrast to the human coders, who, despite their many disagreements, were consistently responsive to the pragmatic and affective context in which words were embedded.

A third pattern concerns systematic omissions among individual coders, both human and AI. Coder A did not identify any instances of SEEKING, RAGE, LUST, or FEAR across either patient. Coder B did not identify any instances of CARE, PLAY, or PANIC/GRIEF across either patient. AI₁ did not identify any instances of PLAY or LUST. These patterns of complete absence of specific emotion categories raise the question of whether individual coders, whether human or AI, may operate with implicit hierarchies of emotional salience that lead them to attend to certain categories and systematically overlook others. This is a finding of some interest for the broader question of what inter-rater reliability means in the context of affective coding, and we return to it in the Section 4.

3.6.1. Inter-Rater Reliability: Cohen’s Kappa and Fleiss’ Kappa

To quantify the degree of inter-rater agreement beyond the descriptive analyses reported above, we computed Cohen’s kappa (κ) for all pairwise coder combinations, and Fleiss’ kappa for the full coder panel. Because the coding task involved free annotation of continuous transcript text rather than rating of pre-defined items, segmentation was evaluated at a range of thresholds, varying from 0.30 to 0.90, as reported in Appendix E. This procedure inevitably introduces a degree of approximation, and the resulting kappa values should be interpreted with this methodological caveat in mind. All kappa values were interpreted according to the benchmarks proposed by Landis and Koch [71]: values below 0.00 indicate less-than-chance agreement, 0.00–0.20 slight agreement, 0.21–0.40 fair agreement, 0.41–0.60 moderate agreement, 0.61–0.80 substantial agreement, and 0.81–1.00 almost perfect agreement.

3.6.2. Overall Agreement: Fleiss’ Kappa

Table 4 presents Fleiss’ kappa values for the full six-coder panel and for the four human coders alone, both overall and by emotion system.

Table 4. Fleiss’ kappa values for overall inter-rater agreement and by emotion system. Negative values indicate agreement below chance level. Colour-coded cells: red = below chance, yellow = slight (0.00–0.20).

	All 6 Coders	Human Only
Overall	−0.046	0.026
SEEKING	0.021	0.048
CARE	−0.033	−0.051
PLAY	−0.112	−0.030
PANIC/GRIEF	−0.069	−0.001
RAGE	−0.059	0.169
LUST	−0.170	−0.095
FEAR	−0.167	−0.111

Fleiss’ kappa for the full six-coder panel was κ = −0.046, indicating agreement at or below chance level. Restricting the analysis to the four human coders did not materially improve the picture: human-only Fleiss’ kappa was κ = 0.026. These values confirm the central descriptive finding: overall inter-rater agreement was effectively absent.

At the level of individual emotion systems, only RAGE reached slight agreement among the human coders (κ = 0.169). The only other system to produce a positive human-only Fleiss’ kappa was SEEKING (κ = 0.048), consistent with SEEKING’s status as the system attracting the most convergent coding. All other systems yielded negative values, with FEAR producing the most strongly negative human-only Fleiss’ kappa (κ = −0.111). The near-zero value for PANIC/GRIEF (κ = −0.001) is consistent with the pattern in which coders converged on the general theme of separation distress but disagreed about the specific segments that exemplified it.

3.6.3. Pairwise Agreement: Cohen’s Kappa

Table 5 presents pairwise Cohen’s kappa values for all 15 coder combinations.

Table 5. Pairwise Cohen’s kappa (κ) for all coder combinations. Human coders (A, B, C, PM) first, then AI (AI1, AI2). Colour-coded cells: red = below chance (κ < 0.00), yellow = slight (0.00–0.20), green = fair (0.21–0.40).

	A	B	C	PM	AI1	AI2
A	—	−0.166	0.018	0.072	0.152	−0.115
B		—	0.101	0.075	0.250	−0.161
C			—	0.057	0.155	−0.281
PM				—	0.170	−0.248
AI1					—	0.078
AI2						—

The pairwise analysis revealed three distinct patterns. First, agreement among the four human coders was uniformly poor. The six human–human pairwise kappa values ranged from κ = −0.166 (A vs. B) to κ = 0.101 (B vs. C), with a mean of κ = 0.026. Although five of the six pairs produced positive values, none exceeded κ = 0.101, and none reached the threshold for even modest agreement.

Second, AI₁ agreed with each of the four human coders at levels that were, remarkably, higher than any human–human pairing. The four Human–AI₁ kappa values ranged from κ = 0.152 (A vs. AI₁) to κ = 0.250 (B vs. AI₁), with a mean of κ = 0.182, placing AI₁’s agreement with the human panel in the modest range. It is of special note that the two highest pairwise kappa values in the entire 15-pair matrix were B–AI₁ (κ = 0.250) and PM–AI₁ (κ = 0.170), the former the only pair in the matrix to reach a fair level of agreement. In other words, AI₁ agreed with individual human coders more reliably than any human coder agreed with any other human coder.

Third, AI₂ diverged systematically from every other coder. All four Human–AI₂ kappa values were negative, ranging from κ = −0.115 (A vs. AI₂) to κ = −0.281 (C vs. AI₂), with a mean of κ = −0.201. AI₂’s agreement with AI₁ was also low (κ = 0.078). This pattern confirms that AI₂ was not merely prolific but was identifying different segments as emotionally salient, a divergence that may reflect the lexical string-matching artefacts noted earlier.

3.7. Summary of Agreement by Coder Group

These data are summarised in Table 6. The overall picture is one of poor agreement across all configurations, with one notable exception: AI₁ achieved slight-to-fair agreement with individual human coders, a level of concordance that exceeded any human–human pairing. This unexpected finding raises the possibility that AI₁’s more conservative coding approach (14 total instances) may have aligned with a subset of the affective content that was also recognised by the human coders, whereas AI₂’s more liberal approach (87 total instances) resulted in systematic divergence.

Table 6. Summary of inter-rater agreement by coder group. Mean κ = mean pairwise Cohen’s kappa. Interpretations follow Landis and Koch [71].

Coder Group	Mean κ	Range	Interpretation
Human–Human	0.026	−0.166 to 0.101	Slight
Human–AI1	0.182	0.152 to 0.250	Slight
Human–AI2	−0.201	−0.281 to −0.115	Less than chance
AI1–AI2	0.078	—	Slight
Fleiss’ κ (all 6)	−0.046	—	Less than chance
Fleiss’ κ (human)	0.026	—	Slight

3.8. Summary of Key Findings

First, inter-rater agreement among the four human coders was poor across all seven basic emotion systems. Fleiss’ kappa for the human coders was κ = 0.026, and no pairwise Cohen’s kappa between human coders exceeded 0.101. The strongest human consensus reached three-out-of-four agreement; no segment attracted unanimous coding by all four human raters. Second, among the human coders, CARE and SEEKING together accounted for 46.0% of all coded instances, while FEAR and LUST together accounted for only 5.1%, a topic we return to in the Discussion. Third, GA attracted substantially more coded instances than JL (94 vs. 43) across all human coders, a finding consistent with clinical observation and which may reflect differences in the nature or accessibility of emotional expression between the two patients. Fourth, the two AI coders diverged dramatically from one another: AI₁ was the most conservative coder in the dataset (14 instances total), while AI₂ was the most prolific (87 instances total). Fifth, AI₁ agreed with individual human coders at levels exceeding any human–human pairing (mean pairwise κ = 0.182, with one pair reaching fair agreement), while AI₂ diverged systematically from all other coders (mean pairwise κ with humans = −0.201). Sixth, AI₂’s output appears, in certain instances, to reflect lexical pattern matching (e.g., “garages” coded as RAGE, “I don’t have any worries” coded as FEAR) rather than contextual interpretation. Seventh, certain patterns of systematic omission were observed among individual coders, both human (e.g., coder A coding zero instances of SEEKING, RAGE, LUST, and FEAR; coder B coding zero instances of CARE, PLAY, and PANIC/GRIEF) and AI (AI₁ coding zero instances of PLAY and LUST), raising questions about implicit hierarchies of emotional salience.

4. Discussion

The data from the various coders produced findings which are both rich and complex. These are synthesised below into a number of themes.

4.1. Consistency Between Human Coders

Remarkably, across the 238 coded instances (137 from the human coders and 101 from the two AI coders), which the item-matching procedure described in Section 3.7. reduced to 136 distinct text segments, there were no examples in which all four human coders identified the same basic emotion for a segment of text. In 28 of these segments (21%), two or three human raters evaluated the same text segment as representing the same emotion; 57 segments were identified by only one human rater, and the remaining 51 were identified by the AI coders alone.

Some raters identified some basic emotions at much higher frequencies than others. However, individual coders varied in which basic emotions they felt were most frequent. For example, in the JL transcript, coders A, C, and PM all identified the statement ‘I lost all my interest after that’ as an instance of PANIC/GRIEF. However, despite this phrase containing well-established features of loss, such as anhedonia and adynamia, B did not identify this as an instance of PANIC/GRIEF.

However, there were many episodes in the sessions that were consistently rated across almost all coders. Notable examples included the presence of classic ‘approach’ language as being representative of instances of SEEKING. In JL’s transcript, a consistent example was a cluster where several coders (AI₁, AI_2, B, C, and PM) marked statements about ‘wanting to see the movie’, and the cluster ‘I keep thinking there’s something out there I’m meant to find’. However, A did not identify these as instances of SEEKING.

Interestingly, there was one finding of reliability between the two patients. Within the basic emotion categories, CARE emerged as the most frequently reported for JL, and the second most frequently reported for GA. Coders who overlapped in their assessments tended to focus on instances of familial warmth and persistent care. Examples included memories of being loved ‘day after day’, returning home with a parent, or spending time together as a family. Similarly, LUST and FEAR were the least frequently reported for both JL and GA.

Across the coders, there was consistent marking of affectionate, everyday references to caring. These instances were clearly labelled with language aligning with the Panksepp synonym list, and were often associated with emphatic terms such as ‘very’ or ‘a lot’ to highlight the strength of the emotion. The expressions of CARE often persisted across extended passages, reinforcing their significance in the transcripts. The emotion of CARE was colloquially understood within the context as an authentic emotional experience, typically relating to the self or events that had happened to the individual. The timeframe referenced was usually the present, as indicated by statements like “I am” rather than future or past considerations. Notably, these references to CARE did not include negative emotions; for example, they were explicitly distinguished from feelings of sadness.

4.2. Low Reliability in Judging Basic Emotions

The central finding of the paper is poor reliability between coders in classifying basic emotions. There was no emotion class in which all coders agreed about the number or specificity of items. At the item-level (including AI), no basic emotion achieved unanimous agreement (0/7 systems reached 6/6). The maximum convergence on any single utterance was 5/6 (e.g., CARE in GA; SEEKING in JL). Several systems showed very weak overlap (e.g., FEAR 2/6 in GA and 1/6 agreement in JL; LUST 2/6 in GA and 1/6 in JL).

The number of events reported in each session also varied enormously between coders: in GA, the total number of coded instances ranged from 8 (AI₁) to 47 (AI₂), and in JL from 6 (AI₁) to 40 (AI₂), indicating markedly different coder thresholds for what counted as an emotion ‘event’. Relatedly, divergence is also evident at the level of whole systems, with some coders recording no instances at all (e.g., coder A coded no SEEKING; coder B coded no CARE), which necessarily inflates between-coder differences in overall event counts.

These totals also bear on the question raised in Section 3.2 of how often emotion is expressed in a session, which, as noted in the Introduction, the published literature has not addressed. Because the figures in Table 1 aggregate the identifications of four coders, they overstate the number of discrete events. Expressed per coder, a single clinical judge identified affective expression roughly once every two minutes in GA and once every four to five minutes in JL. This is the rate at which a clinical judge marks emotion, rather than the rate at which emotional events occur, and the two need not coincide. Even so, one identification every two minutes has face validity as a description of a psychotherapy session. Two features of the present study may inflate the rate. The Pankseppian taxonomy counts as emotion expressions of SEEKING and PLAY that might not ordinarily be described as emotional, and the repetition characteristic of profound amnesia means that the same affective content can recur several times within a session. Whether comparable rates are found in other patient groups, and under other taxonomies, is a question for the larger study discussed in Section 4.5.

The specific events that were coded were also inconsistent. Even where coders converged on a general thematic understanding of an emotion system, they frequently diverged at the utterance level, with the same segments being coded by only a subset of raters (e.g., the JL “wanting to see the movie” SEEKING cluster, coded by five coders but not A; GA “head away home with my dad” coded as CARE by all except A). In some domains (notably FEAR in JL), no single transcript instance elicited any cross-coder agreement at all.

Some coders showed huge absences in category, in several instances recording no events for an entire basic-emotion system across a full session transcript (e.g., coder A coded no SEEKING in both GA and JL and no RAGE in both transcripts; coder B coded no CARE in both transcripts and no PANIC/GRIEF in GA; and multiple human coders coded no FEAR in GA and JL, with near-complete absence of LUST in JL for all coders except AI₂).

In JL, several coders converged on prototypical approach-oriented language as SEEKING, most clearly the cluster around “wanting to see the movie” and “I keep thinking there’s something out there I’m meant to find”, which was marked by AI₁, AI₂, B, C, and PM, while A did not code it as SEEKING. A second consistent SEEKING exemplar in JL was “I should get away with it”, tagged by AI₁, AI₂, and C. In GA, the Results describe coders commonly marking forward-moving, goal-oriented themes (for example, wanting to leave the residential unit, moving toward school-related goals, plans, “wanting to do things” and curious or hopeful statements), plus recurrent thematic clusters (hunting, the residential unit, school). Against that broader pattern, the Results also provide single-coder SEEKING “one-offs” (for example, B coding “bring me out with my dad’s dogs” and C coding “up at the institute”). A coded no SEEKING whatsoever in GA, and likewise no SEEKING whatsoever in JL.

The findings in relation to RAGE are especially interesting, because of the clarity of RAGE synonyms. In GA, there was multi-coder convergence around irritation and anger-tinged complaints, most clearly “mad for going here”, coded as RAGE by AI₁, AI₂, B, and C. There was also strong human convergence on the more violent phrasing “Just catching and killing” and “I’d kill somebody”, which were coded as RAGE by B, C, and PM. However, A did not code these as RAGE. In JL, agreement was rarer, but still present for a couple of disgruntled or rejecting evaluations, for example, PM and B agreeing on “I’m getting it done tomorrow no matter what, I don’t care” and AI₁, B, and PM selecting “I just don’t like her” as RAGE. The Results note explicitly that A coded no RAGE in GA and again no RAGE in JL.

4.3. Causes of Low Reliability

The low level of agreement between raters is the central finding of the paper. However, there are multiple possible explanations for this finding. For example, poor reliability amongst coders may have resulted from unfamiliarity with Panksepp’s categories, despite training. However, all the human coders were experienced psychotherapists with experience of affective neuroscience. All were also provided with training on the Panksepp synonyms, though it is possible that their prior knowledge and the training provided were insufficient. As a counter-argument, no respondents claimed that they were undertrained.

A second hypothesis is that the coders may have ‘skim-read’ the content. On this argument, coders paid insufficient attention to the content of the session, or that there is something limiting about reading psychotherapy transcripts without the presence of a patient in the room. Indeed, coding from a transcript would be a relatively novel experience for all of our human coders. Nevertheless, discussion with the human coders did not suggest that they had been flippant in their concentration on the task. However, several did describe that it was a process with which they were unfamiliar.

4.4. Comparison of Human and AI Coding in the Present Dataset

In the case of the initial prompting condition (AI₁), its overall event counts were of the same order of magnitude as the human coders, albeit at the low end. In GA, AI₁ identified eight instances, compared with the human coders’ totals of 18 (B), 19 (A), 21 (PM), and 36 (C). In JL, AI₁ identified six instances, compared with human totals of four (A), six (B), 12 (C), and 21 (PM).

However, in the second of these prompting conditions, AI₂ produced a much higher number of items than the human coders. This begs the question of what ChatGPT was classifying that humans did not observe. One explanation would be around this issue of ‘silent affect’, as reported by ChatGPT, which AI₂ suggested was some degree of ‘inference’ about emotion, which may not have been an accurate conclusion. Notably, the AI coders, especially AI_2, appeared to operationalise Pankseppian categories by treating certain linguistic markers (e.g., danger appraisals, negation/denial forms, probes of loss) as sufficient evidence of an affective ‘event’. This may have been described as ‘silent affect’ but may also increase false positives when lexical cues are ambiguous. Thus, an important cause of AI misclassification (or confabulation, see [72]) was a similarity based on text ‘near misses’, such as the classification of RAGE in a sentence containing ‘garage’. However, we note that some elements of the psychoanalytic community have drawn conclusions about exactly this sort of textual near miss [73].

Across both transcripts, the Results show that variance is a property of the whole rating system rather than a uniquely ‘AI’ issue. Both human and AI coders differed markedly in how many instances they identified, and in what they treated as a codable event, suggesting different thresholds and criteria. These patterns support the claim that, in practice, AI is neither systematically more nor less ‘reliable’ than human judges, but rather occupies one position on the same continuum of coder selectivity. Where convergence did occur, the Results show that the AI coders frequently aligned with human judges on prototypical, high-salience segments within the transcripts.

These observations about AI performance should, however, be treated with considerable caution. Large language models are highly sensitive to the way in which a task is specified, and their output varies with the prompt employed, the version of the model in use, and the definition of the task itself [74]. The present study offers a direct illustration of the point. AI₁ and AI₂ were not different systems, but the same model under two different prompts, and yet their event counts differed considerably. Nothing in the present data therefore allows for a general claim about the capacity of AI systems to identify basic emotions in psychotherapy transcripts. What the data show is narrower, though still of some interest: that, within a single model, the formulation of the prompt had a larger effect upon coding output than the difference between any two human coders. It would be of some interest to observe whether prompts developed and refined upon independent material, rather than applied directly to the transcripts of interest, would produce more stable results. The same question, of special note, applies to the human coders, whose written instructions were also a form of prompt, and whose variability may in part reflect the way in which the task was framed for them. The design and refinement of ‘prompts’, for human and artificial coders alike, appears to us a potentially fruitful area of further research.

From these findings, there are several clear recommendations for the field.

Firstly, the field needs to pay more attention to the issue of classification, as suggested by the Solms’ quote opening this paper: “The psychoanalyst’s initial task is not analysis, then, but classification” [1] (p. 120). This includes clearly specifying by which criteria an event constitutes a Panksepp-synonym emotion category. For example, does it contain explicit use of a Panksepp synonym? How to treat negations (is the phrase “I am not afraid” an instance of FEAR?). How to treat historical references (“I was afraid a long time ago”)? How to treat references to others (“They are afraid”)? Explicit clarification on these issues would be helpful, and may lead to some interesting lines of research.

Secondly, the field needs to pay better attention to training. This seems especially important because psychotherapists are used to classifying emotions based on live in-person encounters, rather than written transcripts. Potential coders would need to undergo some form of explicit preparation, including working through multiple examples, perhaps culminating in a formally scored assessment of classification ability.

These principles will lead to greater reliability, and greater data will lead to research opportunities. With a more predictable ability to classify basic emotions, including explicit measures of reliability between humans and AI coders, comes the possibility of investigations at scale. The field is currently dominated by small-scale anecdotal findings, for which this paper seeks to be a shift in direction. As elsewhere in science [75,76], there will be substantial advantages to working with much larger data sets, provided the groundwork of reliability is in place initially.

4.5. The Exploratory Status of the Present Study

Several features of the present study require that its conclusions be framed with caution. It is best understood as an exploratory, proof-of-concept investigation, designed to establish whether the coding of basic emotions in psychotherapy transcripts is feasible, and to identify the methodological problems that a larger study would need to solve. This paper is not intended to support general claims about the reliability of psychotherapists, about patients with amnesia, about the frequency of emotional expressions per session, or the performance of AI systems. Instead, these data suggest that this approach to coding is feasible, and one might readily imagine a number of interesting lines of future enquiry.

5. Conclusions

This has been an unusual project in several respects. It is the first attempt to measure the reliability of the classification of basic emotions in psychotherapy. It uses remarkable content, derived from therapy transcripts of a particular type of neurological patient, as well as being the first paper that has attempted to do this process with AI and human coders. This is therefore not a place to identify conclusions, but instead to squarely recognise the many challenges which face the field. These are undoubtedly substantial, but also appear to be eminently achievable.