Natural Language Processing and Machine Learning Supporting the Work of a Psychologist and Its Evaluation on the Example of Support for Psychological Diagnosis of Anorexia

Abstract

Objective: This study sought to address the use of computer-aided diagnosis and therapy for anorexia nervosa. This paper presents the means by which the use of natural language processing methods can augment the work of psychologists. Method: We evaluated this method based on its efficacy when diagnosing anorexia nervosa. Using natural language processing and machine learning, we developed methods for analyzing five basic emotions, analyzing a patient’s body perception, and detecting six potential areas of difficulties for computer support of psychological diagnosis of anorexia. We surveyed 43 psychologists to obtain feedback on these tools. Results: We evaluated efficacy in terms of patient relationship, substantive aspects of the diagnosis, and diagnostic procedures. In terms of patient relationship, we found a noticeable decrease in the patient’s resistance and better support in verifying the substantive scope of the diagnostic thesis. Discussion: The presented methods can be a supporting tool for monitoring the diagnostic process and increasing the degree of self-diagnosis and self-reflection by the patient. This tool can increase the accuracy of the diagnostic process by reducing patient resistance. This will increase access to the patient’s psychopathology.

1. Introduction

Recently, the demand for psychologists has substantially increased. The number of professionals is limited, and as a consequence, a psychologist cannot devote an appropriate amount of time to an individual patient. Therefore, it seems to be crucial to adopt techniques and technologies to better meet the needs of patients. One solution is modern computer-assistive techniques supporting psychologists [1,2].

With the growing need for psychological support, there is also the growing need for reliable psychological diagnosis. The correct diagnosis is the basis for appropriate treatment. Presently, clinical psychological diagnosis is based on empirical evidence which is the result of recommended standards [3]. The standards of clinical practice are evidence-based [4,5]. Such practice assumes that clinical diagnosis and therapeutic influences should focus not only on a therapist’s experiences but mainly on reliable, verified data and modern research [6]. The principles of this attitude come from the classic differentiation of Hans Reichenbach [7].

There is a two-part process to a diagnosis. Making a diagnosis begins with generating a differential diagnosis and then formulating hypotheses based on knowledge and clinical experience that support or refute those hypotheses based on a patient’s signs and symptoms. To validate diagnoses, we examine these hypotheses based on scientific evidence. Collecting information for a diagnosis includes information from interview, observation, projective psychological methods such as a child’s drawing or the technique of free association. When using these methods, the rules for interpreting the material are not tightly formalized and a patient is more important than statistics. In the validation context, on the other hand, we must use the right psychometric evidence-based tools. The methodology to establish these tools was rigorous (these studies are standardized based on a particular population and have adequate reliability, accuracy, and standards referring to age and sex) [2,5,6,8].

A unique challenge for a psychologist is diagnosing children and teenagers, where we need to be aware of their development (e.g., that they are still going through puberty). The quality of their cognitive, emotional, and social functioning depends on the harmonious shaping of psychological and physical factors, including gross and fine motor development [2,7]. We should also consider the environmental context in which children and teenagers develop. This environment comprises the presence or lack of adequate environmental stimulation. Therefore, the diagnosis of a subject in the developing period must include the detailed description of a patient and their broader environment [9].

Related Work Section

Computer science is poised to have a tremendous impact on psychology. Besides experiments and questionnaires, it establishes the observation of human device interaction on a vast scale as a third fundamental research technique [10]. Markowetz et al. [11] and Yarkoni [12] describe psycho-informatics as an emerging discipline that uses tools and techniques from the computer and information sciences to improve the acquisition, organization, and synthesis of psychological data. One of the methods of applying IT tools is CAI (computer-assisted interviewing). One article [13] explores the use of CAI as a tool for consulting with children with autism spectrum disorders (ASD). It was considered within the context of a research study that utilized one CAI program. Based on a process of reflection through dialogue, the authors reported on the perceived value of this methodology compared with traditional semi-structured interviewing. The researchers concluded that the CAI approach offers potential benefits in interviewing children with ASD. Today, a computer often at least partially analyzes data generated from psychological assessments. In another article [14], the author focused on the interaction between the decision-making processes of human clinicians and the test interpretations that are computer-based. The author concluded that both clinical judgment and computer-based test interpretation have weaknesses. However, clinicians can optimize their decision making by considering computer-based test result interpretations.

Computer vision can also support clinical decision making. This technique allows the psychologist to monitor a patient’s face during the psychological interview and to detect hidden emotions. The authors of another paper [15] described approaches for psychological characteristic recognition from facial images such as physiognomy, phase facial portrait, and ophthalmo-geometry. Such information could be much more objective than questionnaires and neuropsychological testing and could be obtained remotely using a person’s facial portrait with the clinician needing to be in the same room as the patient.

This discussion suggests how vital computer methods are and could be in supporting psychologists’ work. These tools may be especially helpful when needing to make a diagnosis quickly. The issue of time limitation has been highlighted in the current coronavirus pandemic. One paper [16] described Structured Letter Therapy. It is a new method of consultation that has developed during the current COVID pandemic. In addition to routine face-to-face psychological counselling, remote written counselling may become a new type of psychological counselling in the context of public health emergencies. Structured Letter Therapy is a kind of feasible psychological intervention approach.

The historical review of methods for natural processing have been described previously [17]. There are different methodologies of processing text documents. For example, there are data clustering of a document, document classification, and sentiment analysis. NLP methods are used for a randomized controlled trial to check whether a particular therapy is beneficial for patients. By analyzing the vast number of therapist-client interactions, we can identify interactions by therapists that are particularly helpful to clients [18,19,20]. Some researchers, who were unsatisfied with the lack of transcripts, developed an automated system for speech recognition, which analyses patient–therapist audio records and then sends the feedback to the therapists [1].

Facebook uses NLP for detecting suicidal language and provides the users with a list of resources that they can use for support [21]. NLP is crucial, especially when it comes to the screening of people’s future mental health. The screening tools may include passive analysis of collected data or actively administering and interpreting the interviews. Chat bots are another way of applying NLP. Woebot, one of the most recognizable chat bots, uses scripted responses to protect the users from getting hurt as the result of learned answers. Woebot is integrated within Facebook Messenger. Randomized controlled research has shown a greater reduction of depression symptoms among participants who use Woebot than among those who read eBooks on self-help [22]. Automated text data processing has some challenges that must be overcome to provide information privacy and the ethical agreement of users on participating in such research but is a promising avenue for the future.

It is worth noticing that according to the International Test Commission (ITC), each procedure, quantitative or qualitative, used in tests during the diagnostic process, should be treated as a test (International Test Commission (ITC) 2001). The proposed new criteria for evaluating diagnostic tools and standards for applying psychological tests can generally refer to the tool itself and the psychologist’s expertise (Anon n.d.). When it comes to using psychological tests, it is essential for a psychologist to have a good knowledge of measurement rules, validation procedures, and other alternative test procedures [23]. Tests must be seen as reliable and evidence-based by practicing psychologists to be useful. Hence, it is evident that we need feedback from experienced psychologists about using computational methods for NLP during the diagnostic process.

Nowadays, it is considered important to use NLP methods to improve our understanding of people and culture [24]. However, the complexity of some methods creates challenges in terms of preparing models, their learning and the method of assessing their effectiveness, which is a current topic in the literature [25]. In addition, an important issue is also the method of coding domain knowledge appearing in texts [26]. Taking into account the above-mentioned challenges known from the literature, the authors based their approach on longer-known and better-tested NLP methods. Dictionary methods and parameter-limited deep recursive networks were primarily used.

Based on the above state of the art, we propose a method of automatic sentiment assessment of patients’ statements and techniques for recognizing emotions which aims at fast detection of the potential causes of psychological disorders. The research can help to answer whether these methods are valid and improve the work of psychologists. We aim to present the possibilities of computer support using natural language-processing methods and machine learning in the work of psychologists. We evaluate this method with the example of the diagnostic process of anorexia nervosa.

The developed method allows for the quantitative assessment of the patient’s condition. The assessed areas include the patient’s feelings of body dysmorphia, measurements of the five basic emotions (happiness, anger, sadness, fear, and disgust), and potential areas of difficulty based on the patient’s description of his or her body. The patient’s statement is analyzed in the form of text. This requires creating a note or recording the patient’s statement and converting it into text. Technically, the method is implemented in the form of Python scripts and requires the input of the path to a text file containing the content of the patient’s note. At the initial stage of diagnosing anorexia diagnosis, there is a need for minimally invasive diagnosis of the patient, due to the severe condition of the body and the psychological resistance manifested by avoiding topics related to the disease. The developed method meets these challenges, allowing patients to prepare a short note about their bodies. The patient can prepare the note based on their preference for having other people around (preparation alone or in the presence of a trusted person or a psychologist) and its form (written note or audio recording). At the same time, the burden to the patient is minimized by avoiding invasive psychological tests and questionnaires of topics that the patient may not wish to talk about.

2. Materials and Methods

Computer methods of NLP and machine learning are especially useful in establishing diagnostic hypotheses. Within the discovery context, a psychologist must build a proper relationship with a patient based on common trust and a sense of security. Moreover, a patient should be motivated by a therapist to reveal their experiences, which are very often painful and shameful. Thus, it is vital to use appropriate tools when making a diagnosis. When in the early phase of the diagnostic process, by allowing the patient to use free association technique with NLP, we can reduce their stress level and make them feel more comfortable.

This method, at the same time, thanks to the free associations while making notes, enables us to collect data on various contexts related to a patient’s functioning and environment (e.g., peers, family relationships, attitudes toward school). It is especially handy in child/teenage diagnosis. The elements of NLP we propose for computer support in clinical diagnostic are presented below (diagram in Figure 1). They can especially be applied during the diagnosis of anorexia nervosa. An overview of the proposed machine learning approach is presented in Figure 2.

2.1. Applied NLP and Machine Learning Methods for Computer-Aided Psychological Diagnosis of Anorexia

2.1.1. Sentiment Analysis

The sentiment analysis problem can be presented as a classification of sequences of words to either positive, negative, or neutral categories of undertone. For analyzing the sentiment information available in patients’ body self-descriptions, the authors used a Recurrent Neural Network open in SAS Visual Text Analytics software [27]. The neural network has been trained on a collection of opinions from The Stanford Amazon Dataset service [28]. The selected model has been additionally trained with the specific text corpus.

2.1.2. Emotion Assessments

Natural language processing was used to evaluate the emotion expressed by the patient during body description. Each document was presented as a vector containing the frequencies of the occurrence of certain words. Preparation of the corpus from the papers included removing punctuation marks and creating the list of words present in each document, making a words stop-list, and transforming the words into basic grammatical form based on the Polish language dictionary included in SAS Visual Text Analytics (Anon n.d.). To rate the relation of documents with different aspects of expressing emotions, the Nencki Affective Word List (NAWL) [29] was used. We created a one-level taxonomy based on five emotion groups. Each group referred to one of the five basic emotions, which are happiness, anger, sadness, fear, and disgust. A rule set based on the NAWL dictionary was created to determine the emotion group to which the expressed emotion belonged. The taxonomy that uses dictionary method and a statistical model has been discussed in Ningham et al. [30].

2.1.3. Assessment of Specific Areas of Difficulty

The overall goal of the presented computer-aided diagnosis system was to provide additional support to psychologists in six areas. These areas correspond to clinical areas that require psychological treatment [31]. These areas include body self-perception, reception of the person by the environment, experienced emotions, aggression and self-aggressive behavior, body functionality, and perception of individual parts of the body. Documents were prepared as a list of words in their basic form representation. The detailed dictionaries were based on the sentiment analysis presented by Wilson, Wiebe, and Hofman [32]. The particular categories were formulated by experts specializing in anorexia diagnosis and therapy [31].

2.2. Example Analysis of an Individual Patient

We present the results of the model applied to an individual patient and the patient’s medical review below. Example statement–a note prepared by the patient [33]:

“How did it start? Very simply, one day I understood that since nobody accepts me, I will accept myself, I started losing weight, I was never overweight–I was just chubby, the problem is that I hate being fat! At the beginning of losing weight, I ate very little, after a short time, I realized that if I do not eat, I feel incredible satisfaction–I feel so strong and persistent. So in the beginning I gave up sugars, butter, and fats, now my only meal during the is a salad. I know I need help, I don’t feel good anymore from not eating–I feel caught in a cage without retreat”.

Both methods of sentiment analysis showed an extremely negative sentiment: Recursive Neural Network method showed negative class with probability 0.68, the dictionary method showed a significant quantitative advantage of negative words. Feelings of sadness and fear accompanying negative attitudes were detected in the following intensities: sadness was 0.8 and fear was 0.24. The chart, including the specific areas of psychological difficulty for this patient, is presented in Figure 3 [33].

Psychologist’s analysis: The quoted words of a patient with AN includes statements suggesting the occurrence of typical, axial symptoms of eating disorders such as fear of getting fat, avoiding products with animal fat, satisfaction from maintaining the control, and an obsession with maintaining a restricted diet, losing weight, and feelings of helplessness when unable to lose weight. The obtained results using the proposed method coincide with the psychologist’s opinion.

2.3. Evaluation Methods for Supporting Psychologist

As a part of our research, we developed the following questionnaire (

Table 1. Questionnaire for assessing the influence of a developed NLP method in the work of a psychologist.

I	RELATION WITH A PATIENT
	Can the use of developed diagnostic computer tool for sentiment analysis (DCTSA) reduce the patient’s resistance to psychological examination?
	Can examination with DCTSA be conducive to a greater sense of security and intimacy in the initial phase of psychological diagnosis?
	Does the examination with DCTSA allow the patient to influence answering difficult questions without verbal contact with the researcher?
II	SUBSTANTIVE ASPECTS OF A DIAGNOSIS
	Can the situation of a test with DCTSA through a series of free associations of a patient support his self-reflection and self-diagnosis?
	Is the situation of test with DCTSA conducive to an in-depth exploration of the patient’s problem areas?
	Do you think that the use of DCTSA based on a series of free patient associations on any subject related to his psychopathology can support the assessment of the patient’s dominant affect related to the applied topic?
	Can the situation of testing using DCTSA help the psychologist to make diagnostic hypotheses?
	Can the situation of the DCTSA test influence the accuracy of the psychological diagnosis?
	Can DCTSA be used in the evaluation process of therapeutic interactions for a given patient?
III	DIAGNOSTIC PROCEDURES
	Can the situation of the DCTSA test significantly support the psychological diagnosis process?
	Do you think that the process of psychological diagnosis using DCTSA is significantly different from that without using the mentioned tool?
	In your opinion, can testing using DCTSA increase the comfort of a psychologist during the diagnostic process?

). The purpose of the developed questionnaire is to enable a quantitative assessment of psychologist’s support through the developed methods of natural language processing.

2.4. Experiments

The developed method was used by psychologists who typically work with patients suffering from eating disorders. The psychologists used the CAI method to:

(1)

Support the first phase of the diagnostic procedure—determining the areas of difficulty and the patient’s attitude to his or her body image;

(2)

Evaluate the diagnostic process—monitoring the intensity of the five basic emotions of the patient.

The psychologist first familiarized his or herself with the idea and possibilities of the developed method presented in this paper. Then, during the second diagnostic meeting, after the initial relationship between the therapist and patient was set, the psychologist asked the patient to write a note about how they see their body. This approach can be used to identify psychopathology associated with anorexia nervosa. The patient was free to choose to write the note during the meeting or at home. Next, the prepared note was examined using the above-described NLP tools. A research team member responsible for IT supervised this process.

The analysis of the note generated results in the areas of attitude towards one’s body, the intensity of the 5 basic emotions, and potential areas of difficulty were used by psychologists as auxiliary elements in making a diagnosis. Then, psychologists assessed the usefulness of the NLP results at the diagnosis stage by answering the questions from the prepared questionnaire (Table 1). They evaluated the usefulness in the areas of establishing a relationship with a patient, substantive aspect of diagnosis, and diagnostic procedures. These results were analyzed in the context of the above-mentioned 3 groups of issues.

2.5. Material

The samples of text used for testing were the notes prepared by patients who suffered from anorexia. Within the developed criteria, 51 girls with anorexia (restrictive form) were included. The participants were aged 13–18, an average of 15.8 ± 2. Anorexia was diagnosed according to ICD-10 and DSM V criteria. The average weight of the girls in this study was 36.5 ± 6.0 kg, and BMI was between 13.5 and 19.0, an average 16.2 ± 2.3.

A total of 43 psychologists participated in the survey. Each psychologist had a minimum of 5 years’ work experience in the National Health Service. Additionally, the psychologists each had a specialization in clinical psychology or had been completed their residency training in clinical psychology. The research was conducted with the consent of the Ethical Committee of the Silesian Medical University.

3. Results

Figure 4 presents the results of a survey in the form of a radar chart, normalized to the percent scale.

We also conducted a statistical chi-square test (confidence level = 0.05) on the survey results to mark the relevance of differences in the experts’ responses within particular groups of issues (the compared values are the percentage of yes marks) related to computer support of psychologist’s work. The issues include a relation with a patient, substantive aspects of a diagnosis, and diagnostic procedures. The results indicate the lack of statistical differences in the expert responses within the responses referring to relation with a patient and substantive aspects of a diagnosis (p-value = 0.744). On the other hand, the results showed a significant difference in the responses of experts between diagnostic procedures connected to relation with a patient (p-value = 0.0479) and substantive aspects of a diagnosis (p-value = 0.0086).

4. Discussion

The research results can be analyzed in many different ways. Nearly 95% of respondents in our survey group reported an improvement in establishing a relationship with the patient. This was primarily due to a reduction in patient’s resistance (100%). The improvement in relation is crucial because, generally, in psychology, achieving a proper patient–psychologist relation is always the most challenging part of therapy [34,35].

In the group of survey issues corresponding to substantive diagnosis, around 94% of responses indicated improvement, especially in establishing the diagnostic hypotheses (97%) which is the primary diagnostic challenge [3,36,37].

The most differentiated results we received in the survey group were referring to the diagnostic procedure. All (100%) of the responders indicated the improvement in work comfort and 70% agreed that this method accelerates the diagnostic process. At the same time, 60% of psychologists declared that the method helped them increase their self-confidence in establishing the diagnosis. These results may indicate the need to better understand the tools and the developed method, primarily their technical and IT issues. An independent aspect seems to be the little experience of a psychologist in applying projective methods in the diagnostic process, which requires to be supervised by an expert.

The chi-square test results show the lack of meaningful differences among groups of questions referring to relationship with a patient and the substantive aspects of a diagnosis. The experts agreed on the benefits that come from applying the computer systems in the diagnostic process. The primary benefit based on our study was the reduction in the patient’s resistance towards the examination. Oftentimes, asking the patient difficult questions may affect the practitioner’s relationship with the patient. Moreover, the methods enabled us to increase the patient’s sense of security and intimacy during the first phase of diagnostics. Similarly, in the scope of the merits of the diagnosis, the experts strongly indicated the possible positive impact of using the support platform on the patient’s self-reflection and self-diagnosis, deepening the possibilities of exploring problem areas and facilitating the assessment of the dominant affect related to a given topic, thereby facilitating the diagnosis of hypotheses and determining the effectiveness of possible interventions in the field of psychological intervention.

A significant difference was noticed for issues related to diagnostic procedures. Psychologists who participated in the survey showed considerable indecision in indicating the usefulness of the method to for increasing confidence when establishing a diagnosis and the benefits associated with shortening the diagnostic process. It may be due to psychologists’ disbelief in new technologies compared to classic diagnostic methods that do not use computer-aided systems. We should also note that a psychologist’s fluency in the use of portable devices will have a final impact on the duration of the diagnostic procedure. Thus, IT training of psychologists in the use of diagnostic support systems will also be required.

There are several limitations to the proposed method. The developed NLP method was used for computer-support of the diagnostic process of anorexia. In the future, we plan to develop detailed guidelines for the use of the developed method in other mental illnesses. There are problematic issues that can be encountered in text analysis such as the problem of irony (inverted meanings of words), the ambiguity of expression, and contextual interpretation. To limit the impact of these phenomena in the future, we plan to combine the conducted analysis with speech analysis, which contains additional information about the non-verbal side of speech (e.g., tempo, stressing, changing the pitch, intonation). It is also important to emphasize the possible limitations of the method’s application to patients with a tendency to dissimulate. Hence, an experienced psychologist diagnostician should always use NLP tools in combination with other diagnostic tools.

The computer-aided methods that we developed in this paper supporting the psychological diagnosis of anorexia using natural language-processing methods may complement diagnostic procedures. The methods can be a supporting tool in monitoring the diagnostic process and supporting the process of increasing the degree of self-diagnosis and self-reflection of the patient. The implementation of the method developed in this study may lead to an increase in diagnostic accuracy by reducing patient resistance. This could lead to greater access to the patient’s psychopathology.