Attention Bidirectional Recurrent Neural Zero-Shot Semantic Classifier for Emotional Footprint Identification

Abstract

Exploring emotions in organization settings, particularly in feedback on organizational welfare programs, is critical for understanding employee experiences and enhancing organizational policies. Recognizing emotions from a conversation (i.e., leaving an emotional footprint) is a predominant task for a machine to comprehend the full context of the conversation. While fine-tuning of pre-trained models has invariably provided state-of-the-art results in emotion footprint recognition tasks, the prospect of a zero-shot learned model in this sphere is, on the whole, unexplored. The objective here remains to identify the emotional footprint of the members participating in the conversation after the conversation is over with improved accuracy, time and minimal error rate. To address these gaps, in this work, a method called Attention Bidirectional Recurrent Neural Zero-Shot Semantic Classifier (ABRN-ZSSC) for emotional footprint identification is proposed. The ABRN-ZSSC for emotional footprint identification is split into two sections. First, the raw data from a Two-Party Conversation with Emotional Footprint and Emotional Intensity are subjected to the Attention Bidirectional Recurrent Neural Network model with the intent of identifying the emotional footprint for each party near the conclusion of the conversation and, second, with the identified emotional footprint in a conversation. The Zero-Shot Learning-based classifier is applied to train and classify emotions both accurately and precisely. We verify the utility of these approaches (i.e., emotional footprint identification and classification) by performing an extensive experimental evaluation on two corpora on four aspects, training time, accuracy, precision, and error rate for varying samples. Experimental results demonstrate that the ABRN-ZSSC method outperforms two existing baseline models in emotion inference tasks across the dataset. An outcome of the proposed ABRN-ZSSC method is that it obtains superior performance in terms of 10% precision, 17% accuracy and 8% recall as well as 19% training time and 18% error rate compared to the conventional methods.

1. Introduction

Emotional footprints out the enduring emotional influence a person leaves on others via their interactions (intrinsically, the idea or thought their emotions create in a given environment), while emotional intensity expresses the depth of an individual’s emotional experiences, designating how strongly they feel a particular emotion. Emotional footprints are defined as to the lasting impression an individual or group’s emotions and actions leave on others, similar to a physical footprint. Emotional intensity is referred to as the strength or degree of an emotion experienced. Emotional footprints are based on several criteria, such as strength, duration, physiological arousal (heart rate, voice tone), and impact on the conversation/participants. Also, emotional intensity depends on numerous criteria, namely the type of emotions (positive/negative) and influence on the other conversant. Measurement methods for the emotional footprint consider self-reporting, observer ratings, textual analysis and physiological data. Measurement methods for emotional intensity include follow-up surveys, content analysis and network analysis. Actual conversation data are gathered. After, emotional categories (anger, joy, fear) are determined. Despite the emotional footprint and emotional intensity being used interchangeably, emotional footprint aids in analyzing how somebody’s emotions influence others around them, whereas emotional intensity is employed in expressing how strongly somebody perceives a specific emotion.

A method called Emotional Voice Conversion with Emotion Intensity Control (EMOVOX) was proposed in [1], with the intent of explicitly featuring and controlling emotion intensity. Here, the speaker style was extracted from linguistic content, following which the speaker style was encoded in a continuous space, therefore forming an emotion-embedding prototype. Moreover, to make certain, emotional intelligibility, emotion classification loss and emotion embedding similarity loss were also included, controlling fine-grained emotion intensity in the output speech, therefore reducing emotion classification loss and improving emotion-embedding similarity loss extensively.

Despite the minimization of emotion classification loss and improving emotion-embedding similarity loss, the training time and the accuracy involved were not analyzed. To focus on these two aspects, an Attention Bidirectional Recurrent Neural Network model is designed by using bidirectional RNN, which leads to improved accuracy with minimal training time by taking into consideration both the past and future of a sequence involved in conversation.

A multi-task deep Cross-Attention Network (MTCANet) that simultaneously performs Key Word Spotting (KWS) and Speaker Verification (SV), while efficiently using information pertaining to both tasks, was proposed in [2]. Also, the method combined a KWS sub-network along with an SV sub-network to improve overall performance in demanding circumstances, including shot duration speech, noisy environments and so on.

Intrinsically, the method included three modules, novel deep cross-attention (DCA) for combing KWS and SV tasks, multi-layer stacked shared encoder (SE) to minimize the influence of noise on the recognition rate and, finally, soft attention (SA) to concentrate on relevant information while circumventing gradient vanishing with minimal error rate and improved accuracy. Despite improvement observed with maximal accuracy and minimal error rate, however, the precision rate was not analyzed. To focus on this aspect, a Zero-Shot Learning-based classifier is introduced that, with the aid of semantic features, achieves the objective.

With online learning, it is tricky for teachers to identify the emotional state of students for adjusting their pace and attention to each learner. A machine learning model called TTNet was employed in [3] for gathering the facial data and examining the emotions of each learner in the online classroom, but the accuracy was not enhanced. Humans are prepared to realize each other’s emotions via ingenious facial expressions, body movements, the way they speak, or unambiguously by the tone of voice. Alternately, computationally speaking, numerous methods are being designed for automating the procedure of information analysis from media sources to determine the emotions expressed by users. An ensemble strategy was applied in [4] to classify the emotion observed from a speech with the intent of improving overall classification accuracy.

Yet another method to focus on the accuracy aspect using a deep neural network model was designed in [5] by considering five different emotions, joy, shame, guilt, sadness and fear. However, certain drawbacks were observed with keyword and lexicon-based methods as they concentrate on semantic relations. To address this issue, a hybrid machine and deep learning method to identify emotions intext was proposed in [6]. These hybrid learning methods resulted in an improvement in overall accuracy. Despite improvement in accuracy, loss factors involved in analysis were focused on in [7], employing a convolutional neural network (CNN) with long short-term memory (LSTM).

Research into speech-based emotion recognition in clinical applications has been continuously increasing over the past few years. To this end, a novel algorithm that takes the edge off the feature extraction mechanism and attributes an importance level of selected neurons by using deterministic edge/node embeddings with attention scores was presented in [8], therefore improving accuracy in an extensive manner.

The challenges and opportunities involved in the automatic detection of expression emotion from a speech were investigated in [9] to focus on inter- and intra-reliability aspects. The challenges and progress involved in deep speaker recognition were presented in [10]. However, in the task of emotion inference, the most prevalent issue is the lack of common-sense knowledge, specifically in the context of dialogue, where conventional research has failed to efficiently extract structural features, resulting in minimizing accuracy related to emotion inference. To address this problem, a dialogue emotion inference method employing Common Sense Enhancement and a Graph Model was proposed in [11].

A systematic review of speech synthesis employing deep learning was investigated in [12]. Yet another comparative study of recognizing emotional footprints using machine learning was presented in [13]. A triangulation method for extracting a novel set of geometric features with the intent of classifying six different emotional expressions, namely, anger, fear, disgust, surprise, happiness and sadness, employing computer-generated markers was proposed in [14].

An ensemble of a deep convolutional and recurrent neural network was designed in [15] for emotion recognition in speech to improve the final recognition rate. However, these texts generally contain several hidden emotions that could indirectly contribute significantly to identifying sentiments. The emotion detection problem was addressed in [16] via a two-stage emotion detection methodology with improved accuracy.

The success of deep learning algorithms motivates us to use a recurrent neural network in the emotional footprint classification process. In this study, we report a new method using the Attention Bidirectional Recurrent Neural Network and Zero-Shot Learning-based classifier. First, we identify emotional footprints using the Attention Bidirectional Recurrent Neural Network model. Then, we pass all the identified emotional footprints through a Zero-Shot Learning-based classifier for accurate and precise classification of emotions. This experiment was carried out on a Two-Party Conversation with Emotional Footprint and Emotional Intensity datasets. Our experimental work reveals that the proposed method outperforms some previously published results.

1.1. Innovations and Contributions of the Work

To address the above issues, like handling the error rate and measuring the accuracy rate using the two-party conversation dataset recorded at the end of conversation, the contributions of the Attention Bidirectional Recurrent Neural Zero-Shot Semantic Classifier (ABRN-ZSSC) are listed as given below.

• To improve the precision, accuracy, error rate and training time, the emotional footprint classification method, ABRN-ZSSC, is designed based on an Attention Bidirectional Recurrent Neural Network employing a Zero-Shot Learning-based classifier.
• To achieve computationally efficient emotional footprint, the Attention Bidirectional RNN, taking into consideration both the past and future of a conversation, allows for a more comprehensive understanding of the corresponding conversation. It includes Global Conversation State, Speaker Conversation State and Speaker Conversation Update State. Global Conversation State is employed to capture the overall context of all preceding utterances in the dialogue. Next, the Speaker Conversation State is used to maintain the path of the state of person speakers using fixed-size vectors throughout the conversation, and Speaker Conversation Update State is applied to update the individual speaker’s state based on their current utterance. In this way, the training time is reduced.
• To enhance the precision and accuracy classifier, the Zero-Shot Learning-based classifier algorithm is employed for identifying the emotional footprint accurately.
• To reduce the error rate, the Zero-Shot Learning-based classifier is used for classifying seven different types of emotions via semantic feature mapping. Also, the high correlation results are determined between features and emotional categories via attribute learning.
• Finally, comprehensive experimental assessment is carried out with four unique performance metrics, precision, accuracy, error rate and training time, to illustrate the proposed ABRN-ZSSC method over traditional methods.

1.2. Organization of the Work

This study is organized as follows: In Section 2, we lay out our study by introducing the background and related work pertaining to emotional footprint classification. In Section 3, we present the details of our proposed Attention Bidirectional Recurrent Neural Zero-Shot Semantic Classifier (ABRN-ZSSC) method, and we introduce our experiments in Section 4. In Section 5, we report the experimental results, followed by a discussion and conclusion in Section 6.

2. Related Works

Emotional footprint refers to the impression we leave, such as walking through a garden, associating with family members or interacting with the employees of an organization. Hence, an emotional footprint is the consequence of emotional transmission, be it positive or negative. As a leader, the emotions and strategy relating to other people influence the organizational environment. To the same extent that there is heterogeneity in the mechanisms that pollute the earth, there is an abundance of mechanisms a leader can utilize to tune the emotional environment of an organization.

Deep learning-assisted semantic text analysis was proposed in [17] with the objective of detecting human emotion utilizing big data. A survey of methods concentrating on the recognition of emotions involved during conversations was explored in [18]. An automatic emotion recognition system employing two-level ensemble classifiers was proposed in [19] by making accurate differentiation between high valence versus low valence and high arousal versus low arousal.

A new framework employing temporal dynamics by incorporating LSTM was presented in [20,21] to improve accuracy. Speaking in a low voice can also result in wrong predictions. To address this gap, emotion analysis based on linguistics using an attention score and softmax function was presented in [22]. This, in turn, not only ensured accurate prediction but also minimized the errors involved in emotion analysis.

A multi-task network was designed in [23] for both speaker and command recognition, with minimal execution time. Also, the emotion detection problem as a part of sentiment analysis was presented in [16] using the zero-shot model with an improved accuracy rate. Affective structural embedding was employed in [24] using zero-shot learning and emotional categories to design an accurate classification model. Yet another novel mechanism employing extreme learning to reduce complexity involved in unseen labels was presented in [25]. A systematic review of zero learning and machine learning classifiers was investigated in [26] towards the abstract screening of emotions.

Nevertheless, fine-grained sentiments might integrate similar emotions with a single primary emotion. Trying to address this issue as a classification task can result in performance improvements; however, it does not generate a better comprehension and representation of language. A fine-grained sentiment analysis mechanism employing neutrosophy was proposed in [27] via three different membership functions, positive, negative, and neutral, to model an instance into Single-Valued Neutrosophic Sets (SVNSs) with a higher level of precision and accuracy. A review on emotion detection employing deep learning algorithms was presented in [28]. A novel method for detecting emotion by fine-tuning a distilled zero-shot student model with the objective of classifying emotions in text was presented in [29].

Our study builds upon the existing body of research in emotional footprint identification within deep learning by specifically concentrating on the application of a zero-shot learning-based classifier model to analyze emotional footprints via semantic features. While previous studies have extensively explored emotional footprint identification using various machine and deep learning techniques, our research distinguishes itself by focusing on the emotional footprint of the members participating in the conversation after the conversation is over and the application of zero-shot learning employing semantic features. The elaborate description of the proposed Attention Bidirectional Recurrent Neural Zero-Shot Semantic Classifier (ABRN-ZSSC) for emotional footprint identification is provided in the following subsections.

3. Materials and Methods

While previous research has studied linguistic content and Key Word Spotting in explaining the variation in emotional footprints, a more comprehensive exploration of individual drivers would benefit the development of effective and equitable mitigation policies. Based on fine-tuning, the limitations of existing emotion recognition research are included, such as data scarcity and cost, poor generalization to unseen classes and computational expense. To address this issue, zero-shot learning is employed by enabling models to recognize emotions from previously unseen categories by leveraging prior semantic information.

Table 1. Advantages of zero-shot learning and fine-tuning.

Advantages	Zero-Shot Learning	Fine-Tuning
Adaptability	Highly flexible. It handles new emotion categories instantly without retraining.	Needs resource-intensive retraining cycle when new classes emerge.
Cost and Speed	Lower data collection cost and faster deployment for new tasks.	High costs and time associated with data labeling and training.
Generalization	Designed to generalize to unseen data by using prior knowledge	Performance is degraded
Scalability	More scalable for huge number of emotion categories	Scalability is limited for new data and every condition.

shows the advantages between zero-shot learning and fine-tuning.

This section presents our proposed Attention Bidirectional Recurrent Neural Zero-Shot Semantic Classifier (ABRN-ZSSC) method for emotional footprint identification from a fine-tuned Zero-Shot Learning-based classifier. The proposed method, ABRN-ZSSC, is depicted in Figure 1.

As shown in the above figure, the proposed ABRN-ZSSC method is split into two sections. The data obtained from the Two-Party Conversation, with the Emotional Footprint and Emotional Intensity datasets as input, are first subjected to the Attention Bidirectional Recurrent Neural Network model for emotional footprint identification. In the discussion, we initially give details of the indispensable cleaning and encoding activities performed in data preparation using the Attention Bidirectional Recurrent Neural Network model with the identification of emotional footprint in a conversation. Bidirectional RNN examines the input sequence in both forward and backward directions. This ensures that the model has context from both the past and the future of a conversation and allows for a more comprehensive understanding of the corresponding conversation. An attention layer is added to the Bidirectional RNN. This layer learns to assign different attention weights to different parts of the input sequence, effectively highlighting the most important elements for emotion identification. It is employed to capture long-term dependencies and consider the most relevant words or features (text, audio, or video) for more accurate emotion classification.

Second, with the identified emotional footprint as the basis, a Zero-Show Semantic Classifier is applied to classify the corresponding emotions accurately and precisely. Then, we highlight the fine-tuning experiments conducted employing the fine-tuned Zero-Shot Learning-based classifier to train and classify emotional footprints in an accurate and precise manner. Zero-Shot Learning involves learning paradigms to identify classes it has never encountered. Emotional Footprint Classification refers to the task of identifying or categorizing emotions from data (text, facial expressions, speech, or other physiological signals). This is succeeded by the experimental setup and discussion on the evaluation and testing of the methods.

3.1. Dataset Description

The Two-Party Conversation with the Emotional Footprint and Emotional Intensity datasets [30] is used to conduct experiments. This dataset is a single-modal (text-only) dataset in emotion identification. It is extracted from https://data.mendeley.com/datasets/fvfjp6n3x9/1 (accessed on 15 November 2023). It consists of 1857 tagged conversations. In the dataset, most features are involved for each of the two parties, such as Emotional Footprint and Emotional Intensity, in the conversation. Emotional Footprint refers to the emotional state or a collection of emotional labels associated with each participant. Emotional Intensity is measured as the strength or severity of the emotions expressed by each party. The emotional footprint and intensity data are noted for each participant near the conclusion of the conversation. The dataset is Csv file format. The data are structured with column headers in the first row and individual conversation data to an individual conversation in the subsequent rows. The aim of the dataset is to train and evaluate models on how emotions and their intensity evolve and are perceived within a dialogue for providing specific metrics for researchers to assess model accuracy and precision.

Table 2. Two-Party Conversation with Emotional Footprint and Emotional Intensity dataset.

S. No	Column headers	Features	Description
1	“0”–“34”	Emotional categories [0–6]	0—no emotion 1—angry 2—disgust 3—fear 4—happiness 5—sadness 6—surprise Length of conversation
2	“35”	Conv_length	Length of conversation
3	“36”–“39”	Inform, question, directive, commissive	Number of utterances within the conversation that fall into their respective categories
4	“40”	Emotion_footprint_first_person	Emotional footprint of the first person in conversation [No emotion, Anger, Disgust, Trust, Happy, Sadness, Surprise Anticipation]
5	“41”	Emotion_footprint_second_person	Emotional footprint of the second person in conversation [No emotion, Anger, Disgust, Trust, Happy, Sadness, Surprise Anticipation]
6	“42”	Conv_intensity_footprint_first_person	Emotional footprint and intensity of the first person in conversation
7	“43”	Conv_intensity_footprint_second_person	Emotional footprint and intensity of the second person in conversation
8	“44”	Emotion_intensity _first_person	Intensity of emotion experienced by first person in conversation
9	“45”	Emotion_intensity_second_person	Intensity of emotion experienced by second person in conversation
10	“46”	Conversation	Complete and unchanged conversation between two parties
11	“47”	Speaker 1	Utterances made by first speaker within the conversation
12	“48”	Speaker 2	Utterances made by second speaker within the conversation

, given below, lists the column header along with the feature name and description.

With the aid of the above dataset, corresponding to the utterance-level emotions of each statement within a conversation between two speakers and values within these fields ranging from 0 to 6, the Emotional Footprint noted for each party (i.e., speaker) near the conclusion of the conversation is analyzed and validated using the proposed method.

3.2. Attention Bidirectional Recurrent Neural Network-Based Emotional Footprint Identification

Emotion footprint identification in conversations has been receiving increasing awareness from the research community owing to its applications in several predominant tasks, including opinion mining over chat history and social media threads on Twitter, Facebook, YouTube, and so on. In this section, we present an algorithm based on an Attention Bidirectional Recurrent Neural Network that can serve these requirements by processing the available conversational data. Previous studies using traditional deep learning or machine learning algorithms were developed for emotion footprint identification. But it faces several drawbacks, such as an inability to capture the temporal context, feature engineering complexity and failure to handle huge datasets. To address this issue, the Recurrent Neural Network (RNN) is well suited and selected for emotional footprint analysis by the inherent temporal and sequential nature of emotional data. RNN is designed to process data in sequences using an internal memory (hidden state) to retain context from prior inputs that is crucial for capturing how emotions evolve and change over time. RNN learns and extracts relevant features from raw data (text data) for better performance compared to traditional machine learning techniques. RNN is employed to understand the dynamics of emotion. Figure 2 shows the structure of the Attention Bidirectional Recurrent Neural Network-based emotional footprint identification model.

The Attention Bidirectional Recurrent Neural Network model shown in Figure 2 revolves around three characteristics as follows: each party is designed, employing a speaker state that changes as and when that speaker utters an utterance. This makes the model track the speaker’s emotion footprint swings throughout the conversations that are associated with the emotion behind the utterances. Moreover, the factors of an utterance are designed employing a global state, where the foregoing utterances and the speaker states are cooperatively encoded for statement characterization requisites for accurate speaker state representation. Finally, the Attention Bidirectional Recurrent Neural Network model hypothesizes emotion footprint representation from the party state of a speaker along with the foregoing speakers’ states as statements. This emotional footprint representation is then used for the final identification of emotional footprint in a conversation.

Let there be ‘$Mspeakers,{Sp}_1,{Sp}_2,\dots ,{Sp}_M$’, (‘$M=2$’ for the dataset ‘$DS$’ we used) in a conversation. The objective is to identify the emotional footprint labels (no emotion, angry, disgust, fear, happiness, sadness and surprise) of the constituent utterances ‘$U_1,U_2,\dots ,U_m$’, where utterance ‘$U_i$’ is uttered by speaker ‘${Sp}_{map}\left(U_i\right)$’, while ‘$map$’ is the mapping between utterance and index of its corresponding speaker. Here, ‘$Sp,U_i\in DS$’, and to update states and representation, GRU cells are employed. Here, each GRU cell evaluates hidden state designated as ‘$H_t=GRU\times \left(H_{t-1},S_t\right)$’ where ‘$S_t$’ represents the current input sample and ‘$H_{t-1}$’ represents the foregoing or former GRU state with ‘$H_t$’ referring to the current GRU output. The emotional footprint representation of the current utterance is modeled as a function of the emotional footprint representation of the former utterance. Finally, this resultant emotion footprint representation is sent to a fine-tuned zero-shot learned model for emotion classification.

3.2.1. Global Conversation State

Global Conversation State intends to capture the statement of a given utterance by cooperatively encoding utterance and speaker state. Addressing these states eases the inter-speaker and inter-utterance reliance to impart improved statement representation. The prevailing utterance ‘$U_t$’ replaces the speaker’s state from ‘$P_{map\left(U_t\right),t-1}$’ to ‘$P_{map\left(U_t\right),t}$’. We reproduce this change with GRU cell ‘${GRU}_G$’ with output size ‘${GCS}_G$’, employing ‘$U_t$’ and ‘$P_{map\left(U_t\right),t-1}$’ as given below.

$$G_t={GRU}_G\left(G_{t-1},\left(U_t⨁P_{map\left(U_t\right),t-1}\right)\right)+W_{G,S}+W_{G,H}+B_{G,S}+B_{G,H}$$

(1)

From Equation (1), ‘${GCS}_G$’ denotes the size of Global Conversation State vector with weight and bias denoted as ‘$W_{G,S},W_{G,H}$’ and ‘$B_{G,S},B_{G,H}$’ denoting the strength of association between neurons and constant value, permitting the neuron to be active when all inputs are close to zero via concatenation ‘$⨁$’. Finally, during training, the Attention Bidirectional Recurrent Neural Network fine-tunes both weights and biases through forward and backward processes with the objective of optimizing its emotional footprint predictions on the training data.

3.2.2. Speaker Conversation State

The proposed method keeps track of the state of individual speakers employing definite fixed-size vectors ‘$P_1,P_2,\dots ,P_M$’ throughout the conversation. These states are characteristic of the state of speakers during a conversation, pertaining to emotional footprint classification. These states are said to be fine-tuned based on the current role of a party in the conversation (i.e., listener or speaker) and the subsequent observed utterance ‘$U_t$’. The foremost motive of this model is to make certain that the Speaker Conversation State model is aware of the speaker of each utterance and handles it accordingly.

3.2.3. Speaker Conversation Update State

Finally, the speaker usually mounts the feedback on the basis of the statement that is the foregoing or earlier utterances near the conclusion of the conversation. Therefore, we encapsulate statement ‘${Stat}_t$’ relevant to the utterance ‘$U_t$’ as given below.

$$\alpha =f_{\theta }⨁f_{{\theta }^{\prime }}^{\prime }\left(U_t^T{W}_{\alpha }\left[G_1,G_2,\dots ,G_{t-1}\right]\right)$$

(2)

$$f_{\theta }⨁f_{{\theta }^{\prime }}^{\prime }=\left[\left({Res}_0,{Res}_0^{\prime }\right),\left({Res}_1,{Res}_1^{\prime }\right),\dots \left({Res}_N,{Res}_N^{\prime }\right)\right]$$

(3)

$$f_{\theta }\left(S_0,H_0\right)=\left({Res}_0,H_1\right),f_{\theta }\left(S_1,H_1\right)=\left({Res}_1,H_2\right)$$

(4)

$$f_{{\theta }^{\prime }}^{\prime }\left(S_N,H_N^{\prime }\right)=\left({Res}_N^{\prime },H_{N-1}^{\prime }\right),f_{{\theta }^{\prime }}^{\prime }\left({Res}_{N-1},H_{N-1}^{\prime }\right)=\left({Res}_{N-1}^{\prime },H_{N-2}^{\prime }\right)$$

(5)

$${Stat}_t=\alpha {\left[G_1,G_2,\dots ,G_{t-1}\right]}^T$$

(6)

From Equations (2) to (6), ‘$G_1,G_2,\dots ,G_{t-1}$’ denotes the former ‘$t-1\prime$ Global Conversation State vector results. Moreover, in Equation (2), attention outcomes ‘$\alpha$’ over the former global conversation states representative of the former utterances are calculated. Finally, in Equation (6), the statement vector ‘${Stat}_t$’ is calculated by combining the former Global Conversation State vector results with ‘$\alpha$’. Finally, the identified emotional footprint near the conclusion of the conversation is represented as given below.

$$P_{map\left(U_t\right),t}={GRU}_{Upd}\left(P_{map\left(U_t\right),t-1,}\left(U_t⨁{Stat}_t\right)\right)$$

(7)

Equation (7) encodes the information on the current utterance along with its statement from the global GRU into the speaker’s state ‘$P_{map\left(U_t\right)}$’ that aids in further emotional footprint classification, shown in the next section. The pseudocode representation of Attention Bidirectional Recurrent Neural Network-based emotional footprint identification is given below.

Algorithm 1 describes a step-by-step process of obtaining computationally efficient emotional footprints. The raw data obtained from the Two-Party Conversation along with Emotional Footprint and Emotional Intensity datasets are applied to identify the emotional footprint of the members participating in the conversation near the conclusion of the conversation. The overall process is split into three states, namely, Global Conversation State, Speaker Conversation State and Speaker Conversation Update State. Initially, in the Global Conversation State, the statement of a given utterance is obtained cooperatively by encoding utterance and speaker state, following which, the Speaker Conversation State, in turn, keeps track of the state of individual speakers using definite fixed-size vectors throughout the conversation.

Algorithm 1 Attention Bidirectional Recurrent Neural Network-based emotional footprint identification

Input: Dataset ‘$DS$’, Samples ‘$S=\left\{S_1,S_2,\dots ,S_N\right\}$’, Features ‘$F=\left\{F_1,F_2,\dots ,F_n\right\}$’, Speaker ‘$S_1,S_2,\dots ,S_M$’, utterances ‘$U=\left\{U_1,U_2,\dots ,U_m\right\}$’

Output: mputationally efficient emotional footprint ‘$P_{map\left(U_t\right),t}$’

1: Initialize ‘N = 1857’, ‘n = 48’, ‘M = 2’, ‘m = 6’ 2: Begin 3: For each Dataset ‘DS’ with Speaker ‘$S_1,S_2,\dots ,S_M$’ Samples ‘S’ and utterances ‘U’ //Global Conversation State 4: Formulate Global Conversation State according to (1) //Speaker Conversation State 5: Keep track of individual speakers using fixed size vectors ‘$P_1,P_2,\dots ,P_M$’ around conversation //Speaker Conversation Update State 6: Formulate Speaker Conversation Update State according to (2), (3), (4), (5) and (6) 7: Obtain emotional footprint according to (7) 8: Return emotional footprint ‘$P_{map\left(U_t\right),t}$’ 9: End for 10: End

By combining these states, the Attention Bidirectional Recurrent Neural Network model keeps track of the individual speaker states all around the conversation and utilizes this information for emotion classification, therefore minimizing the training time extensively. Attending over this attention score and employing bidirectional aspects give statement representations that contain information of all preceding utterances by speaker during the conversation. This, in turn, makes certain that at each time instance, the speaker state obtains information from the speaker’s previous state and Global Conversation State that has information on the preceding parties. Finally, the Speaker Conversation Updated State is fed to decode emotion footprint representation of given utterance that is used for further emotion footprint classification, therefore improving the overall accuracy involved.

3.3. Zero-Shot Learning-Based Classifier

Emotion footprint classification is the task of connecting a conversation with a human emotion. State-of-the-art methods are typically learned using hand-crafted affective lexicons. In this work, we want to detect emotion footprints and their evolution for each party near the conclusion of the conversation. This situation results in numerous issues, ranging between small and mostly unlabeled datasets to identify and adapt methods for such situations. To handle this issue, a Zero-Shot Learning-based classifier is applied for emotional footprint classification in a precise manner with minimal error. Figure 3, given below, shows the structure of Zero-Shot Learning-based classifier model.

A comparison with the traditional Zero-Shot Learning-based classifier concentrates on transferring knowledge between speakers to impart supplementary information in recognizing unobserved classes. Nevertheless, it is laborious and cumbersome in using these mechanisms owing to the reasons that emotion appears in a latent layer in conversation, bringing about trials and tribulations in concluding emotional descriptors, and distinct affective emotions may also result in different emotional descriptors. In emotional footprint classification, zero-shot failures on low-intensity emotions are addressed by using attribute-learning phase to measure the relationship between generic semantic features and specific emotional categories. Hence, a Zero-Shot Learning-based classifier, as shown in Figure 3, using emotional footprints to associate semantic features (i.e., conv_length including, inform, question, directive and commissive) and emotional states is designed. The Zero-Shot Learning-based classifier using attribute learning harnesses semantic features (i.e., conv_length including, inform, question, directive and commissive) to learn emotional categories ‘$EC$’ (i.e., no emotion, angry, disgust, fear, happiness, sadness and surprise).

The attribute-learning phase aims at fitting emotional category values (known emotion labels, i.e., no emotion, anger, disgust, fear, happiness, sadness and surprise) to conversation samples utilizing corresponding semantic features (i.e., conv_length included, inform, question, directive and commissive). This attribute-learning phase models the correlation between ‘$l_F$’ features and ‘$l_{EC}$’ emotional categories. The ‘$m^{\left(S\right)}$’ utterance samples belonging to observed classes (known) and emotional states utilized in attribute learning are then mathematically expressed as given below.

$$P^{\left(OC\right)}={\left[P_1^{\left(OC\right)},P_2^{\left(OC\right)},\dots ,P_{m^{\left(OC\right)}}^{\left(OC\right)}\right]}^T\in {\mathbb{R}}^{m^{\left(OC\right)}\times l_F}$$

(8)

From Equation (8), ‘$l_F$’ denotes the feature dimensionality. The subsequent emotional categories ‘${EC}^{\left(OC\right)}\in {\mathbb{R}}^{m^{\left(OC\right)}\times l_{EC}}$’ of ‘$P^{\left(OC\right)}$’ are mathematically represented as given below.

$${EC}^{\left(OC\right)}={\left[{EC}_1^{\left(OC\right)},{EC}_2^{\left(OC\right)},\dots ,{EC}_{m^{\left(OC\right)}}^{\left(OC\right)}\right]}^T=\left[{\alpha }_1^{\left(OC\right)},{\alpha }_2^{\left(OC\right)},\dots ,{\alpha }_{l_{EC}}^{\left(OC\right)}\right]$$

(9)

Thus, the task of attribute learning is to learn the correlation between ‘$P^{\left(OC\right)}$’ and each category. By defining the mapping of category ‘$i$’ from ‘$P^{\left(OC\right)}$’ as

$$f_i\left(P^{\left(OC\right)}\right)={\left[f_i\left(P_1^{\left(OC\right)}\right),f_i\left(P_2^{\left(OC\right)}\right),\dots ,f_i\left(P_{m^{\left(OC\right)}}^{\left(OC\right)}\right)\right]}^T$$

(10)

Then, the mapping function for prediction task is represented as given below.

$$\widehat{f_i}\left(.\right)=argmaxSM\left(f_i\left(P^{\left(OC\right)}\right),{\alpha }_i^{\left(OC\right)}\right),suchthat{\phi }^{\left(OC\right)}\left(f_i\right)\in {\mathsf{\Omega }}^{\left(OC\right)}$$

(11)

From Equation (11), ‘$f_i$’ denotes the semantic feature mapping of the corresponding ‘$i-th$’sample, ‘$SM$’ denoting the similarity measurement between two vectors, ‘$f_i\left(P^{\left(OC\right)}\right)$’ and ‘${\alpha }_i^{\left(OC\right)}$’ via regularization term ‘${\phi }^{\left(OC\right)}$’ subject to a conditional vector set ‘${\mathsf{\Omega }}^{\left(OC\right)}$’, respectively. Finally, with the assumption that similar mappings are shared from semantic features to emotional categories, the ‘$l_{EC}$’ dimensional predicted emotional categories are obtained are given below.

$${\widehat{P}}^{\left(UOC\right)}={\left[{\widehat{f}}_1\left(P^{\left(UOC\right)}\right),{\widehat{f}}_2\left(P^{\left(UOC\right)}\right),\dots ,{\widehat{f}}_{l_{EC}}\left(P^{\left(UOC\right)}\right)\right]}^T$$

(12)

From Equation (12), results at test time (i.e., testing), speaker observes samples from classes that were not observed during training and are required to predict the class state of emotional footprint that they belong to, both precisely and with minimal error. The pseudocode representation of Zero-Shot Learning-based classifier is given below.

Algorithm 2 describes a step-by-step process of classifier model employing Zero-Shot Learning for improving precision with minimal error. Here, with the identified emotional footprint at the end of a conversation, with Speaker, Samples, Emotional Categories and utterances obtained as input, attribute learning is modeled using Zero-Shot Learning model. Here, initially, utterance samples belonging to observed classes (known) and emotional states are obtained, following which, subsequent emotional categories evolved. Third, a mapping function is formulated to finally obtain dimensional predicted emotional categories with the resultant classified results in a precise manner with minimal error. In this manner, depending on semantic attribute vectors to classify unseen emotional footprints using attribute learning realizes the requirement of comparing their semantic attribute features to the attributes of observed classes.

Algorithm 2 Zero Shot Learning-based classifier

Input: Dataset ‘$DS$’, Samples ‘$S=\left\{S_1,S_2,\dots .,S_N\right\}$’, Features ‘$F=\left\{F_1,F_2,\dots ,F_n\right\}$’, Speaker ‘$S_1,S_2,\dots ,S_M$’, utterances ‘$U=\left\{U_1,U_2,\dots ,U_m\right\}$’, Emotional Categories ‘$EC=\left\{{EC}_1,{EC}_2,\dots ,{EC}_l\right\}$’

Output: Precise and accuracy classifier

1: Initialize ‘$N=1857$’, ‘$n=48$’, ‘$M=2$’, ‘$m=6$’, ‘$l=6$’, regularization term ‘${\phi }^{\left(S\right)}=0.01$’ 2: Begin 3: For each Dataset ‘$DS$’ with Speaker ‘$S_1,S_2,\dots ,S_M$’, Samples ‘$S$’ and utterances ‘$U$’ //Attribute Learning 4: Formulate ‘$m^{\left(S\right)}$’ utterance samples belonging to observed classes (known) emotional states according to (8) 5: Formulate subsequent emotional categories according to (9) 6: Define mapping function according to (10) and (11) 7: Obtain ‘$l_{EC}$’ dimensional predicted emotional categories according to (12) 8: Return classified results ‘$CR$’ 9: End for 10: End

4. Experimental Setup

In this section, elaborate experiments are performed to validate the efficiency of the Attention Bidirectional Recurrent Neural Zero-Shot Semantic Classifier (ABRN-ZSSC) for emotional footprint identification using Python (version 3.11.2),with the aid of the dataset, Two-Party Conversation with Emotional Footprint and Emotional Intensity dataset, extracted from https://data.mendeley.com/datasets/fvfjp6n3x9/1 (accessed on: 15 November 2023). Comparison is made using two existing methods, Emotional Voice Conversion with Emotion Intensity Control (EMOVOX) [1] and Multi-task deep Cross-Attention Network (MTCANet) [2] and TTNet [3]. To ensure fair comparison, validation is made using the same dataset for all four methods, ABRN-ZSSC, EMOVOX [1], MTCANet [2] and TTNet [3].

The objective of the proposed ABRN-ZSSC method is to discover the emotional footprint of the members participating in the conversation with improved accuracy, time and minimal error rate. Based on the objective, the existing methods, such as the EMOVOX [1] and MTCANet [2] and TTNet [3], are taken as base paper. Existing EMOVOX [1] was employed for voice/keyword conversation. Existing MTCANet [2] was used for multi-task or cross-attention network with higher accuracy. These three base papers are relevant and compared to understand the proposed method. The proposed method concept is derived by considering the problems of these base papers. The drawbacks of these methods are effectively avoided by implementing the proposed method.

While performing experiments, the hardware and software requirements are included on a computer with an Intel(R) Core (TM) i5-7200 CPU @2.50 GHz and 8.00 GB of RAM. To validate the experiments, cross-validation is employed to compute the performance of the model ability or efficiency to utilize hidden data. By using cross-validation, the dataset is divided into two sets such as training and testing. Most samples (70%) were used for training, and the remaining (30%) were taken for testing. In experiments, adaptability is quantified by using several performance metrics, such as accuracy, precision and recall, as well as time for emotional footprint identification performance. Accuracy refers to a model or system correctly identifying and classifying emotions expressed in input data samples taken from the dataset. Precision is a performance metric that measures the accuracy of emotion-specific classification made by a model. Recall, also known as sensitivity, measures the ability of the model to correctly identify actual positive cases. Time is defined as an amount of time consumed by the algorithm for emotional footprint identification. To measure the adaptability using these above metrics, 10-cross-validation is used to enhance the emotional footprint identification performance. Hyperparameters are described in

Table 3. Hyperparameter details.

Hyperparameters	Description
Optimizer	Adam
Recurrent_dropout	0.4
Batch size	40
Epochs	50
Learning rate	0.0001
Activation function	Attention mechanism, Attribute learning

.

Hyperparameter selection is the process of choosing the best configuration for the parameters that control the learning algorithm’s performance. During learning, the rationale for these selections is rooted in a balance of model performance, computational efficiency, and preventing common drawbacks such as overfitting or underfitting. Learning rate or batch size is crucial for the stability and speed of the optimization process. The balancing bias and variance goal aims to find an optimal balance that generalizes well to unseen data. The rationale for hyperparameters during the inference phase is focused on fixed performance, speed, and reliability in a production environment.

5. Performance Comparison Analysis

In this section, the proposed ABRN-ZSSC method with the existing EMOVOX [1] and MTCANet [2] and TTNet [3] is discussed by using various parameters, such as accuracy, precision, recall, time and error rate. Contrary to conventional methods, the proposed ABRN-ZSSC method aims to obtain superior performance for the desired output. As shown in Table 4, the implementation of the Zero-Shot Learning-based classifier algorithm led to a notable enhancement in both accuracy and precision. Table 5 further demonstrates a significant reduction in processing time when the model is employed. This efficiency gain can be attributed to the more accurate estimation of state evaluation results, facilitated by incorporating both the Global Conversation State and the Speaker Conversation State. Table 6 provides evidence of a reduced error rate. Finally, Table 7 illustrates an increased recall rate achieved with the ABRN-ZSCC method.

5.1. Performance Analysis of Precision and Accuracy

Accuracy and precision are two performance measures with respect to error. On one hand, accuracy is how close a given set of measurements (observations or readings) are to their true value, whereas, on the other hand, precision is how close the measurements are to each other. Accuracy is also utilized as a statistical measure of how well a classification test (i.e., emotional footprint identification test) correctly identifies or excludes a condition. In other words, accuracy is defined as the ratio of correct predictions (i.e., including both true positives and true negatives) among the total number of samples being examined. Accuracy is represented as given below.

$$Acc={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(13)

From Equation (13), accuracy or emotional footprint identification accuracy ‘$Acc$’ is measured employing the true-positive value ‘$TP$’ (i.e., anger emotional footprint identified as anger), true-negative value ‘$TN$’ (i.e., anger emotional footprint identified as disgust), false-positive value ‘$FP$’ (i.e., disgust emotional footprint identified as disgust) and the false-negative value ‘$FN$’ (i.e., disgust emotional footprint identified as anger). The precision is measured based on the number of true positives and false positives. Precision is defined as given below.

$$Pre={\displaystyle \frac{TP}{TP+FP}}$$

(14)

From Equation (14), precision ‘$Pre$’ is measured using the true-positive instances ‘$TP$’ and the false-positive instances ‘$FP$’. To verify the effectiveness of the proposed ABRN-ZSSC method in the emotion footprint classification task, we conducted experimental comparisons between three baseline methods, EMOVOX [1] method, MTCANet [2] and TTNet [3], employing the abovementioned dataset in terms of precision and accuracy, as shown in Table 4.

Table 4. Experimental results of precision and accuracy using ABRN-ZSSC: EMOVOX [1], MTCANet [2] and TTNet [3].

Samples	Precision				Accuracy
	ABRN-ZSSC	EMOVOX [1]	MTCANet [2]	TTNet [3]	ABRN-ZSSC	EMOVOX [1]	MTCANet [2]	TTNet [3]
180	0.95	0.9	0.87	0.93	0.93	0.87	0.84	0.9
360	0.92	0.86	0.76	0.9	0.9	0.8	0.63	0.84
540	0.89	0.83	0.73	0.87	0.85	0.75	0.63	0.8
720	0.85	0.78	0.68	0.82	0.87	0.77	0.65	0.83
900	0.83	0.77	0.67	0.8	0.9	0.8	0.68	0.86
1080	0.84	0.78	0.67	0.82	0.85	0.75	0.63	0.81
1260	0.87	0.81	0.71	0.84	0.81	0.71	0.59	0.77
1440	0.9	0.84	0.74	0.87	0.81	0.72	0.6	0.77
1620	0.9	0.84	0.74	0.88	0.85	0.75	0.63	0.8
1800	0.91	0.85	0.75	0.89	0.87	0.77	0.65	0.82

Table 4. Experimental results of precision and accuracy using ABRN-ZSSC: EMOVOX [1], MTCANet [2] and TTNet [3].

Samples	Precision				Accuracy
	ABRN-ZSSC	EMOVOX [1]	MTCANet [2]	TTNet [3]	ABRN-ZSSC	EMOVOX [1]	MTCANet [2]	TTNet [3]
180	0.95	0.9	0.87	0.93	0.93	0.87	0.84	0.9
360	0.92	0.86	0.76	0.9	0.9	0.8	0.63	0.84
540	0.89	0.83	0.73	0.87	0.85	0.75	0.63	0.8
720	0.85	0.78	0.68	0.82	0.87	0.77	0.65	0.83
900	0.83	0.77	0.67	0.8	0.9	0.8	0.68	0.86
1080	0.84	0.78	0.67	0.82	0.85	0.75	0.63	0.81
1260	0.87	0.81	0.71	0.84	0.81	0.71	0.59	0.77
1440	0.9	0.84	0.74	0.87	0.81	0.72	0.6	0.77
1620	0.9	0.84	0.74	0.88	0.85	0.75	0.63	0.8
1800	0.91	0.85	0.75	0.89	0.87	0.77	0.65	0.82

A comparison of the precision rate for emotional footprint classification across Two-Party Conversation with Emotional Footprint and Emotional Intensity dataset is shown in Figure 4. From the above figure, it is inferred that using the above-mentioned dataset, the precision rate of all methods is relatively high, with the proposed ABRN-ZSSC method showing a certain advantage over the EMOVOX [1] method. Moreover, the proposed ABRN-ZSSC method exhibits a greater advantage compared to the MTCANet [2] method and TTNet [3]. Additionally, compared to the methods [1,2,3], the proposed ABRN-ZSSC method demonstrates robust performance, adapting well to more real-world data environments. The reason is due to the application of the Zero-Shot Learning-based classifier algorithm. By applying this algorithm, the identified emotional footprint at the end of a conversation was subjected to the input of the classifier along with the Speaker, Samples and Emotional Categories via attribute learning. Here, utterance samples belonging to observed classes (known) and emotional states were obtained; next, the subsequent emotional categories were used; and, finally, a mapping function was performed that, in turn, improved the overall precision of the proposed ABRN-ZSSC method by 7% compared to [1] 21% compared in [2] and 3% in [3], respectively.

Figure 5, given above, illustrates the accuracy rate results of the proposed ABRN-ZSSC method, estimated under a different number of samples. The accurate results of the proposed ABRN-ZSSC method analyzed with samples of 180 confirmed an improvement from 87% to 93% under respective increases in the number of samples from 180 to 1800. Similarly, the accurate rate of emotional footprint classifier results of the EMOVOX [1] method with 180 samples confirmed an improvement from 77% to 87%, an improvement from 65% to 84% using [2], and an improvement from 82% to 90% using [3] over an increasing number of samples. This realizable improvement in accuracy rate efficiency enabled by the proposed ABRN-ZSSC method is owing to the utilization of the Attention Bidirectional Recurrent Neural Network-based emotional footprint identification algorithm that sustains the balance between intensification and extensification processes via the Speaker Conversation Updated State. This, in turn, improves the overall accuracy rate efficiency of the proposed ABRN-ZSSC method by 12% in comparison with [1], 33% compared to [2] and 5% compared to [3], respectively.

5.2. Performance Analysis of Training Time

Training time is measured as the amount of time consumed during the process of emotional footprint identification based on the total number of samples ‘$S_i$’. The training time is represented as given below.

$$TT={\sum }_{i=1}^N{S}_i\times Time\left(P_{map\left(U_t\right),t}\right)$$

(15)

From Equation (15), the training time or emotional footprint identification time ‘$TT$’ is measured based on the total number of tagged conversations or number of parties near the conclusion of the conversation ‘$S_i$’ and the time involved in identifying the emotional footprint near the conclusion of the conversation ‘$Time\left(P_{map\left(U_t\right),t}\right)$’. It is measured in terms of seconds (sec). To verify the efficiency of the proposed ABRN-ZSSC method in the emotion footprint classification task, we conducted experimental comparisons between three baseline methods, EMOVOX [1] method, MTCANet [2] and TTNet [3], employing the abovementioned dataset in terms of training time as shown in Table 5.

Table 5. Experimental results of training time using ABRN-ZSSC, EMOVOX [1] and MTCANet [2] and TTNet [3].

Samples	Training Time (s)
	ABRN-ZSSC	EMOVOX [1]	MTCANet [2]	TTNet [3]
180	63	77.4	99	70
360	85	115	135	97
540	105	135	155	122
720	135	165	185	148
900	142	172	192	159
1080	155	185	205	174
1260	185	215	235	197
1440	200	230	250	213
1620	215	245	265	226
1800	245	275	295	258

Table 5. Experimental results of training time using ABRN-ZSSC, EMOVOX [1] and MTCANet [2] and TTNet [3].

Samples	Training Time (s)
	ABRN-ZSSC	EMOVOX [1]	MTCANet [2]	TTNet [3]
180	63	77.4	99	70
360	85	115	135	97
540	105	135	155	122
720	135	165	185	148
900	142	172	192	159
1080	155	185	205	174
1260	185	215	235	197
1440	200	230	250	213
1620	215	245	265	226
1800	245	275	295	258

Figure 6 shows the proposed ABRN-ZSSC evaluation using training time under a varying number of samples during processing. The training time of the proposed ABRN-ZSSC method confirmed an increase from 63 s to 245 s in the number of samples from 180 to 1800. Similarly, the training time of the EMOVOX [1] confirmed an increase from 77.4 s to 275 s for an increasing number of samples. Furthermore, the training time of the MTCANet [2] also confirmed a significant increase from 99 s to 295 s. Furthermore, the training time of the TTNet [3] also confirmed a significant increase from 70 s to 258 s. This indispensable enhancement in the mean training time promoted by the proposed ABRN-ZSSC method is mainly due to the appropriate estimation of state evaluation results via the Global Conversation State and Speaker Conversation State and then by combining using the Attention Bidirectional Recurrent Neural Network during the process of emotional footprint identification. The statistical tests proved the competence and preference of the proposed ABRN-ZSSC method rather than the other considered methods [1,2,3], minimizing the training time by 21% compared to [1], 26% compared to [2] and 9% compared to [3], respectively.

5.3. Performance Analysis of Error Rate or Misclassification Error

Finally, in this section, the misclassification error or error rate is defined as the number of samples incorrectly identified and classified as emotion expressions to the total samples. Error rate refers to the inaccurate detection of emotional footprints (i.e., anger samples detected as disgust and vice versa) and is mathematically formulated as given below.

$$ER={\sum }_{i=1}^N{\displaystyle \frac{S_{IAD}}{S_i}}\times 100$$

(16)

From Equation (16), the error rate or the false identification of emotional footprint at the end of a conversation ‘$ER$’ is measured based on the samples involved in the simulation process ‘$S_i$’ and the samples’ inaccurate detection of emotional footprint ‘$S_{IAD}$’. It is measured in terms of percentage (%). Finally, we conducted experimental comparisons between ABRN-ZSSC and three baseline methods, EMOVOX [1] method, MTCANet [2] and TTNet [3], employing the abovementioned dataset in terms of error rate time, as shown in Table 6.

Table 6. Experimental results of error rate using ABRN-ZSSC, EMOVOX [1] MTCANet [2] and TTNet [3].

Samples	Error Rate (%)
	ABRN-ZSSC	EMOVOX [1]	MTCANet [2]	TTNet [3]
180	6.66	8.33	11.11	7.7
360	7.85	9.55	12.35	8.5
540	8.35	12	14.85	10.35
720	9.25	13.35	15.55	11.25
900	12.15	14.15	15	12.95
1080	11.55	13.25	14.35	12.25
1260	11	12.55	14	11.85
1440	11	12	13.85	11.45
1620	10	11.55	12.55	11
1800	12	13.55	14.35	12.65

Table 6. Experimental results of error rate using ABRN-ZSSC, EMOVOX [1] MTCANet [2] and TTNet [3].

Samples	Error Rate (%)
	ABRN-ZSSC	EMOVOX [1]	MTCANet [2]	TTNet [3]
180	6.66	8.33	11.11	7.7
360	7.85	9.55	12.35	8.5
540	8.35	12	14.85	10.35
720	9.25	13.35	15.55	11.25
900	12.15	14.15	15	12.95
1080	11.55	13.25	14.35	12.25
1260	11	12.55	14	11.85
1440	11	12	13.85	11.45
1620	10	11.55	12.55	11
1800	12	13.55	14.35	12.65

Finally, Figure 7 shows the error rate involved in the emotional footprint classifier with respect to 1800 samples obtained at different time instances. From the above figurative representation, two inferences are made. First, the error rate of all three methods was not found to be either directly or inversely proportional to the samples provided as input. Second, the error rate of the proposed ABRN-ZSSC method was found to be lower in comparison to [1,2,3]. This is evident from the simulation setup with 180 samples, where 165 samples had an anger emotional footprint and 15 samples had a disgust emotional footprint provided as input. The inaccurate detection using the three methods, ABRN-ZSSC [1,2,3], was observed to be 12, 15, 20 and 14, with an overall error rate of 6.66%, 8.33%, 11.11% and 7.7%, respectively. From this result, the error rate of the ABRN-ZSSC method was found to be comparatively less than [1,2,3]. The reason for error rate minimization using the ABRN-ZSSC method can be contributed to the application of the Zero-Shot Learning-based classifier algorithm. By applying this algorithm in addition to the utilization of the identified emotional footprints, semantic features and emotion categories were applied during the classification stage. Identifying high correlation results between features and emotional categories via attribute learning aided in minimizing misclassification error or error rates using the ABRN-ZSSC method by 17% in comparison with [1], 28% in comparison with [2] and 10% in comparison with [3].

5.4. Performance Analysis of Recall

Recall ‘$Rec$’ is defined as the ratio between the numbers of the true-positive rate accurately identifying the emotional footprint as anger to the total number of positive samples. It is formulated as follows

$$Rec=\left[{\displaystyle \frac{TP}{TP+FN}}\right]$$

(17)

To verify the efficiency of the proposed ABRN-ZSSC method in the emotion footprint classification task, we conducted experimental comparisons between three baseline methods, EMOVOX [1] method, MTCANet [2] and TTNet [3], employing the abovementioned dataset in terms of training time, as shown in Table 7.

Table 7. Experimental results of recall using ABRN-ZSSC, EMOVOX [1], MTCANet [2] and TTNet [3].

Samples	Recall
	ABRN-ZSSC	EMOVOX [1]	MTCANet [2]	TTNet [3]
180	0.97	0.95	0.94	0.96
360	0.94	0.88	0.82	0.92
540	0.92	0.86	0.81	0.9
720	0.9	0.84	0.78	0.87
900	0.89	0.82	0.75	0.86
1080	0.87	0.8	0.73	0.84
1260	0.91	0.85	0.72	0.88
1440	0.92	0.86	0.75	0.89
1620	0.93	0.87	0.77	0.9
1800	0.95	0.89	0.79	0.92

Table 7. Experimental results of recall using ABRN-ZSSC, EMOVOX [1], MTCANet [2] and TTNet [3].

Samples	Recall
	ABRN-ZSSC	EMOVOX [1]	MTCANet [2]	TTNet [3]
180	0.97	0.95	0.94	0.96
360	0.94	0.88	0.82	0.92
540	0.92	0.86	0.81	0.9
720	0.9	0.84	0.78	0.87
900	0.89	0.82	0.75	0.86
1080	0.87	0.8	0.73	0.84
1260	0.91	0.85	0.72	0.88
1440	0.92	0.86	0.75	0.89
1620	0.93	0.87	0.77	0.9
1800	0.95	0.89	0.79	0.92

Figure 8 shows the recall rate achieved by the proposed ABRN-ZSSC and existing EMOVOX [1], MTCANet [2] and TTNet [3]. The horizontal axis denotes the number of samples, while the vertical axis represents recall performance. Among the four methods, ABRN-ZSSC exhibits comparatively higher recall performance than EMOVOX [1], MTCANet [2] and TTNet [3]. The reason for the higher recall is to apply the Attention Bidirectional Recurrent Neural Network. To evaluate the overall improvement, the percentages were calculated in relation to the existing methods. Considering 8000 samples for experimentation, the ABRN-ZSSC achieved a recall performance of 0.95, while the existing methods [1], [2] and [3] achieved 0.89, 0.79 and 0.82, respectively. The overall performance results of the ABRN-ZSSC indicate that the recall is improved by 7%, 14% and 3% compared to [1], [2] and [3], respectively.

5.5. Ablation Study

Am ablation study is utilized to eradicate the sections of inputs systematically, elements that are inputs similar to output. The ablation study is the concept of eradicating a certain part of the network to gain a better understanding of the network behavior. The ablation study is carried out in ABRN-ZSSC to find the emotional footprint to enhance the overall performance. To conduct ablation experiments, ABRN-ZSSC is developed in

Table 8. Ablation study for comparison of methods.

Methods	WSNBFSF Dataset
	Precision	Accuracy	Recall	Training Time (ms)	Error Rate (%)
Attention mechanism	0.66	0.72	0.8	175	17.65
Bidirectional RNN	0.71	0.78	0.83	164	15.45
Zero-Shot Learning-based classifier	0.75	0.82	0.85	158	12.3
Proposed ABRN-ZSSC method	0.83	0.87	0.9	145	9.23

.

Table 8 compares the performance of four methods, namely the attention mechanism, Bidirectional RNN, and ZSL components, using Two-Party Conversation with Emotional Footprint and Emotional Intensity dataset. The best results were achieved using the full ABRN-ZSSC method. Contrary to other methods, the proposed ABRN-ZSSC method attained higher accuracy, precision, recall by 0.87, 0.83 and 0.9 and reduced time and error by 145 ms and 9.23%.

5.6. T-Test

A t-test is an arithmetical tool employed in the proposed ABRN-ZSSC method to validate and quantify differences in data related to emotion identification. A t-test is applied in the proposed ABRN-ZSSC method to statistically compare the performance results of a Zero-Shot Learning classifier for emotional footprint identification when using different sets of semantic features (conv_length including inform). Figure 9 shows the t-test outcome for the proposed ABRN-ZSSC method. In this figure, the x-axis denotes the groups of semantic features, and the y-axis indicates the values of the proposed ABRN-ZSSC method. Also, the p-value is 0.84.

6. Conclusions

In this study, a novel deep learning method named ABRN-ZSSC was developed for the identification and classification of emotional footprints for each party near the conclusion of a conversation. First, the training time is reduced by using Bidirectional RNN for consideration of the past and future of a conversation. Also, the emotional footprint is discovered employing a Zero-Shot Learning-based classifier algorithm with higher accuracy and lower error. The proposed ABRN-ZSSC method is assessed on various numbers of samples. The performance of the proposed method is compared to the popular emotional footprint classification methods, such as EMOVOX, MTCANet and TTNet, using different performance metrics. Experimental results prove that the ABRN-ZSSC method efficiently identifies and classifies emotional footprints near the conclusion of the conversation, achieving more stable and acceptable lasting impressions on the listener. From the experimental results, the following summary key finds are achieved: the proposed ABRN-ZSSC method achieved higher precision of 10%, accuracy of 17% and recall of 8% when compared to EMOVOX [1], MTCANet [2] and TTNet [3]. The ABRN-ZSSC method also minimizes the training time by 19% and the error rate by 18% when compared to existing methods. The limitations of the proposed method are given as follows: it failed to select optimal features using optimization, and several metrics, such as specificity and space complexity, were not considered or estimated. In future work, the proposed method will be extended to apply the novel deep learning and optimization method for emotional footprint identification. Also, the specificity and space complexity will be measured to enhance the emotional footprint identification performance.