3D Facial Expression Recognition for Defining Users’ Inner Requirements—An Emotional Design Case Study

Abstract

This study proposes a novel quality function deployment (QFD) design methodology based on customers’ emotions conveyed by facial expressions. The current advances in pattern recognition related to face recognition techniques have fostered the cross-fertilization and pollination between this context and other fields, such as product design and human-computer interaction. In particular, the current technologies for monitoring human emotions have supported the birth of advanced emotional design techniques, whose main focus is to convey users’ emotional feedback into the design of novel products. As quality functional deployment aims at transforming the voice of customers into engineering features of a product, it appears to be an appropriate and promising nest in which to embed users’ emotional feedback with new emotional design methodologies, such as facial expression recognition. This way, the present methodology consists in interviewing the user and acquiring his/her face with a depth camera (allowing three-dimensional (3D) data), clustering the face information into different emotions with a support vector machine classificator, and assigning customers’ needs weights relying on the detected facial expressions. The proposed method has been applied to a case study in the context of agriculture and validated by a consortium. The approach appears sound and capable of collecting the unconscious feedback of the interviewee.

1. Introduction

Donald A. Norman, in his book “Emotional design: Why we love (or hate) everyday things” and in other publications has stated that a design is a success only when the final product is successful in making customer buy it, use it, and enjoy it, and spread the word of the product to others [1,2]. He claimed that designers need to ensure that the design satisfies people’s needs in terms of function, usability, and the ability to deliver emotional satisfaction, pride, and delight. In order to design a product that elicits positive emotions, it is necessary to define and be able to predict the target users’ emotional responses early in the design process.

Emotional Design occurred with the object to promote positive emotions [1] or pleasure in users [3,4] thanks to design properties of products and services [5]. According to Van Gorp and Adams [6], since emotions influence decision making, affect memory and attention, and generate meaning, they can deeply affect the overall user experience. The essential part of the design process is the capability of understanding the user’s feelings and emotions [7], and for this reason, user experience (UX) is recognized as a significant advantage in product design by more designers and companies [8,9,10]. In studies that investigated specific constructs of UX and their interrelations, such as hedonic quality and emotion [11], users’ ratings of these constructs, such as users’ preference, usage context, and product features are often obtained and analysed for UX [10]. In the literature, UX is considered a multifaceted phenomena that involves subjective feelings (change in core affect) [12], behavioural, expressive, and physiological reactions [13,14]. Today a number of different methods, such as quality function deployment (QFD) [15], semantical environment description (SMB or SD) [16], conjoint analysis [17], Kano model (Sauerwein, E., Bailom, F., Matzler, K., & Hinterhuber, H. H., 1996), Kano [18], and Kansei engineering [19,20] exist and are used in practical applications to capture the customer’s considerations and feelings of products and translate these emotional aspects into concrete product design [21,22,23,24,25,26,27,28].

Indeed, in the design of a product or service, it is important to translate and transform the emotional features of users to the product characteristics. Emotional engineering is a recent discipline that has introduced methods for predicting and likely being able to control users’ emotional responses with respect to product attributes in order to be able to design and engineer them [29]. The three main method groups for capturing emotional feedback and using it product classification are questioning methods, physiological measurements, and observing methods. Questioning methods aim to capture the customer’s written or oral emotional feedback in the form of answers to certain questions. Closed questions allow for only a limited number of answers, mostly between five and seven, placed on an increasing or decreasing Likert scale. Closed questions are restricting and require intensive conscious cognitive processing of affective reactions. Open questions allow free, creative answers. Physiological measurements are based on the quasi-continuous gathering of central (electroencephalography) signals or peripheral physiological signals (electromyography, blood volume pulse, electro-dermal activity, etc.) of individuals. Observing methods (or behavioural methods), relies on objective or subjective insights gained from the careful observation of the individuals experiencing the product. Measures of expressive behaviour include the description of facial expressions, bodily movements, gesture, and posture [30], and vocal nonverbal parameters. Pattern recognition of facial expressions can be used for emotion analysis, but it cannot be the only measurement on emotions as there are large amounts of emotional information besides what is elicited on human faces [31]. Therefore, these methods are usually used as complementary tools in, for example, Kansei studies.

The aim of the methodology proposed in this paper is to define 1) how to capture human emotions so that consumer’s emotional requirements can be identified; 2) how to identify the relationships between products and emotional needs; 3) how to improve products in such a way that new products better fit consumers’ emotional needs. The proposed approach has been specifically applied to QFD, which is a quality tool to improve customers’ satisfaction by translating their requirements into appropriate technical measures [32,33]. The method relies on recent three-dimensional (3D) face analysis techniques based on the study of the geometrical shape of the facial surface [34,35].

The suggested methodology has been applied in the design of an e-learning path to be used by farmers who want to experience new technologies within the context of the European Erasmus+ project “Farmer 4.0”.

2. Method

The QFD approach we propose in this work is a four-phase process that specifically introduces a novel method for assigning weights to customers’ needs. The first phase concerns the identification of the customer’s needs with an interview. Phase 2 defines of the weights to be assigned to the customer’s needs through the adoption of the Russell model and emotion recognition. Phase 3 is about the definition of the design requirements, namely services to be offered to respond to customers’ needs. The application of the QFD is in the fourth phase. The method is schematized in Figure 1.

The assignment of specific numerical values was undertaken with matching between the answers given by a potential customer in a tailored interview and the results of a facial expression recognition (FER) procedure on the customer’s face acquired via depth camera during this interview. All the interviews were both audio-recorded and recorded with the depth camera.

The distance sensing for the depth camera is mapped using coded-light technology (or structured-light). The coded-light technology consists of projecting a known pattern on the surface of an object, a face, or any other element and obtaining information on the depth of the surface according to the way in which the pattern is deformed. This technology is designed specifically for applications at close range and the kind of sensor specifically works best within the range 0.2–1.5 m. The projected pattern is of the IR (infrared) type, and is therefore out of the visible spectrum. The depth video format, as well as the frame-rate, is adjustable. Typically, the recommended resolution is the best available (640 × 480) while the frame-rate is 30 FPS (Frame per Second). The output of every frame is a depth map.

The reason for involving emotions through facial expression monitoring and evaluation was to measure the emotional involvement in every topic, i.e., so that weights are assigned according to a quantification of engagement of the user.

In a traditional QFD, the scale to be used to attribute the weights to the needs is usually a Likert scale of points 1 to 5. Differing from the standard approach, the degree of importance of needs was not directly asked to users through a questionnaire but obtained as output of the facial expression recognition algorithm. A model was searched for in the literature for matching weights (numerical values or degrees) to specific emotions, especially in the perspective of user involvement. Russell proposed an emotional model, called the “circumplex model”, which is shown in Figure 2, that can be discretized as it is placed on a Cartesian plane [36,37]. The x-axis quantifies the positivity/negativity of the emotion; the y-axis quantifies the emotional activation and involvement.

Considering the canonical emotional tone of a professional interview regarding a product to be conceptualized, we considered for our model only the “positive quadrants”, the first and the fourth. Weights 1, 2, 3, 4, and 5 have respectively been assigned to emotions “deactivation”, “contentment”, “pleasure”, “excitement”, and “arousal”, according to the circumplex model. We also took into consideration a simplified model with weights 1, 3, and 5 assigned respectively to “deactivation”, “pleasure”, and “arousal”, which could be adopted instead of the complete one, depending on the emotional involvement degree of the scenario, the topic, and the interviewed person. The two models are shown in Figure 3.

During the interview, which is organized with the potential user to define his/her needs, a depth camera should be placed in front of him/her to acquire the face frame-by-frame. The camera is started when the interview starts, together with a vocal recording. Then, the vocal recording is analysed and notes should be taken about the exact range of seconds in which the interviewer’s question is about to finish and the interviewee starts answering. This is supposed to be the very moment in which the micro-expression of the inner emotion (the “ground truth feeling”) is displayed by the face. Thus, these are the moments in which the degree of actual emotional involvement of the user could be measured to define the numerical values of the needs weight in QFD.

Facial data undergo a selection of the significant frames, which are then post-processed so that a depth map remains, framing the face alone. Then, an algorithm is run on the facial depth map to automatically localize 17 landmarks [38], shown in Figure 4, with a thresholding methodology [39] based on differential geometry descriptors [40].

Euclidean distances are calculated between landmarks by adopting the pronasale as an anchor point, as shown in Figure 5. Thus, distances between every landmark and the pronasale have been computed.

These distances are adopted as features to be inputted in a classification methodology based on the support vector machine (SVM). Other classificators could also be adopted, depending on the cardinality of the dataset. The classification is obtained by providing in advance a number of clusters in which to funnel the data. These clusters represent the emotions; thus, they will be three or five in number, depending on the chosen model, namely the simplified or the complete model.

Once the list of needs is defined according to the interview topics and answers, every facial frame to be adopted in the facial expression recognition methodology will be labelled with a specific need, so that the result of the emotion analysis will provide an emotional involvement outcome for that specific customer’s need. If a facial frame, labelled with need X, is in cluster 4, representing excitement, then value 4 will be added in the set of possible weight values of need X. When the final weight of need X is to be assigned to proceed with the QFD, a median is computed among the available values obtained with the clustering. The whole methodology concerning needs is summed-up in Figure 6.

Case Study

The method has been applied to a case study in the agricultural field. The final objective is to design an e-learning path for farmers so that they could embed recent technologies in their daily farming practices. The QFD adoption appeared suitable for understanding the current needs in the field and for correlating needs with services.

Thirteen people coming from the field of interest have been interviewed with an interview form, which has been developed in the European project “Farmer 4.0”. The interview, reported in Appendix A, is approximately 40 min long and is composed of 74 questions. All the interviews were audio-recorded; one interviewed person was recorded with the depth camera Intel RealSense SR300. Thus, the 13 audio recordings supported the conceptualization of the needs list and the 3D video-recording was adopted for assigning weights. The person who was recorded with the depth camera was the most representative of the whole group, as he was used to interviews for agriculture projects, to public speaking, and in general with social events and media. He was evaluated as the most suitable candidate for emotion involvement analysis.

Figure 7 shows the unprocessed depth map given by the camera and a picture taken during the interview.

138 frames and subsequent facial depth maps were selected and post-processed on MATLAB, relying on the identification of the key moments of the interview between every question and every answer. They have been “need labelled” so that the results of the FER method could support the assignment of weights accordingly. Figure 8 shows some processed depth maps in which the face has been framed.

The landmark localization algorithm was then run on the faces (Figure 9) and Euclidean distances were computed.

According to the type of data and the emotions displayed by the interviewee’s face, the complete model with weights 1, 2, 3, 4, and 5 was chosen in this case. Thus, five clusters were adopted. Clustering results allowed us to “cluster label” every facial frame so that the median was computed among the obtained weights for every need. The resulting QFD matrix is shown in Figure 10.

3. Discussion

The obtained final weights underwent a validation, relying on the competencies of the project consortium. The results were examined and evaluated as suitable to every need taken into consideration for this QFD. In the literature, the development of a QFD model that provides technical design elements derived from psychological and behavioural needs was studied in several fields of application [42,43,44,45].

Generally speaking, the information obtained by QFD permits the determining of product improvement directions and sets targets and specifications for the product. The QFD approach decreases time for market and costs as the probability of mid-course corrections and implementation errors is reduced [43,46,47,48,49].

In particular, the advantage of the present methodology compared with other traditional QFD approaches is that the weight assignation does not require any work from the interviewee, as the method determines the weight values relying only on the detected facial expressions. The counterpart is that the proposed technique requires a depth camera, which, despite its low cost, may not be available for the research group. As well as this, it requires some processing of the data and knowledge about clustering techniques. Nonetheless, several software and routines in this sense are already open and available for the community.

The methodology is still preliminary and not ready for all application contexts yet. The next steps of research will strengthen the FER methodology by adding new geometrical features so that the clustering can be more robust. A psychologist will be asked to evaluate the emotions of the costumer during the interview so that a validation of the SVM classification can be provided.

4. Conclusions

A new QFD methodology is proposed in this work, in which needs weights are assigned relying on detected facial expressions. Russell’s circumplex model of emotions is used as theoretical background to discretize the emotion interval and make them correspond to numerical values representing degrees of emotional involvement. During interviews, the face of the user should be acquired frame-by-frame with a depth camera, which allows for the obtaining of 3D models of the face. These data are processed and classified with a support vector machine and weights are assigned according to the result of this classification.

The approach has been applied to a case study of a European project in agriculture in order to understand the needs of agricultural entrepreneurs. The method, validated by the project consortium, appears to accurately weight the inner needs, and the resulting QFD worked appropriately for the project purposes.

This technique is new in the field of product design, as it merges knowledge from the user-centred design field and pattern recognition techniques coming from a computer vision background. Even if the method is still preliminary, this approach could open a vein for designing new products which respond to the inner needs of the customer.