Next Article in Journal
Vacant Parking Slot Detection in the Around View Image Based on Deep Learning
Previous Article in Journal
A Method for Human Facial Image Annotation on Low Power Consumption Autonomous Devices
Article

Tensor-Based Emotional Category Classification via Visual Attention-Based Heterogeneous CNN Feature Fusion

1
Graduate School of Information Science and Technology, Hokkaido University, N-14, W-9, Kita-ku, Sapporo, Hokkaido 060-0814, Japan
2
Office of Institutional Research, Hokkaido University, N-8, W-5, Kita-ku, Sapporo, Hokkaido 060-0808, Japan
3
Faculty of Information Science and Technology, Hokkaido University, N-14, W-9, Kita-ku, Sapporo, Hokkaido 060-0814, Japan
*
Authors to whom correspondence should be addressed.
Sensors 2020, 20(7), 2146; https://doi.org/10.3390/s20072146
Received: 12 March 2020 / Revised: 1 April 2020 / Accepted: 7 April 2020 / Published: 10 April 2020
(This article belongs to the Section Internet of Things)
The paper proposes a method of visual attention-based emotion classification through eye gaze analysis. Concretely, tensor-based emotional category classification via visual attention-based heterogeneous convolutional neural network (CNN) feature fusion is proposed. Based on the relationship between human emotions and changes in visual attention with time, the proposed method performs new gaze-based image representation that is suitable for reflecting the characteristics of the changes in visual attention with time. Furthermore, since emotions evoked in humans are closely related to objects in images, our method uses a CNN model to obtain CNN features that can represent their characteristics. For improving the representation ability to the emotional categories, we extract multiple CNN features from our novel gaze-based image representation and enable their fusion by constructing a novel tensor consisting of these CNN features. Thus, this tensor construction realizes the visual attention-based heterogeneous CNN feature fusion. This is the main contribution of this paper. Finally, by applying logistic tensor regression with general tensor discriminant analysis to the newly constructed tensor, the emotional category classification becomes feasible. Since experimental results show that the proposed method enables the emotional category classification with the F1-measure of approximately 0.6, and about 10% improvement can be realized compared to comparative methods including state-of-the-art methods, the effectiveness of the proposed method is verified. View Full-Text
Keywords: tensor analysis; visual attention; change with time; feature fusion; convolutional neural network tensor analysis; visual attention; change with time; feature fusion; convolutional neural network
Show Figures

Figure 1

MDPI and ACS Style

Moroto, Y.; Maeda, K.; Ogawa, T.; Haseyama, M. Tensor-Based Emotional Category Classification via Visual Attention-Based Heterogeneous CNN Feature Fusion. Sensors 2020, 20, 2146. https://doi.org/10.3390/s20072146

AMA Style

Moroto Y, Maeda K, Ogawa T, Haseyama M. Tensor-Based Emotional Category Classification via Visual Attention-Based Heterogeneous CNN Feature Fusion. Sensors. 2020; 20(7):2146. https://doi.org/10.3390/s20072146

Chicago/Turabian Style

Moroto, Yuya, Keisuke Maeda, Takahiro Ogawa, and Miki Haseyama. 2020. "Tensor-Based Emotional Category Classification via Visual Attention-Based Heterogeneous CNN Feature Fusion" Sensors 20, no. 7: 2146. https://doi.org/10.3390/s20072146

Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop