Trial Analysis of the Relationship between Taste and Biological Information Obtained While Eating Strawberries for Sensory Evaluation

Abstract

This paper presents a trial analysis of the relationship between taste and biological information obtained while eating strawberries (for a sensory evaluation). This study used the visual analog scale (VAS); we collected questionnaires used in previous studies and human brain activity obtained while eating strawberries. In our analysis, we assumed that brain activity is highly correlated with taste. Then, the relationships between brain activity and other data, such as VAS and questionnaires, could be analyzed through a canonical correlation analysis, which is a multivariate analysis. Through an analysis of brain activity, the potential relationship with "taste" (that is not revealed by the initial simple correlation analysis) can be discovered. This is the main contribution of this study. In the experiments, we discovered the potential relationship between cultural factors (in the questionnaires) and taste. We also found a strong relationship between taste and individual information. In particular, the analysis of cross-loading between brain activity and individual information suggests that acidity and the sugar-to-acid ratio are related to taste.

1. Introduction

Strawberries were first bred on Dutch farms in the 18th century and quickly spread worldwide. Currently, over nine million tons of strawberries are grown globally every year [1]. Since strawberries are used as fresh fruits and processed foods, various cultivars and methods of cultivation have been developed [2]. Thus, various studies have been conducted. For example, researchers have studied methods to improve cultivars [3], as well as cultivate [4], preserve [5,6], and process them [7,8].

Recently, studies on taste have received significant attention [1,9,10]. Ikegaya et al. [11] clarified the effects of sugar and organic acids (the main components in strawberry fruit juice) on the taste of strawberries. It has also been reported that the palatability of fruit is influenced by several factors, such as size, color, flavor, and texture [12,13,14]. Using multiple types of data is important for sensory evaluations. However, collecting the biological data of humans (related to texture and flavor) is difficult due to heavy burdens. To the best of our knowledge, no research has been conducted on the sensory evaluation of strawberries using heterogeneous data, including sugar and organic acid content, taste, the sizes of strawberries, and human perceptions.

Recent advances in the Internet of Things technologies have remarkably "transcended" traditional environmental sensing [15,16]. Various sensors can easily be used; we can also obtain biological data from humans. Thus, many studies using sensors have been conducted. In studies using wearable sensors, healthcare monitoring systems using vital data, such as blood pressure, heart rate, weight, and blood glucose, have been proposed [17,18]. Additionally, several studies estimated the interests of content involving human behavior [19,20] and personalized saliency (and its prediction) using gaze data [21,22,23]. Furthermore, the analysis of behavior, gaze data, and brain activity contribute to the solutions to several tasks, such as brain decoding [24,25,26,27] and certain applications [28,29,30,31,32]. Onuma et al. [31,32] showed that human brain activity calculated from functional near-infrared spectroscopy (fNIRS) is related to the taste of food [31]. fNIRS is a useful and non-invasive technique for brain imaging that measures blood flow in neural regions near the cortical surface. Minematsu et al. also showed that pleasant and unpleasant tasting foods tended to elicit decreased and increased oxyhemoglobin levels by monitoring changes in oxygenated hemoglobin levels in the anterior prefrontal cortex in response to the intake of hedonically different edibles in healthy adults [32]. From the above studies, it is expected that the taste of strawberries can be measured by brain activity through fNIRS.

It is possible to analyze the taste of strawberries from a new perspective, i.e., we hypothesize that there is a relationship between brain activity and the taste of strawberries. To illustrate this advantage, Figure 1 shows an overview of the differences between the analyses conducted in previous studies and the analysis that this paper focuses on. As shown by the black arrow in Figure 1, previous studies have used taste, questionnaires (e.g., reward and culture), and individual information (e.g., size and sugar content). In the analysis of the relationship between taste and questionnaires, participants filled out various questionnaires [33]. In these analyses, it is assumed that there is a correlation between the items in the questionnaire and taste. However, if the participants cannot find this correlation, it is difficult to grasp the relationship with the taste. Additionally, since individual data of strawberries consist of numerical data, such as sugar content, it is difficult to find a quantitative relationship with the visual analog scale (VAS) assigned by human senses. Based on the hypothesis represented by the blue arrow with the broken line, we analyzed the relationships between brain activity and questionnaires, and between brain activity and individual information. We expect to extract new questionnaires that have a high correlation with taste. In the same way, clarification of the existence of potential relationships between taste and individual numerical data can be expected. These relationships correlate with the blue arrow in Figure 1. Therefore, this study clarifies the potential relationships between tastes, questionnaires, and individual information using brain activity.

In this study, we conducted a trial analysis of relationships between heterogeneous data related to strawberries based on a multivariate analysis. For the preparation of the analysis, we collected four kinds of data: VAS, questionnaires, brain activity, and individual information. Several participants ate strawberries and evaluated their taste using the VAS. We measured fNIRS data from participants; participants completed questionnaires for reward and cultural factors. Individual information consisted of information about each strawberry, such as size and sugar content. Then, we performed two kinds of analyses on the relationships between the data. Previous studies focused on the relationship between taste and other data; however, in this study, we first conducted a simple correlation analysis as the first analysis, i.e., a traditional approach. In the first analysis, we analyzed the correlation between taste and three other kinds of data. Then, we identified the data that were highly correlated with taste. This analysis was similar to previous analyses conducted by researchers since it directly focused on taste and other data. The second analysis was the canonical correlation analysis (CCA) [34]; this was conducted to find the potential relationships that could not be revealed by the first simple correlation analysis. It is a method that projects different kinds of features into a common latent space so that their correlations can be maximized. Specifically, we verified the canonical correlation between brain activity and other data (taste, questionnaires, and individual information). Through the analysis of “brain activity and taste”, we clarified that they were related. Then, in the analyses of “brain activity and questionnaire data” and “brain activity and individual information”, we identified the data that were highly correlated with brain activities by calculating the cross-loading between the data. In other words, the identified data could be regarded as potentially related to the taste. Consequently, we obtained new knowledge about the analysis of the taste of strawberries via questionnaires that were potentially highly correlated with taste; we found a strong relationship between taste and individual information. Therefore, discovering the potential relationships that could not be revealed by the first simple correlation analysis is the main contribution of this study.

2. Procedure of Data Acquisition and Constructed Dataset

In this section, we explain the procedure of data acquisition and details of the constructed dataset.

2.1. Procedure of Data Acquisition

Nineteen participants (fourteen men and five women) participated in this experiment. The average age was 24.7 years. The minimum age of the participants was 21 years old and the maximum age was 55 years old. All participants were healthy and did not report any olfactory or gustatory disorders. Verbal and written explanations about the experiment were given to the participants and written informed consent was obtained. These experiments received approval from the Ethics Committee of Hokkaido University, Japan. Participants were seated in comfortable chairs in a room at a constant temperature (23 °C). Participants were given verbal and written instructions for the experimental procedure and fitted with the fNIRS headset. In this experiment, there were sequential blocks of rest periods and task periods, as shown in Figure 2.

During the rest periods (300 s), the participants sat on their chairs; they had their eyes opened, and were relaxed. Then, the participants watched one strawberry and its name (15 s); there was a rest period (45 s). After that, the participants took the strawberry into their mouths, bit into it, and swallowed it immediately; there was also a rest period (80 s). Finally, the participants began the evaluation of the VAS and the questionnaires, as explained in the “VAS obtained through user evaluation”. After the evaluation, participants rinsed their mouths with a cup of pure water and rested. This procedure was constructed based on previous studies [33,35]. Similar to the previous study [36], the mouthwash was performed with pure water at a constant temperature. In this study, we calculated fNIRS features, as explained in “fNIRS data”, when the participants bit and swallowed the strawberries. The above task was performed for each strawberry; thus, the participants repeated the task four times. Note that the flow of the experiments was explained to the participants. Then, in the actual experiment, the staff provided strawberries and questionnaires to the participants, and the participants performed the task. Thus, the experimental staff directly controlled the instructions. During the fNIRS measurement, the participants were filmed by a video camera, as shown in Figure 2; the fNIRS signal synchronized with the time of eating strawberries. The number of strawberries was not large, but we compensated for the small number of trials by attracting about 20 participants.

Figure 2. Schematic diagram of the rest and task sequences in the experiment. The signal of the temporal brain area was recorded by fNIRS during the biting and swallowing of the strawberry. After recording, evaluations of the VAS and some questionnaires were conducted. These tasks were repeated for four strawberries per participant. Note that a five-minute gap between tasks was the same condition as in the literature [37].

Figure 2. Schematic diagram of the rest and task sequences in the experiment. The signal of the temporal brain area was recorded by fNIRS during the biting and swallowing of the strawberry. After recording, evaluations of the VAS and some questionnaires were conducted. These tasks were repeated for four strawberries per participant. Note that a five-minute gap between tasks was the same condition as in the literature [37].

2.2. Dataset

This subsection explains the dataset used in the correlation analysis of heterogeneous data. We collected four kinds of data: VAS, questionnaires, brain activity, and individual information. We measured the taste of the strawberries using VAS (Section 2.2.1), and some factors related to strawberries were obtained through questionnaires (Section 2.2.2). We obtained fNIRS data from participants as brain activity (Section 2.2.3) and individual information from each strawberry (Section 2.2.4).

2.2.1. VAS Obtained via User Evaluation

In this experiment, participants ate four strawberries and filled out two types of questionnaires about taste and other factors. Figure 3 shows examples of the strawberries eaten by the participants. In this experiment, we used two kinds of strawberries (Skyberry and Tochiotome). Skyberry and Tochiotome originated from the Tochigi prefecture, which is the most famous prefecture in Japan for strawberry cultivation. Tochiotome is a strawberry that was registered in 1996. Meanwhile, Skyberry is a new strawberry that was registered in 2014 as a successor to Tochiotome. Additionally, we adopted large-sized and small-sized strawberries to analyze the taste differences, depending on the size of the strawberry. Thus, we used four types of strawberries.

In the first questionnaire, participants evaluated the taste of the strawberries using the VAS, which is one of the most commonly used scales. They rated taste using the VAS for each strawberry. The VAS measures the intensity of pain or taste; it has been widely used in previous studies [38,39,40]. Since it is also used in the evaluation of the taste of strawberries [41,42], this index was adopted. The length of the VAS in this experiment was 100 mm. The evaluated results are presented in

Table 1. The VAS, which is a score in millimeters of each strawberry evaluated by each participant. “Ave” and “Std” mean the average and standard deviations of the VAS, respectively.

ID	Skyberry		Tochiotome
	(Large)	(Small)	(Large)	(Small)
1	88	77	93	89
2	84	10	63	56
3	79	59	89	81
4	75	78	88	74
5	73	69	76	77
6	83	76	6	6
7	70	70	82	80
8	74	86	61	56
9	80	81	74	68
10	52	39	64	71
11	88	69	96	96
12	50	70	74	74
13	41	18	29	11
14	58	55	52	48
15	71	60	83	53
16	83	73	91	80
17	57	53	68	47
18	76	55	48	42
19	48	64	67	65
Ave	70	61	69	62
Std	15	20	23	24

.

2.2.2. Questionnaires Obtained via User Evaluation

Participants rated four factors related to the strawberries. Specifically, five items from each of the three factors (reward, cultural, and information factors) were selected based on the literature [33]. Additionally, a factor related to the appearance of strawberries was newly adopted. Each factor and its contents are described as follows.

1.

Reward factor

(1-1)

The degree of desire caused by the addictiveness of the food;

(1-2)

The degree of level of difficulty in inhibiting urges to eat;

(1-3)

The degree of level of difficulty in inhibiting eating the food;

(1-4)

The degree of sense of satiety recognized by eating the food;

(1-5)

The degree of sense of rewarding ingredients perceived by eating the food.

2.

Cultural factor

(2-1)

The degree of repeated exposure to the food;

(2-2)

The degree of dietary accustomedness to the food;

(2-3)

The degree of similarity with an accustomed food;

(2-4)

The degree of embeddedness of the food as a home-cooked taste;

(2-5)

The degree of entrenched preference for a certain food.

3.

Information factor

(3-1)

The degree of visual information of the food;

(3-2)

The degree of publicity of the food;

(3-3)

The degree of health information of the food;

(3-4)

The degree of perceived safeness of the food;

(3-5)

The degree of perceived value for the price of the food.

4.

Appearance factor

(4-1)

The degree of the size of the food;

(4-2)

The degree of brilliance of the food;

(4-3)

The degree of gloss of the food;

(4-4)

The degree of scent of the food.

The above 19 contents were evaluated via 19 questionnaires, as shown in

Table 2. The 19 questionnaires for the 4 factors.

Q1	Reward factor
Q1-1	Does it taste like a habit?
Q1-2	Does it taste like something you would want to pick up?
Q1-3	Does one bite make you want to take another bite?
Q1-4	Are you satisfied with the taste?
Q1-5	Do you think that the food tastes good
	because of its fatty sweetness and delicious taste?
Q2	Cultural factor
Q2-1	Does it taste familiar to you?
Q2-2	Have you ever eaten food that tasted the same or similar to you?
Q2-3	Have you eaten food like this many times?
Q2-4	Do you think your family will like the taste of this food?
Q2-5	Did you like the taste since you were a child?
Q3	Information factor
Q3-1	Does the food look good to you?
Q3-2	Have you seen this food in advertisements
	or heard about it by word of mouth?
Q3-3	Have you ever heard of the health benefits?
Q3-4	Do you feel comfortable with the ingredients?
Q3-5	Do you think it looks expensive?
Q4	Appearance factor
Q4-1	Is the size of this food desirable?
Q4-2	Does it look colorful and delicious?
Q4-3	Does it look shiny and delicious?
Q4-4	Do you think it smells good?

. Participants evaluated them at five levels (5 (strong agreement), 4 (agreement), 3 (neutral), 2 (disagree), and 1 (strongly disagree)).

2.2.3. fNIRS Data

Light in the near-infrared region (620 to 1000 nm) of the electromagnetic spectrum penetrates most living tissues, such as bone and skin. However, hemoglobin, which is an oxygen carrier in the blood, absorbs near-infrared light. Taking advantage of this effect, fNIRS measures cortical activity by irradiating the head with near-infrared light from a light source and measuring the intensity of the light transmitted through the head with a detector. In this experiment, the temporal lobe involved in the perception of visual and gustatory stimuli was measured using LIGHTNIRS (Shimadzu Corporation Japan (http://www.shimadzu.com/, accessed on 1 November 2022)), to measure taste induced by both visual and gustatory stimuli. The measurement method was a three-wavelength absorbance calculation, which measures changes in Oxy-Hb, Deoxy-Hb, and Total-Hb. The laser wavelengths of LIGHTNIRS were 780, 805, and 830 nm. Thus, when recording fNIRS signals, we used LIGHTNIRS with 20 channels on the temporal lobe, as shown in Figure 4. We set up eight emitters and eight detectors of a near-infrared ray alternately to cover the regions in the shape of a 2 × 4 rectangle. We recorded fNIRS signals from participants and calculated five-dimensional features from each hemoglobin concentration. Specifically, we measured the average, standard deviation, root-mean-square, maximum and minimum values of the oxygenated hemoglobin, and deoxygenated hemoglobin concentrations. The time interval differs for each participant since the brain activity during the whole time of eating strawberries is targeted. In this paper, the above five types of features were calculated from each channel from the signal during the time period to avoid the influence of the time difference, and these features were calculated from each channel. Consequently, we calculated fNIRS features $\mathit{f}\in {\mathbb{R}}^{200}$ as five-dimensional features × 20 channels × 2 (oxygenated/deoxygenated). The time interval was different for each participant because the brain activity was targeted during the entire time period of eating strawberries. In this paper, five types of features were calculated for each channel from the signals during the entire time period in order to reduce the influences of differences among participants.

2.2.4. Individual Information

The individual information of strawberries was collected and converted into features. In this experiment, we collected four types of individual information: sugar content, acidity, sugar-to-acid ratio, and size (large/small).

Table 3. Sugar content, acidity, and sugar-to-acid ratio of the strawberries.

ID	Sugar Content(%)				Acidity (%)				Sugar to Acid Ratio
	Skyberry		Tochiotome		Skyberry		Tochiotome		Skyberry		Tochiotome
	Large	Small	Large	Small	Large	Small	Large	Small	Large	Small	Large	Small
1	8.3	9.7	14.7	10.5	0.76	0.71	0.72	0.61	10.9	13.7	20.4	17.2
2	7.7	11.1	9.0	9.5	0.76	0.57	0.60	0.61	10.1	19.5	15.0	15.6
3	10.0	9.0	8.8	11.5	0.98	0.58	0.66	0.60	10.2	15.5	13.3	19.2
4	10.2	10.8	12.7	12.0	0.66	0.62	0.77	0.63	15.5	17.4	16.5	19.0
5	8.9	11.7	9.7	9.6	0.59	0.56	0.65	0.71	15.1	20.9	14.9	13.5
6	9.9	10.8	13.4	10.8	0.64	0.70	0.68	0.70	15.5	15.4	19.7	15.4
7	9.7	10.3	10.8	11.3	0.58	0.64	0.67	0.62	16.7	16.1	16.1	18.2
8	9.2	9.3	9.4	8.4	0.54	0.63	0.52	0.82	17.0	14.8	18.1	10.2
9	9.1	9.0	9.9	9.1	0.58	0.58	0.72	0.58	15.7	15.5	13.8	15.7
10	8.7	9.2	11.6	10.6	0.59	0.57	0.82	0.58	14.7	16.1	14.1	18.3
11	10.1	9.5	9.7	8.6	0.58	0.54	0.48	0.52	17.4	17.6	20.2	16.5
12	9.3	10.0	10.8	10.2	0.50	0.59	0.72	0.55	18.6	16.9	15.0	18.5
13	8.5	9.2	10.4	9.4	0.57	0.74	0.64	0.59	14.9	12.4	16.3	15.9
14	10.2	9.7	11.8	11.7	0.51	0.50	0.66	0.54	20.0	19.4	17.9	21.7
15	8.5	10.4	10.0	8.4	0.57	0.51	0.58	0.88	14.9	20.4	17.2	9.5
16	10.4	7.7	11.4	10.4	0.78	0.90	0.78	0.69	13.3	8.6	14.6	15.1
17	9.3	10.1	10.8	8.3	0.56	0.70	0.72	0.63	16.6	14.4	15.0	13.2
18	9.1	7.0	10.4	9.4	0.68	0.47	0.67	0.60	13.4	14.9	15.5	15.7
19	9.1	8.7	9.5	10.9	0.75	0.60	0.63	0.63	12.1	14.5	15.1	17.3
Ave	9.27	9.64	10.8	10.0	0.64	0.62	0.67	0.64	14.9	16.0	16.2	16.1
Std	0.72	1.11	1.50	1.15	0.12	0.10	0.08	0.09	2.63	2.85	2.08	2.93

presents the obtained individual information. This table shows that characteristics, such as the sugar content, differ depending on the type and size. Note that the strawberries used for evaluation by the participants were the same strawberries used to measure individual information.

3. Correlation Analysis between Heterogeneous Data

In this study, we conducted two analyses. The first analysis investigated the correlation between “taste” and “questionnaire, brain activity, and individual information” and identified the data that were highly correlated with taste. This analysis clarifies the direct relationships between the taste and other data through a simple correlation analysis. In the second analysis, we used CCA to analyze the potential correlation between “brain activity and taste”, “brain activity and questionnaire data”, and “brain activity and individual information”. Then we calculated the canonical correlation between the heterogeneous data and the cross-loading between them to identify the factors that maximize their correlations. The second analysis could discover questionnaires that were potentially highly correlated with taste; a strong relationship between taste and individual information was found.

3.1. Analysis of Correlation between Taste and Other Data

To identify the data that were highly correlated with taste, we calculated a correlation between the VAS representing taste and other data (questionnaires, brain activity, and individual information, such as sugar content and acidity). Before calculating the correlation, we extracted features from these data. First, from Table 1, using the VAS value $v_{p,s}$ of the strawberry $s\in \{s_1:\mathrm{Skyberry}\left(\mathrm{large}\right),s_2:\mathrm{Skyberry}\left(\mathrm{small}\right),s_3:\mathrm{Tochiotome}\left(\mathrm{large}\right)\mathrm{and}{s}_4:\mathrm{Tochiotome}\left(\mathrm{small}\right)\}$ evaluated by participant $p(=1,2,...,19)$, we obtained features ${\mathit{v}}_s={[v_{1,s},v_{2,s},\dots v_{19,s}]}^{\top }\in {\mathbb{R}}^{19}$. Then we aligned the features ${\mathit{v}}_s$ and obtained the VAS features $\mathit{v}={[{\mathit{v}}_{s_1}^{\top },\dots ,{\mathit{v}}_{s_4}^{\top }]}^{\top }\in {\mathbb{R}}^{76}$.

Second, we obtained questionnaire features from questionnaires, as shown in Table 2. Specifically, using the answer value $q_{p,s}^r$ for questionnaire item $r(\in \{1,2,...,R;R$ being the number of questionnaires}) of the strawberry s evaluated by the participant p, we obtained features ${\mathit{q}}_s^r={[q_{1,s}^r,q_{2,s}^r,...,q_{19,s}^r]}^{\top }\in {\mathbb{R}}^{19}$. To calculate the correlation between each item and VAS in the first analysis, we aligned the features ${\mathit{q}}_s^r$ and obtained questionnaire features ${\mathit{q}}^r={[{{\mathit{q}}_{s_1}^r}^{\top },\dots ,{{\mathit{q}}_{s_4}^r}^{\top }]}^{\top }\in {\mathbb{R}}^{76}$.

Third, as explained in the “fNIRS data”, we acquired fNIRS data from the participant p while eating the strawberry s and calculated 200-dimensional features; the elements were various statistics, such as average and standard deviation. In this analysis, we applied the principal component analysis (PCA) [43], one of the simplest dimensionality reduction methods to remove various noises included in the fNIRS data. We adopted PCA as feature preprocessing instead of data preprocessing. Consequently, low-dimensional features ${\tilde{\mathit{f}}}_{p,s}\in {\mathbb{R}}^d$ were calculated. Note that we set the dimension d after the PCA so that the cumulative contribution rate was larger than $90\%$. In this analysis, the dimension d was set to 9. Dimensionality reduction of fNIRS features is commonly used in studies using machine learning. For example, in previous studies, i.e., [44,45,46], a local Fisher discriminant analysis [47]-based approach was adopted to perform supervised dimensionality reduction on fNIRS data. However, in [30], dimensionality reductions of fNIRS data were conducted based on PCA. Since this experiment investigated whether there was a correlation between each dimension for each feature, we calculated fNIRS features ${\mathit{f}}^d={[{\tilde{f}}_{1,s_1}^d,{\tilde{f}}_{2,s_1}^d,\dots ,{\tilde{f}}_{19,s_1}^d,{\tilde{f}}_{1,s_2}^d,\dots ,{\tilde{f}}_{19,s_4}^d]}^{\top }\in {\mathbb{R}}^{76}(d=1,2,...,9)$.

Finally, by employing $i_{p,s}^u$ for the individual information $u(\in \{$sugar content, acidity, sugar-to-acid ratio, and size$\left\}\right)$ of the strawberry s eaten by the participant p, we obtained features ${\mathit{i}}_s^u={[i_{1,s}^u,i_{2,s}^u,...,i_{19,s}^u]}^{\top }\in {\mathbb{R}}^{19}$. Then, we aligned the features ${\mathit{i}}_s^u$ and obtained individual features ${\mathit{i}}^u={[{{\mathit{i}}_{s_1}^u}^{\top },\dots ,{{\mathit{i}}_{s_4}^u}^{\top }]}^{\top }\in {\mathbb{R}}^{76}$. Note that when u = “size”, we set $i_{p,s}^u$ to one if the size of the strawberry was large; otherwise, $i_{p,s}^u$ was set to zero.

In this analysis, we calculated Pearson’s correlation coefficient, defined as follows:

$$\begin{array}{cc}\hfill {\eta }^{ab}=& Corr(\mathit{a},\mathit{b})\\ \hfill =& \frac{{\sum }_{j=1}^{76}(a_j-\overline{a})(b_j-\overline{b})}{\sqrt{{\sum }_{j=1}^{76}{(a_j-\overline{a})}^2}\sqrt{{\sum }_{j=1}^{76}{(b_j-\overline{b})}^2}},\end{array}$$

(1)

where $\mathit{a}$ and $\mathit{b}$ are features; $\overline{a}$ means the average of $\mathit{a}$. In this analysis, since the correlation between taste and other data can be calculated, we set $\mathit{a}$ to $\mathit{v}$. Additionally, $\mathit{b}$ was set to ${\mathit{q}}^r$, ${\mathit{f}}^d$, or ${\mathit{i}}^u$. Thus, we can identify data that were highly correlated with taste by searching ${\eta }^{v{q}^r}$, ${\eta }^{v{f}^d}$, and ${\eta }^{v{i}^u}$ in this analysis.

3.2. Analysis of Potential Correlation between Heterogeneous Data through CCA

In this analysis, we searched for the potential correlation between taste and brain activity based on CCA, which projects different types of features into the common latent space. In CCA, given two kinds of feature matrices $\mathit{A}=[{\mathit{a}}^1,{\mathit{a}}^2,...,{\mathit{a}}^{D_a}]\in {\mathbb{R}}^{76\times D_a}$ and $\mathit{B}=[{\mathit{b}}^1,{\mathit{b}}^2,...,{\mathit{b}}^{D_b}]\in {\mathbb{R}}^{76\times D_b}$, we estimated the projection vectors ${\mathbf{\varphi }}_a$ and ${\mathbf{\varphi }}_b$ that satisfy the following correlation:

$$\begin{array}{c}\hfill arg\underset{{\mathbf{\varphi }}_a,{\mathbf{\varphi }}_b}{max}\frac{{\mathbf{\varphi }}_a^{\top }{\mathit{C}}_{AB}{\mathbf{\varphi }}_b}{\sqrt{{\mathbf{\varphi }}_a^{\top }{\mathit{C}}_{AA}{\mathbf{\varphi }}_a}\sqrt{{\mathbf{\varphi }}_b^{\top }{\mathit{C}}_{BB}{\mathbf{\varphi }}_b}},\end{array}$$

(2)

where

$$\begin{array}{c}\hfill {\mathit{C}}_{AB}={\mathit{A}}^{\top }\mathit{B},{\mathit{C}}_{AA}={\mathit{A}}^{\top }\mathit{A},{\mathit{C}}_{BB}={\mathit{B}}^{\top }\mathit{B}.\end{array}$$

(3)

To solve Equation (2), we solved the following Lagrange problem:

$$\begin{array}{c}\hfill \mathcal{L}({\mathbf{\varphi }}_a,{\mathbf{\varphi }}_b)={\mathbf{\varphi }}_a^{\top }{C}_{AB}{\mathbf{\varphi }}_b-\frac{{\lambda }_a}{2}({\mathbf{\varphi }}_a^{\top }{C}_{AA}{\mathbf{\varphi }}_a-1)-\frac{{\lambda }_b}{2}({\mathbf{\varphi }}_b^{\top }{C}_{BB}{\mathbf{\varphi }}_b-1),\end{array}$$

(4)

where ${\lambda }_a={\lambda }_b(=\lambda )$, and $\lambda$ is defined as follows. Then, we solved the following eigenvalue problems:

$$\begin{array}{c}\hfill C_{AA}^{-1}{C}_{AB}{C}_{BB}^{-1}{C}_{AB}^{\top }{\mathbf{\varphi }}_a={\lambda }^2{\mathbf{\varphi }}_a,C_{BB}^{-1}{C}_{AB}^{\top }{C}_{AA}^{-1}{C}_{AB}{\mathbf{\varphi }}_b={\lambda }^2{\mathbf{\varphi }}_b,\end{array}$$

(5)

where $\lambda$ correlates to the eigenvalue of this problem. Since we can obtain multiple eigenvalues $\lambda$ and their correlating eigenvectors ${\mathbf{\varphi }}_a$ and ${\mathbf{\varphi }}_b$ as solutions to the above problem, we obtain the projection matrices as ${\Phi }_a=[{\mathbf{\varphi }}_a^1,{\mathbf{\varphi }}_a^2,...,{\mathbf{\varphi }}_a^{D_{cca}}]\in {\mathbb{R}}^{D_a\times D_{cca}}$ and ${\Phi }_b=[{\mathbf{\varphi }}_b^1,{\mathbf{\varphi }}_b^2,...,{\mathbf{\varphi }}_b^{D_{cca}}]\in {\mathbb{R}}^{D_b\times D_{cca}}$ by aligning $D_{cca}$ eigenvectors, respectively. We can obtain the canonical features $\widehat{\mathit{A}}=[{\widehat{\mathit{a}}}^1,{\widehat{\mathit{a}}}^2,\cdots ,{\widehat{\mathit{a}}}^{D_{cca}}]\in {\mathbb{R}}^{76\times D_{cca}}$ and $\widehat{\mathit{B}}=[{\widehat{\mathit{b}}}^1,{\widehat{\mathit{b}}}^2,\cdots ,{\widehat{\mathit{b}}}^{D_{cca}}]\in {\mathbb{R}}^{76\times D_{cca}}$, which can consider their relationships as follows:

$$\begin{array}{c}\hfill \widehat{\mathit{A}}=\mathit{A}{\Phi }_a,\widehat{\mathit{B}}=\mathit{B}{\Phi }_b.\end{array}$$

(6)

In the second analysis, to reveal the relationships between brain activity and other data, we applied CCA to the correlating features. First, we reveal the existence of the relationship between taste and brain activity through the analysis between them. We used VAS features $\mathit{V}={[{\mathit{v}}_{s_1}^{\top },\dots ,{\mathit{v}}_{s_4}^{\top }]}^{\top }\in {\mathbb{R}}^{76\times 1}$ and a brain feature matrix, as $\mathit{F}=[{\mathit{f}}^1,{\mathit{f}}^2,...,{\mathit{f}}^9]\in {\mathbb{R}}^{76\times 9}$. Given $\mathit{V}$ and $\mathit{F}$, we estimate the projection matrices ${\Phi }_v$ and ${\Phi }_{f_v}$. Consequently, we obtain canonical features $\widehat{\mathit{V}}=[{\widehat{\mathit{v}}}^1,{\widehat{\mathit{v}}}^2,\cdots ,{\widehat{\mathit{v}}}^{D_{cca}^{vf}}]\in {\mathbb{R}}^{D_{cca}^{vf}}$ and $\widehat{{\mathit{F}}_v}=[{\widehat{\mathit{f}}}_v^1,{\widehat{\mathit{f}}}_v^2,\cdots ,{\widehat{\mathit{f}}}_v^{D_{cca}^{vf}}]$, which consider their relationships as follows:

$$\begin{array}{ccc}\hfill \widehat{\mathit{V}}=& \mathit{V}{\Phi }_v,{\widehat{\mathit{F}}}_v=& \hfill {\mathit{F}}_v{\Phi }_{f_v}.\hfill \end{array}$$

(7)

Note that the dimension of the canonical features is limited to the minimum dimensions among original features. Since dimensions of the original VAS and fNIRS features are set to 1 and 9, respectively, $D_{cca}^{vf}$ is set to 1. Finally, we measure the canonical correlation ${\eta }_{\mathrm{cca}}^{{\widehat{\mathit{v}}}^1{\widehat{\mathit{f}}}_v^1}=Corr({\widehat{\mathit{v}}}^1,{\widehat{\mathit{f}}}_v^1)$ to verify the existence of canonical correlation between taste and brain activity.

Similar to the above motivation, we revealed the existence of relationships between “brain activity and questionnaire data” and “brain activity and individual information” using CCA. We calculated the cross-loading between them and identified the data that were highly correlated with brain activity. Thus, we discovered questionnaires that were potentially highly correlated with taste and found a strong relationship between taste and individual information. In this analysis, we calculated the canonical correlation and identified the data to maximize the canonical correlations by calculating the cross-loading between the information. Before the detailed approach for calculating the cross-loading, we defined the feature matrices of CCA. We used a questionnaire feature matrix, as $\mathit{Q}=[{\mathit{q}}^1,{\mathit{q}}^2,...,{\mathit{q}}^R]\in {\mathbb{R}}^{76\times R}$, and a feature matrix $\mathit{I}=[{\mathit{i}}^1,{\mathit{i}}^2,...,{\mathit{i}}^4]\in {\mathbb{R}}^{76\times 4}$ for individual information. Similar to the above analysis of taste and brain activity, we obtain the canonical features $\widehat{\mathit{Q}}=[{\widehat{\mathit{q}}}^1,{\widehat{\mathit{q}}}^2,\cdots ,{\widehat{\mathit{q}}}^{D_{cca}^{qf}}]\in {\mathbb{R}}^{76\times D_{cca}^{qf}}$ and $\widehat{{\mathit{F}}_q}=[{\widehat{\mathit{f}}}_q^1,{\widehat{\mathit{f}}}_q^2,\cdots ,{\widehat{\mathit{f}}}_q^{D_{cca}^{qf}}]\in {\mathbb{R}}^{76\times D_{cca}^{qf}}$, which consider their relationships as follows:

$$\begin{array}{ccc}\hfill \widehat{\mathit{Q}}=& \mathit{Q}{\Phi }_q,{\widehat{\mathit{F}}}_q=& \hfill {\mathit{F}}_q{\Phi }_{f_q}.\hfill \end{array}$$

(8)

In this analysis, we measured the canonical correlation $Corr({\widehat{\mathit{q}}}^1,{\widehat{\mathit{f}}}_q^1)$ for the first canonical feature and identified the questionnaire r that was highly correlated with the brain activity based on the cross-loading $Corr({\mathit{q}}^r,{\widehat{\mathit{f}}}_q^1)$. Furthermore, following the same procedure, we obtained the canonical features $\widehat{\mathit{I}}=[{\widehat{\mathit{i}}}^1,{\widehat{\mathit{i}}}^2,\cdots ,{\widehat{\mathit{i}}}^{D_{cca}^{if}}]\in {\mathbb{R}}^{76\times D_{cca}^{if}}$ and $\widehat{{\mathit{F}}_i}=[{\widehat{\mathit{f}}}_i^1,{\widehat{\mathit{f}}}_i^2,\cdots ,{\widehat{\mathit{f}}}_i^{D_{cca}^{if}}]\in {\mathbb{R}}^{76\times D_{cca}^{if}}$, which considered their relationships as follows:

$$\begin{array}{ccc}\hfill \widehat{\mathit{I}}=& \mathit{I}{\mathbf{\Phi }}_i,{\widehat{\mathit{F}}}_i=& \hfill {\mathit{F}}_i{\mathbf{\Phi }}_{f_i}.\hfill \end{array}$$

(9)

In this analysis, we measured the canonical correlation $Corr({\widehat{\mathit{i}}}^1,{\widehat{\mathit{f}}}_i^1)$ and identified the information u that was highly correlated with the brain activity based on the cross-loading $Corr({\mathit{i}}^u,{\widehat{\mathit{f}}}_i^1)$.

4. Experimental Results

4.1. Results of Correlation between Taste and Other Data

Figure 5 shows the results of the correlation between taste and other data. This figure shows that questionnaires had a high correlation with taste among all participants. In particular, the values of the reward factor (Q1) were high, and four of the top five items were considered as reward factors. This phenomenon of a strong correlation between the reward factor and taste has also been confirmed in a previous study [33]; thus, the results are reasonable.

Next, we focus on the correlation coefficient between brain activity and taste. Half of the features obtained from brain activity were about 0.2, but it was not clear that they were correlated. To analyze the correlation coefficients in more detail, we calculated the p-value, which indicates whether the correlation coefficients are significant or not. The p-values correlating to the correlation coefficients in Figure 5 are shown in

Table 4. p-value for correlation between taste and other data.

Data		p-Value
Questionnaires	Q1-1	0.000
	Q1-2	0.000
	Q1-3	0.000
	Q1-4	0.000
	Q1-5	0.000
	Q2-1	0.047
	Q2-2	0.025
	Q2-3	0.115
	Q2-4	0.000
	Q2-5	0.002
	Q3-1	0.076
	Q3-2	0.243
	Q3-3	0.575
	Q3-4	0.507
	Q3-5	0.000
	Q4-1	0.159
	Q4-2	0.039
	Q4-3	0.084
	Q4-4	0.022
Brain activity	$d=1$	0.105
	$d=2$	0.130
	$d=3$	0.122
	$d=4$	0.416
	$d=5$	0.655
	$d=6$	0.271
	$d=7$	0.855
	$d=8$	0.127
	$d=9$	0.558
Individual information	sugar content	0.965
	acidity	0.446
	sugar-to-acid ratio	0.565
	size	0.098

. If the p-value is low, the correlating correlation coefficient is not zero, i.e., these data are correlated. Table 4 shows that the p-values of the correlation coefficients between brain activity information and taste are about 0.1 for $d=1,2,3,8$, confirming the existence of the correlation between these data.

In this present analysis, which focused on direct correlations, the reward factors (Q1) in the questionnaires strongly correlated with taste. A part of the brain activity was also correlated with taste. However, other data needed further verification; thus, we will analyze the potential correlation through brain activity in the next subsection.

4.2. Results of the Potential Correlation between Heterogeneous Data via CCA

Table 5. Canonical correlation between heterogeneous data.

Data	Canonical	Canonical	p-Value
	Feature	Correlation
“Brain activity and taste”	1st	0.40	0.000
“Brain activity and questionnaires”	1st	0.81	0.000
	2nd	0.75	0.000
	3rd	0.68	0.000
	4th	0.59	0.000
	5th	0.58	0.000
	6th	0.52	0.000
	7th	0.47	0.000
	8th	0.42	0.000
	9th	0.21	0.073
“Brain activity and individual information”	1st	0.53	0.000
	2nd	0.41	0.000
	3rd	0.31	0.007
	4th	0.15	0.187

presents the results of the canonical correlation between heterogeneous data. As shown in this table, “feature” represents the dimension of the canonical feature, and “canonical correlation” represents the value calculated with the canonical features. The first canonical feature is projected by the eigenvector correlating to the largest eigenvalue. Additionally, the “p-value” indicates whether the canonical correlation is significant or not. The table shows that the canonical correlation between brain activity and taste is 0.4, and its correlating p-value is zero, confirming that brain activity is potentially correlated with taste. Additionally, the canonical correlations of “brain activity and questionnaires” and “brain activity and individual information” are also high, and their p-values are low. Therefore, there is a latent relationship between taste, questionnaires, and individual information through brain activity. Although the first analysis shows that the individual information is not related to taste due to the higher p-value shown in Table 4, the p-values of the canonical correlation between brain activity and individual information are very low. Thus, the results suggest that individual data, such as numerical data, can be correlated with taste. Furthermore, by clarifying which data from questionnaires and individual information have high canonical correlations with brain activity, it is expected that it will be possible to clarify which data are potentially correlated with taste.

Figure 6 and Figure 7 show the results of the analysis to identify the factors that are highly correlated with brain activity, which clarifies the above point. Figure 6 shows the cross-loading between brain activity and questionnaires. This figure shows that the cultural factor (Q2) is particularly correlated with brain activity. In other words, taste and cultural factors are potentially correlated. In the first analysis, th reward factor (Q1) was highly correlated with taste; however, this analysis can provide some other results. From these results, we consider that this is a discovery that can be obtained through brain activity. Furthermore, Figure 7 shows the cross-loading between brain activity and individual information. It can be seen that the sugar-to-acid ratio is correlated with brain activity. Thus, the balance between sweetness and acidity is potentially correlated with taste. Therefore, analyzing canonical correlations between heterogeneous data is effective since we can discover the potential relationships that cannot be revealed by the first simple correlation analysis.

The experimental results confirm that brain activity is potentially correlated with other data about strawberries. The above results verify that taste is likely to have a potential correlation between the cultural factor and the sugar-to-acid ratio. In other words, the estimation of the VAS representing taste becomes feasible using these data. Although it is revealed that our analysis is valid and fruitful, it has several limitations. It is well-known that brain activity often contains noise. In this study, we addressed this problem by employing PCA to calculate the fNIRS features. However, the extraction of principal components from features is likely insufficient, which is a limitation of this analysis. In future studies, we will calculate more reliable canonical correlations by applying denoising methods for fNIRS data, such as the previously reported method [45]. In the second analysis, when the canonical correlation between brain activity and the taste was calculated using CCA, the canonical features were limited to one dimension since those of the features representing taste were one dimension. Generally, there is a limit to the expressive power of one-dimensional features. Therefore, we will also increase the number of dimensions of features expressing taste by adding evaluations other than VAS and will calculate more reliable canonical correlations. As a solution to this problem, multivariate analysis models that introduce label dequantization have been proposed [48,49], and their use will be considered in future work. In addition, while general CCA is based on two types of data, methods based on multi-view CCA are effective when various types of data are used [50,51], as in our experiment in this paper. Capturing more complex correlations will become feasible by introducing the above methods.

5. Conclusions

In this paper, we conducted a trial analysis of the relationship between taste and biological information obtained while eating strawberries (for a sensory evaluation). We collected four types of data: taste, questionnaire, brain activity, and individual information. We also compared their potential correlations based on CCA. The experiments suggested that brain activity data were associated with information about taste. Furthermore, it was revealed that the information about food culture and the sugar-to-acid ratio was correlated with brain activity. This confirms that the cultural factors in questionnaires are potentially highly correlated with taste, and there is a strong relationship between taste and individual information. Therefore, these results can lead to the construction of a machine learning model for estimating taste using heterogeneous data.