Comparative Evaluation of Sensory Attributes of Coffee Using Best–Worst Scaling and Pairwise Comparison Methods ^†

Abstract

Understanding consumer preferences is vital for rational decision-making in the agri-food sector and for effective product development. This study examines two comparative evaluation methods, Best–Worst Scaling (BWS) and Pairwise Comparison via the Analytic Hierarchy Process (AHP), focusing on the sensory attributes of coffee. The objective is to explore which attributes influence the preferences of students from the International Hellenic University in Sindos and assess the effectiveness of each method in capturing these preferences. Primary data were collected through structured questionnaires where participants ranked six attributes: taste, aroma, aftertaste, body, acidity and intensity. Taste emerged as the most significant attribute across all methods. However, discrepancies in the ranking of the remaining attributes revealed methodological differences. This research contributes to the applied evaluation of qualitative attributes in coffee and proposes the combined use of BWS and AHP for a more comprehensive understanding of consumer behavior.

1. Introduction

Understanding consumer preferences is a key issue in agri-food economics, product development and marketing. Decision-making in this context often involves multiple criteria that must be evaluated and prioritized in a rational and transparent way. Multi-Criteria Decision Analysis provides a solid framework for such evaluations, enabling a systematic comparison of alternatives across both quantitative and qualitative attributes [1,2].

Among the most widely applied MCDA techniques are Pairwise Comparison through the Analytic Hierarchy Process [3] and Best–Worst Scaling [4]. AHP allows structured pairwise judgments of attributes and generates priority weights based on consistency, while BWS requires respondents to identify the most and least important attributes in balanced sets, thereby reducing cognitive burden. Both methods have been applied extensively in consumer research, though their comparative performance in the evaluation of sensory attributes remains underexplored.

Coffee represents an ideal case study for applying these methods. As one of the most consumed beverages worldwide, its sensory attributes—“taste, aroma, aftertaste, body, acidity and intensity”—play a central role in consumer choices [5]. Previous research has highlighted the dominant role of taste, but less is known about how methodological approaches affect the relative importance of other sensory dimensions.

The objective of this study is therefore twofold: (1) to evaluate the importance of six coffee sensory attributes among university students and (2) to compare the strengths and limitations of BWS and AHP in capturing consumer preferences. By integrating both methods, this research contributes to the methodological discussion on preference elicitation and provides insights for agri-food product development and marketing.

2. Methodology

2.1. Study Design and Sample

The study was conducted among undergraduate students of the Department of Agriculture, International Hellenic University, Sindos, Greece. A structured questionnaire was designed and distributed during April 2025. A total of 122 respondents participated, providing primary data for the comparative evaluation of coffee sensory attributes.

2.2. Questionnaire Structure

The questionnaire consisted of two main sections. In the first section, participants evaluated six sensory attributes of coffee—“taste, aroma, aftertaste, body, acidity and intensity”—using the BWS method. In the second section, the same attributes were compared using the PWC framework of the AHP. Demographic questions were also included to characterize the sample.

2.3. Best–Worst Scaling (BWS)

In the BWS section, respondents were presented with balanced sets of attributes based on Balanced Incomplete block Design (BIBD). For each set, they were asked to identify the most important (the best) and least important (the worst) attribute. From these responses, the Best–Worst Index (BWI) was calculated according to the difference between best and worst counts, normalized by the number of appearances of each attribute [4,5].

2.4. Pairwise Comparison and AHP

In the AHP section, participants performed pairwise comparisons of the six attributes using Saaty’s 1–9 scale [6]. For each matrix of judgments, consistency ratios (CRs) were calculated to ensure the reliability of responses. Attributes’ weights were then derived using the eigenvalue method. Cases with a CR above 0.1 were excluded from the analysis, following Saaty’s guidelines [7,8].

2.5. Data Analysis

Data was coded and analyzed using Microsoft Excel (Microsoft 365, ver. 2605) and R studio (ver. 2024.12.1). For BWS, normalized scores were computed for each attribute and aggregated across respondents. For AHP, weight vectors were calculated and compared with BWS scores to highlight similarities and differences between methods.

3. Results and Discussion

3.1. Sample Characteristics

The demographic profile of respondents is summarized in

Table 1. Demographic characteristics of respondents.

Category	n (%)
Gender
Male	39 (32%)
Female	81 (66.4%)
Other	2 (1.6%)
Age
≤20	42 (34.4)
21–22	28 (23%)
23–25	33 (27%)
>25	19 (15.6%)
University
IHU/Geotechnical Science/dept of Agriculture	122 (100%)
Place of origin
Urban area	65 (53.2%)
Semi-urban area	28 (23%)
Rural area	29 (23.8%)
Employment status
Full-time student	71 (58.2%)
Part-time (up to 16 h/week)	7 (5.7%)
Part-time (17–40 h/week)	17 (13.9%)
Full-time (40 h/week)	27 (22.2%)
Monthly Income
<EUR 300	62 (50.8%)
EUR 301–EUR 600	31 (25.4%)
EUR 601–EUR 900	16 (13.1%)
>EUR 901	13 (10.7%)

Source: Authors’ research.

. The sample consisted predominantly of female students (66.4%) aged ≤20 (34.4) and enrolled at the Department of Agriculture, IHU Sindos. Most participants were full-time students, with half indicating a monthly income below EUR 300.

3.2. Best–Worst Scaling Results

The BWS analysis highlighted clear differences in the relative importance of sensory attributes. Taste consistently emerged as the most influential attribute, with a BWI exceeding 0.9. This confirms previous evidence that taste is the primary driver of consumer choice in coffee [4,9]. Aftertaste occupied a middle position, showing a moderate positive score with a BWS ≈ 0.35, while aroma ranked close to neutral. By contrast, body, acidity and intensity were evaluated as less important, all with negative BWI values. These findings indicate that students prioritize flavor above all other sensory characteristics, but the degree of importance attached to secondary attributes is less clear.

The BWS analysis revealed clear differences in the relative importance of sensory attributes, as presented in

Table 2. Best–Worst Scaling results.

Attributes	Best	Worst	Net Score	Total	BWI
Taste	359	16	343	375	0.915
Aroma	133	156	−23	289	−0.080
Body	66	262	−196	328	−0.600
Aftertaste	210	100	110	310	0.355
Intensity	106	197	−91	303	−0.300
Acidity	116	229	−113	345	−0.454

Source: Authors’ research.

.

These findings confirm that taste dominates consumer perception, while secondary attributes vary considerably in importance

3.3. Analytic Hierarchy Process Results

The AHP analysis produced a similar but not identical ranking of attributes, as also presented in

Table 3. Attribute weights.

-	Taste	Aroma	Body	Acidity	Aftertaste	Intensity
Taste	44%	68%	38%	24%	19%	15%
Aroma	12%	18%	47%	30%	25%	20%
Body	11%	4%	10%	32%	25%	21%
Acidity	12%	4%	2%	6%	26%	21%
Aftertaste	10%	3%	2%	7%	4%	19%
Intensity	11%	3%	2%	1%	1%	4%

Source: Authors’ research.

. Again, taste received the highest weight (approximately 40–45%), confirming its dominance across methods. However, in contrast to BWS, aroma received a relatively higher weight (≈18%), suggesting that when respondents evaluate attributes in pairwise comparisons, aroma gains greater significance. Aftertaste and body were assigned lower weights, while acidity and intensity were consistently the least valued. Importantly, most respondents’ matrices achieved a CR below 0.1, indicating reliable pairwise judgment [6].

These results show that while taste dominates in both methods, the relative ranking of secondary attributes differs, with aroma gaining more importance in AHP compared to BWS. To better illustrate the relative weights and the associated consistency ratios, Figure 1 presents the combined results. The figure confirms that all CR values remain well below the 0.1 threshold, ensuring robustness of the findings.

4. Conclusions

Although both methods identified taste as the primary attribute, discrepancies emerged in the ranking of secondary attributes. In BWS, aftertaste was perceived as more relevant, while in AHP, aroma ranked higher. These differences highlight the methodological effects on preference elicitation. BWS has the advantage of lower cognitive burden, as respondents only identify the most and least important attributes in each set, making it particularly suitable for studies involving non-expert participants. In contrast, AHP requires more effort but provides additional validation through consistency checks, offering a more structured approach to decision-making.

The divergence between methods suggests that consumer preferences are sensitive to the evaluation framework. For practitioners, this implies that relying on a single method may overlook important nuances. Combining BWS and AHP can therefore yield a more comprehensive picture of consumer behavior. In the context of coffee, such insights are valuable for product development, sensory profiling and targeted marketing strategies.