An Approach to Developing Likert Scale Survey Results Based on the Example of a Research Study Involving a Limited Number of Students

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The article analyzes the method of calculating the feature significance index, in the context of Likert scale surveys performed on a limited number of respondents. Although a series of improved approaches were described, the conclusion fails to highlight the importance of this intermediate stage of research. The aim of this research could be better highlighted and brought to relevance if better connected with the practical situations in which such survey results can influence dairy fabrication or distribution approaches.

Moreover, if an extended mathematical model can be developed to adapt the presented methodology for surveys on a larger scale of respondents, it should be mentioned, at least as a further development direction.

A suggestion of integrating the developed formula within statistical software tools or hypothetical algorithm could better highlight the applied character of the research.

Author Response

Reviewer:

The article analyzes the method of calculating the feature significance index, in the context of Likert scale surveys performed on a limited number of respondents. Although a series of improved approaches were described, the conclusion fails to highlight the importance of this intermediate stage of research. The aim of this research could be better highlighted and brought to relevance if better connected with the practical situations in which such survey results can influence dairy fabrication or distribution approaches.

Authors' response:

Thank you for your detailed feedback on the article, including your critical comments and suggestions, which helped us present the main ideas of the survey study more clearly and accessible to potential readers.

As suggested, we have further developed the justification for the survey research, highlighting the practical importance of dairy production in assessing the distribution of dairy products and consumer preferences in the dairy market. This justification, or rather confirmation of the importance of research in the field of dairy production, is included in the final, concluding section of the article. We have added citations of scientific publications in this section.

Reviewer:

Moreover, if an extended mathematical model can be developed to adapt the presented methodology for surveys on a larger scale of respondents, it should be mentioned, at least as a further development direction.

A suggestion of integrating the developed formula within statistical software tools or hypothetical algorithm could better highlight the applied character of the research.

Authors' response:

In the final section of the article, we also added a paragraph discussing possible further directions for research using the proposed method for processing survey results and their practical application.

Reviewer 2 Report

Comments and Suggestions for Authors

The authors present a simple procedure for processing the results of a questionnaire with a 5-point Likert scale, the core of which is the Feature Significance Index (FSI) defined as the ratio of high ratings (4–5) to low ratings (1–2). When applied in practice to a questionnaire with 20 students and 10 questions, a problem arises: several questions did not receive ratings of 1–2, so the FSI cannot be calculated. The authors propose modified versions of the index (FSIbis and the supplementary FSIbis1.0) and demonstrate how this can be used to create a ranking of questions.

Recommendation:

• Define what the index should represent (e.g., "strength of agreement" vs. "polarization" vs. "prevalence of positive over negative").
• Explain why the ranking should be based on the ratio of extremes (1–2 vs. 4–5) and not, for example, on the median, the average with caution, or the proportion (4–5) itself.

When FSIbis = 1.0 for multiple questions, the authors introduce another index (FSIbis1.0) for differentiation. However, its definition and interpretation are unclear: the text mentions the use of "ps4.5 / ps1..5," but since ps1..5 = 100%, this term is essentially equal to ps4.5/100. The resulting index is therefore not a fundamentally new concept, but only the (multiplied) proportion of top-box responses.

Recommendation:

• Either explicitly acknowledge that at FSIbis=1.0, decisions are made solely on the basis of the 4–5 (top-box) ratio and adapt the methodology accordingly,
• or propose a uniform index that works continuously for all cases without "switching" rules.

 

With N=20, a slight shift in responses can change the order of questions. The article treats the ranking as a fixed result, without intervals of uncertainty or sensitivity analysis. Although the authors mention that the small number of respondents is a limitation, this needs to be quantified methodologically.

Recommendation (at least one of the following):

• bootstrap (for each index, estimate the distribution and probability that the question is in the top 3, etc.),
• simple confidence intervals for ps4.5 and ps1.2 proportions,
• "leave-one-out" sensitivity: how the ranking changes when 1 respondent is removed.

 

The text states that Excel contains the sums of responses for each value 1–5 and question, but the article does not publish the complete contingency tables (only derived indices and rankings). Without this, it is difficult to verify the calculations and interpretation.

Recommendation:

• Insert a 10×5 table with the number of responses in the appendix (or supplementary material).
• Add a short pseudocode/formula for the calculation.

 

1. Inconsistency in the number of respondents: the text mentions a group of 23 and then 20, whereas it is necessary to state precisely how many were approached, how many responded, and, if applicable, why three did not respond.
2. Terminology: "grades" and "impact ratings" are mixed in places; I recommend using "ratings/response categories" consistently.
3. Tables 1–3: add not only the index values but also the basic shares (ps4.5; ps1.2; or ps3) so that the reader can see what the index actually summarizes.
4. Conclusion on dairy farming: the claim that modernity has "the greatest impact on the development of dairy farming" is empirically weak given the small sample size and specific student population; I recommend phrasing it more cautiously (as an illustrative result).

Author Response

Reviewer:

The authors present a simple procedure for processing the results of a questionnaire with a 5-point Likert scale, the core of which is the Feature Significance Index (FSI) defined as the ratio of high ratings (4–5) to low ratings (1–2). When applied in practice to a questionnaire with 20 students and 10 questions, a problem arises: several questions did not receive ratings of 1–2, so the FSI cannot be calculated. The authors propose modified versions of the index (FSIbis and the supplementary FSIbis1.0) and demonstrate how this can be used to create a ranking of questions.

Recommendation:

• Define what the index should represent (e.g., "strength of agreement" vs. "polarization" vs. "prevalence of positive over negative").
• Explain why the ranking should be based on the ratio of extremes (1–2 vs. 4–5) and not, for example, on the median, the average with caution, or the proportion (4–5) itself.

Authors' responses:

Thank you for your detailed feedback on the article, your critical comments, and suggestions, which helped us present the study's main ideas in a clearer, more accessible way for potential readers.

Thank you for your recommendations, which helped us reflect further on the index and its interpretation. Given the FSI's structure, its purpose is to show the predominance of positive ratings over negative ratings for each question (issue) in the survey. Of course, this "predominance" can vary greatly, which provides the basis for discussion of each question (in the survey) based on the distribution of responses (respondent ratings), which determine the FSI value.

The importance of the FSI in developing survey results lies in its ability to rank the issues considered in the survey by FSI value. This allows us to demonstrate, to some extent, the "strength" of the importance of individual issues addressed in the survey. We have added a few sentences to the article to elaborate on these points.

Including extreme values ​​(1-2 and 4-5) in the FSI index (formula) is my idea – a way to approach the analysis of the results (data) of a study using a Likert scale survey. Of course, a different index structure, the mean, median, or just 4-5 responses, could be proposed to calculate the index used to rank responses. I believe that in a future survey, these options (proposals) could be used to develop the results and compare them. This would be valuable material for analysis and discussion.

The assumption that extreme Likert scale scores were used in constructing the FSI index was intended to demonstrate the diversity of respondents' perspectives on the issue at hand. While average values ​​can certainly be used, in my opinion, these represent a certain "average" approach. While the "average" also has its value, it's easier to engage in and develop a discussion of the study results based on extreme values.

I considered the rationale for using extreme (Likert-scale) ratings for each of the survey's issues in the analysis. My proposed approach was inspired by the fact that we often operate within a sphere of positive and negative opinions. I wanted to demonstrate the prevalence of positive opinions over negative (weak) opinions. This was my intuitive approach. All of this is, of course, controversial, so I welcome the discussion, as it motivates me to continue my research and exploration in the  ​​analysis and description of research results using various indicators.

Reviewer:

When FSIbis = 1.0 for multiple questions, the authors introduce another index (FSIbis1.0) for differentiation. However, its definition and interpretation are unclear: the text mentions the use of "ps4.5 / ps1..5," but since ps1..5 = 100%, this term is essentially equal to ps4.5/100. The resulting index is therefore not a fundamentally new concept, but only the (multiplied) proportion of top-box responses.

Recommendation:

• Either explicitly acknowledge that at FSIbis=1.0, decisions are made solely on the basis of the 4–5 (top-box) ratio and adapt the methodology accordingly,
• or propose a uniform index that works continuously for all cases without "switching" rules.

Authors' responses:

The transition to calculating the FSI_bis and FSI_bis1.0 indexes was considered when the FSI index could not be calculated. The formulas for calculating FSI_bis and FSI_bis1.0 are our proposal and, at the same time, an inspiration for further exploration of methods for presenting survey results that take into account responses provided using a Likert scale. Thank you for your substantive comment regarding the FSI_bis1.0 index and its calculation. Indeed, this stage of the calculation can be simplified, and we have done so. Instead of the previous formula (3), we have provided a new, simplified version that accounts for the ratio of the percentage shares of answers 4 and 5, divided by 100.

Reviewer:

With N=20, a slight shift in responses can change the order of questions. The article treats the ranking as a fixed result, without intervals of uncertainty or sensitivity analysis. Although the authors mention that the small number of respondents is a limitation, this needs to be quantified methodologically.

Recommendation (at least one of the following):

• bootstrap (for each index, estimate the distribution and probability that the question is in the top 3, etc.),
• simple confidence intervals for ps4.5 and ps1.2 proportions,
• "leave-one-out" sensitivity: how the ranking changes when 1 respondent is removed.

Authors' responses:

I agree with the Reviewer that the very small number of respondents participating in the survey increases the likelihood (risk) that some answers on the issue under consideration will not be selected by respondents. In the research study, the number of respondents (students) was limited by the number of students enrolled in a given field of study. Nevertheless, the observation that the number of students affects the ability to fully process the research results confirms the importance of the study population's size.

Thank you for your suggestions regarding methodological quantification for the sample of 20 respondents under consideration. We have calculated the confidence interval and included it in the text (in the Results section), along with methodological details (in the Materials and Methods section).

Reviewer:

The text states that Excel contains the sums of responses for each value 1–5 and question, but the article does not publish the complete contingency tables (only derived indices and rankings). Without this, it is difficult to verify the calculations and interpretation.

Recommendation:

• Insert a 10×5 table with the number of responses in the appendix (or supplementary material).
• Add a short pseudocode/formula for the calculation.

Authors' responses:

An Excel table with the survey results (sums of responses for each value 1–5 and each question) is available in the repository. Access to the repository is available via the following link: https://doi.org/10.18150/LKPDOD. The link address is provided in the Data Availability Statement section. The table with survey results (in the repository) also includes formulas for calculating the FSI and FSI_bis indices.

Reviewer:

1. Inconsistency in the number of respondents: the text mentions a group of 23 and then 20, whereas it is necessary to state precisely how many were approached, how many responded, and, if applicable, why three did not respond.

Authors' response:

The group consisted of 23 students. This group was asked to respond to a survey (rating the importance of the indicated factors on a scale of 1 to 5). Twenty students completed the survey. Three did not complete the survey due to personal reasons. A relevant statement correcting this information is included in the Materials and Methods section.

Reviewer:

2. Terminology: "grades" and "impact ratings" are mixed in places; I recommend using "ratings/response categories" consistently.

Authors' response:

As suggested, we have verified the terminology in the article within the specified scope.

Reviewer:

3. Tables 1–3: add not only the index values but also the basic shares (ps4.5; ps1.2; or ps3) so that the reader can see what the index actually summarizes.

Authors' response:

We have added underlying data (shares) in tables to make it easier to identify calculation results.

Reviewer:

4. Conclusion on dairy farming: the claim that modernity has "the greatest impact on the development of dairy farming" is empirically weak given the small sample size and specific student population; I recommend phrasing it more cautiously (as an illustrative result).

Authors' response:

As suggested, we have reworded the indicated paragraph in the Conclusions.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The justification and presentation of the survey research and the development approach have been improved.

Author Response

The justification and presentation of the survey research and the development approach have been improved.

Answer:

Thank you for your feedback on the article. Taking into account the suggestions highlighted in the section "Are the conclusions supported by the results? – can they be improved...", I added a sentence in the Conclusions section in the second paragraph that expands on future research.

Reviewer 2 Report

Comments and Suggestions for Authors

I have any comments.

Author Response

I have any comments.

Answer:

Thank you for the information.