Assessing Overall Diet Quality: Development and Evaluation of the Performance of a Short Self-Administrated Questionnaire SCASA

Several tools assessing diet quality have been developed over the last decades, but their use in public health and clinical practice is limited because they necessitate detailed quantitative assessment of food intake. Our goal was to develop and validate a score (Score d’Alimentation Saine, SCASA) based on a short self-administrated online questionnaire to assess overall diet quality. SCASA targets the adult population in French-speaking Switzerland, but it was designed in a way enabling its adaptation for other regions. The choice of the items involved experts and lay volunteers. Construct validation and inter-method reliability were assessed by screening meal plans and by comparing the self-rated scores with food-record derived scores (kappa and Bland–Altman). SCASA (17 components) discriminated adequately balanced from imbalanced meal plans (93–95% and 44–46% of maximal score). Agreement between self-assessed and food record-based scores ranged between >90% (3 items), 80–89% (3 items), 70–79% (4 items), and <70% (5 items). The Bland–Altman plot showed a mean difference of −1.60 (95% CI −2.36 to −0.84), indicating a slight overestimation of the self-assessed diet quality compared to the food record. SCASA offers a reliable way to assess overall diet quality without requiring burdensome data collection or nutrient calculations.


Introduction
Diet quality plays a large role in health and disease [1]. Despite some controversies, the evidence-based dietary priorities related to the prevention of several major chronic diseases are widely agreed on: they include the increased consumption of vegetables, fruits, whole grains, legumes, and nuts; the wise choice of oils; and the decrease of (red) meat and highly processed foods [2][3][4][5][6]. Public health policies, health promotion programs and (primary or secondary) prevention practices targeting individuals share the goal of reducing the burden of diet-related conditions. The evaluation of their efficacy on diet mostly relies on dietary assessments [7][8][9][10][11][12][13]. After decades of focusing on nutrient intake, a practice inherited from an era where food shortage and nutritional deficiencies were a major threat [2], the emphasis has shifted toward the assessment of the whole diet [6,14,15]. Since the late 1990s, nutritional epidemiologists have warned against the evaluation of diet quality based solely on nutrient intake, which can hide paradoxical situations, for example when nutritional requirements are fulfilled by excessive intake of ultra-processed foods [16][17][18].

Materials and Methods
SCASA (score d'alimentation saine, or healthy eating score in French) targets primarily the adult population in French-speaking Switzerland, but we designed it in a way that enables its adaptation and use in other regions or countries (Appendix A). The development of the score involved a multistage process (Figure 1), with adjustments after each stage according to the outcomes. The Geneva Cantonal Ethics Committee on Research Involving Humans reviewed and approved this study (Project 73,457). All participants signed an informed consent form.

Stage 1: Construction of SCASA
The item selection (i.e., score components), their cut-off values, and the scoring method are key steps in the construction of a diet quality score [41]. To construct SCASA, we used the RAND/UCLA appropriateness method, a guided process for medical decision making in the face of limited evidence [42,43]. First, we identified the potential components to include (e.g., consumption of fruits, consumption of meat, physical activity) based on a review of existing scores [41,44] and the national food-based Swiss dietary guidelines [45,46]

Stage 1: Construction of SCASA
The item selection (i.e., score components), their cut-off values, and the scoring method are key steps in the construction of a diet quality score [41]. To construct SCASA, we used the RAND/UCLA appropriateness method, a guided process for medical decision making in the face of limited evidence [42,43]. First, we identified the potential components to include (e.g., consumption of fruits, consumption of meat, physical activity) based on a review of existing scores [41,44] and the national food-based Swiss dietary guidelines [45,46].
We submitted the list of potential components (n = 19) with their definitions (i.e., included foods and their impact on health) to a panel of four national experts in nutritional epidemiology and public health. The experts expressed their opinion about the appropriateness of each component in two rounds: the first individually, the second during a panel meeting moderated by a person not involved in the project. During the panel meeting, which was audio-recorded and transcribed, the experts discussed the relevance of each potential component and voted for its (non-)inclusion in the score. Five components were excluded (caloric beverages, fruit juice, candy, processed food, food variety) and two were split into several distinct items (starchy foods and whole grains, all meat and red meat). Corpulence was added as a component.
For the chosen 17 components, we defined the cut-offs based on international [47,48] and national [45,46] dietary guidelines, as well as national population-based food consumption data [49,50]. Sixteen components were evaluated in terms of quantity and/or frequency (e.g., "3 portions of 120 g a day", "up to 3 times a week"), one component (i.e., type of fats used) was assessed qualitatively, and one component characterized corpulence as a proxy for energy intake. We then phrased the items (i.e., questions, possible answers, and examples of included foods and portion sizes) and established the scoring method. In this first stage, corpulence was categorized using either body mass index categories or the Lorentz formula [51], in order to test both models. Following the procedure described by Estaquio et al. [23], we established a two-to-four-point scale within items: +2 or +1 (guideline fully respected), 0 (guideline partially respected), and −1 (guideline not respected). Each item had the same weight in the overall score computation. Total score was expressed as a proportion of the maximal score.

Stage 2: Content and Face Validation
We interviewed each of the four experts individually, asking them to review the items, cut-offs and scoring method, which led to refining the wording of some items but no major changes. We followed the suggestion of one expert to include a rating of self-perceived diet quality on a Likert scale, and an estimation of physical activity level for descriptive purposes (not included in the score calculation).
We recruited by word of mouth 15 volunteers with various characteristics, in terms of age, gender, socioeconomic status, and corpulence, and without specific nutritional knowledge. During an individual, semi-directed interview of approximately 45 min, each individual completed a SCASA on paper, during which they explained aloud their understanding of each item and their reflections about how to answer the questions. Each interview was audio-recorded. This stage led to the simplification of the questionnaire's introduction, the modification of several items' wording, and more precise examples of portion sizes (e.g., for vegetables). The corrected version of SCASA was then put online on a secure platform (EvaSys, Stat'Elite, Yens, France, version 7.1).

Stage 3: Pre-Test and Internal Consistency Assessment
This first version of SCASA was pre-tested in a sample of 30 volunteers recruited by an email sent to all second-year bachelor students at the Geneva School of Health Sciences (nursing, midwifery, nutrition and dietetics, physiotherapy, and radiology technology). The students were asked to fill the questionnaire online and provide written comments regarding the completion process. We analyzed the distribution of the responses to each item and of the total score in order to detect a ceiling or floor effect (i.e., grouped responses at the top or bottom of the distribution, respectively) or unclear questions leading to numerous missing answers and comments from students. One item (i.e., candy) was removed after this stage, leaving the final score with 17 items, for a score ranging from −19 to 19.

Stage 4: Construct Validation
We modelled the process of the construct validation proposed by Guenther et al. for the HEI [20] by assessing the ability of the score to discriminate balanced from imbalanced meal plans (Appendix B). The investigators (i.e., dietitians), created six weekly meal plans: three balanced plans according to the Swiss dietary guidelines [45][46][47], and three imbalanced plans: (1) western-type diet rich in ultra-processed foods [52,53]; (2) low-carb western-type diet rich in protein-based foods; (3) very low caloric weight loss diet. Out of the six meal plans, two were vegetarian (one balanced and one imbalanced). Three experienced dietitians external to the project reviewed all six meal plans, and three other dietitians, blinded to the project, completed SCASA for each meal plan (Appendix B). The average score of each plan was compared with the maximum score obtainable. Our hypothesis was that the balanced and imbalanced meal plans would obtain ≥80% and ≤30% of the maximum score, respectively.

Stage 5: Inter-Method Reliability Assessment
In order to assess the reliability of SCASA, we evaluated the concordance of the scores obtained by self-rating (i.e., when an individual fills the questionnaire) with the scores obtained with a reference method (i.e., calculation by nutrition experts on the basis of a 5-to-7-day food record) ( Figure 2). Food records were considered preferable to repeated 24-h dietary recalls because they can capture the weekly consumption of foods that are not consumed very often. Albeit burdensome, food records also provide more accurate and precise information on food consumption than a food-frequency questionnaire (FFQ) [40,[54][55][56].
We modelled the process of the construct validation proposed by Guenther et al. for the HEI [20] by assessing the ability of the score to discriminate balanced from imbalanced meal plans (Appendix B). The investigators (i.e., dietitians), created six weekly meal plans: three balanced plans according to the Swiss dietary guidelines [45][46][47], and three imbalanced plans: (1) western-type diet rich in ultra-processed foods [52,53]; (2) low-carb western-type diet rich in protein-based foods; (3) very low caloric weight loss diet. Out of the six meal plans, two were vegetarian (one balanced and one imbalanced). Three experienced dietitians external to the project reviewed all six meal plans, and three other dietitians, blinded to the project, completed SCASA for each meal plan (Appendix B). The average score of each plan was compared with the maximum score obtainable. Our hypothesis was that the balanced and imbalanced meal plans would obtain ≥80% and ≤30% of the maximum score, respectively.

Stage 5: Inter-Method Reliability Assessment
In order to assess the reliability of SCASA, we evaluated the concordance of the scores obtained by self-rating (i.e., when an individual fills the questionnaire) with the scores obtained with a reference method (i.e., calculation by nutrition experts on the basis of a 5-to-7-day food record) ( Figure 2). Food records were considered preferable to repeated 24-h dietary recalls because they can capture the weekly consumption of foods that are not consumed very often. Albeit burdensome, food records also provide more accurate and precise information on food consumption than a food-frequency questionnaire (FFQ) [40,[54][55][56]. For this inter-method reliability assessment, we recruited a sample of 105 volunteers from the target population (73% women, mean age 30 ± 13.7), who completed SCASA online (no feedback provided). One week after completion, they received oral (by phone) and written instructions (by email) on how to fill in a paper-based food record. To analyze For this inter-method reliability assessment, we recruited a sample of 105 volunteers from the target population (73% women, mean age 30 ± 13.7), who completed SCASA online (no feedback provided). One week after completion, they received oral (by phone) and written instructions (by email) on how to fill in a paper-based food record. To analyze the food records we grouped the reported food items according to the items defined by SCASA, and assigned points to each record-derived item using the same cut-offs as in SCASA. We assessed agreement between each self-rated and record-derived item using quadratic weighted kappa statistics. Agreements were classified as follows: weak <70%; fair 70-79%; moderate 80-89%; and strong ≥90%. The participants did not report consistently the type of oils and fats in their food record, and therefore the item "Fats and oils" could not be included in the quadratic weighted kappa statistics. The item "Corpulence" was also not included in the food record. We then assessed agreement between the total score obtained with SCASA and with the food record using Bland-Altman plots. Limits of agreement were set at 1.96 × standard deviations above and below the mean difference [36,40,57]. As the goal of SCASA is to assess the overall quality of the diet and not to assess nutrient intake, we chose not to show nutritional intake data.

Adaptations for Other Regions
SCASA is mainly based on the Swiss national dietary guidelines [45,46]. In order to facilitate the adaptation for other countries, we have tabulated the recommendations next to those of Great Britain, France, Germany, Belgium, and the Netherlands in Appendix A. Table 1 describes the 17 components, cut-offs, and scoring method of SCASA after content and face validation, and pre-testing. In addition to these items, the respondents were asked to give a general estimation of their diet quality (Likert scale from 0 to 10) and physical activity level (sedentary, light, active, and very active), and to state their sex, age, weight, and height. It takes 15 to 20 min to fill in the online version of SCASA.

Construct Validation
The balanced (n = 3) and imbalanced (n = 3) weekly meal plans obtained, respectively, 93 to 95% and 44 to 46% of the maximal score, showing a good discrimination between balanced and imbalanced dietary patterns.

Inter-Method Reliability
The agreement between self-assessed and food-record based SCASA-scores for each item is shown in Table 2. Agreement was considered as strong for three items: "Vegetables", "Starchy foods", and "Alcoholic beverages". Mismatches (low agreement and kappa value <0.2) between the self-assessed and food-record based scores were observed for two items in particular "Total meat" and "Nuts and seeds".
The Bland-Altman plot ( Figure 3) showed that the mean difference of the scores according to the assessment method was −1.60 (95% CI −2.36 to −0.84), indicating a slight overestimation of the diet quality with the self-assessment (SCASA) compared to the food record, especially when the diet was of lower quality, i.e., when the total score was closer to negative values ( Figure 3).   Figure 3. Bland-Altman plot for overall agreement between the total score obtained with SCASA (Score d'Alimentation Saine) and with the 5-to-7-dayfood record (i.e., reference method) (n = 105). The black line represents the mean bias between the two measures and the grey lines are the 95% limits of agreement.

Discussion
Our goal was to develop a score assessing the overall diet quality of Swiss adults (SCASA).
SCASA discriminated adequately between healthy and unhealthy diets, as shown by the consistent results obtained when applying the score to optimal and suboptimal meal Figure 3. Bland-Altman plot for overall agreement between the total score obtained with SCASA (Score d'Alimentation Saine) and with the 5-to-7-dayfood record (i.e., reference method) (n = 105). The black line represents the mean bias between the two measures and the grey lines are the 95% limits of agreement.

Discussion
Our goal was to develop a score assessing the overall diet quality of Swiss adults (SCASA). SCASA discriminated adequately between healthy and unhealthy diets, as shown by the consistent results obtained when applying the score to optimal and suboptimal meal plans. The imbalanced meal plans obtained higher scores than hypothesized (45% of maximum score obtainable vs. 30%). These rather generous scores can be explained by the fact that SCASA does not consider foods as "unhealthy" per se, but penalizes only very inadequate consumption. Another explanation is that the "unhealthy" food items were grouped into four items (i.e., sweets, salty snacks, and fatty dishes representing one item), whereas the "healthy" foods were detailed into 12 items, resulting in a lower influence of unhealthy foods on the final score.
Reliability of SCASA was fair, similarly to other instruments, such as the recently developed short healthy eating index survey (sHEI) which shows correlations with a 24 h recall score ranging from 0.44 to 0.64 for individual food group items [58]. Meat consumption was underestimated by SCASA compared to the food records. This might be explained by the fact that any consumption of meat reported in the food records, regardless of quantity, was taken into account, whereas the respondents might not have counted the very small quantities of meat in preparations when assessing their overall weekly intake. For example, when consuming a mixed salad with small pieces of salami, a respondent might not count this as an occasion of meat consumption, whereas it will be counted during the rating based on their food record. The consumption of nuts and seeds was either overestimated (e.g., when the participants counted salted peanuts in this category whereas it belonged to the "Snacks" item), or underestimated (when the participants did not count the nuts present in composed dishes or as ingredients). This observation led us to clarify the definition of the item in the final version. The fair agreement obtained by the item "Whole grains" can be explained by their infrequent consumption, and that for "Red meat" could be related to a temporal bias. Indeed, the food records of the discrepant pairs showed an increased consumption of grilled meat, related to the barbecue-favorable weather that appeared between the two assessments.
Two items "Starchy foods" and "Alcoholic beverages" presented a kappa paradox (i.e., a low kappa despite excellent agreement). Indeed, kappa tests are less reliable when tables are symmetrically imbalanced, i.e., when most people rate themselves similarly [59]. This was the case for starchy foods, which most participants reported eating at least twice a day.
The limitations of SCASA are those that are inherent to dietary assessment. Very strong agreement is rare when comparing dietary assessments methods, because of the complexity of the diet (large number of foods available, variability over weekdays and seasons, variation in nutrient composition of similar foods and recipes) and the related difficulties of self-assessing diet [60,61]. Food records, although considered as the reference method, are subject to measurement errors, notably because they reflect consumption during a short time frame. We overcame some of these limitations by combining several validation methods.
SCASA offers a complementary tool to those already existing. Similarly to the sHEI [58], it does not require burdensome data collection or nutrient calculation, and focuses solely on overall, food-based, diet quality. The authors of the sHEI noted that some of their questions might have been difficult to understand for the respondents (i.e., "How many servings of saturated fat do you consume on average per day?") [58]. The strength of SCASA is that is has been tested among a panel of lay people, which increased its intelligibility and hence the reliability of the answers. Communication about respondents' current diet is key to improve awareness and trigger change [33]. As part of the implementation process, feedback on total score, fats and oils, plant-based foods, meat, dairy products and other sources of calcium, snacks and sweets, and alcoholic beverages was developed and tested. The texts, following a structure inspired by the health belief model [59] are in French and may be obtained from the authors upon request.

Conclusions
SCASA offers a reliable way to assess overall diet quality without requiring burdensome data collection or nutrient calculations in the French-speaking part of Switzerland, with the possibility to adapt it to other regions in Western Europe.  Data Availability Statement: Data available on request from the corresponding author due to privacy and ethical restrictions.

Conflicts of Interest:
The authors declare no conflict of interest.