Validation and Reliability of the Alzheimer’s Disease-Commonwealth Scientific and Industrial Research Organisation Food Frequency Questionnaire

Accuracy in measuring intake of dietary constituents is an important issue in studies reporting the associations between diet and chronic diseases. We modified a Commonwealth Scientific and Industrial Research Organisation (CSIRO) food frequency questionnaire (FFQ) to include foods of interest in the field of Alzheimer’s disease (AD) research. The aim of the current study was to determine the reliability and validity of the AD-CSIROFFQ in 148 cognitively normal older adults. The AD-CSIROFFQ was completed before and after completion of a four-day weighed food record. Of the 508 food and beverage items reported, 309 had sufficient consumption levels for analysis of reliability. Of the 309 items, over 78% were significantly correlated between the two questionnaire administrations (Spearman’s rank correlations). We used two additional methods to assess absolute nutrient intake agreement between the AD-CSIROFFQ and the weighed food records (Pearson’s correlation coefficients and Bland–Altman plots) and quintile rankings to measure group level agreement. The adequate correlations observed between questionnaire responses suggest that the AD-CSIROFFQ is reliable. All nutrient intakes were acceptable for ranking of individuals on a group level, whilst the agreement levels with respect to the weighed food records for 11 of the 46 nutrients show validity in terms of their individual level absolute intake. The AD-CSIROFFQ makes an important contribution to the tools available for assessing usual dietary intake in groups of older adults with respect to AD research.


Introduction
Accurate assessment of usual dietary intake is crucial to understanding the association between diet and chronic diseases; incorrect information may lead to false associations between dietary factors and disease risk and prevention. There is a need to develop and validate methods of assessing food and beverage intake to facilitate the development of guidelines for the prevention of diet-related chronic diseases.

Alzheimer's Disease Food Frequency Questionnaire
The AD-CSIROFFQ is an online FFQ with over 280 food and beverage items which includes serving sizes that can be altered by the respondent. The FFQ also contains options for each item related to food preparation and cooking techniques, as well as options for selection of specific type of the item (e.g., type of milk used, type of bread consumed, type of fats used as spread or in cooking, type of fish consumed, etc.), which yields a plethora of final options of the 280 initial food and beverage items. To minimise fatigue, the FFQ was designed to have the foods arranged into categories and automatically saves data entered, allowing participants to leave the questionnaire and return later to complete it in multiple sittings. Participants cannot move to the next section until all questions are fully answered in the current section, ensuring the questionnaire is complete. The AD-CSIROFFQ takes approximately 45 min to complete, requires access to the Internet, and assesses usual daily intake over the preceding 12 months. Grams per day intake of foods and beverages is provided following completion of the questionnaire. This intake was then analysed using FoodWorks Professional version 7.0 (Xyris Software Pty Ltd., Brisbane, Queensland, Australia.) and the AUSNUT 2007 database of Australian Foods to yield grams per day of nutrients consumed.

Four-Day Weighed Food Record
Participants attended an orientation session, during which detailed verbal and written instructions regarding the completion of the four-day weighed food records were given by an Accredited Practising Dietitian. Digital electronic kitchen scales (model 1348, Propert) weighing to the nearest gram and modular measuring cup and spoon sets were provided to each participant, as was a sample of a completed food record. Each participant was requested to record the weight of all food and beverages consumed during the four-day period from Sunday through Wednesday (at home or away from home) and to record brand names, methods of food preparation, and ingredients of recipes in a specially designed diary. In addition, participants were asked to record the amount of any leftover food at the end of a meal or snack, either at home or away from home, and to report any dietary supplements consumed. All food records were reviewed face-to-face with the participant upon completion of the four-day period to clarify details or missing information. Four-day weighed food records were analysed using FoodWorks Professional version 7.0 and the AUSNUT 2007 database of Australian Foods to yield grams per day of nutrients consumed.

Statistical Analysis
Statistical analyses were performed using the IBM Statistical Package for the Social Sciences, Version 25.0 (SPSS Inc., Chicago, IL, USA). A p-value of <0.05 determined a significant result. Mean

Alzheimer's Disease Food Frequency Questionnaire
The AD-CSIROFFQ is an online FFQ with over 280 food and beverage items which includes serving sizes that can be altered by the respondent. The FFQ also contains options for each item related to food preparation and cooking techniques, as well as options for selection of specific type of the item (e.g., type of milk used, type of bread consumed, type of fats used as spread or in cooking, type of fish consumed, etc.), which yields a plethora of final options of the 280 initial food and beverage items. To minimise fatigue, the FFQ was designed to have the foods arranged into categories and automatically saves data entered, allowing participants to leave the questionnaire and return later to complete it in multiple sittings. Participants cannot move to the next section until all questions are fully answered in the current section, ensuring the questionnaire is complete. The AD-CSIROFFQ takes approximately 45 min to complete, requires access to the Internet, and assesses usual daily intake over the preceding 12 months. Grams per day intake of foods and beverages is provided following completion of the questionnaire. This intake was then analysed using FoodWorks Professional version 7.0 (Xyris Software Pty Ltd., Brisbane, Queensland, Australia.) and the AUSNUT 2007 database of Australian Foods to yield grams per day of nutrients consumed.

Four-Day Weighed Food Record
Participants attended an orientation session, during which detailed verbal and written instructions regarding the completion of the four-day weighed food records were given by an Accredited Practising Dietitian. Digital electronic kitchen scales (model 1348, Propert) weighing to the nearest gram and modular measuring cup and spoon sets were provided to each participant, as was a sample of a completed food record. Each participant was requested to record the weight of all food and beverages consumed during the four-day period from Sunday through Wednesday (at home or away from home) and to record brand names, methods of food preparation, and ingredients of recipes in a specially designed diary. In addition, participants were asked to record the amount of any leftover food at the end of a meal or snack, either at home or away from home, and to report any dietary supplements consumed. All food records were reviewed face-to-face with the participant upon completion of the four-day period to clarify details or missing information. Four-day weighed food records were analysed using FoodWorks Professional version 7.0 and the AUSNUT 2007 database of Australian Foods to yield grams per day of nutrients consumed.

Statistical Analysis
Statistical analyses were performed using the IBM Statistical Package for the Social Sciences, Version 25.0 (SPSS Inc., Chicago, IL, USA). A p-value of <0.05 determined a significant result. Mean values, standard deviations (SD), and percentages are provided for the demographics for the reliability and validation study cohort.

Reliability Study
The raw data for the majority of food items was not normally distributed; therefore, Spearman's rank correlation coefficients were used to compare the correlation between food and beverage intakes from the first and second administration of the AD-CSIROFFQ.

Validation Study
The raw data for the majority of nutrients were not normally distributed; logarithmic transformation produced normalised data that maximised the Kolmogorov-Smirnov test statistic. Following log transformation, the nutrient data were energy-adjusted using linear regressions. Raw data were used to calculate mean and SD intakes for the 46 nutrients analysed. Bivariate Pearson's correlation coefficients were calculated for the log-transformed and energy-adjusted nutrient data to estimate the associations between the two methods, i.e., the AD-CSIROFFQ and the weighed food record.
Agreement between both methods was assessed using Bland-Altman plots [17], which represent the individual differences between the two measurements graphed against the mean of the measurements. Interpretation of the Bland-Altman results was based on three categories: "good agreement", "fairly good agreement", and "poor agreement", depending on whether the difference between the two measurements is approximately equal to 1, 2, or 3 standard deviations of the average nutrient intake, respectively.
The agreement between methods of the relative rankings of nutrient intakes was assessed by classifying subjects into quintiles and cross-tabulating, using the log-transformed and energy-adjusted data. Weighted kappa statistics (κ) were calculated to measure the strength of agreement with 0.61-0.80 being "substantial", 0.41-0.60 "moderate", 0.21-0.40 "fair", and less than 0.20 "slight" [18].

Results
Of the cohort of 148 participants, 136 completed the two administrations of the AD-CSIROFFQ and were included in the reliability study analysis. This cohort included 44 males (32%) and had a mean and SD age of 73.19 ± 6.28 years. All 148 individuals participated in the validation study; of these, 48 were male (32%) and the mean age was 73.20 ± 6.43 years.

Reliability Study
To investigate the reliability of the AD-CSIROFFQ, we assessed the correlation of intakes of 508 food and beverage items from two administrations of the questionnaire (Table 1). Of the 508 items, 182 were highly correlated (p < 0.001), 27 were moderately correlated (p < 0.01), and 34 were weakly correlated (p < 0.05). Sixty-six items were not significantly correlated; these items comprised mostly individual fruits and vegetables, and takeaway foods for which low consumption levels were reported in both administrations of the questionnaire. Spearman's rank correlation coefficients ranged from −0.866 for subs/wraps with seafood as the main filling (three participants reporting consumption) to 1.000 for packet soup-mushroom. One hundred and ninety-nine items could not be analysed due to their low reporting rate; 19 of these items were from the AD-related questions added to the FFQ, and included intakes of chicken and beef liver, beef and veal kidney, lamb heart, and palm oil. Intake of herbs and spices reported following completion of the AD-CSIROFFQ on two occasions was compared using binary scores assigned to "yes" or "no" answers to questions relating to whether an individual had consumed each of 19 herbs or spices in the last fortnight. Out of a possible 2584 responses, 2083 (80.61%) matched between the two administrations. The herbs and spices most commonly inconsistent in their responses were ginger, parsley, and rosemary. Only 19 of the 136 participants had all 19 answers identical.
The intake of a total of 964 dietary supplements was reported overall; 288 were reported in one FFQ administration and not the other, whilst 338 were reported in both administrations. Of the 288 reported in only one administration, 235 had a consumption frequency of "yearly" or "rarely". Of the 338 supplements reported in both administrations, 50 had inconsistent consumption frequency stated, 120 had different brands of supplements stated, and 22 had different supplement strength reported (only fish oil and krill oil had supplement strength requested). The final question concerning supplement intake was an open-ended question regarding intake of any other supplements not listed previously within the questionnaire (e.g., magnesium, iron, and probiotics): 37 participants stated that they consumed an additional supplement in both administrations of the questionnaire; 27 participants stated that they consumed an additional supplement in one administration of the questionnaire only.

Validation Study
Bivariate Pearson's correlation coefficients estimating the association of the nutrient intakes between the AD-CSIROFFQ and weighed food records ranged from −0.092 (vitamin B6) to 0.570 (alcohol). Of the 46 nutrients analysed, 41 were significantly positively correlated, with 34 at the p < 0.001 level, 5 at the p < 0.01 level, and 2 at the p < 0.05 level ( Table 2). Nutrients that were not correlated were vitamin B6, vitamin B12, sodium, zinc, and energy.
Agreement between both methods with respect to nutrient intake was assessed using Bland-Altman plots ("Level of Agreement"; Table 2). Six of 46 nutrients showed "good agreement", 29 nutrients showed "fairly good agreement", and 11 nutrients showed "poor agreement". Example plots depicting "good", "fairly good", and "poor agreement" are shown in Figure 2.
Quintiles of nutrient intake levels were also compared between the AD-CSIROFFQ and the weighed food records. The ranking analysis indicated that between 54% and 88% of subjects (mean 63%) were classified into either the same or adjacent nutrient quintiles in both methods (% in same quintile + % in adjacent quintile), with the highest percentages observed for niacin, docosahexaenoic acid, eicosapentaenoic acid, and alcohol. The two methods grossly misclassified (disagreement by three or four quintiles) between 3% and 22% of subjects (average 15%), with the highest misclassification seen for water, carbohydrate, and energy. The strength of quintile agreement between the two methods was "substantial" for 4 nutrients, "moderate" for beta carotene intake, "fair" for 7 nutrients, and "slight" for the remaining 34 nutrients (Table 3). an plots ("Level of Agreement"; Table 2). Six of 46 nutrients showed "good agreemen ients showed "fairly good agreement", and 11 nutrients showed "poor agreement". Exa depicting "good", "fairly good", and "poor agreement" are shown in Figure 2.   Bold indicates statistical significance (* p < 0.05, ** p < 0.01, *** p < 0.001); n = 148. a Uses raw data. b Uses log-transformed and energy-adjusted data. c "Good agreement": the difference between the two methods is approximately equal to one standard deviation of the average nutrient intake. "Fairly good agreement": the difference between the two methods is approximately equal to two standard deviations of the average nutrient intake. "Poor agreement": the difference between the two methods is approximately equal to three standard deviations of the average nutrient intake. Abbreviations: CSIRO, Commonwealth Scientific and Industrial Research Organisation; FFQ, food frequency questionnaire; SD, standard deviation; VLC N3, very long chain n-3 polyunsaturated fatty acids; WFR, weighed food record.

Discussion
The reliability and comparative validity of the customised online AD-CSIROFFQ were assessed by comparing actual food intake data from the AD-CSIROFFQ at two administrations (reliability), and by comparing nutrient intake documented using four-day weighed food records with data generated from the first administration of the AD-CSIROFFQ (validity).
Of the 309 food and beverage items that had sufficient consumption levels for analysis of reliability, over 78% were significantly correlated between questionnaire administrations. The items lacking significant correlation were mainly takeaway foods and individual fruit and vegetable items that were consumed at low levels. These results indicate that the overall repeatability, and thus reliability, of the AD-CSIROFFQ is good.
Validity of an FFQ is the degree to which the instrument measures the diet of the subjects it was designed to study. To determine validity, we used two methods to assess agreement of absolute nutrient intake between the AD-CSIROFFQ and four-day weighed food records. First, Pearson's correlation coefficients showed that 41 of the 46 nutrients analysed were significantly positively correlated, with 34 of these highly correlated, providing evidence that intake levels in the AD-CSIROFFQ are associated with intakes documented via the weighed food records. Second, Bland-Altman plots assessing individual agreement of nutrient intakes between the two methods indicated that 35 of the 46 nutrients showed "good" or "fairly good" agreement.
We also examined the ranking of individuals into quintiles of nutrient intake and compared quintiles between the AD-CSIROFFQ and weighed food records. The AD-CSIROFFQ demonstrated an acceptable ability to rank subjects into the same or adjacent quintile as the weighed food records. On average, for all nutrients, 63% of participants were classified into the same or adjacent nutrient intake quintile, with the highest observed agreement of 88% for alcohol intake. On average, only 15% of participants were grossly misclassified by three or four quintiles, with the lowest value of 3% again for alcohol intake. Our quintile agreement results are comparable to those reported in a previous validation study using the paper-based CSIROFFQ in 62 women aged 31-60 years, where approximately 70% of reported intakes were in the same or adjacent quintile [6]. Yet, our group level agreement results are lower than values reported in a validation study of 14-year-old adolescents (n = 785), where the percentage agreement was 80%-90%. Notably, however, the adolescent study utilised a three-day weighed food record as the reference tool, and data were assigned to tertiles rather than quintiles [3]; these methodological differences increased the likelihood of group level agreement in the adolescent study.
In the present study, due to the quintile agreement being similar to that reported in other studies [3,6], we conclude that the AD-CSIROFFQ is a suitable tool for determining high or low intakes of nutrients, thereby yielding valuable information at the group level. For epidemiological purposes, ranking of individuals may be more relevant than determining absolute intakes. FFQs are commonly used in studies to examine the association between diet and chronic diseases [2], and for these studies, the ability of the FFQ to rank individuals on usual nutrient intakes may be more important than the estimation of absolute intake. For example, dietary patterns utilise individuals' intake rankings to investigate the association between diet and chronic diseases; dietary patterns frequently use median intake values to assign individuals to a binary value for inclusion into a dietary pattern score.
Overall, the evidence from the current study, particularly from agreement levels, leads to the recommendation that the AD-CSIROFFQ should be used selectively if measuring absolute nutrient intakes at the individual level. Specifically, our results provide evidence that the AD-CSIROFFQ measures VLC N3, alpha linolenic acid, docosahexaenoic acid, and caffeine accurately, based on good agreement and significant correlations, and a further 27 nutrients reasonably accurately, based on fairly good agreement and significant correlations; therefore, the AD-CSIROFFQ is a useful tool in this respect. Available carbohydrate, sugars, and dietary fibre are important nutrients for disease research, and whilst there was high correlation (p < 0.001) of intakes between the AD-CSIROFFQ and weighed food records, poor agreement was observed in the Bland-Altman plots, suggesting that one should proceed with caution in utilisation of the AD-CSIROFFQ for measurement of absolute intake of these nutrients.
Weighed food records are acceptable gold standards because they are thought to provide more accurate and representative food intake data within the period covered by the FFQ. It is important that the errors of both methods are independent to avoid spuriously high estimates of validity. Among the available and feasible comparison methods for validating an FFQ, diet records are likely to have the least correlated errors. The main source of error in the reference method is the possibility of a change in diet due to the measurement process itself, despite requests that participants consume their normal diet, and reassurance from all that this was the case. Conversely, the main sources of error in the questionnaire are restrictions imposed by a fixed list of foods, memory of foods consumed, interpretation of questions, and the assumption of average serving sizes for most foods. By contrast, diet records are collected prospectively and do not depend on memory. They allow a more direct assessment of portion sizes, and errors in interpretation relate to the dietitian coding the records rather than to the subjects. Agreement between the two methods of dietary data collection may be affected by reporting of items in the FFQ which were not consumed during the four-day weighed food record or were consumed by few people (e.g., specific types of fish and breakfast cereals). The day of the week may also influence the result of validation studies. In our study, the greatest proportion of records was collected on weekdays (75%) with 25% collected on a weekend day; this division was deliberate, as it is perceived that usual dietary intake occurs during the week with greater variation observed at weekends [1,2]. Irrespective of the limitations described above, the present study provides evidence of acceptable reliability and validity of the AD-CSIROFFQ as a measure of dietary intake, particularly in the AD field for Australian cohorts. The adequate correlations observed between test-retest responses suggest that the AD-CSIROFFQ is reliable. The agreement levels with respect to the weighed food records for a number of nutrients shows validity in terms of their individual intake, with all intakes being acceptable for ranking of individuals on a group level, and thus providing an acceptable assessment of long-term dietary intake in older adults. In summary, the AD-CSIROFFQ makes an important contribution to the tools available for assessing usual dietary intake in older adults with respect to AD research.