Next Article in Journal
Cross-Sectional Study of the Prevalence of Cobalamin Deficiency and Vitamin B12 Supplementation Habits among Vegetarian and Vegan Children in the Czech Republic
Previous Article in Journal
The Effects of Nutritional Interventions on the Cognitive Development of Preschool-Age Children: A Systematic Review
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Nutrients
Volume 14
Issue 3
10.3390/nu14030537
nutrients-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Validating Accuracy of a Mobile Application against Food Frequency Questionnaire on Key Nutrients with Modern Diets for mHealth Era

by
Joyce D. Kusuma
1,2,*,
Hsiao-Ling Yang
3,
Ya-Ling Yang
3,
Zhao-Feng Chen
4 and
Shyang-Yun Pamela Koong Shiao
5,*
1
Heritage Victor Valley Medical Group, Big Bear Lake, CA 92315, USA
2
Mt San Antonio Gardens, Pomona, CA 91767, USA
3
School of Nursing, College of Medicine, National Taiwan University, Taipei 10051, Taiwan
4
Department of Nursing, Yuanpei University of Medical Technology, Hsinchu 30015, Taiwan
5
Center for Biotechnology and Genomic Medicine, Medical College of Georgia, Augusta University, Augusta, GA 30912, USA
*
Authors to whom correspondence should be addressed.
Nutrients 2022, 14(3), 537; https://doi.org/10.3390/nu14030537
Submission received: 22 November 2021 / Revised: 21 January 2022 / Accepted: 24 January 2022 / Published: 26 January 2022
(This article belongs to the Section Nutritional Policies and Education for Health Promotion)
Browse Figures
Versions Notes

Abstract

In preparation for personalized nutrition, an accurate assessment of dietary intakes on key essential nutrients using smartphones can help promote health and reduce health risks across vulnerable populations. We, therefore, validated the accuracy of a mobile application (app) against Food Frequency Questionnaire (FFQ) using artificial intelligence (AI) machine-learning-based analytics, assessing key macro- and micro-nutrients across various modern diets. We first used Bland and Altman analysis to identify and visualize the differences between the two measures. We then applied AI-based analytics to enhance prediction accuracy, including generalized regression to identify factors that contributed to the differences between the two measures. The mobile app underestimated most macro- and micro-nutrients compared to FFQ (ranges: −5% for total calories, −19% for cobalamin, −33% for vitamin E). The average correlations between the two measures were 0.87 for macro-nutrients and 0.84 for micro-nutrients. Factors that contributed to the differences between the two measures using total calories as an example, included caloric range (1000–2000 versus others), carbohydrate, and protein; for cobalamin, included caloric range, protein, and Chinese diet. Future studies are needed to validate actual intakes and reporting of various diets, and to examine the accuracy of mobile App. Thus, a mobile app can be used to support personalized nutrition in the mHealth era, considering adjustments with sources that could contribute to the inaccurate estimates of nutrients.

1. Introduction

A dietary intake of key essential nutrients is important to promote and maintain human health and to reduce health risks across the life span, which can be achieved with a personalized diet [1,2,3,4,5,6]; however, significant proportions of the population do not consume the recommended daily intake of essential nutrients [7,8]. Inadequate intake of essential macro- and micro-nutrients can lead to mortality, morbidity, and create a burden on healthcare systems [1,2,3,4,5,6]. With the accessibility of modern smartphones, the essential micro-nutrients can be assessed conveniently through internet-based health applications (apps) to enhance healthy nutrigenomics and epigenetics [9,10,11]. In addition to energy-producing macro-nutrients (energy nutrients) of carbohydrates, protein, and fats [12,13], micro-nutrients such as folate (B9), cobalamin (B12), choline, and amino acids, including methionine and glycine, are methyl donors essential for DNA methylation in nutrigenomics pathways [4,14,15,16,17]. Hence, accurate measurement tools in assessing macro- and micro-nutrients are critical for personalized nutrition to improve health outcomes across vulnerable populations [18,19].
The Food Frequency Questionnaire (FFQ) has been comprehensively used to examine the associations between diets and disease risks across populations in many countries with its variations of diets [19,20,21,22,23]. The FFQ includes the frequency and portion size of foods and beverages consumed over time (30 days to 12 months) with additional items to adjust fat intakes from various sources [22,24,25,26]. FFQ is advantageous for its low-cost, relative ease of use, ability to measure dietary intake over time, and ability to measure nutrition consumption in large populations [19,21,26]. FFQ was validated as a dietary measure [19,20,21,22,23,24], against 1-day [21], 2-day [23,24], 3-day [20,22], and 6-day [19] dietary records with the caloric ranges of 500–5000 [27,28], in association with disease risk (506 colorectal cancer patients and 673 controls aged 20–74 years) [27]. As an example, FFQ was validated against a 4-day weighted food record with a convenient sample (female participants aged 16–45 years of Māori, Pacific, or European ethnicity); therefore, this information might not be generalized to other populations [28]. The FFQ presented greater estimates on macro- and micro-nutrients compared to the 3–4 day diet records on macro-nutrients (10–20%) and micro-nutrients (10–30%), particularly fat-soluble Vitamin E (15–30%) and Vitamin A (>50%) [28,29]; people who might consume extreme ranges of calories, <500 and >3500 calories per day, were cautioned for possible inaccurate reporting on dietary intakes [4,12,22,30].
There are numerous diets across the world in the modern era. The culture, climate, soil characteristics, and socio-economic status have nurtured the various traditional diets, including examples in the Mediterranean, Italian, convenient American, Chinese, or Japanese diets [31,32]. With urbanization, globalization, and the aging population, convenient fast foods [33,34], high school foods [8], smoothies [35,36,37,38], liquids [35,39], and dozens of other diet types have been influenced by immigration [31]; however, many diets are reported with insufficient essential nutrients; many Western diets could contain excessive empty calories [40] and rice-based Eastern diets would have insufficient zinc and iron [41]. Accurate assessment of nutrient intakes and more conveniently accessible tools such as mobile apps are critical for personalized nutrition [42].
Bland–Altman plots and correlation coefficients are commonly used univariate analytics to verify the accuracy of a new measure against another established measure [43,44]. The Bland–Altman plots illustrate the % differences between the two measures, where variability increases with increased differences [43,45,46,47]. The correlation coefficients can illustrate the strengths of associations between two measures [45,47]. Additionally, multiple regression analyses could be used to explore the sources of differences between the two measures [4,12]. Newly available artificial intelligence (AI) machine-learning-based multivariate analytics, such as the generalized regression (GR) method, can be used to identify significant factors that could contribute to the differences between two measures with added analytic criteria to enhance prediction accuracy [4,48,49].
The trend on the common use of smartphones and with most mobile-phone users accessing Internet-based health apps, mobile apps might be used to facilitate personalized nutrition in preparation for the mobile health (mHealth) era via an accurate assessment of essential micro-nutrients [42,50,51]. The mHealth apps with data analytics platforms might also help improve access to quality nutrition data for a faster assessment of key micro-nutrients with intakes on fruits, vegetables, fats, and sugar-sweetened beverages to achieve personalized nutrients [52,53,54,55]. As mobile apps offer more versatility with lower respondent burden and faster feedback, they could be more cost-effective in facilitating self-monitoring of health behaviors to improve healthy eating habits [55,56,57,58,59]. The correlations between the mobile app and 24-hour recalls were moderate to high (0.3–0.95) for macro-nutrients, and the difference between the two varied (5 to >50% difference) for macro-nutrients [60,61,62]. Specifically, using 5–10% difference as criteria, mobile apps were not recommended as replacements for the reference method (1–2 days dietary recall) on macro-nutrients of calories, protein [60], and fat [60,61]. Compared to the controlled diets, a mobile app log significantly underestimated calories by 32%, protein by 56%, and fat by 68%, whereas 24-h dietary recalls significantly underestimated fat by 23% [62]. None of the prior studies validated mobile apps in assessing micro-nutrients for personalized nutrition. As a proof-of-concept, we added predictive modeling to examine the sources of the differences in assessing nutrients between the two measures. Thus, in this study, we integrated AI machine-learning-based analytics to validate the accuracy of a mobile app against FFQ on assessing key macro- and micro-nutrients across various modern diets.

2. Materials and Methods

We examined 135 modern human diets of real consumers taken by populations derived from prior studies [4,12,63]. Based on our research team’s [4,12,63] and other studies [22,30], under-reporting dietary intake is a common behavior. Validation with model diets can provide a more accurate estimate of nutrients compared to human reporting and deviating from needed intakes. While simulation with model diets may not represent actual human intake, validation of dietary intake with model diets can provide more accurate and controlled estimates of nutrient intakes [12,64]; therefore, we categorized human diets based on possible liquids and solid foods to enhance the delivery of macro- and micro-nutrients. Baseline daily recipes with variations of added calories, proteins, vegetables, fruits, or fats for all categories were included in these diets. We included four categories of modern diets as (1) liquid diets [35,39]; (2) convenient diets (canned food, high-school café diet, and fast foods) [8,33,34]; (3) ethnic diets of Western (American, Mexican, Italian, and Mediterranean) and Eastern (Japanese, Chinese, and Korean) origins [31,32], and (4) smoothies added to these diets [35,36,37,38].
We examined the macro- and micro-nutrients based on the National Institute of Health’s (NIH) dietary nutrients to meet the needs of 97–98% of healthy adults across all demographics in the US [13] with both the FFQ and the mobile app. The differences between the two measures were assessed across all identified macro- and micro-nutrients. Macro-nutrients included energy-producing carbohydrates, protein, and total fat, as well as saturated fat, cholesterol, and fiber. Micro-nutrients including B vitamins (B1 thiamin, B2 riboflavin, B3 niacin, B6 pyridoxine, B9 folate, and B12 cobalamin), vitamin C, vitamin A, vitamin D, vitamin E, choline; minerals (zinc, calcium, magnesium, iron, and sodium); essential amino acids methionine and glycine [13]. Micro-nutrients that are fat-soluble include vitamins A, D, and E; water-soluble micro-nutrients include vitamins Bs and C. Foods containing meat-based protein also have fat, saturated fat, and cholesterol with high content of vitamin B12, methionine, and glycine. Foods containing carbohydrates also have fiber with high content of vitamins A, B9, and C [13].

2.1. Selected Modern Human Diets

As many diets were reported with insufficient essential nutrients [8,40,41], we have categorized liquids and smoothies [35,36,37,38,39] as important modern diets in healthcare and community settings. Additionally, convenient diets [33,34] are common modern diets in addition to various ethnic diets [31,32]. The following sections present the four categories of modern human diets, including liquid diets, convenient diets, ethnic diets, and smoothie-added diets.
Liquid diets are most commonly used by frail elders or patients with gastrointestinal (GI) disorders or recovering from surgery and illness, or in palliative care settings for terminally ill cancer patients in modern healthcare settings [35,39]. Additionally, liquid diets are essential for humans to maintain basic hydration, and healthy foods are necessary to maintain GI motility and for the microbiome in the human GI system to enhance immune functions for human health [65,66]. The liquid diets evaluated in this study embodied items such as jello, ice pops, teas, coffee, Gatorade, Ensure, fruit juices, vegetable juices, and soups of various kinds, including pure liquids to cream soups, and cream of wheat commonly provided in healthcare settings.
Convenient diets provide access to various nutrients in modern industrial productive societies. The convenient diets evaluated in this study included subtypes of canned-food diets, high-school café diets, and fast-food diets [8,33,34]. The canned-food diets embodied canned soups, corn, beans and peas, potatoes, white bread, ramen noodles, frozen vegetables, and hot dogs. The high-school diets typically contained high calories food mostly associated with increased sodium and saturated fat [8], including school cafeteria items such as pizza, cheese sticks, mini cheeseburgers, corn dogs, grilled cheese sandwiches, chicken wings, and chicken tenders. The fast-food diet encompasses processed foods rich in calories, fat, sodium, and sugar [33,34]. The fast-food diets contained popular value-priced items such as chicken nuggets, fried chicken sandwiches, hamburgers, and fries along with breakfast foods of biscuits, English-muffin-based sandwiches, breakfast wraps, and hash browns [8,33,34].
The ethnic diets evaluated in this study included diets with American, Mexican, Italian, Mediterranean, Japanese, Chinese, and Korean influences [31,32]. The American diets included steaks, fried chicken, burgers, and salads. The Mexican diets included tortillas, tacos with carnitas, enchiladas, nachos with guacamole, tamales, breakfast burritos, quesadillas, tostadas, steak burritos, and chili soups. The Italian diets consisted of lasagna, alfredo, spaghetti, pizzas, calzones, bruschetta, pasta carbonara, and zucchini eggplant lasagna. The Mediterranean diets are rich in whole grains, fruits, vegetables, nuts, olive oil with limited meat, and could include baba ghanoush, falafel, hummus, Greek salads, pita bread, bulghur, shish kebab, and lamb. The Japanese diets contained sashimi, sushi, miso soup, edamame, udon, and tempura. The Chinese diets comprised sweet and sour pork, tofu, wonton, chow mein, duck, kung pao chicken, eggs, steamed buns, and rice. The Korean diets included Kimchi stew, white rice, bulgogi meats, mixed bowls or bibimbap, soups with tofu and seaweed [31,32].
Smoothie-added diet recipes are commonly used when people need added hydration or nutrients with activities such as exercise by athletes or others undergoing physically straining activities. The smoothies evaluated in this study included a variety of fruits and vegetables added to the modern base diets. Fruits comprised apples, mangoes, bananas, strawberries, peaches, grapes, apricots, blueberries, cranberries, and melons. The vegetables included broccoli, cabbage, kale, and spinach. Ethnic Chinese, Italian, and Mediterranean fruit smoothies included ginger roots, baby spinach, beet juice, Japanese matcha smoothies, and Korean multigrain smoothies [35,36,37,38].

2.2. Dietary Measures and Nutrient Intakes

We assessed the selected modern diets using a 144-item for 30-day with FFQ, a 12-page questionnaire with complexities of content adjustments using frequencies and portions sizes of foods [4,67]. The FFQ consisted of three sections: (1) usual food choices, (2) usual food and beverage use, and (3) summary questions. The usual food choices section contained 13 additional questions to adjust fat intakes through detailed inquiry on food preparation methods and added fats both in cooking and at the table. The usual food and beverage use section included 128 foods or food groups with the amount and frequency of foods such as cereals, bread, snacks items, meat, fish, eggs, pasta, mixed dishes, soups items, dairy products, vegetables, grains, fruits, beverages, and alcohol. In the summary questions section, three items on intakes of fruits, vegetables, and fats added to foods and used in cooking were included [67,68]. The nutrient database used to quantify the FFQ was derived from the University of Minnesota Nutrition Coordinating Center (NCC) [69].
Dietary intakes on nutrients were also analyzed for a 3-day average using a mobile app that was developed to assess daily nutrient intakes (GB HealthWatch, San Diego, CA, USA) [42,70]. The mobile app was developed by a digital health and nutritional genomics company that focused on gene–diet interactions and research tool development (https://healthwatch360.gbhealthwatch.com, accessed on 1 June 2021). In 2016, NIH funded the company to develop a mobile app for personalized genetics-based diets to record daily food consumption for dietary management and to empower users to take control of healthy eating for disease prevention. The app had the capacity to extract 30 essential nutrients by analyzing users’ food logs and provided reports on the total amounts and percentages of suggested daily values of nutrients. Furthermore, it offered personalized recommendations based on dietary guidelines and users’ goals to manage weight and to prevent chronic conditions such as metabolic syndrome and Alzheimer’s disease [42,70]. Thus, for each diet, we performed a nutrient analysis using both 30-day FFQ and mobile app by mathematically dividing the 30-day data into a 3-day dietary diary. For example, 1 oz (30 g) of fat intake every week would yield 13 g for 3 days or 4.3 g of fat for 1 day. This calculation is comparable dietary intakes between the two measures, while realistically, many human subjects might neglect to report the minute dietary intakes; therefore, the 3-day dietary diary in this study could be more detailed than human reporting. Before analysis, data entry was checked independently by two research team members for accuracy.

2.3. Data Analysis

All data were analyzed using JMP version 13.0.0 statistical software [64,71,72] (SAS Institute Inc., Cary, NC, USA). We first assessed bias and agreement, and then GR to predict the source of differences between the two measures. Means and standard deviations (SD) for all nutrients [73] were calculated for both mobile App and FFQ. Then, agreement and bias analyses using mean % differences and standard errors (SE) of the differences between the two measures were compared across all nutrient parameters. Pearson correlation coefficients (r) were used to measure the strengths of associations between the two dietary measures (r > 0.80 as very strong, r < 0.60 as moderate) [43]. The Bland–Altman plots were used to visualize the mean % differences with the limits of agreement (LoA: mean difference ± 2 SD) between the two measures [28,72] for variability of the differences, with a good agreement if 95% and greater of the agreement being within ± 2 SDs [20,21,64]. The alpha for all analyses was set at 0.05 for the significance level.
We then utilized GR models to predict the differences between the app and FFQ in assessing essential nutrients by progressively including related factors in the dataset. The analytics and rationales have been reported earlier [4,12,63] and are summarized in the following. We added predictive modeling to examine the sources of the differences between the two measures [63]. JMP software provided default logistic regression (LR) as a baseline and exploratory model to predict dependent variables in categorical values. Following LR, other models could be selected for validation (Elastic Net) and associated validation methods (Leave-One-Out, Validation Column) for further analysis. In effect, GR estimation methods are a confirmatory model to predict the accuracy with a lower misclassification rate for minimal prediction error [4,12,64,74]. We incorporated Elastic Net models for their capacity to handle datasets with many variables, balancing potential interactions from various domain factors [75]. It is important to point out that GR eliminates certain predictors to avoid over-fitting. Conventional statistical procedures are limited by the sample size [63]. If the number of parameters to be estimated exceeds the degrees of freedom, the regression model would be highly unstable. The AI-based analytics use partition in iteration by resampling with machine learning [63]. Both AICc validation and LOO cross-validation methods are effective methods for small sample sizes and handling multiple domains based on the logic of resampling [4,72]. In resampling, observed biases are corrected by such repeated analyses on random subsets [63]. This AI machine learning approach is superior to conventional statistics, including the baseline logistic regression analyses that tend to yield an overfitted model [4,63]. We used Leave-One-Out (LOO) for validations to select significant factors within domains of caloric ranges, the effect of differences from energy nutrients on the differences on micro-nutrients, and diet types [48,49]. Once significant factors were determined through LOO within each domain, we then used AICc validation columns to confirm how well the model fits with unbiased prediction [49].
We identified related factors that could affect the differences between FFQ and the app per categories of (1) caloric ranges (<1000, 1000–2000, or >2000) of total calories; (2) effects of differences from energy-producing carbohydrate, protein, and fat; (3) diet types. We used an 80/20 split for training and validation sets for predictive modeling to further identify significant predictors that contributed to the difference between the app and FFQ. The best models on predicting the sources contributing to the differences were based on three accuracy criteria, including lowest AICc (fitter for more precise model), lower misclassification (smaller for accuracy), and higher area under the receiver operating characteristics (ROC) curve (AUC, >0.80) [12,64]. The prediction and interaction profilers were used to visualize the potential significant interactions among the factors. If significant interactions exist in the models, the interaction terms would be included in the model testing [48,49].

3. Results

3.1. Agreement and Difference: The Bland and Altman Method

We examined the data using the Bland and Altman method to assess the agreement (% differences), bias (SE), and correlations between FFQ and mobile app per three caloric ranges (<1000, 1000–2000, and >2000) and all assessed nutrients (Table 1) using 5–10% differences as criteria [60]. Compared to FFQ, the mobile app presented acceptable differences (<5%) with a caloric range of <1000 but underestimated the caloric range of 1000–2000 calories (>5%) and >2000 (>10%). For macro-nutrients, the app underestimated total calories and protein (>5%); fat, saturated fat, and cholesterol (>10%); overestimated fiber (>10%); presented acceptable differences with carbohydrate (<5%). For micro-nutrients, the app underestimated choline, vitamin D (>5%); vitamin B12, methionine, glycine, vitamin E, zinc, and sodium (>10%); overestimated vitamins A and C (>5%); presented acceptable differences with other B vitamins including B1, B2, B3, B6, B9, and calcium, magnesium, and iron. Thus, we found good agreement between the two measures on many micro-nutrients but not protein-based micro-nutrients that are essential in nutrigenomics pathways.
The bias (SE) increased between the two measures with increased caloric ranges was smallest for 1000–2000 caloric range (1.44), larger for <1000 (2.3), and largest for >2000 (5.91); indicating the greater spread of means. The bias was greater (>2) for fiber, vitamins A and C, calcium, and sodium when compared to all other nutrients. The correlations between the two measures were strong for the mid-range of 1000–2000 caloric range (0.78), moderate for higher calories of >2000 (0.64), and lowest for <1000 calories intake (0.54) (average 0.65) (all p < 0.001). The correlations are strong for most nutrients (average for macro- 0.87 and 0.84 for micro-nutrients) except for calcium (0.53).
In addition, we used Bland–Altman plots to visualize the difference and LoA between mobile app and FFQ. We presented side-by-side comparison of correlation (left panel) and Bland–Altman plots (right panel) on example macro-nutrients of calories (Figure 1a,b), fat (Figure 2a,b), protein (Figure S1a,b), carbohydrate (Figure S2a,b); and micro-nutrients of folate (Figure 3a,b) and cobalamin (Figure 4a,b). Correlation plots presented good agreement between the two measures for all nutrients presented in these figures. Bland–Altman plots displayed a wide variability of differences for both macro- (calories, fat, protein, carbohydrate) and micro-nutrients (folate, cobalamin). Bland–Altman plots further illustrated the underestimation by the app compared to FFQ on total calories (−5.34%, Figure 1b), fat (−17.55%, Figure 2b), protein −9.8%, Figure S1b), and cobalamin (−18.86%, Figure 4b); but higher estimation on carbohydrate (4.55%, Figure S2b) and folate intakes (3.2%, Figure 3b).
Furthermore, we assessed the differences between the mobile app and FFQ with nutrients in association with caloric ranges and diets (Table 2). For total calories with caloric ranges, the app, in comparison to FFQ, presented acceptable differences with <1000 calories (<5%); but underestimated with 1000–2000 calories (>5%) and >2000 calories (>10%) (Table 2). Additionally, the app overestimated total calories with pure liquid diet (>10%), underestimated total calories with high school, Mediterranean, Chinese, and smoothie-added diets (>5%); also canned food and fast-food diets with greater bias (>10%), but presented acceptable differences (<5%) with FFQ on American, Mexican, Italian, Japanese, and Korean diets. Likewise, the app overestimated carbohydrates with a caloric range of <2000, but underestimated >2000 caloric range (>5%); with Italian, Japanese, Korean diets (>5%); pure liquid, Mexican, Chinese diets (>10%); but presented acceptable differences (<5%) with other diets (canned food, high school, fast food, American, Mediterranean, and smoothie-added). Additionally, the app underestimated protein with caloric ranges of 1000–2000 (>5%), and caloric ranges of <1000 and >2000 (>10%); pure liquid, high school, fast food, American, Mexican, Italian, Japanese, Chinese diets (>5%); canned food, Mediterranean, Korean, and smoothie-added diets (>10%). The app underestimated fat with all caloric ranges (>10%) and all diets except liquid diets. For folate, the app presented overestimation with a caloric range of 1000–2000, underestimation with a caloric range of >2000, and acceptable differences (<5%) with a caloric range of <1000; underestimation with Mediterranean diets; overestimation with American and Italian diets (>5%); pure liquid, Mexican, and Chinese diets (>10%); but acceptable differences (<5%) with canned food, high school, fast food, Japanese, Korean, and smoothie-added diets. For cobalamin, the app presented underestimation with all caloric ranges (>10%) and all diets except fast-food diets. Additionally, the app underestimated protein-based and fat-based nutrients, including saturated fat, cholesterol, methionine, choline, glycine, vitamins B12 and E; but overestimated carbohydrate-based nutrients including fiber, vitamins B9, A, and C (Tables S1–S4).

3.2. Predictive Modeling for the Difference of Mobile App against FFQ: Generalized Regression Analysis

For predictive modeling, we progressively examined significant factors per individual domains of caloric ranges (coded as one of the three versus the other two categories for <1000, 1000–2000, and >2000), energy nutrients, and various diets. We included the significant factors of all domain factors in the final combined model (Table S5 progression examples for total calories, Table S6 for folate, Table S7 for cobalamin). For total calories, differences on 1000–2000 over other caloric ranges, carbohydrate, and protein were significant contributing factors to the difference between two measures (misclassification 0.04, AICc 18.2, and AUC 0.99) (Table 3), baseline LR model on the left panel and GR model validation on the right panel. As an example, Figure 5 illustrates the AUC curve with closer to 100% sensitivity and 100% specificity for the accuracy of the selected model with total calories [48]. Through the progression analyses, we noted a higher AICc and less precise model by including the additional factor of the Japanese diet, thus a less favorable model than the selected model (Table S5).
Folate and cobalamin are the most representative essential micro-nutrients needed in the nutrigenomics pathways. Factors that contributed to the differences in folate between the two measures included caloric range (1000–2000 versus other two categories), carbohydrate, fiber, and Mediterranean diet (misclassification 0.11, AICc 30.7, and AUC 0.91) (Table 4). Through the progression analyses, we noted a higher AICc and lower AUC, thus, less precise and less favorable models than the selected model by including additional factors of fiber and Chinese diet (Table S6).
For cobalamin, significant factors that contributed to the differences between the two measures included caloric range (1000–2000 versus other two categories), protein, and Chinese diet (misclassification 0.27, AICc 35.8, and AUC 0.79) (Table 5). With the progression analyses, we observed a higher AICc by including additional factors of fat and Korean diet, which presented less favorable models than the selected model (Table S7).
We also examined significant factors between the two measures for other nutrients with the progression analyses and summarized final models for other nutrients (carbohydrate, protein, fat, saturated fat, cholesterol, and fiber in Table S8; thiamin, riboflavin, niacin, pyridoxine, choline, glycine, and zinc in Table S9; and vitamins A, C, D, E, calcium, magnesium, iron, sodium in Table S10). Significant factors that contributed to the difference between the two measures on carbohydrate included calories, fat, fiber, and Mediterranean diet; for protein: caloric range (1000–2000 versus others), calories, and fat; on fat: calories, saturated fat, and Japanese diet; on saturated fat: fat, cholesterol, and Japanese diet; for cholesterol: protein, saturated fat, and Japanese diet; on fiber: carbohydrate, Japanese diet, and Chinese diet (misclassification 0.04–0.23, AICc 20.4–33.8, AUC 0.74–0.98). Similarly, significant factors for thiamin included caloric range (<1000 versus others), saturated fat, fiber, and canned-food diet; for riboflavin: protein, fiber, and canned-food diet; for niacin: calories, fiber, canned food, and Italian diet; for pyridoxine: calories, fiber, and Japanese diet; for choline: calories, protein, and canned-food diet; for glycine: protein, Mexican, and Japanese diet; and for zinc: protein, canned food, fast-food, and Japanese diet (misclassification 0.04–0.29, AICc 20.5–36.53, AUC 0.82–0.96). Additionally, significant factors for vitamin A included fat, saturated fat, and fast-food diet; vitamin C: caloric range (<1000 versus others), fiber, and canned-food diet; on vitamin D: protein, fat, canned food, and Chinese diet; for vitamin E; carbohydrate, fat, and cholesterol; for calcium; carbohydrate, cholesterol, American, and Italian diet; on magnesium: carbohydrate, fat, cholesterol, and fiber; for iron: calories, protein, fiber, and Italian diet; on sodium: protein, fiber, and high school diet (misclassification 0.18–0.36, AICc 29.6–39.9, AUC 0.75–0.96). For diet types, the Japanese diet was a common contributing factor for the differences between the two measures on fat, saturated fat, cholesterol, fiber, riboflavin, pyridoxine, glycine, and zinc (Tables S8–S10). We did not explicitly test the model for methionine, as methionine was purely dependent on protein. The interaction profiler plots did not present any significant three-way interactions in the final models for all nutrients.

4. Discussion

In preparation for personalized nutrition, we validated the accuracy of a mobile app against the FFQ as a reference method in assessing key nutrients with various modern diets. Using 5–10% difference as criteria [60], compared to FFQ, the app presented acceptable estimation with a caloric range of <1000 but not for total calories and caloric ranges of 1000 or higher with a greater bias for calories of >2000. Specifically, for macro-nutrients, the app presented acceptable estimation for carbohydrates but underestimated protein and greatly underestimated fats (fat, saturated fat, cholesterol). For micro-nutrients, the app presented with acceptable estimation for most B vitamins (B1, B2, B3, B6, and B9) and some minerals (calcium, magnesium, and iron), but underestimated choline and vitamin D and greatly underestimated vitamin B12, methionine, glycine, vitamin E, zinc, and sodium; and greatly overestimated vitamins A and C. Prior research indicated that mobile apps were not recommended as replacements to reference method (1–2 day dietary recall) on macro-nutrients of calories, protein [60], and fat [60,61]. With this study, we further demonstrated that the mobile app presented with greater bias with increased calories with FFQ as a reference method, while biases are similar with total calories, protein, and fat. Although correlations between the two measures were strong, lower correlations were observed with greater calories of the diets and calcium.
Furthermore, we demonstrated that caloric ranges and various diets may be used to examine additional sources that might have contributed to the differences between the two measures. Overall, the app underestimated all macro- and micro-nutrients (except vitamins B1 and A) with calories >2000, underestimated protein-based and fat-based nutrients, but overestimated carbohydrate-based nutrients for calories of ≤2000. Additionally, we noted that with various diets, the app underestimated foods that contain meat-based protein and fat (saturated fat, cholesterol, vitamins B12 and E, methionine, choline, glycine), but overestimated foods that contain carbohydrate nutrients (fiber, vitamins B9, A, and C). We used FFQ as the reference method for its capacity to adjust fat intakes [22,24,25,26]. Hence, additional adjustments with caloric ranges and diet types could be considered for the mobile app. Previous studies used <500 and >3500 calories as the limits for valid caloric ranges of FFQ that were extreme for human health [4,12,22,30]. With the realistic caloric intakes for human health, the challenges of accuracy remain for the mobile app.
As a proof-of-concept, we added predictive modeling to examine the sources that contributed to the differences between the two measures [4,63,64]. We used AI machine-learning-based analytics with criteria to enhance the accuracy of prediction. Thus far, no prior studies have validated the accuracy of a mobile app against FFQ using predictive modeling. With confirmatory predictive modeling, we demonstrated that caloric ranges and source of micro-nutrients in relation to macro-nutrients (folate source of difference from carbohydrates versus cobalamin source of difference from protein) contributed to less accurate estimates of nutrients for the mobile app in reference to FFQ. A specific diet, such as Mediterranean (folate) and Chinese (cobalamin) diets, might further contribute to the differences between the two measures; therefore, caloric ranges and source of macro-nutrients might be considered for adjustments to accurately measure micro-nutrients.
In summary, we noted that fat and protein were major sources for the differences between FFQ and mobile app, with specific caloric ranges and diet types contributing to the differences. The predictive modeling further substantiated the findings that fat and protein were the major sources of differences between the two measures, with specific caloric ranges and diet types contributing to the differences. The challenges remain for the mobile app to accurately measure macro- and micro-nutrients. Further adjustments with caloric ranges, source of nutrients, and diets might help to improve the accuracy of the mobile app. In addition, future studies may include various diets across different human populations with accurate measures on dietary intakes and nutrients with the mobile app. In conclusion, the mobile app has the capability to support personalized nutrition in the mHealth era with the use of AI-based analytics, integrating potential contributing factors to improve its accuracy.

Supplementary Materials

The following are available online at https://www.mdpi.com/article/10.3390/nu14030537/s1, Table S1: Bias and agreement between Mobile Application and Food Frequency Questionnaire per domains of caloric ranges, energy nutrients, and various diets for major nutrients (N = 135). Table S2: Bias and agreement between mobile application and Food Frequency Questionnaire per domains of caloric ranges, energy nutrients, and various diets for vitamins Bs (N = 135). Table S3: Bias and agreement between mobile application and Food Frequency Questionnaire per domains of caloric ranges, energy nutrients, and various diets for methyl-donors and vitamins A, D, and D (N = 135). Table S4: Bias and agreement between mobile application and Food Frequency Questionnaire per domains of caloric ranges, energy nutrients, and various diets for vitamin E and minerals (N = 135). Table S5: Progression on selecting significant factors contributing to the differences between mobile application and Food Frequency Questionnaire on total calories. Table S6: Progression on selecting significant factors contributing to the differences between mobile application and Food Frequency Questionnaire on folate. Table S7: Progression on selecting significant factors contributing to the differences between mobile application and Food Frequency Questionnaire on cobalamin. Table S8: Summary on significant factors contributing to the differences between mobile application and Food Frequency Questionnaire on major nutrients. Table S9: Summary on significant factors contributing to the differences between mobile application and Food Frequency Questionnaire on methyl-donors and co-factors. Table S10: Summary on significant factors contributing to the differences between mobile application and Food Frequency Questionnaire on other vitamins and minerals. Figure S1: (a) Correlation, (b) Bland and Altman plots between mobile application and Food Frequency Questionnaire for protein. Figure S2: (a) Correlation, (b) Bland and Altman plots between mobile application and Food Frequency Questionnaire for carbohydrates.

Author Contributions

Conceptualization and methodology: S.-Y.P.K.S.; acquisition and search of the literature: J.D.K., H.-L.Y. and S.-Y.P.K.S.; data entry and verification of data accuracy J.D.K., S.-Y.P.K.S., Z.-F.C. and H.-L.Y.; analysis and interpretation of data: S.-Y.P.K.S., J.D.K., Z.-F.C. and H.-L.Y.; writing, review and editing: S.-Y.P.K.S., J.D.K., Z.-F.C., H.-L.Y., Y.-L.Y. All authors have read and agreed to the published version of the manuscript ensuring integrity and accuracy.

Funding

Funding supports include the Doctoral Research Council Grants, Azusa Pacific University; Research Start-up fund from Augusta University awarded to the corresponding author.

Institutional Review Board Statement

The parent study was approved by the appropriate Human Subjects Institutional Review Boards (IRB) from the California State Committee for the Protection of Human Subjects for data access through the CCR (CPHS-12-12-1007, approved 2013–2019), and from the local educational institutions (Azusa Pacific University, approved 2013–2015; Augusta University, 806069-7, approved 2015–2018).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the parent study.

Acknowledgments

The authors thank the support from GB HealthWatch (San Diego, CA, USA) for providing the mobile app software. The authors acknowledge the contribution and assistance from Zenab Khan for coding and entering data.

Conflicts of Interest

The authors declare no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

  1. Bruins, M.J.; Van Dael, P.; Eggersdorfer, M. The role of nutrients in reducing the risk for noncommunicable diseases during aging. Nutrients 2019, 11, 85. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  2. Liu, G.; Bin, P.; Wang, T.; Ren, W.; Zhong, J.; Liang, J.; Hu, C.-A.A.; Zeng, Z.; Yin, Y. DNA methylation and the potential role of methyl-containing nutrients in cardiovascular diseases. Oxid. Med. Cell. Longev. 2017, 2017, 1670815. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  3. Mahmoud, A.M.; Ali, M.M. Methyl donor micronutrients that modify DNA methylation and cancer outcome. Nutrients 2019, 11, 608. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  4. Shiao, S.P.K.; Grayson, J.; Lie, A. Personalized nutrition—Genes, diet, and related interactive parameters as predictors of cancer in multiethnic colorectal cancer families. Nutrients 2018, 10, 795. [Google Scholar] [CrossRef] [Scilit]
  5. Cappuccilli, M.; Bergamini, C.; Giacomelli, F.A.; Cianciolo, G.; Donati, G.; Conte, D.; Natali, T.; La Manna, G.; Capelli, I. Vitamin B Supplementation and Nutritional Intake of Methyl Donors in Patients with Chronic Kidney Disease: A Critical Review of the Impact on Epigenetic Machinery. Nutrients 2020, 12, 1234. [Google Scholar] [CrossRef] [Scilit]
  6. Zeng, J.; Gu, Y.; Fu, H.; Liu, C.; Zou, Y.; Chang, H. Association between one-carbon metabolism-related vitamins and risk of breast cancer: A systematic review and meta-analysis of prospective studies. Clin. Breast Cancer 2020, 20, e469–e480. [Google Scholar] [CrossRef] [Scilit]
  7. Mertens, E.; Kuijsten, A.; Dofková, M.; Mistura, L.; D’Addezio, L.; Turrini, A.; Dubuisson, C.; Favret, S.; Havard, S.; Trolle, E. Geographic and socioeconomic diversity of food and nutrient intakes: A comparison of four European countries. Eur. J. Nutr. 2019, 58, 1475–1493. [Google Scholar] [CrossRef] [Scilit]
  8. Liu, J.; Micha, R.; Li, Y.; Mozaffarian, D. Trends in food sources and diet quality among US children and adults, 2003–2018. JAMA Netw. Open 2021, 4, e215262. [Google Scholar] [CrossRef] [Scilit]
  9. Van Horn, L.; Cornelis, M.C. US Dietary Guidance-Is it working? JAMA 2019, 322, 1150–1151. [Google Scholar] [CrossRef] [Scilit]
  10. Mullins, V.A.; Bresette, W.; Johnstone, L.; Hallmark, B.; Chilton, F.H. Genomics in personalized nutrition: Can you “eat for your genes”? Nutrients 2020, 12, 3118. [Google Scholar] [CrossRef] [Scilit]
  11. Ferguson, L.R.; De Caterina, R.; Görman, U.; Allayee, H.; Kohlmeier, M.; Prasad, C.; Choi, M.S.; Curi, R.; De Luis, D.A.; Gil, Á. Guide and position of the international society of nutrigenetics/nutrigenomics on personalised nutrition: Part 1-fields of precision nutrition. Lifestyle Genom. 2016, 9, 12–27. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  12. Shiao, S.P.K.; Grayson, J.; Lie, A. Predictors of healthy eating index and glycemic index in multiethnic colorectal cancer families. Nutrients 2018, 10, 674. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  13. National Institute of Health (NIH). Nutrient Recommendations: Dietary Reference Intakes (DRI). Available online: https://ods.od.nih.gov/Health_Information/Dietary_Reference_Intakes.aspx (accessed on 1 April 2018).
  14. Myte, R.; Gylling, B.; Schneede, J. Components of one-carbon metabolism other than folate and colorectal cancer risk. Epidemiology 2016, 27, 787–796. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  15. Niculescu, M.D.; Zeisel, S.H. Diet, methyl donors and DNA methylation: Interactions between dietary folate, methionine and choline. J. Nutr. 2002, 132, 2333S–2335S. [Google Scholar] [CrossRef] [Scilit]
  16. Amenyah, S.D.; Hughes, C.F.; Ward, M.; Rosborough, S.; Deane, J.; Thursby, S.-J.; Walsh, C.P.; Kok, D.E.; Strain, J.; McNulty, H. Influence of nutrients involved in one-carbon metabolism on DNA methylation in adults-a systematic review and meta-analysis. Nutr. Rev. 2020, 78, 647–666. [Google Scholar] [CrossRef] [Scilit]
  17. Anderson, O.S.; Sant, K.E.; Dolinoy, D.C. Nutrition and epigenetics: An interplay of dietary methyl donors, one-carbon metabolism and DNA methylation. J. Nutr. Biochem. 2012, 23, 853–859. [Google Scholar] [CrossRef] [Scilit]
  18. Bouchard-Mercier, A.; Paradis, A.-M.; Rudkowska, I.; Lemieux, S.; Couture, P.; Vohl, M.-C. Associations between dietary patterns and gene expression profiles of healthy men and women: A cross-sectional study. Nutr. J. 2013, 12, 24. [Google Scholar] [CrossRef] [Scilit]
  19. Palacios, C.; Trak, M.A.; Betancourt, J.; Joshipura, K.; Tucker, K.L. Validation and reproducibility of a semi-quantitative FFQ as a measure of dietary intake in adults from Puerto Rico. Public Health Nutr. 2015, 18, 2550–2558. [Google Scholar] [CrossRef] [Scilit]
  20. Kowalkowska, J.; Slowinska, M.A.; Slowinski, D.; Dlugosz, A.; Niedzwiedzka, E.; Wadolowska, L. Comparison of a full food-frequency questionnaire with the three-day unweighted food records in young Polish adult women: Implications for dietary assessment. Nutrients 2013, 5, 2747–2776. [Google Scholar] [CrossRef] [Scilit]
  21. Pakseresht, M.; Sharma, S. Validation of a quantitative food frequency questionnaire for Inuit population in Nunavut, Canada. J. Hum. Nutr. Diet 2010, 23, 67–74. [Google Scholar] [CrossRef] [Scilit]
  22. Ye, Q.; Hong, X.; Wang, Z.; Yang, H.; Chen, X.; Zhou, H.; Wang, C.; Lai, Y.; Sun, L.; Xu, F. Reproducibility and validity of an FFQ developed for adults in Nanjing, China. Br. J. Nutr. 2016, 115, 887–894. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  23. Zack, R.M.; Irema, K.; Kazonda, P.; Leyna, G.H.; Liu, E.; Gilbert, S.; Lukmanji, Z.; Spiegelman, D.; Fawzi, W.; Njelekela, M. Validity of an FFQ to measure nutrient and food intakes in Tanzania. Public Health Nutr. 2018, 21, 2211–2220. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  24. Athanasiadou, E.; Kyrkou, C.; Fotiou, M.; Tsakoumaki, F.; Dimitropoulou, A.; Polychroniadou, E.; Menexes, G.; Athanasiadis, A.P.; Biliaderis, C.G.; Michaelidou, A.-M. Development and validation of a Mediterranean oriented culture-specific semi-quantitative food frequency questionnaire. Nutrients 2016, 8, 522. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  25. Collins, C.E.; Burrows, T.L.; Rollo, M.E.; Boggess, M.M.; Watson, J.F.; Guest, M.; Duncanson, K.; Pezdirc, K.; Hutchesson, M.J. The comparative validity and reproducibility of a diet quality index for adults: The Australian Recommended Food Score. Nutrients 2015, 7, 785–798. [Google Scholar] [CrossRef] [Scilit]
  26. Fallaize, R.; Forster, H.; Macready, A.L.; Walsh, M.C.; Mathers, J.C.; Brennan, L.; Gibney, E.R.; Gibney, M.J.; Lovegrove, J.A. Online dietary intake estimation: Reproducibility and validity of the Food4Me food frequency questionnaire against a 4-day weighed food record. J. Med. Internet Res. 2014, 16, e3355. [Google Scholar] [CrossRef] [Scilit]
  27. Chen, Z.; Wang, P.P.; Woodrow, J.; Zhu, Y.; Roebothan, B.; Mclaughlin, J.R.; Parfrey, P.S. Dietary patterns and colorectal cancer: Results from a Canadian population-based study. Nutr. J. 2015, 14, 8. [Google Scholar] [CrossRef] [Scilit]
  28. Beck, K.L.; Houston, Z.L.; McNaughton, S.A.; Kruger, R. Development and evaluation of a food frequency questionnaire to assess nutrient intakes of adult women in New Zealand. Nutr. Diet. 2020, 77, 253–259. [Google Scholar] [CrossRef] [Scilit]
  29. Lin, P.I.; Bromage, S.; Mostofa, M.; Allen, J.; Oken, E.; Kile, M.L.; Christiani, D.C. Validation of a dish-based semiquantitative food questionnaire in rural Bangladesh. Nutrients 2017, 9, 49. [Google Scholar] [CrossRef] [Scilit]
  30. Rhee, J.J.; Sampson, L.; Cho, E.; Hughes, M.D.; Hu, F.B.; Willett, W.C. Comparison of methods to account for implausible reporting of energy intake in epidemiologic studies. Am. J. Epidemiol. 2015, 181, 225–233. [Google Scholar] [CrossRef] [Scilit]
  31. Medicine LibreTexts. Diets around the World. Available online: https://med.libretexts.org/@go/page/21470 (accessed on 1 April 2020).
  32. Waldfogel, J. Dining out as cultural trade. J. Cult. Econ. 2019, 44, 1–30. [Google Scholar] [CrossRef] [Scilit]
  33. Kupka, R.; Siekmans, K.; Beal, T. The diets of children: Overview of available data for children and adolescents. Glob. Food Sec. 2020, 27, 100442. [Google Scholar] [CrossRef] [Scilit]
  34. Li, L.; Sun, N.; Zhang, L.; Xu, G.; Liu, J.; Hu, J.; Zhang, Z.; Lou, J.; Deng, H.; Shen, Z. Fast food consumption among young adolescents aged 12–15 years in 54 low-and middle-income countries. Glob. Health Action 2020, 13, 1795438. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  35. Abikusno, N. Importance of nutrition and lifestyle for elderly during the COVID-19 pandemic. J. Biomedika Kesehat. 2020, 3, 109–111. [Google Scholar] [CrossRef] [Scilit]
  36. Batchelor-Murphy, M.K.; Steinberg, F.M.; Young, H.M. Dietary and feeding modifications for older adults. Am. J. Nurs. 2019, 119, 49–57. [Google Scholar] [CrossRef] [Scilit]
  37. Salomonsen, S.J.C.; Berland, A.; Bentsen, S.B. High-energy smoothies for patients in nursing homes. Nor. J. Clin. Nurs. Sykepl. Forsk. 2017, 12, e-62986. [Google Scholar] [CrossRef] [Scilit]
  38. Stoker, D.; Durward, C. Smoothies-Helpful or Harmful? Available online: https://digitalcommons.usu.edu/cgi/viewcontent.cgi?article=2856&context=extension_curall (accessed on 1 April 2021).
  39. Gellrich, N.C.; Handschel, J.; Holtmann, H.; Krüskemper, G. Oral cancer malnutrition impacts weight and quality of life. Nutrients 2015, 7, 2145–2160. [Google Scholar] [CrossRef] [Scilit]
  40. Malinowska, A.M.; Mlodzik-Czyzewska, M.A.; Chmurzynska, A. Dietary patterns associated with obesity and overweight: When should misreporters be included in analysis? Nutrition 2000, 70, 110605. [Google Scholar] [CrossRef] [Scilit]
  41. Zhang, C.M.; Zhao, W.Y.; Gao, A.X.; Su, T.T.; Wang, Y.K.; Zhang, Y.Q.; Zhou, X.B.; He, X.H. How could agronomic biofortification of rice be an alternative strategy with higher cost-effectiveness for human iron and zinc deficiency in China? Food Nutr. Bull. 2018, 39, 246–259. [Google Scholar] [CrossRef] [Scilit]
  42. GB Health Watch. Available online: https://www.gbhealthwatch.com/ (accessed on 1 June 2021).
  43. Saravia, L.; Miguel-Berges, M.L.; Iglesia, I.; Nascimento-Ferreira, M.V.; Perdomo, G.; Bove, I.; Slater, B.; Moreno, L.A. Relative validity of FFQ to assess food items, energy, macronutrient and micronutrient intake in children and adolescents: A systematic review with meta-analysis. Br. J. Nutr. 2020, 18, 1–27. [Google Scholar] [CrossRef] [Scilit]
  44. Sierra-Ruelas, É.; Bernal-Orozco, M.F.; Macedo-Ojeda, G.; Márquez-Sandoval, Y.F.; Altamirano-Martínez, M.B.; Vizmanos, B. Validation of semiquantitative FFQ administered to adults: A systematic review. Public Health Nutr. 2020, 24, 1–20. [Google Scholar] [CrossRef] [Scilit]
  45. Giavarina, D. Understanding bland altman analysis. Biochem. Med. 2015, 25, 141–151. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  46. Twomey, P.J. How to use difference plots in quantitative method comparison studies. Ann. Clin. Biochem. 2006, 43, 124–129. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  47. Harmouche-Karaki, M.; Mahfouz, M.; Obeyd, J.; Salameh, P.; Mahfouz, Y.; Helou, K. Development and validation of a quantitative food frequency questionnaire to assess dietary intake among Lebanese adults. Nutr. J. 2020, 19, 65. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  48. SAS Institute. Fitting Linear Models, 2nd ed.; SAS Institute Inc.: Cary, NC, USA, 2017. [Google Scholar]
  49. Cheng, H.; Garrick, D.J.; Fernando, R.L. Efficient strategies for leave-one-out cross validation for genomic best linear unbiased prediction. J. Anim. Sci. Biotechnol. 2017, 8, 38. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  50. Reiners, F.; Sturm, J.; Bouw, L.J.; Wouters, E.J. Sociodemographic factors influencing the use of eHealth in people with chronic diseases. Int. J. Environ. Res. Public Health 2019, 16, 645. [Google Scholar] [CrossRef] [Scilit]
  51. Rollo, M.E.; Haslam, R.L.; Collins, C.E. Impact on dietary intake of two levels of technology-assisted personalized nutrition: A randomized trial. Nutrients 2020, 12, 3334. [Google Scholar] [CrossRef] [Scilit]
  52. Mandracchia, F.; Llauradó, E.; Tarro, L.; Del Bas, J.M.; Valls, R.M.; Pedret, A.; Radeva, P.; Arola, L.; Solà, R.; Boqué, N. Potential use of mobile phone applications for self-monitoring and increasing daily fruit and vegetable consumption: A systematized review. Nutrients 2019, 11, 686. [Google Scholar] [CrossRef] [Scilit]
  53. El Khoury, C.F.; Karavetian, M.; Halfens, R.J.; Crutzen, R.; Khoja, L.; Schols, J.M. The effects of dietary mobile apps on nutritional outcomes in adults with chronic diseases: A systematic review and meta-analysis. J. Acad. Nutr. Diet. 2019, 119, 626–651. [Google Scholar] [CrossRef] [Scilit]
  54. Zmora, N.; Elinav, E. Harnessing smartphones to personalize nutrition in a time of global pandemic. Nutrients 2021, 13, 422. [Google Scholar] [CrossRef] [Scilit]
  55. Mayordomo-Martínez, D.; Carrillo-de-Gea, J.M.; García-Mateos, G.; García-Berná, J.A.; Fernández-Alemán, J.L.; Rosero-López, S.; Parada-Sarabia, S.; García-Hernández, M. Sustainable accessibility: A mobile app for helping people with disabilities to search accessible shops. Int. J. Environ. Res. Public Health 2019, 16, 620. [Google Scholar] [CrossRef] [Scilit]
  56. World Health Organization (WHO). eHealth at WHO. Available online: https://www.who.int/ehealth/about/en/ (accessed on 1 April 2021).
  57. Svensson, Å.; Larsson, C. A mobile phone app for dietary intake assessment in adolescents: An evaluation study. JMIR Mhealth Uhealth 2015, 3, e4804. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  58. Lee, J.-E.; Song, S.; Ahn, J.S.; Kim, Y.; Lee, J.E. Use of a mobile application for self-monitoring dietary intake: Feasibility test and an intervention study. Nutrients 2017, 9, 748. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  59. Badawy, S.M.; Kuhns, L.M. Texting and mobile phone app interventions for improving adherence to preventive behavior in adolescents: A systematic review. JMIR Mhealth Uhealth 2017, 5, e50. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  60. Bzikowska-Jura, A.; Sobieraj, P.; Raciborski, F. Low Comparability of Nutrition-Related Mobile Apps against the Polish Reference Method—A Validity Study. Nutrients 2021, 13, 2868. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  61. Mistura, L.; Comendador Azcarraga, F.J.; D’Addezio, L.; Martone, D.; Turrini, A. An Italian case study for assessing nutrient intake through nutrition-related mobile apps. Nutrients 2021, 13, 3073. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  62. Lemacks, J.L.; Adams, K.; Lovetere, A. Dietary intake reporting accuracy of the bridge2u mobile application food log compared to control meal and dietary recall methods. Nutrients 2019, 11, 199. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  63. Shiao, S.P.K.; Grayson, J.; Yu, C.H. Gene environment interactions and predictors of colorectal cancer in family-based, multi-ethic groups. J. Pers. Med. 2018, 8, 10. [Google Scholar] [CrossRef] [Scilit]
  64. SAS Institute. Overview of the Generalized Regression Personality. Available online: https://www.jmp.com/support/help/en/16.0/index.shtml#page/jmp/overview-of-the-generalized-regression-personality.shtml (accessed on 27 June 2021).
  65. Alemao, C.A.; Budden, K.F.; Gomez, H.M.; Rehman, S.F.; Marshall, J.E.; Shukla, S.D.; Donovan, C.; Forster, S.C.; Yang, I.A.; Keely, S. Impact of diet and the bacterial microbiome on the mucous barrier and immune disorders. Allergy 2021, 76, 714–734. [Google Scholar] [CrossRef] [Scilit]
  66. Alexander, M.; Turnbaugh, P.J. Deconstructing mechanisms of diet-microbiome-immune interactions. Immunity 2020, 53, 264–276. [Google Scholar] [CrossRef] [Scilit]
  67. Patterson, R.E.; Kristal, A.R.; Tinker, L.F.; Carter, R.A.; Bolton, M.P.; Agurs-Collins, T. Measurement characteristics of the Women’s Health Initiative food frequency questionnaire. Ann. Epidemiol. 1999, 9, 178–187. [Google Scholar] [CrossRef] [Scilit]
  68. Beaton, E.; Wright, J.; Devenish, G.; Do, L.; Scott, J. Relative validity of a 24-h recall in assessing intake of key nutrients in a cohort of Australian toddlers. Nutrients 2018, 10, 80. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  69. Nutrition Coordinating Center. Nutrition Data System for Research; University of Minnesota: Minneapolis, MN, USA, 2014; Available online: http://www.ncc.umn.edu/food-and-nutrient-database/ (accessed on 1 April 2020).
  70. Krebs, P.; Duncan, D.T. Health app use among US mobile phone owners: A national survey. JMIR Mhealth Uhealth 2015, 3, e4924. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  71. Grayson, J.; Gardner, S.; Stephens, M. Building Better Models with JMP®Pro; SAS Press: Cary, NC, USA, 2015. [Google Scholar]
  72. Klimberg, R.; McCullough, B.D. Fundamentals of Predictive Analytics with JMP, 2nd ed.; SAS Press: Cary, NC, USA, 2016. [Google Scholar]
  73. National Coalition for Health Professional Education in Genetics. Family History Educational Aids; National Coalition for Health Professional Education in Genetics: Washington, DC, USA, 2010; Available online: http://www.nchpeg.org/index.php?option=com_content&view=article&id=145&Itemid=64 (accessed on 1 October 2018).
  74. Wu, Y. Elastic net for Cox’s proportional hazards model with a solution path algorithm. Stat. Sin. 2012, 22, 27–294. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  75. Erkkilä, A.T.; Järvinen, R.; Karvonen, H.; Keronen, L.; Tuppurainen, M.T. Validation of a semi-quantitative FFQ using food records as a reference in older women in the Kuopio Fracture Prevention Study (OSTPRE-FPS). Public Health Nutr. 2012, 15, 635–639. [Google Scholar] [CrossRef] [Scilit]
Nutrients 14 00537 g001
Figure 1. (a) Correlation, (b) Bland and Altman plots between mobile application and Food Frequency Questionnaire (FFQ) on total calories.
Figure 1. (a) Correlation, (b) Bland and Altman plots between mobile application and Food Frequency Questionnaire (FFQ) on total calories.
Nutrients 14 00537 g001
Nutrients 14 00537 g002
Figure 2. (a) Correlation, (b) Bland and Altman plots between mobile application and Food Frequency Questionnaire (FFQ) for fat.
Figure 2. (a) Correlation, (b) Bland and Altman plots between mobile application and Food Frequency Questionnaire (FFQ) for fat.
Nutrients 14 00537 g002
Nutrients 14 00537 g003
Figure 3. (a) Correlation, (b) Bland and Altman plots between mobile application and Food Frequency Questionnaire (FFQ) for folate.
Figure 3. (a) Correlation, (b) Bland and Altman plots between mobile application and Food Frequency Questionnaire (FFQ) for folate.
Nutrients 14 00537 g003
Nutrients 14 00537 g004
Figure 4. (a) Correlation, (b) Bland and Altman plots between mobile applications and Food Frequency Questionnaire (FFQ) for cobalamin.
Figure 4. (a) Correlation, (b) Bland and Altman plots between mobile applications and Food Frequency Questionnaire (FFQ) for cobalamin.
Nutrients 14 00537 g004
Nutrients 14 00537 g005
Figure 5. Predicting accuracy of total calories analyses using mobile application against Food Frequency Questionnaire: Area under the receiver operating characteristic curve (AUC) for baseline logistic regression model (a) and Elastic Net with validation model (b).
Figure 5. Predicting accuracy of total calories analyses using mobile application against Food Frequency Questionnaire: Area under the receiver operating characteristic curve (AUC) for baseline logistic regression model (a) and Elastic Net with validation model (b).
Nutrients 14 00537 g005
Table 1. Agreement and bias for the mobile application against Food Frequency Questionnaire (N = 135).
Table 1. Agreement and bias for the mobile application against Food Frequency Questionnaire (N = 135).
Parameters% Difference
M ± SD		FFQ
M ± SD		Mobile
M ± SD		SE± 2 SD%r **
Calories (kcal)−5.34 ** ± 16.421332 ± 904.41201 ± 84641.4194.810.86
 <1000 (N = 54)−0.88 ± 16.87748.3 ± 170.4730.3 ± 182.12.3094.440.54
 1000–2000 (N = 63)−7.08 ** ± 11.461297 ± 290.41180 ± 365.41.4494.330.78
 >2000 (N = 18)−12.63 * ± 25.093206 ± 11142685 ± 14215.9188.890.64
Carbohydrate (g)4.55 ** ± 19.58169.4 ± 132.0176.8 ± 134.71.6891.850.85
Protein (g)−9.80 ** ± 12.2754.84 ± 36.2146.71 ± 32.881.0694.810.87
Fat (g)−17.55 ** ± 15.0850.80 ± 34.2937.20 ± 29.351.3094.070.87
Sat Fat (g)−17.69 ** ± 15.9915.32 ± 10.7211.37 ± 9.281.3895.560.88
Cholesterol (mg)−10.26 ** ± 13.37197.8 ± 131.2172.4 ± 121.61.1595.560.91
Fiber (g)11.39 ** ± 24.0315.97 ± 16.6518.19 ± 18.732.0793.330.85
Thiamin (mg)2.46 ± 15.591.08 ± 0.701.11 ± 0.721.3494.810.86
Riboflavin (mg)−1.75 ± 15.111.27 ± 0.831.23 ± 0.801.3093.330.86
Niacin (mg)−0.91 ± 16.3514.63 ± 9.9314.16 ± 10.301.4191.850.87
Pyridoxine (mg)3.29 * ± 19.291.52 ± 1.291.58 ± 1.401.6693.330.84
Folate (mcg)3.24 * ± 18.78290.7 ± 209.4298.9 ± 214.21.6293.330.85
Cobalamin (mcg)−18.86 ** ± 14.923.71 ± 2.102.69 ± 1.541.2894.070.83
Methionine (g)−11.90 ** ± 12.931.23 ± 0.811.01 ± 0.741.1192.590.88
Choline (mg)−6.23 ** ± 15.42266.4 ± 176.4237.7 ± 160.41.3394.070.86
Glycine (g)−12.13 ** ± 14.552.31 ± 1.581.89 ± 1.441.2592.590.87
Vitamin A (IU)35.19 ** ± 29.4011,037 ± 14,34316,238 ± 16,3592.5397.040.87
Vitamin C (mcg)18.27 ** ± 30.07123.4 ± 137.5148.2 ± 165.82.5992.590.86
Vitamin D (mcg)−6.18 ** ± 12.814.38 ± 2.423.95 ± 2.191.1093.330.90
Vitamin E (mcg)−32.85 ** ± 18.9510.87 ± 7.426.09 ± 4.841.6391.850.83
Zinc (mg)−10.44 ** ± 15.387.39 ± 4.626.15 ± 3.871.3293.330.86
Calcium (mg)−2.66 ± 31.33594.4 ± 382.8569.3 ± 383.82.7093.330.53
Magnesium (mg)3.72 * ± 17.26201.1 ± 150.3208.8 ± 157.61.4993.330.85
Iron (mg)1.46 ± 17.429.00 ± 5.818.91 ± 5.601.5093.330.88
Sodium (mg)−12.60 ** ± 24.942551 ± 17001969 ± 12632.1597.040.83
Note: M: mean; SD: standard deviation; FFQ: food frequency questionnaire; SE: standard error; * p < 0.05; ** p < 0.001.
Table 2. Differences between the mobile application and Food Frequency Questionnaire per domains of caloric ranges and various diets for key macro- and micro-nutrients (N = 135).
Table 2. Differences between the mobile application and Food Frequency Questionnaire per domains of caloric ranges and various diets for key macro- and micro-nutrients (N = 135).
Parameters (N)Calories,
kcal		Carbohydrate,
g		Protein,
g		Fat,
g		Folate,
mcg		Cobalamin,
mcg
%diff M ± SD%diff M ± SD%diff M ± SD%diff M ± SD%diff M ± SD%diff M ± SD
Caloric ranges
<1000 (54)−0.88 ± 16.877.40 ** ± 19.79−10.77 ** ± 10.43−11.75 ** ± 13.662.06 ± 20.12−21.02 ** ± 16.02
1000–2000 (63)−7.08 ** ± 11.465.91 ** ± 14.34−6.93 ** ± 9.31−22.60 ** ± 11.587.04 ** ± 12.56−14.87 ** ± 12.47
>2000 (18)−12.63 * ± 25.09−8.78 ± 28.79−16.93 ** ± 21.09−17.28 ** ± 22.83−6.53 ± 27.99−26.31 ** ± 15.94
Diet Types
Pure liquid (10)16.28 ± 26.7023.27 * ± 29.82−9.14 ± 12.82−3.23 ± 18.0220.44 ± 35.87−20.13 * ± 25.65
Convenient Diet (30)−12.39 ** ± 9.61−0.34 ± 10.15−9.63 ** ± 11.74−26.68 ** ± 10.200.76 ± 13.84−9.82 ** ± 10.84
  Canned Food (10)−12.98 ** ± 5.62−3.99 * ± 5.39−12.68 ** ± 7.27−27.05 ** ± 5.42−3.11 ± 14.39−9.95 ** ± 1.72
  High School (10)−8.33 ** ± 2.992.24 ± 3.46−8.40 ** ± 4.78−18.58 ** ± 5.204.51 ± 8.19−15.43 ** ± 6.48
  Fast Food (10)−15.86 ** ± 15.010.72 ± 16.35−7.80 ± 18.79−34.40 ** ± 11.820.87 ± 17.63−4.07 ± 16.18
Ethnic Food (73)−5.00 ** ± 9.946.21 ** ± 15.43−9.27 ** ± 8.46−17.50 ** ± 9.684.11 ** ± 12.80−19.96 ** ± 11.24
  Western Diet (40)−4.49 ** ± 9.775.20 ± 16.46−8.47 ** ± 8.00−14.25 ** ± 8.383.64 ± 13.74−16.46 ** ± 11.64
  American (10)−4.16 ± 14.423.87 ± 22.58−5.14 ± 7.60−15.16 ** ± 6.975.43 ± 17.35−9.73 ** ± 6.53
  Mexican (10)−4.90 ± 10.6910.81 ± 20.60−6.99 ** ± 4.79−22.07 ** ± 3.9611.42 * ± 13.22−12.92 ** ± 4.78
  Italian (10)−3.43 ** ± 3.186.61 ** ± 4.69−6.10 * ± 6.86−14.07 ** ± 3.277.29 ** ± 1.77−15.71 ** ± 9.93
  Mediterranean (10)−5.45 ± 8.88−0.48 ± 11.92−15.66 ** ± 8.43−5.69 ± 8.87−9.60 ** ± 7.61−27.48 ** ± 14.94
  Eastern Diet (33)−5.63 ** ± 10.267.44 ** ± 14.24−10.24 ** ± 9.01−21.45 ** ± 9.804.68 * ± 11.74−24.20 ** ± 9.24
  Japanese (10)−3.02 ** ± 1.295.05 ** ± 2.17−5.08 ** ± 0.77−12.29 ** ± 1.19−2.25 ** ± 1.14−16.76 ** ± 0.43
  Chinese (10)−9.74 ** ± 5.3312.34 ** ± 7.03−9.44 ** ± 1.85−32.50 ** ± 3.7515.64 ** ± 2.62−20.75 ** ± 0.84
  Korean (13)−4.47 ± 15.395.52 ± 21.72−14.82 ** ± 12.96−20.00 ** ± 8.391.57 ± 14.52−32.58 ** ± 9.74
Smoothie-added (22)−6.67 ± 25.56−2.83 ± 28.95−12.09 * ± 21.17−14.70 ** ± 22.68−4.09 ± 25.77−26.94 ** ± 18.78
Note: M: mean; SD: standard deviation; * p < 0.05; ** p < 0.001.
Table 3. Significant factors contributing to the differences between the mobile application and Food Frequency Questionnaire on total calories.
Table 3. Significant factors contributing to the differences between the mobile application and Food Frequency Questionnaire on total calories.
ParametersLogistic Regression Original ModelGeneralized Regression
Elastic Net Model Validation
Estimatep (χ2)Estimatep (χ2)
(Intercept)1.660.01601.660.0035
1000–2000 caloric range−2.790.0003−2.79<0.0001
Carbohydrate % Difference3.24<0.00013.24<0.0001
Protein % Difference−3.05<0.0001−3.05<0.0001
MR0.0455 0.0455
AICc18.18 18.19
AUC0.9957 0.9957
Note: MR: Misclassification rate; AICc: Akaike’s information criterion with corrections; AUC: Area under the curve.
Table 4. Significant factors contributing to the differences between mobile application a nd Food Frequency Questionnaire on folate.
Table 4. Significant factors contributing to the differences between mobile application a nd Food Frequency Questionnaire on folate.
ParametersLogistic Regression Original ModelGeneralized Regression
Elastic Net Model Validation
Estimatep (χ2)Estimatep (χ2)
(Intercept)−5.590.9386−3.83<0.0001
1000–2000 caloric range−1.810.0053−1.810.0084
Carbohydrate % Difference−2.350.0006−2.350.0003
Fiber % Difference−2.070.0008−2.070.0008
Mediterranean9.110.90017.35<0.0001
MR0.1154 0.1154
AICc30.71 30.73
AUC0.9125 0.9125
Note: MR: Misclassification rate; AICc: Akaike’s information criterion with corrections; AUC: Area under the curve.
Table 5. Significant factors contributing to the differences between mobile application and Food Frequency Questionnaire on cobalamin.
Table 5. Significant factors contributing to the differences between mobile application and Food Frequency Questionnaire on cobalamin.
ParametersLogistic Regression Original ModelGeneralized Regression
Elastic Net Model Validation
Estimatep (χ2)Estimatep (χ2)
(Intercept)11.630.87833.87<0.0001
1000–2000 caloric range2.06<0.00011.86<0.0001
Protein % Difference−1.220.0094−1.050.0140
Chinese−12.260.8718−4.45<0.0001
MR0.2727 0.2727
AICc35.84 35.78
AUC0.7906 0.7906
Note: MR: Misclassification rate; AICc: Akaike’s information criterion with corrections; AUC: Area under the curve.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Kusuma, J.D.; Yang, H.-L.; Yang, Y.-L.; Chen, Z.-F.; Shiao, S.-Y.P.K. Validating Accuracy of a Mobile Application against Food Frequency Questionnaire on Key Nutrients with Modern Diets for mHealth Era. Nutrients 2022, 14, 537. https://doi.org/10.3390/nu14030537

AMA Style

Kusuma JD, Yang H-L, Yang Y-L, Chen Z-F, Shiao S-YPK. Validating Accuracy of a Mobile Application against Food Frequency Questionnaire on Key Nutrients with Modern Diets for mHealth Era. Nutrients. 2022; 14(3):537. https://doi.org/10.3390/nu14030537

Chicago/Turabian Style

Kusuma, Joyce D., Hsiao-Ling Yang, Ya-Ling Yang, Zhao-Feng Chen, and Shyang-Yun Pamela Koong Shiao. 2022. "Validating Accuracy of a Mobile Application against Food Frequency Questionnaire on Key Nutrients with Modern Diets for mHealth Era" Nutrients 14, no. 3: 537. https://doi.org/10.3390/nu14030537

APA Style

Kusuma, J. D., Yang, H.-L., Yang, Y.-L., Chen, Z.-F., & Shiao, S.-Y. P. K. (2022). Validating Accuracy of a Mobile Application against Food Frequency Questionnaire on Key Nutrients with Modern Diets for mHealth Era. Nutrients, 14(3), 537. https://doi.org/10.3390/nu14030537

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop