Next Article in Journal
Induction of p53 Phosphorylation at Serine 20 by Resveratrol Is Required to Activate p53 Target Genes, Restoring Apoptosis in MCF-7 Cells Resistant to Cisplatin
Previous Article in Journal
Influence of Maternal Fish Intake on the Anthropometric Indices of Children in the Western Amazon
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Exploring the Feasibility of Use of An Online Dietary Assessment Tool (myfood24) in Women with Gestational Diabetes

1
Nutritional Epidemiology Group, School of Food Science and Nutrition, University of Leeds, Leeds LS2 9JT, UK
2
Sheffield Diabetes and Endocrine Centre, Sheffield Teaching Hospitals NHSF Trust, Sheffield S5 7AU, UK
3
School of Healthcare, University of Leeds, Leeds LS2 9JT, UK
4
School of Social and Health Sciences, Leeds Trinity University, Leeds LS18 5HD, UK
5
Department of Nursing, Midwifery & Health, Northumbria University, Newcastle upon Tyne NE7 7XA, UK
6
Leeds Institute of Cardiovascular and Metabolic Medicine, University of Leeds, Leeds LS2 9JT, UK
7
Academic Unit of Primary Care and Population Sciences, Faculty of Medicine, University of Southampton, Southampton General Hospital, Southampton SO16 6YD, UK
8
NIHR Southampton Biomedical Research Centre, University of Southampton and University Hospital Southampton NHS Foundation Trust, Southampton SO16 6YD, UK
9
Leeds Institute for Data Analytics, School of Medicine, University of Leeds, Leeds LS2 9JT, UK
10
Department of Food Science and Nutrition, King Abdul-Aziz University, P.O. Box 42807, Jeddah 21551, Saudi Arabia
*
Author to whom correspondence should be addressed.
Nutrients 2018, 10(9), 1147; https://doi.org/10.3390/nu10091147
Submission received: 15 July 2018 / Revised: 18 August 2018 / Accepted: 21 August 2018 / Published: 23 August 2018

Abstract

:
myfood24 is an online 24 hr dietary recall tool developed for nutritional epidemiological research. Its clinical application has been unexplored. This mixed methods study explores the feasibility and usability of myfood24 as a food record in a clinical population, women with gestational diabetes (GDM). Women were asked to complete five myfood24 food records, followed by a user questionnaire (including the System Usability Scale (SUS), a measure of usability), and were invited to participate in a semi-structured interview. Of the 199 participants, the mean age was 33 years, mean booking body mass index (BMI) 29.7 kg/m2, 36% primiparous, 57% White, 33% Asian. Of these, 121 (61%) completed myfood24 at least once and 73 (37%) completed the user questionnaire; 15 were interviewed. The SUS was found to be good (mean 70.9, 95% CI 67.1, 74.6). Interviews identified areas for improvement, including optimisation for mobile devices, and as a clinical management tool. This study demonstrates that myfood24 can be used as an online food record in a clinical population, and has the potential to support self-management in women with GDM. However, results should be interpreted cautiously given the responders’ demographic characteristics. Further research to explore the barriers and facilitators of uptake in people from ethnic minority and lower socioeconomic backgrounds is recommended.

1. Introduction

Gestational diabetes mellitus (GDM) has onset or detection in pregnancy, most commonly in the third trimester. Universal screening for GDM is conducted in some countries such as Australia. However, in the UK, screening targets those with one or more established risk factors. i.e., women with body mass index (weight kg/height m2) above 30, previous large baby >4.5 kg, previous GDM, ethnic family origin with a higher prevalence of diabetes, or a family history of diabetes [1]. It is performed at around 24–28 weeks gestation and uses an oral glucose tolerance test following fasting, to detect raised blood glucose levels.
The main treatment aim is to maintain healthy blood glucose levels throughout pregnancy to reduce the risks of complications for the mother and child; for example: macrosomia, stillbirth, and birth-related trauma [2]. After giving birth, blood glucose levels usually revert to normal; however, 50% of women with GDM will develop Type 2 diabetes within five years [1]. As with Type 2 diabetes, the treatment of GDM initially focuses on dietary and lifestyle changes with a healthy diet being recommended [1]. To achieve blood glucose targets, emphasis is placed on optimising the type of carbohydrate by encouraging low glycaemic index choices; and on reducing the portion size. If within one to two weeks of dietary modification, blood glucose levels are above the target range for pregnancy (fasting blood glucose below 5.3 mmol/L and one hour after meals below 7.8 mmol/L), women will then require medication for diabetes for the remainder of pregnancy; for example, metformin tablets or insulin injections. The self-management behaviours that are required to achieve these levels, including dietary modification, and frequent self-blood glucose monitoring, are demanding and are associated with increased levels of distress [3].
Standard care at diagnosis for women with GDM includes a dietary assessment, usually by a dietitian, from which education and negotiations for dietary changes will be based; progress may be reviewed at subsequent appointments with further dietary assessments [1]. As with other clinical populations requiring dietary management, typical dietary assessment methods include 24 h dietary recalls that are undertaken with a dietitian or with self-completion of paper food records. The level of detail acquired is variable, and it may include the frequency, amount, and type of food. Gathering and interpreting this information requires time and skills for both health professionals and patients. Such dietary assessment methods are known to have limitations. For example, interviewer-led 24 h dietary recalls can introduce reporting bias as subjects under-report certain foods due to social desirability [4].
The development and implementation of technology-assisted dietary assessment, including, for example the use of websites, mobile phone cameras, and mobile apps to log food intake, is proving to be an alternative to these traditional methods [5,6,7,8,9]. Such technologies in healthy populations are being shown to reduce costs [10], improve completion rates, and increase the accuracy of the dietary assessment [5,7,10,11,12]. Automated self-administered dietary assessment tool provides participants with more privacy than interviewer-led assessments; hence reducing judgment bias, which is associated with omissions in reporting and underestimation of portion sizes of unhealthy foods [13]. Therefore, minimising contact with the interviewer may encourage more participants to report all food items [14].
Given these favourable findings when using technology-assisted dietary assessment in healthy populations, their application in the management of medical conditions has begun to be explored [15,16]. This data can be instantly analysed and provide feedback to the user and health professional. These features can provide additional support and education outside of the clinic setting [15,17,18], and a study in primary care reported that technology-assisted dietary assessment may promote self-efficacy and increase empowerment in users [17]. Such tools also enable remote monitoring and follow up, potentially reducing the number of face-to-face appointments; therefore, there could be substantial clinical benefits and cost reductions associated with their use.
myfood24 is a self-completed online 24 h dietary recall tool, incorporating elements of an automated multiple pass methodology [19]. Foods consumed are entered on the system, by selecting from a list of options; portion size is then selected, and the food is added to a meal event or time. Prior to submitting the completed recall, it prompts the user to check for omissions, e.g., snacks. The recall is completed at one time, inputting all information from the previous 24 h period. The myfood24 food database contains 45,000 foods including both generic and branded items, 5600 with associated portion size images, and provides the user with an immediate nutritional analysis of their dietary intake. The tool was developed for large scale epidemiological studies in the UK population. Usability and acceptability testing of myfood24 has shown it is suitable to be used in healthy UK teenager (System Usability Score (SUS) Median score 80) and adult populations (SUS Median score 73) and relative validity testing in teenagers showed good agreement with interviewer-administered 24 h multiple-pass recall (MPR)s [19,20,21]. A validation study in healthy adults that includes the use of biomarkers has been undertaken [22]. Whilst myfood24 is currently used as a 24 h recall tool for nutritional assessment in epidemiological research, its application in healthcare as a food record has not been explored.
This study examined the introduction of myfood24 in a clinical population for the first time. Women with GDM, who are routinely required to complete food records, were requested to use myfood24 as an alternative tool for self-recording dietary intake. The aim of this observational study was to explore the feasibility of using the tool as an electronic food record and the usability of the tool software. Therefore, providing direction for the ongoing development and evaluation of this tool, both for use in clinical research, and ultimately, as an electronic food diary in routine clinical practice. The study also addressed wider aims, including the association between dietary components and blood glucose levels, and the broader experiences of women with GDM, which are the subject of other publications.

2. Materials and Methods

2.1. Ethics

All subjects gave their informed consent for inclusion before they participated in the study. The study was conducted in accordance with the Declaration of Helsinki, and the protocol and subsequent amendments were approved by the South Central Oxford Research Ethics Committee C (reference 14/SC/1267).

2.2. Eligibility

Women attending for their initial appointment following the diagnosis of GDM at Leeds Teaching Hospitals NHS Trust Diabetes in Pregnancy Clinic, were eligible to participate, provided they could read and understand English and were not to be commenced on any diabetes medication.

2.3. Recruitment

Women were invited to take part by a research midwife who provided a participant information sheet, addressed any questions relating to the study and secured informed consent to participate. The sample size of 200 was based on the sample size calculation for other research questions beyond the scope of this paper, examining the association between dietary components and blood glucose levels. This was a considerably larger sample than what was required for usability testing and examining the differences in usability based on sample characteristics [23].
A pragmatic approach was adopted for interviews, using a convenience sample of women who had experienced myfood24 in the main study. Guided by other qualitative research, a minimum of 12 interviews was anticipated as sufficient to provide saturation [24]. As the aim was to interview both participants who had used the tool and those who chosen not to, different recruitment methods were used. For those who had used myfood24, their interest could be indicated via the user questionnaire, and following a substantial amendment, those who had never completed myfood24, were sent an email. Informed consent for interviews was secured separately by the interviewer (ZD).

2.4. Study Design and Data Collection

This was a mixed method prospective observational study. The stages of the study and the participation at each stage can be seen in Figure 1. Participants consented for relevant demographic and clinical data to be retrieved from their health records.

2.4.1. Completing a Food Record Using myfood24

Antenatal care provision followed standard care guidelines for diabetes in pregnancy, which included ongoing self-monitoring throughout the pregnancy with blood glucose monitoring seven times a day and paper food diaries. By taking part in the study, participants were also requested to complete five 24 h food records in a two-week period using myfood24. The process of accessing and completing a myfood24 food record is illustrated in Figure 2. The request for five days of records was a pragmatic decision. This would allow women to record a large proportion of their habitual intake and to become more familiar with using myfood24; thus, providing more opportunities to test out the usability and functions of the website.
The feasibility of using the tool as a food record was investigated by measuring any online submission of a myfood24 food record and the total number of submissions per participant.

2.4.2. User Questionnaire

Those who completed myfood24 on at least one occasion were emailed a link to complete the user questionnaire, administered by Bristol Online Survey [25]. The questionnaire consisted of multiple choice questions, Likert scales, yes/no responses and open-ended questions which covered demographic information, previous experience of using technology and food diaries, attitudes to technology and usability of myfood24. It included the SUS, a validated, reliable tool for measuring usability of the software [26]. This is a 10-item scale, with the users asked to rate their level of agreement with 10 usability statements (1 = strongly agree; 5 = strongly disagree) which gives a total score from 0–100. Scores below 50 are unsatisfactory with 50 to 70 are judged as marginal and above 70 considered good [27]. In practice, a SUS score above 68 is considered to be above average and anything below 68 is considered to be below average, and therefore the aim was for the SUS score to be at least 68 [28].
Usability considers user experience of the system, including the ease of understanding and ease of use of the software [29]. The SUS score from the user questionnaire, along with the interview data was the main measure used to assess usability of myfood24, when utilised as an online food record.

2.4.3. Interviews

Interviews were semi-structured and followed a pre-designed flexible topic guide, exploring women’s views and experiences of using myfood24. The interviews sought to contextualise the completion rates and findings from the user questionnaire in the words of the women, exploring beyond the practical aspects relating to feasibility and usability. Women were interviewed at a time and location of their choice. All were interviewed by telephone with interviews lasting on average 35 min (range 20–54).

2.5. Data Analysis

2.5.1. Statistical Analyses

Analyses were performed using Stata IC14 statistical software. Descriptive statistics were used to define the sample characteristics. Comparisons were made using chi-squared test for categorical data and independent t-test for continuous data. For all inferential statistics the significance level was two-sided and set at 0.05.
Further analysis, using data from the health records of the study population along with additional data from the user questionnaire, considered the secondary aim. This was to identify specific factors that may influence the usability of myfood24, such as demographics, other participant characteristics including online literacy, and features of the tool.

2.5.2. Analysis of Interviews

Interviews were audio-recorded and transcribed verbatim in an anonymous format. Data were analysed using thematic analysis, using the principles that were outlined by Braun and Clarke [30], familiarising selfwith the data, generating initial codes, searching for potential themes, reviewing themes, defining and naming themes, and producing the report.

3. Results

3.1. Characteristics of Participants

Two hundred women consented to the study, and one of these withdrew. The mean age was 33.3 years (SD 5.0). According to health records, 76% of the women were overweight or obese at booking (first formal antenatal appointment), and the mean body mass index (BMI) was 29.7 kg/m2 (SD 6.5). Most women were White (57%) and one-third (33%) were Asian, and 36% women were primiparous.

3.2. Completion of myfood24 Recalls

Women were asked to use myfood24 to complete and submit five online food records over a two-week period. For the full sample (n = 199), the mean number of days completed was 2.3 (SD 2.2) days; however only 61% (121/199) completed myfood24 at least once. In the 121 women who completed myfood24 at least once, the mean number days completed was 3.8 (SD 1.4); 98 (81%) completed it at least three times and 58 (48%) completed it all five times. Further details are shown in Figure 3.
There were statistically significant demographic differences between those who did (121/199; 61%) and did not (78/199; 39%) complete myfood24 at least once, as shown in Table 1. Women who used myfood24 as a food record were statistically more likely to have lower fasting blood glucose levels (p = 0.008), be of White ethnicity (p = 0.001) and primiparous (p = 0.02).
Within the women who completed myfood24, no statistically significant differences in demographic characteristics were found according to the number of times that it was completed, with comparison between women who completed it one to two times (n = 23), versus three times or more (n = 98).

3.3. User Questionnaire

User questionnaires were completed by 73 of the 121 women who used myfood24, a response rate of 60%. Comparisons were made between these women (n = 73) and those who did not return the questionnaire (n = 48), as shown in Table 2. Statistically significant differences in demographics were found; responders were older (p = 0.01) and had lower BMI at booking (p = 0.008). The mean number of myfood24 food records completed was significantly higher (p < 0.001) in those who completed the questionnaire.
The user questionnaire contained the usability questionnaire, which provided a SUS for measuring the usability of the software [26]. Therefore, this was only accessible to 121/199 participants, and of these, only 73 completed it; hence SUS was completed by 73/199 (37%) of the participants. The mean SUS score for myfood24 in this sub-group was considered to be good at 70.9 (95% CI 67.1, 74.6). Further questions on the user questionnaire provided the following demographic details on this sub-group. Of the 73 women, 47% were employed in managerial or professional occupations, and 58% were educated to degree level or above. Most women (90%) described themselves as being confident with using technology, with good access and the ability to use the internet (97%). Previous use of technology to record food intake was reported by 45%.

3.4. Interviews

3.4.1. Characteristics of Interviewees

Despite a substantial amendment part-way through the study, to ensure that the invitation for interview was sent to those who did not complete any myfood24 food records, there were no expressions of interest to be interviewed in this group. Sixteen women who had completed myfood24 expressed an interest, resulting in 15 interviews with one declining due to time commitments. Characteristics of those interviewed were similar to the other participants who completed myfood24. The mean age was 35.1 years (SD 4.5); five were primiparous. Most women were White (13 White, two Asian). The mean BMI at booking was 28.3 (SD 4.7). All had completed myfood24 at least once, and 10 had completed it five times. The mean SUS score was 74.0 (SD 23.0).
Most women reported completing the record at the end of the day in the evenings, with a number of them using their clinic paper food records as an aid to recall that day’s food intake.

3.4.2. Themes

Thematic analysis relating to usability and acceptability of using myfood24 as a food record identified the following themes: (1) ease of use, (2) impact on food choices, (3) comparisons with clinic paper records, and (4) future developments. Due to the characteristics of the interview sample, these themes represent only the views of women who had completed myfood24.
(i) Getting Acquainted: Ease of Use
Many of the women had used similar technologies before to record their diet and reported finding myfood24 to be comparable once they had become acquainted with the technology. Most found myfood24 ‘straightforward’ and that inputting data became faster with practice, initially taking approximately 20 min.
‘I found it quite easy… I mean, you just typed in a word and it would bring what you wanted up and then you chose from the list’.
(Interview 3)
Several features were reported that helped or hindered the ease of use of myfood24. Features of the tool that were most commonly identified as being useful for input were the food photographs to help with portion estimation, and the use of reminder prompts. Some also used the recipe function and favourites list; most reported that they would use these functions on a longer-term basis. The food database received mixed feedback. Some found the food database too restrictive, whereas others found the choice overwhelming. Some reported finding food entry to be burdensome when cooking from raw ingredients, and the process was too time consuming to be feasible.
‘Obviously you don’t weigh your food… it was nice to see actually a portion size’
(interview 6)
‘I didn’t have to go back and correct anything really because it reminded me’
(interview 3)
‘For something that was quite simple, it would take actually a long time to find it’
(interview 4)
‘It was quite time consuming having to kind of search for things and put everything in in its own, you know, like all the different components that made up a meal’.
(interview 11)
One reported disadvantage was the need to record all food entries for the 24 h period in one go, rather than being able to add data after each meal, making myfood24 less easy to use than paper diaries. Interviews identified that accessibility was hindered by problematic initial access to the myfood24 website, offering insight into the completion rates observed. As shown in Figure 1, myfood24 uses a web link accessed from the participant’s email account, with each recall requiring a separate email. Women reported that the process had been confusing and it was difficult to keep track of the multiple emails, all of which were sent on the day of entry to the study and without any indication as to which of the five days they referred to.
(ii) Feedback: Impact on Food Choices and Behaviours
On completion of a food record, myfood24 provides an instant summary of its nutritional composition, including energy, protein, fat, carbohydrate, fibre, salt, and micronutrients. This is compared to a dietary reference value.
All of the interviewees reported finding this feature useful to obtain feedback on their food intake and associated eating behaviours, although a minority questioned the accuracy. Women described how this feedback increased their knowledge of their dietary intake, how it could provide reassurance, and how it could enable women to consider what changes were required. Some women reported that the summaries were a motivating factor, influencing future choices about food types and portion sizes.
‘I did find it frustrating, to be honest, in terms of just trying to find what matched what I was eating, I don’t know how accurate it was for me’
(Interview 4)
‘a week’s worth of days in front of you it does make you think about what you’re eating and how much’
(Interview 3)
‘I was like ooh I shouldn’t have eaten that or oh, I’ve had a really good day today.’
(Interview 12)
Most women focused on the data relating to total carbohydrates and sugars—key to the management of GDM—but some also noted using the summaries to understand their fat and salt consumption. In addition, for some, the summary information highlighted that they had actually been too restrictive with their diet, in response to their diagnosis.
‘Once you can see it in numbers and can see the picture of it, it’s harder just to shrug off and think I’m fine...I couldn’t just go on with it’ (interview 11).
(iii) Comparing Online Self-Monitoring to Clinic Paper Diaries, and the Use of Real-Time Tracking
Comparing myfood24 with paper food diaries that women were required to complete as standard care, some felt myfood24 led to more accurate recording of food intake due to its use of reminders (for example prompting about snacks between meals, and the use of condiments) and recording of quantities, which had not been requested in as much detail in their paper food diaries.
‘I thought that [myfood24] made me remember things. It was more specific. I think it was easy to forget when you’re writing it down. You know because it reminded you—have you remembered to put a drink down here, have you remembered to put a snack down there.’
(Interview 9)
Women felt that whilst myfood24 was more accurate than paper-based diaries, in its current format it was also less convenient than paper-based diaries, particularly given the need to track blood glucose levels alongside the dietary intake in the paper-based diaries. Hence, although women valued the myfood24 summary information highly, the utility of myfood24 was limited by being produced at the end of a 24 h period and without any connection to blood glucose readings, limiting its use as a self-monitoring tool for improving blood glucose levels.
(iv) Future Developments: Suggestions based on Experience
Several suggestions were made by women, the most common being to improve accessibility via an app so that data could be conveniently logged throughout the day using a smart phone or tablet. Other suggestions for practical aspects relating to usability and improving its acceptability as an electronic food record were a larger food database, an option to scan bar codes, and easier retrieval of favourite foods. Women made several suggestions to adapt myfood24 to develop its accessibility, and also to develop its usefulness in promoting self-management and supporting behaviour change. Of greatest priority for this clinical population was the need to record blood glucose readings alongside dietary information, in order for the information to be meaningful and to develop knowledge of the impact of different types and quantities of food on blood glucose levels.
Some also identified the need for improved visual displays of the summary nutritional information, to support behaviour change. Suggestions included the ability to track information across time and individualised targets for nutritional intake, rather than reference values. Women also emphasised their perceived additional benefit from being able to use the tool in a diary format, rather than solely as a 24 h recall. This related both to aiding recall, but also to the way in which women used feedback to guide future food choices and behaviour.
‘I think a phone app would be a lot easier…you’d just keep updating it as the day went along… it’d make it more helpful like that’
(Interview 12)
‘I don’t know if there was an option where you could save that information, and maybe it was something that I missed, but I would have found that useful as an ongoing thing, if you could save it…’
(Interview 7)
‘I thought what might be quite useful would be more like a graph … that shows like you’re here and this is what you really should be getting, aiming for.’
(Interview 2)

4. Discussion

This study has demonstrated that it is feasible to use myfood24, a tool previously only used for nutritional epidemiology studies, as an electronic food record in a clinical population. Usability of the software was scored as good within a sub-group of the study population. However, the lower response rates throughout the study and the lack of sample representativeness mean that these results should be interpreted cautiously.
The study has generated a number of suggested areas for improvement which could increase the uptake of myfood24 as a dietary assessment tool in clinical populations. It is important that these are applied where possible, and their impact further evaluated. These changes, along with the ideas for improving the myfood24 software, should be considered prior to future studies examining its use as a dietary management tool to support behaviour change, in a healthcare setting.

4.1. Feasibility of Using myfood24 as An Electronic Food Record

The feasibility of using the tool as an electronic food record was investigated by measuring the total number of submissions of a completed myfood24 food record per participant.
In this study, 78/199 women (39%) did not submit any myfood24 food records. Reasons for this are not known and may extend beyond the acceptability of the tool as an electronic food record. It is noted that statistically significant differences existed between those who did and did not complete myfood24, some of which may be indicative of levels of motivation and self-management; unfortunately, further comparisons were precluded by certain data only being available for those who completed myfood24 and who went on to complete the user questionnaire.
Of those who completed the user questionnaire (n = 73), 45% of participants had used technology previously to log food, so this indicates it is common for women in this group to have similar previous experiences, potentially influencing acceptability in this is population. After participants used myfood24 at least once, the majority went on to use it again, with almost half of women completing it the five times as requested; which does suggest that it is feasible to use this tool as an electronic food record. There did not appear to be any demographic differences regarding number of times completed, however a larger sample size may be needed to determine this. Findings from the interviews suggested that there was confusion experienced with the email system. This system had remained unchanged from its application in nutritional epidemiology data collection and appeared not to suit this setting. As a consequence, this could have impacted on the uptake. In contrast, the data suggests that its perceived provision of a more complete and accurate record and the receipt of the nutritional intake summaries immediately following the diary submission, made a valuable contribution to its acceptability, and enhanced its use as a food record. This is supported in other studies [15,18]. Future work should prioritise these areas for development.
Women continued to have to complete paper-based food and blood glucose diaries as part of standard care. Some women found the paper diary more convenient, due to the ease of access and having all the data in one place, enabling links to be made between food intake and blood glucose levels. Studies in populations with diabetes and renal disease support the suggestion that being able to input foods in real time via a phone app and adding other clinical data such as blood glucose levels can improve acceptability [15,16]. The absence of this functionality may have affected completion rates, as may have the requirement to dual record data on both paper and electronic records.
myfood24 user data can be shared with both the user and the provider/health professional; hence it has the potential to promote user–provider interactions. This function was not utilised in this study. Significant benefits in diabetes outcomes have been found when patients were provided with analysis or feedback from clinicians on their data [15]. Additionally, acceptability and effectiveness of a smartphone nutritional assessment tool for healthy pregnant women, was enhanced by using a similar feature [18]. Therefore, utilisation of this function should be a priority in future studies and its impact on the uptake of the tool assessed.

4.2. Usability

Usability considers the user experience of the system, including its ease of understanding and ease of use [29]. This was measured using the SUS and interview data. The SUS score was classed as good (SUS = 70), and it was comparable to the scores obtained in the general population in other myfood24 evaluations with adults (SUS = 80) and adolescents (SUS = 74) [19,20]. However, it was only obtained in 61% of participants who used the system, and it is noted that this sub-group lacked representativeness compared to the whole study population. This lack of representativeness was also present in the group of interviewees, from whom further data on usability of myfood24 software was collected. Therefore, although this study has generated valuable insights into the usability of the software, that can be further explored and considered for future development of myfood24, no conclusions on the usability of myfood24 for this clinical population can be made at present.

4.3. Improving Uptake of Use of the Tool and Usability of Software in the Future

A number of suggestions were made by interviewees regarding how to improve myfood24 software. Many of these are applicable to all dietary assessment tools/ food logging applications [6,17] and include: improvements to speed up food data entry, extensions to the food database, including ethnic foods, chain restaurant meals and popular homemade recipes, and more support to estimate portion sizes. Specifically related to myfood24, women wished to input food in real time and not have to enter the entire day’s food intake in one entry. They requested being able to undertake food records as frequently as they wanted, to access previous records, to be able to utilise food lists and to also make comparisons between days. Recent myfood24 developments on the visual displays of nutrient summary feedback, could be adapted for the healthcare setting to address some of these suggestions. Further thought into how the website is accessed and the setup of personal accounts for clinical rather than research purposes, is crucial to achieving this, and improving uptake. This would reduce the confusion experienced with the current email system and allow women to undertake dietary recalls as frequently as they choose to; furthermore, personal accounts could be used to promote self-monitoring during the pregnancy and beyond.
The findings highlight the importance of tailoring tools to the needs of population, illustrated here by the importance of real-time tracking for women with GDM and need for integration of dietary intake with other data; here for blood glucose, but other conditions would benefit from recording other health information (e.g., weight, other blood test results, blood pressure, activity data). Continued development by the myfood24 consortium and website developers, taking into account suggestions for further improvements from this and other studies [6] will enhance the usability further.

4.4. Opportunities for Use of myfood24 in a Clinical Setting

This study has established that it is feasible to use myfood24 as an electronic food record in the GDM population, and provides helpful insights into how to optimise its use in this population. There could be significant benefits of introducing myfood24 in this clinical setting. Diabetes ante-natal care includes frequent follow up at lengthy multi-professional clinic appointments and by telephone. Therefore, health apps have been developed specifically for GDM to enable health professionals to manage case-loads using remote monitoring of blood glucose levels [31]. Online food diaries could be used to share data in the same way, and these could be beneficial for a number of reasons. In diabetes care, such tools have been shown to be a more engaging and accurate way of capturing a dietary assessment [5,16] and are preferred to traditional methods [32]. Women in our study commented that they began to consider changes in their food intake from performing the recalls and receiving the feedback; this has been observed in other populations [15,17], and has been shown to improve the uptake of self-monitoring and lead to dietary changes [33]. Following initial education, women could continue to monitor their dietary intake and receive feedback from the website which could result in increased self-efficacy and self-management skills. Additionally, as demonstrated in other studies, remote monitoring and provision of feedback by the health care professional could reduce workload during clinics and enhance service provision by freeing up time for education and support [12,34]. In other chronic conditions, use of dietary assessment technologies in this way has had a beneficial effect on clinical outcomes [16,35].
The interviews highlighted the emotional aspects of living with GDM, and the breadth of behaviours that women with GDM are tasked with managing (reported in full elsewhere: [36], in preparation) indicating that if myfood24 were to be optimised for mobile devices, there would likely be other beneficial components that may support women with GDM. From diagnosis of GDM, women are expected to begin to make immediate changes to their lifestyle to meet the blood glucose target recommendations. However, there will be a wide range of levels of knowledge, skills and confidence around healthy eating. From women’s feedback provided in interview, it appears that myfood24 has the potential to improve the health literacy and eating behaviours of women with GDM. This is something that would require further testing. Reducing dependence on the health care professional to advise on dietary changes, and increasing knowledge and skills, has been shown to increase an individual’s sense of control. Receiving prompts as cues for action and receiving feedback on their nutritional intake alongside biofeedback through blood glucose levels are recognised behaviour change techniques that can support healthy behaviours [37]. This increased level of control over their condition and perceived treatment efficacy may additionally help to reduce the levels of distress experienced [38,39].

4.5. Strengths and Limitations of the Study

There are several strengths to this study. There was a large study population, which provided a complete data set on 73 women. The qualitative element of the study provided valuable insight into the trial of the tool in everyday life, adding depth that has been reported as lacking in other studies of this type [6].
Limitations arose from the data collection methods in the study. The user questionnaire provided valuable information on the demographics and technological skills of the women, but it was only collected from 37% of the study population. In hindsight, if this had been completed at baseline, without the SUS, this would have provided significantly more information about the women who did not respond at the different stages of the study. Furthermore, there was a lack of evaluation built into the study to routinely establish reasons for their not using myfood24 even once. This could in part reflect the methods for accessing myfood24 (i.e., through automated emails), or challenges with usability. However, other reasons should have been considered such as how the tool was introduced to women. Increasing individuals’ understanding of the benefits of using myfood24, such as those highlighted in this study around accuracy and the receiving of feedback, could be important for increasing uptake, and should be built into future interventions. Additionally, it was not established how much of an impact being in the late stages of a clinical complex pregnancy affected uptake.
Another significant weakness was the lack of representativeness of the data. This included a lack of ethnic diversity, which diminished with each stage of the research. A smaller proportion of women from black and minority ethnic (BME) groups completed the user questionnaire and took part in the interviews, limiting transferability of the study findings. As GDM is more prevalent in women from BME groups, this is not representative of a typical GDM population in the UK [40], and the barriers and facilitators for using myfood24 in BME groups needs to be explored in future studies. Over-represented in the group who completed the user questionnaire were women who were well educated and who worked in professional jobs. Based on those completing the questionnaire, the study population appeared to be confident in using technology and had good access to the internet, as well as previous experience with similar technologies. Again, this limits the conclusions that can be drawn from the results.
Finally, once women had begun to use myfood24 they may have been willing to continue to use it throughout the pregnancy; however the system did not allow them to continue to access myfood24 beyond their five records. The opportunity for continued use would have provided valuable further insights into its use by participants with a health condition [16].
This study has generated a number of suggestions for the future development of myfood24 for clinical populations. Real-time data entry assisted by a phone app format, could improve acceptability and usability. Adding further features such as improving the presentation and content of the feedback, and tracking pertinent biomedical data alongside nutritional information, would increase its potential beyond dietary assessment, and are ready for testing in the context of self-management of health conditions. The feasibility of its application within the health service should be explored in a feasibility study that also captures the views of users and health professionals on its use as an alternative tool in dietetic care. Ultimately a definitive trial considering the clinical and cost-effectiveness of a refined myfood24 tool in clinical practice is recommended.

5. Conclusions

This study has demonstrated that it is feasible to use myfood24, an online 24 h dietary recall tool developed for nutritional epidemiology studies, as an electronic food record in women with GDM. Usability of the software was assessed as good by a subgroup of the women. However, a higher response rate could have potentially been possible with a more diversely representative sample; therefore, the barriers against achieving this should be explored in future research.
The findings have generated a number of suggestions to improve acceptance and to enhance the usability of such tools in clinical populations. Women were motivated to use the tool, largely due to the feedback on their nutritional intake it provided, although being able to record food intake and input clinical data (blood glucose levels) in real time using mobile-optimised technology was deemed necessary to be able to replace paper-based diaries. Issues arose from employing the tool in a different context to the one in which it had been designed for, causing practical problems which need to be addressed.
Following improvements to the tool, suggested in this study and from other research findings, myfood24 should then be further evaluated within a healthcare setting, exploring its use as a dietary management tool to enhance behaviour change.

Author Contributions

Conceptualization, N.A.A. and J.E.C.; Data curation, Z.D., M.A.M. and E.M.S.; Formal analysis, C.G., Z.D., N.A.A., R.H., L.M.G. and D.M.S.; Funding acquisition, S.A. and J.E.C.; Investigation, Z.D., M.C., M.A.M. and E.M.S.; Methodology, S.A., N.A.A. and M.A.M.; Project administration, M.C. and M.A.M.; Supervision, J.E.C.; Writing—original draft, C.G. and Z.D.; Writing—review & editing, C.G., Z.D., S.A., N.A.A., M.C., R.H., L.M.G., M.A.M., E.M.S., D.M.S. and J.E.C. All authors reviewed and approved the final version of the manuscript.

Acknowledgments

This study was funded by the Medical Research Council (ref: G1100235) and supported by the National Institute of Health Research Clinical Research Network (NIHR CRN). Research was conducted by the University of Leeds in collaboration with Leeds Teaching Hospitals NHS Trust. The qualitative data collection and transcription was supported by pump priming funds awarded by the University of Leeds School of Healthcare Research Pump Priming Fund. Funds for open access publishing were provided through the project being funded by the MRC. JC, MC, MAM and the development of myfood24 was funded by the Medical Research Council (GB) grant MRC G1100235. We acknowledge: Neil Hancock and Helen Brown for their database support for this project; Del Endersby for recruiting patients; Jayne Hutchinson for her work and advice with the data analysis and; the myfood24 consortium for contributions to development of myfood24.

Conflicts of Interest

Janet E. Cade is Director of Dietary Assessment Ltd., the company that runs myfood24, and Michelle A. Morris and Michelle Carter are shareholders. The funding sponsors had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, and in the decision to publish the results.

References

  1. NICE. Diabetes in Pregnancy: Management from Preconception to the Postnatal Period; NICE Guidelines [NG3]; NICE: London, UK, 2015. [Google Scholar]
  2. Farrar, D.; Simmonds, M.; Bryant, M.; Sheldon, T.A.; Tuffnell, D.; Golder, S.; Lawlor, D.A. Treatments for gestational diabetes: A systematic review and meta-analysis. BMJ Open 2017, 7, e015557. [Google Scholar] [CrossRef] [PubMed]
  3. Noctor, E.; Dunne, F. A practical guide to pregnancy complicated by diabetes. Diabetes Prim. Care 2014, 16, 146–153. [Google Scholar]
  4. Bingham, S.; Gill, C.; Welch, A.; Day, K.; Cassidy, A.; Khaw, K.T.; Sneyd, M.J.; Key, T.J.; Roe, L.; Day, N.E. Comparison of dietary assessment methods in nutritional epidemiology: Weighed records v. 24 h recalls, food-frequency questionnaires and estimated-diet records. Br. J. Nutr. 1997, 72, 619–643. [Google Scholar] [CrossRef]
  5. Illner, A.-K.; Freisling, H.; Boeing, H.; Huybrechts, I.; Crispim, S.P.; Slimani, N. Review and evaluation of innovative technologies for measuring diet in nutritional epidemiology. Int. J. Epidemiol. 2012, 41, 1187–1203. [Google Scholar] [CrossRef] [PubMed]
  6. Timon, C.M.; van den Barg, R.; Blain, R.J.; Kehoe, L.; Evans, K.; Walton, J.; Flynn, A.; Gibney, E.R. A review of the design and validation of web- and computer-based 24-h dietary recall tools. Nutr. Res. Rev. 2016, 29, 268–280. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  7. Sharp, D.B.; Allman-Farinelli, M. Feasibility and validity of mobile phones to assess dietary intake. Nutrition 2014, 30, 1257–1266. [Google Scholar] [CrossRef] [PubMed]
  8. Rusin, M.; Arsand, E.; Hartvigsen, G. Functionalities and input methods for recording food intake: A systematic review. Int. J. Med. Inform. 2013, 82, 653–664. [Google Scholar] [CrossRef] [PubMed]
  9. Long, J.D.; Littlefield, L.A.; Estep, G.; Martin, H.; Rogers, T.J.; Boswell, C.; Shriver, B.J.; Roman-Shriver, C.R. Evidence Review of Technology and Dietary Assessment. Worldviews Evid. Based Nurs. 2010, 7, 191–204. [Google Scholar] [CrossRef] [PubMed]
  10. Foster, E.; Hawkins, A.; Delve, J.; Adamson, A.J. Reducing the cost of dietary assessment: Self-completed recall and analysis of nutrtion for use with children (SCRAN24). J. Hum. Nutr. Diet. 2014, 27 (Suppl. 1), 26–35. [Google Scholar] [CrossRef]
  11. Thompson, F.E.; Subar, A.F.; Loria, C.M.; Reedy, J.L.; Baranowski, T. Need for technological innovation in dietary assessment. J. Am. Diet. Assoc. 2010, 110, 48–51. [Google Scholar] [CrossRef] [PubMed]
  12. Probst, Y.C.; Faraji, S.; Batterham, M.; Steel, D.G.; Tapsell, L.C. Computerized dietary assessments compare well with interviewer administered diet histories for patients with type 2 diabetes mellitus in the primary healthcare setting. Patient Educ. Couns. 2008, 72, 49–55. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  13. Touvier, M.; Kesse-Guyot, E.; Mejean, C.; Pollet, C.; Malon, A.; Castetbon, K. Comparison between an interactive web-based self-administered 24 h dietary record and an interview by a dietitian for large-scale epidemiological studies. Br. J. Nutr. 2011, 105, 1055–1064. [Google Scholar] [CrossRef] [PubMed]
  14. Ngo, J.; Engelen, A.; Molag, M.; Roesle, J.; García-Segovia, P.; Serra-Majem, L. A review of the use of information and communication technologies for dietary assessment. Br. J. Nutr. 2009, 101 (Suppl. 2), S102–S112. [Google Scholar] [CrossRef]
  15. Porter, J.; Huggins, C.E.; Truby, H.; Collins, J. The Effect of Using Mobile Technology-Based Methods That Record Food or Nutrient Intake on Diabetes Control and Nutrition Outcomes: A Systematic Review. Nutrients 2016, 8, 815. [Google Scholar] [CrossRef] [PubMed]
  16. Campbell, J.; Porter, J. Dietary mobile apps and their effect on nutritional indicators in chronic renal disease: A systematic review. Nephrology 2015, 20, 744–751. [Google Scholar] [CrossRef] [PubMed]
  17. Bonilla, C.; Brauer, P.; Royall, D.; Keller, H.; Hanning, R.M.; Dicenso, A. Use of electronic dietary assessment tools in primary care: An interdisciplinary perspective. BMC Med. Inform. Decis. Mak. 2015, 15, 14. [Google Scholar] [CrossRef] [PubMed]
  18. Ashman, A.M.; Collins, C.E.; Brown, L.J.; Rae, K.M.; Rollo, M.E. Validation of a Smartphone Image-Based Dietary Assessment Method for Pregnant Women. Nutrients 2017, 9, E73. [Google Scholar] [CrossRef] [PubMed]
  19. Carter, M.C.; Albar, S.A.; Morris, M.A.; Mulla, U.Z.; Hancock, N.; Evans, C.E.; Alwan, N.A.; Greenwood, D.C.; Hardie, L.J.; Frost, G.S.; et al. Development of a UK Online 24-h Dietary Assessment Tool: Myfood24. Nutrients 2015, 7, 4016–4032. [Google Scholar] [CrossRef] [PubMed]
  20. Albar, S.A.; Carter, M.C.; Alwan, N.A.; Evans, C.E.L.; Cade, J.E. Formative evaluation of the usability and acceptability of myfood24 among adolescents: A UK online dietary assessments tool. BMC Nutr. 2015, 1, 29. [Google Scholar] [CrossRef]
  21. Albar, S.; Alwan, N.; Evans, C.; Greenwood, D.; Cade, J. Agreement between an online dietary assessment tool (myfood24) and an interview-administered 24-h dietary recall in British adolescents aged 11–18 years. Br. J. Nutr. 2016, 115, 1678–1686. [Google Scholar] [CrossRef] [PubMed]
  22. Wark, P.; Hardie, L.; Frost, G.; Alwan, N.; Carter, M.; Elliott, P.; Ford, H.E.; Hancock, N.; Morris, M.A.; Mulla, U.Z.; et al. Validity of an online 24-hour recall tool (myfood24) for dietary assessment in population studies: Comparison with biomarkers and standard interviews. BMC Med. 2018, 16, 136. [Google Scholar] [CrossRef] [PubMed]
  23. Virzi, R.A. Streamlining the Design Process: Running Fewer Subjects. Available online: http://journals.sagepub.com/doi/abs/10.1177/154193129003400411 (accessed on 22 August 2018).
  24. Guest, G.; Bunce, A.; Johnson, J. How Many Interviews Are Enough?: An Experiment with Data Saturation and Variability. Field Methods 2006, 18, 59–82. [Google Scholar] [CrossRef]
  25. Bristol Online Survey. Available online: https://www.onlinesurveys.ac.uk/ (accessed on 22 August 2018).
  26. Brooke, J. SUS—A Quick and Dirty Usability Scale. Available online: https://hell.meiert.org/core/pdf/sus.pdf (accessed on 22 August 2018).
  27. Bangor, A.; Kortum, P.T.; Miller, J.T. An empirical evaluation of the system usability scale. Int. J. Hum. Comput. Interact. 2008, 24, 574–594. [Google Scholar] [CrossRef]
  28. Bangor, A.; Kortum, P.; Miller, J. Determining what individual SUS scores mean: Adding an adjective rating scale. J. Usability Stud. 2009, 4, 114–123. [Google Scholar]
  29. Petrie, H.; Bevan, N. The Evaluation of Acceptability, Usability and User Experience; The Universal Access Handbook; CRC Press: Boca Raton, FL, USA, 2009; pp. 10–20. [Google Scholar]
  30. Braun, V.; Clarke, V. Using thematic analysis in psychology. Qual. Res. Psychol. 2006, 3, 77–101. [Google Scholar] [CrossRef] [Green Version]
  31. NICE. Health App: GDm-Health for People with Gestational Diabetes. Available online: https://www.nice.org.uk/advice/mib131/chapter/The-technology#current-care-pathway (accessed on 22 April 2018).
  32. Rollo, M.E.; Ash, S.; Lyons-Wall, P.; Russell, A.W. Evaluation of a Mobile Phone Image-Based Dietary Assessment Method in Adults with Type 2 Diabetes. Nutrients 2015, 7, 4897–4910. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  33. Lieffers, J.R.; Hanning, R.M. Dietary Assessment and Self-monitoring: With Nutrition Applications for Mobile Devices. Can.J. Diet. Pract. Res. 2012, 73, e253–e260. [Google Scholar] [CrossRef] [PubMed]
  34. Probst, Y.C.; Tapsell, L.C. Computerised dietary assessment interviews: Health professionals and patients’ opinions about web communications. Nutr. Diet. 2012, 69, 56–63. [Google Scholar] [CrossRef]
  35. Carter, M.C.; Burley, V.; Nykjaer, C.; Cade, J. “My Meal Mate” (MMM): Validation of the diet measures captured on a smartphone application to facilitate weight loss. Br. J. Nutr. 2013, 109, 539–546. [Google Scholar] [CrossRef] [PubMed]
  36. Darwin, Z.; Smith, D.; Haddrill, R.; Scott, E.; McGowan, L.; Cade, J. Women’s experiences of pregnancy and the management of gestational diabetes: A qualitative interview study. BMC Pregnancy Childbirth 2018. submitted. [Google Scholar]
  37. Michie, S.; Ashford, S.; Sniehotta, F.F.; Dombrowski, S.U.; Bishop, A.; French, D.P. A refined taxonomy of behaviour change techniques to help people change their physical activity and healthy eating behaviours: The CALO-RE taxonomy. Psychol. Health 2011, 26, 1479–1498. [Google Scholar] [CrossRef] [PubMed]
  38. Sturt, J.; Dennick, K.; Hessler, D.; Hunter, B.M.; Oliver, J.; Fisher, L. Effective interventions for reducing diabetes distress; systematic review and metaanlysis. Int. Diabetes Nurs. 2015, 12, 40–55. [Google Scholar] [CrossRef]
  39. Fisher, L.; Hessler, D.; Glasgow, R.E.; Arean, P.A.; Masharani, U.; Naranjo, D.; Strycker, L.A. REDEEM: A pragmatic trial to reduce diabetes distress. Diabetes Care 2013, 36, 2551–2558. [Google Scholar] [CrossRef] [PubMed]
  40. Jenum, A.K.; Mørkrid, K.; Sletner, L.; Vangen, S.; Vange, S.; Torper, J.L.; Nakstad, B.; Voldner, N.; Rognerud-Jensen, O.H.; Berntsen, S.; et al. Impact of ethnicity on gestational diabetes identified with the WHO and the modified International Association of Diabetes and Pregnancy Study Groups criteria: A population-based cohort study. Eur. J. Endocrinol. 2012, 166, 317–324. [Google Scholar] [CrossRef] [PubMed]
Figure 1. Study flow chart.
Figure 1. Study flow chart.
Nutrients 10 01147 g001
Figure 2. Process of accessing and completing an online myfood24 food record.
Figure 2. Process of accessing and completing an online myfood24 food record.
Nutrients 10 01147 g002
Figure 3. Number of myfood24 food records completed by participants (n = 199).
Figure 3. Number of myfood24 food records completed by participants (n = 199).
Nutrients 10 01147 g003
Table 1. Comparison of characteristics between women who completed myfood24 at least once (n = 121) and those who did not (n = 78).
Table 1. Comparison of characteristics between women who completed myfood24 at least once (n = 121) and those who did not (n = 78).
Completed myfood24 (n = 121)Not Completed myfood24 (n = 78)p Value
nMeannMean
Maternal Age, years (SD)12133.5 (4.6)7833.2 (5.6)0.7
Booking body mass index (BMI), kg/m² (SD)11929.5 (6.2)7730.0 (7.0)0.7
Fasting Blood Glucose, mmol/l (SD)1154.9 (0.6)615.2 (0.8)0.008
Ethnicity0.001
White (%)7965.33544.9
Asian (%)3125.62329.5
* Other (%)119.12025.6
Parity0.02
Primiparous (%)52432126.9
Multiparous (%)69575773.1
* Other ethnicities included Black, Chinese and Mixed race.
Table 2. Characteristics of myfood24 users who completed the user questionnaire (n = 73) and those who did not (n = 48).
Table 2. Characteristics of myfood24 users who completed the user questionnaire (n = 73) and those who did not (n = 48).
Completed User QDid Not Return User Qp Value
nMeannMean
Age, Years (SD)7334.3 (4.3)4832.2 (4.7)0.01
Pre-Pregnancy BMI, kg/m2 (SD)7228.3 (5.0)4731.4 (7.5)0.008
Fasting Blood Glucose, mmol/L (SD)724.9 (0.7)435.0 (0.5)0.5
Ethnicity
White (%)5271.22756.30.4
Asian (%)1520.61633.3
Other (%)68.2510.4
Parity 0.9
Primiparous (%)3142.52143.8
Multiparous (%)4257.52756.2
Number of days completed myfood24 (SD)734.2 (1.1)483.2 (1.6)<0.001
System Usability Scale (SUS) score (95% CI)7370.9 (67.1, 74.6)---
Occupation68 ---
Managerial & professionals (%)3247
Intermediate & lower supervisory 1725
Semi routine & routine (%)46
Not employed (%)1522
Education level70 ---
Degree or above (%)4258.3
No degree (%)3041.7
Had previously filled in food diary (%)2635.6---
Previously used technology to record food (%)3345.2---
Internet ability good to excellent (%)7197.3---
Access to internet (%)7298.6---
Use internet daily (%)6994.5---
Confident in using technology (%)6690.4---

Share and Cite

MDPI and ACS Style

Gianfrancesco, C.; Darwin, Z.; McGowan, L.; Smith, D.M.; Haddrill, R.; Carter, M.; Scott, E.M.; Alwan, N.A.; Morris, M.A.; Albar, S.A.; et al. Exploring the Feasibility of Use of An Online Dietary Assessment Tool (myfood24) in Women with Gestational Diabetes. Nutrients 2018, 10, 1147. https://doi.org/10.3390/nu10091147

AMA Style

Gianfrancesco C, Darwin Z, McGowan L, Smith DM, Haddrill R, Carter M, Scott EM, Alwan NA, Morris MA, Albar SA, et al. Exploring the Feasibility of Use of An Online Dietary Assessment Tool (myfood24) in Women with Gestational Diabetes. Nutrients. 2018; 10(9):1147. https://doi.org/10.3390/nu10091147

Chicago/Turabian Style

Gianfrancesco, Carla, Zoe Darwin, Linda McGowan, Debbie M. Smith, Roz Haddrill, Michelle Carter, Eleanor M. Scott, Nisreen A. Alwan, Michelle A. Morris, Salwa A. Albar, and et al. 2018. "Exploring the Feasibility of Use of An Online Dietary Assessment Tool (myfood24) in Women with Gestational Diabetes" Nutrients 10, no. 9: 1147. https://doi.org/10.3390/nu10091147

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop