Technological innovations in dietary assessment have been pursued to reduce researcher and respondent burden, while enabling the collection of high-quality dietary data [1
]. One such innovation is the Automated Self-Administered 24-h (ASA24) Dietary Assessment Tool, developed by the U.S. National Cancer Institute [6
]. In addition to allowing self-administration of 24-h recalls (24HRs), ASA24 incorporates automated coding and thus negates the need for coders. As a result, it is possible to collect 24HRs in studies in which this would previously have been cost-prohibitive. Studies conducted among adults in the US have suggested high rates of accuracy of recalled intake relative to true consumption, as well as comparability with interviewer-administered 24HRs [7
]. Research also indicates high levels of acceptance of ASA24 among adults with Internet access [8
]. Evaluations with children have highlighted challenges among younger children and pointed to the need for additional research to determine the age at which it is possible to use a self-administered tool such as ASA24 [9
]. There has been little evaluation to date with older adults.
A Canadian adaptation of ASA24 has been developed to reflect the Canadian food supply [11
]. Modifications included adding foods and beverages unique to Canada and removing those not available in this country, changes to reflect differences in brand names in Canada compared to the U.S., and the addition of metric units. ASA24-Canada has been freely available since the spring of 2014 [11
]. An updated interface, released in the fall of 2016, expanded usage to mobile devices in addition to laptop and desktop computers, provided the capacity to collect food records as well as recalls, and introduced a French language option [11
]. As a result, ASA24 represents a potentially viable tool for the collection of dietary data in a range of research studies within the Canadian context.
Several studies to evaluate the feasibility and validity of ASA24 have been undertaken with Canadian populations, including preschoolers (with parental reporting), school-aged children, adults, and older adults. The objective of this paper is to discuss experiences in implementing ASA24 and lessons learned from these studies, with the aim of informing the use of technology-enabled dietary assessment tools with diverse samples.
3.1. Receptivity to Completing ASA24
Overall, participants within the studies were quite receptive to completing ASA24, even when they encountered technological challenges (elaborated upon below). Within the childcare setting (S1), participating families demonstrated a commitment to the study, with a low attrition rate: only 2 of 40 parents did not fully complete ASA24-Canada (both started the recall but did not complete all of the passes). In the school-based studies (S2 and S3), participating schools, teachers and children were enthusiastic about the study (particularly the validity study for which the provision of a free lunch was an exciting incentive for students). Within the classroom, students were generally willing to engage with the ASA24 system and the vast majority of those with parental consent and who assented to participate at least started the recall. Of students participating in the feasibility study (S2) who submitted a user experience survey (14%), the majority indicated they found completing ASA24 “very easy”, “easy”, or “neutral”. However, a number of children in that study either did not provide valid email addresses (21%) or attempt to complete the at-home ASA24 (62%).
In the feasibility study of adults (S4), participants also appeared keen to complete ASA24. Several participants contacted the study centre to inform staff about upcoming periods of unavailability (e.g., due to travel), seeking to remain part of the study nonetheless. For recalls conducted using ASA24-Canada-2014, participants spent considerable amounts of time navigating issues with the Silverlight plug-in, highlighting their willingness to complete the study elements. Further, despite some frustrations about the time involved in completing 24HRs, participants made it clear they wanted to remain in the study. During the course of the 16-week study, 50 participants (15% of the sample) actively withdrew; just under half did not provide a reason, while approximately one-fifth cited challenges completing ASA24 (other reasons included being too busy, travel, and too much time required to complete the assessments).
In the study of older adults (S5), respondents were eager to learn how to complete ASA24 and indicated they did not mind spending time doing so. Some older adults viewed this as a way to maintain and enhance their technological skills (though the extent to which study staff could assist in this regard was limited due to feasibility). Participants also shared accounts of spouses, children and friends helping them navigate technological issues to allow completion of ASA24. Across the age and sex strata (younger versus older, men versus women), fewer than 20% of those who completed the first recall did not complete the full study. In this study, recruiting participants in person appeared to have a beneficial effect on completion compared to recruiting by phone, perhaps due to the upfront time investment and rapport established during the first study visit and/or the training on the use of ASA24, which may have increased motivation and/or mitigated technological issues.
3.2. User Experiences with the ASA24 Interface
Given the self-administered nature of ASA24, some participants experienced challenges in navigating the passes. In particular, participants were sometimes unclear as to the next step to complete. For example, some students in the school-based studies (S2 and S3) assumed they were finished after describing a single eating occasion or after reporting the foods and drinks consumed at multiple meals but without providing details about these foods and drinks. In these cases, researchers encouraged students to continue by prompting them to “keep clicking, try your best, scroll down” (depending on the screen size on the device used, the entire frame may not show on a single screen, requiring scrolling to reach the next step). Further, some students did not understand that the search function required entering a single term (e.g., “banana”) at a time, sometimes entering multiple items (e.g., “banana cereal orange juice”), which does not return the desired results. Some did not understand how to filter using ASA24’s option for sorting search results by category. As well, some students had difficulty interpreting response options to the question “How much did you actually eat?”, perceiving the portion size images to show the plate or bowl once they were finished eating (i.e., an almost empty plate was interpreted to mean they finished most of their meal rather than representing the amount they actually ate). Finally, the language used is not necessarily child-friendly (e.g., “pat” of butter), leading to some confusion.
In the feasibility study of adults (S4), some respondents had difficulty finding the foods they wanted to add, particularly if they searched for a generic food item (e.g., “bread”) that sometimes resulted in a very long list of foods to consider. In the community-based study of older adults (S5), older participants were observed to be more likely to browse for foods using the categories (rather than searching), resulting in longer completion times. Some older participants had difficulties understanding how the search functioned and how to navigate the results returned, as well as how to move selected foods to their recall. Younger adults within this study (S5) appeared more comfortable using the search, speeding up the entry of foods. However, older adults were observed to be more patient than younger adults with the repetition in completing the multiple passes.
In the study of preschoolers (S1), some parents recalled only the dinner meal for which they were present with their children, despite ASA24’s repeated prompts to report all foods and drinks consumed the prior 24-h period. This suggests that parents perhaps did not understand they were to report all eating occasions, even those for which they were not present, or that they may not have felt confident reporting meals for which they did not have detailed knowledge.
3.3. Technological Challenges and Provision of Support
For studies using ASA24-Canada-2014, challenges were encountered related to the Microsoft Silverlight plug-in. Additionally, some participants in the various studies experienced “freezing” of ASA24 midway through completion, sometimes resulting in the loss of data. It was unclear whether this arose due to problems with Internet connectivity or other issues.
For the childcare centre- and school-based studies (S1, S2 and S3), researchers were available on site. Given the study objectives related to feasibility and validity, participants were encouraged to complete ASA24 independently to the extent possible in the face of issues that did not require intervention to address technological issues, such as problems entering passwords or “freezing”. In the community-based studies of adults and older adults, for which recalls were completed at home, there were relatively frequent requests for assistance from study staff.
In the feasibility study of adults (S4), participants were not provided with prior training in the use of ASA24, but assistance was available via phone or email. Over half of the participants in this study contacted the study centre for help at least once, and over one in ten made three or more contacts for assistance. Password issues, specifically forgotten or misplaced login credentials, were common. Staff were generally able to assist with these issues, but in some situations, had difficulty verbally describing non-alphanumeric characters used in passwords (e.g., @, !, and particularly & (ampersand)). The capacity to assist with requests for help by email or phone depended upon participants’ computer literacy, as well as the nature of the technical problem. As the study progressed, staff became familiar with common issues and, in some cases, sought the advice of the information technology team to find solutions that could be communicated to participants by phone or email. Staff members reported anecdotally that informing participants that similar issues had been encountered by others seemed to provide comfort and increased willingness to continue with the study.
In the community-based study of older adults (S5), most recalls were completed at home and assistance was available by email and telephone. Some participants experienced problems copying and pasting the link to ASA24 and with login credentials. It was observed that technological troubleshooting capacity was more limited in older participants compared to younger participants. However, compared to younger adults within the sample, older participants were willing to spend more time troubleshooting. Further, some older adults reported vision problems and used special computers and software that did not always display ASA24 correctly. Participants who were unable to complete recalls independently were given the opportunity to do so over the phone. Using ASA24 as an interviewer guide allowed research staff to collect recall data in a standardized manner. Most telephone-administered recalls required less than 20 min to complete; this was deemed a reasonable investment in terms of study resources to avoid drop out. Many older participants reported having smart phones or tablets, which they were more comfortable using than laptop/desktop computers. This study used ASA24-2014 but this observation speaks to the potential for ASA24-Canada-2016, which can be completed on mobile devices, to expand uses of ASA24.
3.4. Time to Complete
In the study of preschoolers (S1), time to complete ASA24 typically fell between 25 and 45 min (median, 35 min). Given that data collection was conducted during drop-offs to the childcare centre, some parents found it difficult to devote the time to complete ASA24-Canada due to work schedules. Some exceptions were made to allow parents to complete the recall later in the day from another location (e.g., work, home); this led to some technological challenges with the Silverlight plug-in. Time to complete ASA24 also emerged as a challenge within the school setting (S2 and S3), in which limited classroom time (a 50-min period, with ~35–40 min for ASA24) was allotted to the study activities. For school-aged children, the median time for completion was 35 min for the feasibility study and 34 min for the validity study (few students in the validity study completed ASA24-Kids, thus, these estimates are limited to ASA24-2016). Younger students were observed to take longer and many did not finish the recalls in the given time. Given our research interests, we opted to enable the ASA24 source module, which asks about the source (e.g., grocery store) of the main ingredients for each food and beverage; this likely added significantly to completion times.
In the feasibility study of adults (S4), the median time to complete recalls was 41 min; this decreased with successive recalls. The time to complete in the sample of older adults (S5) ranged from 15 min to 2 h; we do not report median completion times for this study since both self-administered and interviewer-administered ASA24 recalls were completed (these results will be detailed elsewhere). Based on interviewer-administered ASA24 recalls completed over the phone, time to complete appeared to depend on how complex the participant’s meals were, how long it took the participant to remember details, and the extent to which he or she discussed the details of what was consumed. For initial recalls completed in the presence of a researcher, it was observed that the reaction and processing time of older adults for each step was considerably longer than for younger adult participants.
3.5. Costs Associated with the Use of ASA24 in Research
The ASA24 tool itself is freely accessible to researchers to use. However, the implementation of ASA24 does require resources. For example, in the studies described here, there were costs associated with supporting participants (e.g., provision of assistance by telephone and email, completion of initial recalls in the presence of a researcher, interviewer-administration for some participants). As with any study, there are also resources associated with recruitment, contacting and monitoring participants, and data analyses. The costs associated with collecting recalls using ASA24 will vary in relation to sample size, the number of recalls to be completed, the level and types of support provided, and other study-specific factors.
3.6. Considerations in Conducting ASA24 Studies in Institutional Settings
The study of parents’ ability to report their children’s intakes was conducted within a childcare setting (S1) and provided the opportunity to assess the accuracy of parental reporting for meals both for which the parent was and was not present. However, as noted above, time challenges were sometimes an issue for busy parents. For the studies of school-aged children (S2 and S3), working within the school setting meant numerous constraints, including a lack of control over scheduling and time allotment for ASA24 completion, as well as interruptions (e.g., due to announcements). Further, some classroom computers had Internet Explorer only, which seemed to perform less optimally than Chrome for ASA24. Lastly, using email as a means of contacting students recruited within the school setting was not effective in terms of completion of subsequent at-home recalls.
3.7. Considerations in Administering ASA24 in Community-Based Samples
Practical issues influencing participants’ experiences of ASA24 emerged in the community-based studies. For example, the scheduling of recalls had a bearing on the availability of support for participants. In the feasibility study of adults (S4), support available by phone or by email was limited to office hours on weekdays. Thus, participants who required assistance on a Friday evening did not receive support until Monday morning, by which time the window for ASA24 completion had closed. Researchers can choose to allow unscheduled recalls (the participant can log in at any time to complete a recall) or use the scheduled recall setting (the participant must log in on a specific day to complete a recall). With the scheduled setting, some participants reported they would have liked to complete the recall, but owing to time constraints or technological issues for which they needed assistance, they missed the window. Changing to an unscheduled setting increased the proportion of participants who completed recalls, but increased the risk of reactivity as participants knew they were to complete a recall but may not have done so until the following day, providing an opportunity to modify their intake for the recalled period. Each of the studies involving at-home completion of ASA24 included multiple attempts as a means of increasing completion rates given that sometimes participants were willing to complete recalls but unable to do so in a given time period.
Misplaced passwords also posed challenges. In the feasibility study with adults (S4), passwords were mailed to participants in a letter shortly after they consented to participate, and in some cases, may have arrived up to three weeks prior to the first invitation to complete a recall. Participants who lost passwords had to contact study staff before they could proceed with ASA24. Passwords were also an issue for some participants in the study with older adults (S5).
3.8. Unique Issues in Validity Feeding Studies
Both feeding studies were conducted with children, one with preschoolers (with parent reporters) and the second with school-aged children. The childcare setting (S1) allowed for control over study food: researchers were present during meal preparation and ensured items were prepared according to the study protocol and accurately weighed prior to serving. Collecting and weighing plate waste was challenging given that young children can be messy eaters. For example, the research team had to collect spillage from the floor and bibs (Figure 2
). In addition, young children tend to mix foods together, making it difficult in some cases to decipher and weigh plate waste (Figure 3
In the feeding study with school-aged children (S3), conducting the study within classrooms offered the advantage of completing data collection with a number of children simultaneously, but this also posed challenges. On day one, despite being asked to leave all remaining items including packaging on trays and the watchful eyes of the researchers, some students took items with them for the recess break that followed lunch. Further, because participating schools were geographically dispersed, foods and drinks were ordered from a central caterer but delivered by local operators. Variation in items (e.g., standardization of carrots by number rather than weight, different brands of yogurt) complicated the study protocol in terms of calculating amounts consumed. Spillage and mixing of foods were not major problems as in the study of preschoolers and researchers monitored the lunch meal to avoid, or at least minimize, trading of foods and drinks.
Comparing and contrasting the experiences of researchers evaluating ASA24 in diverse samples provides insights into challenges that might be commonly encountered in research using technology-based tools and potential strategies to mitigate these. Overall, we found participants were receptive to completing ASA24, and in some cases, willing to invest significant effort to do so. However, technology does not negate all challenges in collecting dietary intake data and indeed, may introduce new issues, for example, due to the need for computer literacy. As a result, it is critical to carefully consider the implementation of technology-based approaches into research and the potential need for tailored supports for study participants; Figure 4
summarizes considerations emerging from the experiences of the research teams leading these five studies in implementing ASA24.
The research teams observed that the ASA24 interface and the steps in completing recalls were not necessarily intuitive for all respondents. Generally, challenges appeared greater for younger versus older children, presumably due to less developed cognitive skills. Further analysis of the validity study data for school-aged children will provide insights into the age at which children might be expected to complete ASA24 independently. As well, older versus younger adults sometimes exhibited less comfort with computers and the Internet. In the study of older adults, having participants complete an initial recall in the presence of a staff member appeared to be beneficial in supporting future independent completion of ASA24.
These studies also illustrated that the time to complete ASA24 can be variable. This was evident in those that included both younger and older children and younger and older adults, respectively, with younger children and older adults needing more time to complete ASA24 than others. Pilot testing is critical to ensure that the time allotted to ASA24 within a study is sufficient. Attention to the optional modules enabled within ASA24 is important from the perspective of time. In the school-based studies, the source module was enabled to allow collection of information on acquisition of foods from fast food restaurants, for example. However, given that this module results in an extra probe for most foods and drinks reported, it is likely to have added quite significantly to completion times. As with any assessment tool, it is necessary to consider the extent of data to be collected along with the respondent burden and other study constraints.
In some cases, technological challenges encountered in these studies relate to factors no longer relevant to ASA24, including problems with the Silverlight plug-in used in the 2014 and earlier versions, now superseded by ASA24-2016. Instances of “freezing” may have been related to bugs in the 2016 version shortly after release that have since been remedied. However, with the use of web-based tools, it is virtually impossible to prevent all possible technology-related issues given that the reliability of Internet connections can vary and completion of ASA24 can be affected by issues outside the system itself. This is likely a particular issue in studies including participants who may not have access to high-speed Internet. Given that it is difficult to predict all potential issues that participants may encounter when interacting with technology, the provision of some level of support from research staff is likely to be important in most studies. Comfort with different devices is also a consideration; for example, some older participants reported having smart phones or tablets, which they were more comfortable using than laptops or desktop computers. ASA24-2016 is operational on both mobile devices and laptops and desktops, offering flexibility to researchers and study participants.