Childhood Immunization in Ethiopia: Accuracy of Maternal Recall Compared to Vaccination Cards

Health surveys conducted in low- and middle-income countries typically estimate childhood vaccination status based on written vaccination cards, maternal recall (when cards are not available), or a combination of both. This analysis aimed to assess the accuracy of maternal recall of a child’s vaccination status in Ethiopia. Data came from a 2016 cross-sectional study conducted in the Southern Nations, Nationalities, and Peoples’ (SNNP) Region of Ethiopia. Vaccine doses received by a given 12–23-month-old child were recorded from both a vaccination card and based on maternal recall and then compared. Concordance, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and Cohen’s Kappa were calculated. Estimates of full immunization coverage were similar when collected via vaccination card (75%) and maternal recall (74%). For fully vaccinated children, comparison of maternal recall versus vaccination card showed high concordance (96%), sensitivity (97%), specificity (93%), PPV (98%), NPV (92%), and Kappa (90%). Accuracy of maternal recall of a child’s vaccination status is high in the SNNP region of Ethiopia. Although determination of vaccination status via vaccination card is preferred since it constitutes a written record, maternal recall can also be used with confidence when vaccination cards are not available.


Introduction
Since its inception in the 1970s, the World Health Organization's Expanded Program on Immunization (EPI) has aimed to increase childhood vaccination coverage globally by making recommendations for childhood vaccination schedules and working with ministries of health to provide those vaccines, for free, to children [1]. Public health programming for these childhood vaccination services relies on valid estimates of children's vaccination status in order to develop accurate estimates of population-level coverage. Health surveys conducted in low-and middle-income countries often assess vaccination status using maternal (caregiver) recall when vaccination cards are unavailable and, as such, it is important to understand the validity of caregiver recall compared to a print vaccination card in documenting a child's immunization status.
Prior studies have reported mixed findings regarding the accuracy of caregiver recall of child vaccination status. Reports from Tanzania [2], Sudan [3], India [4], Egypt [5], Bangladesh [6], and Canada [7] all reported high accuracy and/or validity of parental recall of child vaccination status. However, a 2018 systematic review of the quality of caregiver recall in low-and middle-income

Materials and Methods
Data for this analysis came from a cross-sectional survey conducted in the summer of 2016 in Worabe town in the Southern Nations, Nationalities, and Peoples' Region of Ethiopia.  Ethiopian census reported 27,852 residents in Worabe town; 15% of these residents were under five years of age [27]. Details of the survey methodology are published elsewhere [27]. Briefly, community health workers traveled to each of the two kebeles (equivalent to neighborhoods) in Worabe town and determined a random starting point using a random number chart. From this starting point, all households in each kebele were enumerated. Interviewers then started at the random starting point and visited each home to see if at least one child aged 12-23 months lived there and interviewed the children's mother. Subsequent neighboring households were visited until the sample quota of 270 homes per kebele was obtained. Data collected included sociodemographic characteristics, health services utilization, child characteristics, child vaccination status, and maternal knowledge regarding vaccines and vaccine-preventable diseases. Information about the child's receipt of individual vaccines and number of doses was obtained from both maternal recall and vaccination cards, if available [27].
Study participants were initially asked to recall their child's vaccination status (i.e., maternal recall) and then to provide the child's vaccination card; only those participants who did both were included in the study. Several variables were created for this analysis. When the vaccination card indicated that a child received a specific dose but no date was recorded, the child was considered vaccinated. For the oral polio vaccine, the pentavalent vaccine, the pneumococcal conjugate vaccine, and the rotavirus vaccine, a new variable was generated that summed the number of doses the child received of each vaccine, according to the vaccination card. A binary variable was used to indicate whether the child was fully immunized; a child was considered fully vaccinated if he or she received one dose of BCG, three doses of OPV, three doses of the pentavalent vaccine, three doses of PCV, two doses of the rotavirus vaccine, and one dose of the measles-containing vaccine. Child's vaccination status according to maternal recall was treated similarly. Variables were crafted to indicate whether vaccination status via recall and card were the same in children who were fully vaccinated and for each vaccine. Some categories of sociodemographic variables (religion, occupation, parity, marital status, etc.) were collapsed to allow sufficient cell sizes to conduct the appropriate analyses.
Vaccination coverage was calculated by source (i.e., recall vs. card) and compared. Net reporting bias was defined as the difference between vaccination coverage obtained via recall versus vaccination card. To determine the validity of maternal recall in correctly identifying a child's vaccination status, concordance, sensitivity, specificity, positive predictive value, negative predictive value, and Kappa were computed using vaccination status obtained from the vaccination card as the gold standard. Accuracy of maternal recall was assessed as the percent of mothers who correctly recalled their child's vaccination status (compared to the vaccination card). False positives were the percent of mothers who reported their child received a vaccine when the card indicated the child had not received that vaccine (i.e., mothers who overestimated their child's vaccination status, Table 1). False negatives were the percent of mothers who reported their child did not receive a vaccine when the vaccination card designated the child had received the vaccine (i.e., mothers who underestimated their child's vaccination status). All analyses were conducted for full vaccination and each vaccine type. Due to small sample sizes, for vaccines provided in a series of multiple doses (i.e., the pentavalent vaccine), analyses only compared whether maternal recall and the vaccination card reported receipt of any doses of the vaccine rather than maternal recall of specific doses of the vaccine. Associations between maternal recall accuracy and demographic factors were examined using a chi-square analysis. Analyses were conducted using SAS version 9.4 (SAS Institute, Cary, NC, USA).
Approval for data collection was granted by the Institutional Review Board at St. Paul's Hospital Millennium Medical College and the Worabe Health Bureau approved study activities in their region. Verbal informed consent was attained by data collectors prior to participant enrollment. This analysis was approved by the University of Michigan Health Sciences and Behavioral Sciences Institutional Review Board.

Results
Interviewers contacted 540 mothers and 56 declined to participate, leaving a total of 484 (90%) mothers in the original survey. Of these, 247 participants were excluded from this analysis because they did not have their child's vaccination card and 5 individuals were eliminated because their child's age was outside of the 12-23 months age range. These exclusions resulted in a final sample size of 232 individuals for this analysis. By design, only mothers were interviewed (i.e., no other family members were interviewed). Mothers' mean age was 28 years and the majority of women were married (97%), Muslim (87%), and worked as homemakers (60%, Table 2). Half of the children were females (53%) and child's mean age was 17 months. Table 2. Descriptive statistics of sample of women and children included in accuracy analysis (n = 232).

Variable Frequency (Mean) Percent (Standard Deviation)
Child characteristics Estimates of full vaccination coverage were high based on both vaccination cards (75% fully vaccinated) and maternal recall (74%, Table 3). Net reporting bias was negative for estimates of full vaccination and receipt of all recommended OPV doses, indicating estimates of vaccination status from cards were higher than from maternal recall. Conversely, net reporting bias was positive for receipt of BCG vaccine, all recommended pentavalent vaccine doses, all recommended PCV doses, both recommended rotavirus doses, and measles vaccine, which suggests mothers overestimated receipt of individual vaccines, though the magnitudes of these measures were small (Table 3). Most vaccinated children received the recommended number of doses of each vaccine: one dose of the BCG vaccine, three (or four) doses of the OPV vaccine, three doses of the PCV vaccine, and two doses of the rotavirus vaccine (Table 4).  Concordance between maternal recall and vaccination card was high, with all measures over 95%. Kappa, however, was quite variable; Kappa values were high for full vaccination and receipt of the measles vaccine but modest for receipt of all other vaccines (Table 5). Maternal recall was highly sensitive with measures for all vaccines over 97%, indicating that of the children whose vaccination cards showed they had received a vaccine, over 97% of mothers correctly identified their child as vaccinated with that vaccine. Specificity of maternal recall was high for full vaccination (93%) and the measles vaccine (92%) but generally low for receipt of all other vaccines. This suggests that of the children whose vaccination cards showed they had not received a vaccine, few mothers correctly identified their child as unvaccinated. Similar to sensitivity, positive predictive values were over 97% for all vaccines; of children whose mothers indicated they were vaccinated, over 97% were truly vaccinated according to their vaccination card. Negative predictive value was also generally high, indicating that of children whose mothers indicated that they were not vaccinated, most were not vaccinated based on the card. Recall accuracy was extremely high for full vaccination (96%) and receipt of each type of vaccine (Table 6). False positive and false negative reports of vaccine receipt were uncommon for full vaccination (2% of reports were false positives and 2% were false negatives) and each vaccine. No statistically significant associations were found between recall accuracy and demographic factors (Table 7). Table 6. Accuracy, recall false positives, and recall false negatives by vaccine.

Accuracy of Recall Frequency Percent
Accuracy of caregiver recall compared to vaccination card-full vaccination

Discussion
This analysis examined the accuracy of maternal recall of child vaccination status in a southern region of Ethiopia. We found generally high levels of concordance between maternal recall and vaccination card data. While mothers were more likely to overestimate the number of vaccine doses their child had received, they paradoxically tended to slightly underestimate whether their child was fully vaccinated with all recommended vaccine doses. A recent study from Tanzania that compared vaccination status via vaccination card and maternal recall similarly found mothers tended to overestimate receipt of individual doses of vaccine [2]. It has been hypothesized that overestimation in these circumstances may be due to social desirability bias; that is, women may feel that having a vaccinated child is more favorably perceived and may therefore report receipt of vaccine doses irrespective of their child's actual vaccination status [2]. Given that female community health workers were collecting data from mothers in our study, it may be that the sense of social pressure to report children as vaccinated was particularly acute. Conversely, it is also possible that the mothers' recall was accurate rather than representing an overestimation if a child received vaccinations that were not recorded on the vaccination card. This scenario might be more likely, for example, if a caregiver forgets to bring the vaccination card when their child is vaccinated, if perhaps in situations where the child is vaccinated at an appointment when the primary purpose of the visit was for something other than vaccination, or if the child was immunized as part of a special immunization activity (e.g., catch-up campaign or national immunization day), which may reduce the probability that vaccination(s) will be appropriately recorded on the card. In fact, when asked if their child had received vaccines other than those listed on the card, 99% of the women in our study responded affirmatively. Other studies have reported limitations of vaccination-card-reported data. One study from India that examined prospectively obtained vaccination information compared to vaccination cards and retrospective recall of vaccination status found that the vaccination cards of 70% of the sampled children were incompletely and/or inaccurately filled out [13]. Another study from Egypt noted problems with transcription of correct information onto the vaccination cards and suggested that some inconsistencies in maternal recall accuracy were due to vaccination card errors rather than faulty maternal recall [5].
Sensitivity of maternal recall was extremely high in this study, above 97% for full vaccination and also for receipt of specific vaccines. However, specificity was more variable, at over 90% for full vaccination status and for receipt of the measles vaccine, but much lower (25-50%) for receipt of BCG, pentavalent, PCV, and rotavirus vaccines. Low specificity is especially concerning in this context as it potentially represents a worst-case scenario-a mother who thinks her child is vaccinated with a specific vaccine (and therefore less likely to vaccinate with that same vaccine in the future) when the child is actually unvaccinated and therefore unprotected against disease. These findings align with low recall specificity of BCG, OPV, and DPT which was found in a prior Tanzanian study of maternal recall [2]. It may be that women conceptualize their child's vaccination status in terms of whether future visits to the health clinic will be needed or are necessary. Full vaccination of their child(ren) is a commonly articulated goal by women and, as such, it may be they are well informed about whether their child has received all necessary vaccines (and therefore do not need to return for future vaccinations) but may not know which specific vaccines are included in the series nor which her child has received. It is also likely that social desirability bias may play a role here, too. A woman may feel more comfortable telling a health worker that her child is not fully vaccinated (potentially interpretable as vaccination delay but with intentions to complete) as opposed to informing a health worker about the specific recommended vaccines and/or doses that her child has not yet received.

Strengths and Limitations
Like all studies, this one has limitations. The small sample size was a barrier to conducting analyses with sufficient power to support conclusions and the findings should be interpreted in that context. This may explain why statistically significant associations between sociodemographic factors and accuracy of maternal recall that have been identified in other studies (e.g., caregiver age and education [2]) were not significant in this analysis. Vaccines with multiple doses in a series only compared whether maternal recall and the vaccination card reported receipt of any doses of the vaccine, which limits conclusions regarding a mother's ability to correctly identify the number of doses her child received. Due to the small number of women who inaccurately identified their child's vaccination status, we were unable to run more nuanced analyses examining validity of recall of particular numbers of doses of vaccine. Women who could not produce a vaccination card for their child (51% of the initial sample) were excluded from this study. Vaccination card retention is related to a child's immunization status and may demonstrate a stronger maternal interest in vaccination services and thus potentially introducing selection bias in the study population [28]. Worabe, the site of this study, is a predominantly Muslim town-whereas the majority population in Ethiopia is Ethiopian Orthodox. Previous studies have found religion to be associated with both full vaccination [29] and vaccine nonreceipt [30] in Ethiopia. A larger, nationally representative survey comparing vaccination status via maternal recall and vaccination card would improve upon the sample size and the limits to generalizability present in this study. Additionally, studies have shown health provider records of vaccination receipt are generally more accurate than vaccination cards [14]. A comparison of maternal recall to health clinic records (instead of to vaccination cards) may provide a better estimate of the accuracy of maternal recall. However, Worabe health clinic records were not available to the study team. This study also has the important strength of being the first to examine the validity of maternal recall of a child's vaccination status in Ethiopia. Also, in addition to examining validity of maternal recall of full vaccination, this study examined the validity of maternal recall of each vaccine included in the Expanded Program on Immunization series.

Conclusions
Given that many health surveys rely on both vaccination cards and maternal recall to estimate childhood vaccination coverage and maternal recall validity, it is important to understand if maternal recall is a valid reflection of true vaccination status. The results of this study suggest that the accuracy of maternal recall of a child's vaccination status was generally high. Although determination of vaccination status via vaccination card is preferred, maternal recall should be used when cards are not available. The results of this and future related studies could help researchers and public health officials correctly estimate vaccination coverage in Ethiopia and elsewhere, which is necessary for appropriate planning of vaccination programs and outreach.