The Influence of the Early Childhood Education and Care Environment on Young Children’s Physical Activity: Development and Reliability of the PLAYCE Study Environmental Audit and Educator Survey

(1) Background: Participation in physical activity is crucial for the healthy growth and development of young children. More robust measurement of environmental influences on children’s physical activity in early childhood education and care (ECEC) settings may help resolve inconsistencies in the literature. This study evaluated the reliability of an environmental audit and educator practice survey for assessing ECEC physical, policy, and social environments related to young children’s physical activity. (2) Methods: A convenience sample of 20 ECEC centres participated in this PLAYCE (Play Spaces and Environments for Children’s Physical Activity) sub-study. Trained auditors conducted audits and educators completed surveys, two weeks apart. Test-retest reliability of the survey (n = 32), inter-rater (n = 20 pairs) and intra-rater reliability (n = 38) of the audit was assessed using intra-class correlation coefficients (ICCs), Kappa statistics and percent agreement. (3) Results: Intra-rater and inter-rater reliability ICCs for outdoor equipment, spaces and features were good to excellent (ICC = 0.70–0.94), while ratings for indoor equipment, media and spaces varied from fair to excellent (ICC = 0.46–0.78). The majority of items were rated by Kappa as moderate or above for intra-rater, inter-rater and survey test-retest reliability. (4) Conclusions: The PLAYCE Study instruments provide reliable measures of ECEC physical activity environments which can help to better understand influences on young children’s physical activity.


Introduction
Physical activity is vital for the healthy growth and development of young children [1]. For pre-school aged children (1-5 years), participation in regular physical activity is associated with better health and developmental outcomes including a healthy weight status, fitness, motor development and bone health [2,3]. However, many preschool children do not meet recommendations for physical activity, although estimates vary considerably [4].
The majority of young children in Europe aged 3-4 years, over half in Australia aged 2-3 years, and almost one quarter of US children under 5 years of age, attend some type of formal early childhood education and care (ECEC) [5][6][7]. Overall, a large proportion of young children spend a significant amount of time in ECEC. ECECs play an important role in promoting and providing physical activity opportunities for young children enabling them to meet the recommended 180 minutes of physical activity per day [8,9]. There is evidence that ECEC environments have a greater influence on preschooler's physical activity than socio-demographic factors [10]. Thus, the ECEC setting is an ideal target for further research and intervention to increase preschoolers' physical activity.
In line with the socio-ecological framework [11], a range of amenable environmental factors have been positively associated with preschoolers' physical activity whilst attending ECEC [12,13]. Physical environmental factors include the availability of play equipment and size of play space [14][15][16][17][18], and social environmental factors include educator practices such as providing children with opportunities to be active and participating in physical activity education and training [19][20][21]. A 2016 review of the correlates of children's physical activity and sedentary behaviour in ECECs found the most significant influence within the ECEC setting was the outdoor physical environment [12]. However, inconsistencies exist as it also concluded that there is inconclusive or limited evidence for most social, physical and policy environmental factors, except for educators providing active opportunities for children and the size of play space [12]. This may be attributed, in part, to varied and imprecise measurement. Therefore valid and reliable measures of the ECEC social, physical and policy environment have been recommended [22].
Robust assessment of ECEC environments is important both for determining influences on young children's physical activity, and the monitoring and assessment of ECEC regulatory standards [23]. Of the various environmental measurement instruments available, the Environment and Policy Assessment and Observation (EPAO) tool has been most extensively used to assess the social, physical and policy environment of ECEC services [24]. While US and northern European studies show associations between EPAO sub-scales and physical activity [19,25], recent studies in the UK and Australia found no relationship [26,27], raising concerns about applicability across different ECEC environments. Furthermore, aspects of the natural environment, the availability of electronic media, and other features of the indoor and outdoor space, may all potentially influence physical activity for children in ECEC [28]. Measurement of these aspects of the environment may be better addressed in instruments that assess ECEC environments for physical activity.
The purpose of this study was to develop and test the reliability of two measurement instruments-an environmental audit and an educator practice survey-for assessing the ECEC physical, policy and social environment related to young children's physical activity.

Materials and Methods
This research was part of the PLAYCE (Play Spaces and Environments for Children's Physical Activity) study which sought to determine features of the ECEC environment associated with preschoolers' physical activity and inform ECEC regulatory standards in Australia. This cross-sectional observational study collected accelerometry data from 1596 children aged 2-5 years attending ECEC in Perth, Western Australia. Potential individual, social and physical environmental influences on children's physical activity at the ECEC, home and neighborhood level, were collected using a parent survey and geographic information systems measures, in addition to the instruments described in this manuscript. The study methodology has been published previously [28].

Ethical Consideration
Ethics approval was granted from The University of Western Australia Human Research Ethics Committee (approval number: RA/4/1/7417).

Instrument Development
The PLAYCE Study ECEC environmental audit and educator practice survey were informed by a review of measurement tools used to measure the correlates of physical activity and sedentary behaviour of children attending ECEC and findings from previous pilot work [29]. Established items with good validity and reliability in international contexts were chosen where possible and adapted to suit the Australian ECEC environment if necessary. New items were drafted in consultation with the research team [28]. Each instrument was pilot tested by six people in three ECEC centres. The environmental audit was completed by six research project team members who provided feedback on the study protocol, including practical implementation, and identified items that were unclear or missing. The educator practice survey was completed by six ECEC educators who provided written feedback on the relevance and clarity of questions. Feedback was collated and both instruments were revised and finalised in discussion with study investigators.
PLAYCE ECEC environmental audit-the audit was designed to objectively assess elements of the physical and policy environment of ECEC centres that may support (or hinder) children's physical activity. The physical environmental component assessed the indoor and outdoor environment including spaces, equipment, and built and natural features and included: (1) outdoor physical activity equipment (number, type and condition of portable and fixed); (2) outdoor play environment (number and type of play spaces, natural features, paths); (3) indoor physical activity equipment (number, type and condition of portable and fixed); (4) indoor media equipment (number and type of portable and fixed); and (5) indoor play environment (presence and connection of play spaces). The policy environmental component of the audit addressed physical activity, screen time and sun-protection policy. The majority of items were modified from previous instruments [17,[30][31][32] for the Australian ECEC environment (Table 1 footnotes).  PLAYCE ECEC educator practice survey-the survey assessed the practices of ECEC staff to promote and support children's physical activity including: (1) time provided for physical activity (duration and frequency of indoor and outdoor physical activity); (2) educator role in physical activity (lead, supervise, encourage, join in); (3) educator physical activity practices (incorporate in other activities, use as behaviour management, talk about it, use promotional materials); (4) educator professional development; (5) provision of physical activity equipment (availability and offering); and (6) indoor spaces for play (accessibility, availability and use). In relation to sedentary behaviour the survey assessed: (7) time provided for screen time (duration and opportunity for TV time and interactive screen time; use as a reward). All items were taken from the Nutrition and Physical Activity Self-Assessment for Child Care (Go NAP SAAC) instruments [31], except availability and use of indoor play space items [30] and new items ( Table 2). Most items were updated and modified for the Australian ECEC context; for example, using the term 'educators' instead of teachers, referring to the Australian specific 'Get up and Grow' physical activity resources, separating indoor and outdoor physical activity frequency, and including interactive whiteboards as an educational screen time example.

Study Participants and Protocol
Data were collected to assess intra-rater reliability and inter-rater reliability for the PLAYCE ECEC environmental audit and test-retest reliability for the PLAYCE ECEC educator practice survey. The PLAYCE principal investigator and project coordinator briefed a team of ten auditors, consisting of PLAYCE investigators, project staff and postgraduate students on how to administer the environmental audit. The 90-minute briefing introduced the instruments, providing examples and considerations for completing, and presented the research protocol for conducting the environmental audit in ECECs. A convenience sample of 12 ECEC centres in the Perth metropolitan area (Western Australia) were invited, and eight provided written consent to participate.
To assess the intra-rater reliability of the PLAYCE ECEC environmental audit, each centre was visited at Time 1 by between three and nine auditors who completed the physical environmental sections by walking through each indoor room and outdoor space and recording their answers. Auditors accessed a copy of the centre policies to complete the policy environmental sections. Two weeks later the same auditors returned to the centre and conducted the audit a second time using the same protocol (Time 2). Auditors completed the audit independently. Each visit took 45-90 minutes depending on the size and features of the centre. Centre visits were completed during March-April 2015 prior to commencement of the main PLAYCE study. Auditors were assigned to each centre based on their availability at the centre's preferred visiting times and the size of the centre to audit, providing a total of 38 paired audits at Time 1 and 2, across the eight participating centres.
To assess inter-rater reliability of the PLAYCE ECEC environmental audit, data from the only two auditors who assessed all eight centres at Time 1 as part of the intra-rater reliability data collection, were combined with data from another pair of auditors. This second pair of auditors conducted the audit at the first 12 ECEC centres that agreed to take part in the main PLAYCE study [28] during September-December 2015, to provide an overall sample of 20 centres for assessing inter-rater reliability. Both sets of auditors followed the same procedure for completing the environmental audit as outlined above.
To assess test-retest reliability of the PLAYCE ECEC educator practice survey, the centre director and educators who cared for children 2-5 years were invited to complete the first survey (Time 1) during the audit team visit, and the second survey (Time 2) two weeks later.

Statistical Analysis
Socio-economic status (SES) of the ECEC services' location was determined by the Socio-Economic Indexes for Areas (SEIFA) suburb ranking [33] and categorised into tertiles. Size of centres was determined according to the number of approved child places and divided into quartiles-small (≤ 41 places), small/medium (42-57 places), medium/large (58-74 places) and large (≥ 75 places).
For the PLAYCE ECEC environmental audit, individual play equipment item scores were truncated at 20 and the number of types or individual pieces of equipment, features and spaces were totalled. Intra-rater reliability was assessed by comparing the completed audits for each auditor at Time 1 and Time 2, reporting single measure intraclass correlation coefficients (ICCs) using a one-way random absolute model for continuous variables and Cohen's Kappa with percent agreement for categorical variables. Inter-rater reliability was assessed by comparing completed audits between the two pairs of raters, using the same statistics. Test-retest reliability for the PLAYCE ECEC educator survey was assessed by comparing the responses for each Educator at Time 1 and Time 2. Categorical items and ordinal scales were assessed using a weighted Kappa statistic and percent agreement.

Sample Characteristics
Of the eight centres that participated in the environmental audit intra-rater reliability testing, 38% were from the lowest SES tertile, 12% were from the middle and 50% were from the highest. Half of the centres were large in size and 25% were medium-to-large. On average, five auditors visited each centre (range: 3-9), and audits were completed seven days apart.
Of the 20 centres included in the environmental audit inter-rater reliability testing, 50% were from the lowest SES tertile, 15% were from the middle, and 35% were from the highest SES tertile. The majority of the centres were very either small (35%) or large (50%).
A total of 32 educators from seven of the eight centres participating in the environmental audit testing completed the educator practice survey at Time 1 and Time 2. Educators were aged 20-57 years (mean 33 years; SD 11.1), 97% were female and 28% held a diploma or higher degree. Surveys were completed on average 13 days apart (SD 7.13).

Intra-and Inter-Rater Reliability of Play Spaces and Environments for Children's Physical Activity Early Childhood Education and Care (PLAYCE ECEC) Environmental Audit
Intra-rater reliability ICCs for outdoor equipment, spaces and features were excellent (ICC = 0.80-0.94) ( Table 1). The results for indoor equipment, media and spaces were more varied with indoor fixed play equipment the least reliable (ICC = 0.48-fair) and indoor fixed media equipment the most reliable (ICC = 0.77-excellent). Kappa ratings of moderate agreement or above were achieved for more than three quarters of the items. Intra-rater reliability showed slight agreement for the two "condition of outdoor equipment" items, although percent agreement was 79% and 69% for condition of fixed and portable outdoor equipment, respectively.
Inter-rater reliability ICCs for outdoor equipment, spaces and features were good to excellent (ICC = 0.70-0.79) ( Table 1). Results for indoor equipment, media and spaces varied from excellent for indoor fixed play equipment (ICC = 0.78) to fair for indoor portable play equipment (ICC = 0.46) and indoor portable media equipment (ICC = 0.47). For categorical items half showed moderate or substantial inter-rater reliability. The items addressing "condition of fixed outdoor equipment", "outdoor running space" and "outdoor path shape" demonstrated poor or slight agreement (κ ≤ 0.20) and percent agreement was 74%, 55% and 42% respectively.

Test-Retest reliability of the PLAYCE ECEC Educator Practice Survey
Kappa statistics across the 57 items ranged from −0.07 to 0.82 and percent agreement ranged from 38% to 94% (Table 2). Using the Kappa rating, more than two thirds of items were rated moderate agreement or above. Due to large imbalances in the distribution of responses, three items recorded negative Kappa statistics-"frequency that educators supervise during physical activity time" (κ = −0.04), "duration of TV time per week" (κ = −0.03) and "frequency that TV time is used as a reward" (κ = −0.07), although percentage agreement scores for these items ranged from 85% to 88%, indicating a high level of consistency.

Discussion
Reliable measures of environmental attributes are vital for understanding how the ECEC environment influences young children's physical activity and for addressing inconsistent findings to date. This study evaluated the reliability of the PLAYCE study ECEC environmental audit and educator practice survey, designed to assess the environments that influence young children's physical activity while at ECEC as part of a socio-ecological approach. The results show that the environmental audit provides excellent intra-rater reliability and good to excellent inter-rater reliability for outdoor equipment, natural features and play spaces. Audit items that consisted of presence or observed counts generally performed better than those that required a quality assessment. Furthermore, the majority of items measuring educators' physical activity-related practices were rated as moderate agreement and above. The PLAYCE study instruments extend previous tools by including the assessment of ECEC natural features and indoor play spaces, to provide a reliable and comprehensive assessment of the physical, social and policy environment related to physical activity in ECEC.
The PLAYCE study instruments were developed to improve upon previous tools to comprehensively assess the ECEC environment for physical activity in the Australian context [31,32]. The environmental audit differentiates itself from other observational audits by comprehensively assessing both the indoor and outdoor ECEC environment. Physical activity equipment and the size of play space have been included in most instruments using an ecological approach to measure the ECEC physical environment [34,35]. There has been less focus on other elements of the physical environment such as outdoor and indoor design features and spaces for different types of play. While these elements have been associated with physical activity within ECEC settings in individual studies [12], they have not been routinely measured alongside other physical, social and policy environment measures. Furthermore, the two PLAYCE study instruments complement each other by providing an assessment of the presence of physical activity related equipment and spaces, as well as aligned educator practices (e.g., frequency equipment and spaces are available, and used for physical activity). This will allow further exploration of how ECEC physical and social environments may interact to facilitate or inhibit children's physical activity.
Robust measurement of ECEC environments can also play an important role in monitoring and assessing ECEC quality standards [23]. In Australia, the Children's Education and Childcare Quality Authority (ACECQA) regularly assess ECEC centres on meeting quality, standards and elements, including 'children's health and safety' and 'physical environment'. However, these quality area standards, their assessment and ECEC accreditation are not currently aligned to physical environmental factors known to influence young children's physical activity in ECEC settings. The data collected by the environmental audit and educator practice survey during the PLAYCE study can be used to inform future ACECQA standards and their assessment [28].
This study had several limitations. While written and pictorial descriptors were included to provide greater clarity on what constituted a particular piece of equipment or play space, items assessing the condition of equipment may also have benefited from further description and auditor training as these performed most poorly. Furthermore, large imbalances in the distribution of responses to some items resulted in high levels of percent agreement but low Kappa values. Also, while the study tested three types of reliability, validity of the instruments was not tested. Finally, the PLAYCE study instruments have been implemented in the Australian ECEC context and testing in other countries with differing physical, social and policy environments is recommended. Despite these limitations, the instruments described in this study are both comprehensive and reliable and enhance current methods of environmental data collection in ECECs. The PLAYCE instruments may help with the benchmarking of current environmental standards within ECECs and facilitating further research to identify influences and support the improvement of ECEC environments for young children's physical activity.

Conclusions
The ECEC setting provides an important intervention opportunity for increasing physical activity and improving the health and development of young children. The PLAYCE study instruments provide reliable measures of the social, physical and policy environment for physical activity in ECECs, and can help to better understand the factors that influence physical activity and sedentary behaviour in early childhood and resolve current inconsistencies in the literature. The PLAYCE study instruments have been used to assess the ECEC environment and contribute to recommendations on meeting regulatory quality standards for children's health and the environment for ECEC in Australia.

Funding:
The PLAYCE study is supported by funding from the Western Australian Health Promotion Foundation (Healthway; #24219). HC is supported by an Australian National Heart Foundation Future Leader Fellowship (#100794). GT is supported by a NHMRC Early Career Research Fellowship (ID1073233).