Counting Bees: Learning Outcomes from Participation in the Dutch National Bee Survey

Abstract

Citizen science approaches to data collection are growing in popularity, in part because of their potential for achieving both scientific and educational objectives. Evaluating the impacts of participation on citizen scientists is important, yet such evaluations are still relatively rare. In addition, recent literature reviews indicate that existing studies often focus on content learning, make limited use of existing scales, and rarely report null results. This paper reports an evaluation of the demographic profile, motivations and learning outcomes of participants in the Dutch National Bee Survey, a national-scale citizen science project involving citizens in collecting observational data of wild bees. Using a repeated measures survey study, we assessed the impact of participation on respondents’ attitudes and behavioural intentions regarding bees, and their attitudes towards nature, citizenship, and citizen science. Our baseline (N = 373) and follow-up (N = 208) surveys indicate that our respondents are of a relatively high age and education level, have a pre-existing affinity with nature, and are strongly motivated by conservation concerns and learning about bees. Assessment of learning outcomes indicated a significant difference between two types of self-report questions: respondents reported significant gains in knowledge and appreciation of wild bees, yet attitudinal scales indicated no significant shifts in attitudes towards bees, nature, citizen science, or citizenship. In our discussion, we suggest several explanations for this finding, including respondents’ pre-existing affinity with nature, and advance suggestions for future research into citizen science learning outcomes.

1. Introduction

As we take steps around the world to combat biodiversity loss and species extinction, the importance of protecting our planet’s insect population is increasingly recognised. Recent studies paint a worrying picture, with a German study using data spanning 63 German nature areas over 27 years indicating a 76–82% decline in flying insect biomass [1]. Considering the crucial role insects play in processes such as pollination and food webs, the need to take action is abundantly clear. Bees in particular play an important role in pollination [2], and halting their decline thus receives increasing attention.

Data on the distribution of different types of bees, and understanding the factors that affect their decline or recovery, are crucial for formulating effective strategies for action. Citizen science approaches to monitoring bees hold great potential for improving our understanding of bee populations, while also engaging society in supporting this crucial species. In fact, the aforementioned study [1] drew on a dataset collected over 27 years by dedicated citizen scientists of the Entomological Society Krefeld. Other authors also assessed pollinator data collected by citizen scientists or through crowdsourcing [3,4,5,6].

1.1. The Netherlands Buzzes

It is within this context of increasing attention to pollinators and citizen science that the Dutch National Bee Survey (DNBS), our case study, was initiated. The DNBS is organised within the context of The Netherlands Buzzes (tNB), a multifaceted national project launched in 2018 by four partners: Naturalis Biodiversity Center, Natuur & Milieu, IVN Nature Education and LandschappenNL. The goal of tNB is to improve nesting opportunities and access to food for wild bees across the country [7]. These goals align well with those of the Dutch National Bee Strategy, signed in January 2018 by 43 partners, to contribute to nesting areas and food (referred to in the policy document as ‘bed and breakfast’) for pollinators, especially wild bees [8]. To realise these aims, tNB encompasses a wide variety of projects, including education programmes for both professionals and schools, a national ‘bee working day’, museum exhibitions, and an opportunity for municipalities to become recognised as a ‘bee-friendly municipality’.

It should be noted that tNB is focussed specifically on wild bees, not honey bees. Concerns have also been raised about significant regional declines in honey bee populations, with some sources documenting colony losses of over 50% in the USA ([2] p. 345). However, while the role of honeybees in providing pollination services is well-recognised, there has been increasing attention paid to the contribution of wild bees. As such, the project specifically targeted this important species.

One of the projects organised within tNB is the DNBS, aimed at involving the public in counting wild bees. During a single weekend in April, Dutch citizens are asked to spend half an hour in their garden to count and classify wild bees. Participants receive a checklist to help identify and count the bees, and submit their collected data using the project website. The DNBS celebrated its first edition on the 21st and 22nd of April 2018, and has become a yearly event. During its first edition, more than 4000 participants submitted over 40,000 bee observations [9].

The main aim of the DNBS is to support wild bee populations in the Netherlands, e.g., by contributing to large-scale data collection on their occurrence. However, it also specifically opts for a citizen science approach with the aim to improve knowledge about bees, stimulate positive attitudes and adoption of bee-friendly behaviours, and increase participants’ broader scientific literacy and concerns about biodiversity [10] (pp. 3–4). As such, the DNBS aims for both scientific and educational outcomes.

The authors of this manuscript were tasked with evaluating the first edition of the DNBS. While the DNBS team itself carries out evaluations of the pollinator data reported by participants, in line with their secondary aims noted above they also wanted an evaluation of the social dimension of the DNBS. We thus designed a repeated measures survey study to gain insight into the profile, motivations, and experiences of the DNBS participants. In addition, based on the project aims, we assessed to what degree we could identify an impact of participation on respondents’ attitudes and behavioural intentions. In the next section, we place our study in the context of previous work on citizen science learning outcomes and evaluation.

1.2. Evaluation of Citizen Science Learning Outcomes

For the DNBS, as for many similar projects, the choice to involve citizen scientists revolves around the potential to achieve both scientific outcomes and learning outcomes for participants; in other words, a win-win scenario. However, such dual-aim programmes may also face significant barriers in achieving both objectives, and in some cases, they may be conflicting or in competition for limited resources [11]. In a recent report, Edelson et al. [12] emphasise that achieving both outcomes is possible, but does require careful design and planning to anticipate and handle tensions. The authors also describe several design strategies that may aid in achieving both outcomes, such as clear data collection protocols and opportunities for social interaction between participants and scientific staff. However, in order to assess to what degree these outcomes are achieved, evaluation is crucial. Evaluation of project outcomes is important for a project itself [13], but for the emergent field of citizen science such evaluations can also be highly instructive in terms of sharing experiences and lessons learned for future projects.

Before being able to evaluate learning outcomes, it is important to be clear about the kinds of learning in which a project is interested. While the word ‘learning’ might evoke the memorization of information (e.g., recognising types of bees), this is only one type of learning outcome that may be of interest. For instance, Phillips et al. [14] distinguish between six different categories of individual learning outcomes of citizen science: interest, self-efficacy, motivation, behaviour and stewardship, skills of science inquiry, and knowledge of the nature of science. Edelson et al. [12] group these under the larger categories of cognitive, affective, and behavioural learning outcomes.

Many potential citizen science learning outcomes are claimed or hypothesised (for an overview, see [15] p. 67), and evaluations of different outcomes can be found in the literature. For instance, in terms of content knowledge, Masters et al. [16] demonstrate how participation in online projects may contribute to better performance in a project-specific science quiz. Dem et al. [17] identified improved awareness of the importance of butterflies and dragonflies among Filipino citizen science participants. In addition, Everett and Geoghegan [18] show how natural history can boost participants’ confidence through contributing to scientific data collection.

However, it may be just as (if not more) important to consider more transformative forms of learning, which Bela et al. [19] (p. 992) define as “a deep, structural shift in awareness that alters one’s way of being in the world and how one views interconnectedness among the universe, the natural environment, one’s personal world, and the human community.” In other words, these learning outcomes include perspective shifts and changing relationships between participants and their social or natural environments, and may include deeper reflection and behaviour change [19,20,21]. Cosquer et al. [22] studied the impacts of a French butterfly project on participants, and argued that beyond knowledge gains, citizen science participation inspired some participants to exchange their knowledge with others and could lead to strengthened feelings of community (see also [23]). In terms of promoting behaviour change, Toomey and Domroese [24] found that participants in two projects on bees and coyotes reported more positive attitudes towards these species, as well as increased interest in conservation actions, while Lewandowski and Oberhauser [25] found that 95% of surveyed butterfly citizen scientists reported increased engagement in one of several conservation actions. Other authors have emphasised the potential of citizen science participation to encourage scientific modes of thinking, improve an understanding of the scientific process, or alter perceptions of science or scientists [26]. Finally, some evaluations have indicated gains in one type of learning, but few to none in others [27,28,29,30].

Rather than providing a comprehensive overview of previous studies evaluating citizen science learning outcomes, here we instead build on insights from several recent reviews of the current state of the evaluation literature [31,32,33,34]. Important insights drawn from these reviews include the following:

• Being a relatively nascent field, citizen science evaluation would gain much from increased clarity regarding the types of learning outcomes projects may aim for [33] (p. 1), informed by the literature. Phillips et al. [33] note that one important current limitation may be a lack of insight into relevant dimensions of learning or appropriate measurement instruments (see also [31] p. 5), as well as a lack of resources (staff, time, money) for conducting social science evaluations;
• Linked to this issue, the nascent nature of the published evaluation literature has also raised the concern that existing scales are rarely re-used ([33] p. 12). Drawing on items used in previous studies would provide more opportunities for synthesis and cross-project comparability of findings;
• Current evaluations often emphasise measuring content knowledge gains, despite other learning outcomes (e.g., skill development or behavioural change) being of more interest for the stated goals of the project [33,34]. Peter et al. ([32] p. 13) note that this could be explained by several factors, amongst which might be funders emphasising content learning, or a higher familiarity among project leaders with assessing content learning compared to other learning outcomes;
• In terms of question formats, Phillips et al. ([33] p. 13) note that there should not be a complete reliance on self-report questions. Peter et al. ([32] p. 13) suggest the enrichment of evaluation studies with other assessment approaches, which might include those that measure actual rather than perceived impacts;
• Few evaluation studies publish null or negative results regarding project outcomes for participants [32]. Stepenuck and Green [34] argue that this may be caused both by biases against such results in the scientific publication process, as well as potential risks of losing project support if project aims are not achieved. However, such results can be highly instructive to scholars and staff of other projects ([35] p. 39).

Our aim with this evaluation study was to incorporate several of these recommendations into our research design. As noted in the previous section, the DNBS team had formulated diverse project goals, which informed our selection of learning outcomes to include in our study. When available and possible, we made use of existing scales from previous studies. If scales that fit our design were not available, we made sure to include items based on previous studies, or we developed items ourselves informed by previous studies (see Section 2). Finally, in terms of null or negative results, we made the explicit commitment before initiating the evaluation to report the results regardless of outcomes.

1.3. Research Goals and Questions

An important aim of our study was to evaluate the learning outcomes of participation in the first edition of the DNBS. In addition, other key points of interest were participants’ profiles (in terms of demographics), their motivations for participating, and their evaluation of the DNBS. Aside from the general importance of knowing the background and motivations of participating citizen scientists [36], profile, motivations, and project experiences may also affect learning in citizen science [37]. For instance, Dem et al. [17] demonstrate how, e.g., gender and motivations were related to differing learning outcomes, and Druschke and Seltzer [30] discuss the impact of project experiences on learning. A recent report by the National Academies of Sciences, Engineering, and Medicine [38] also argues the importance of citizen science projects providing insights into their participants’ demographics (p. 44), and one of its key conclusions regarding learning opportunities is that “science learning outcomes are strongly related to the motivations, interests, and identities of learners” (p. 148). Profile, motivations, and project evaluation were thus included as research questions in this study.

Based on these key dimensions of our study, and the insights from the literature described above, our repeated measures inquiry was guided by the following research questions:

(1)

What is the profile of the DNBS participants in terms of demographics, motivations, and further engagement with green volunteering?

(2)

How did they evaluate their participation in the DNBS?

(3)

What impacts of participating in the DNBS do respondents report in terms of changes in knowledge on, attitudes towards, and behavioural intentions to support bees?

(4)

What impacts of participating in the DNBS can be measured in terms of attitudes towards bees, nature, citizenship, and citizen science?

2. Materials and Methods

For our assessment of learning outcomes from participating in the DNBS, we opted for a repeated measures survey study, with a baseline measurement before participation and a second survey completed after participation. While not a true experimental setting (thus caution must be exercised in attributing any changes in responses to participating in the DNBS), a repeated measures design does allow for more structured insights compared to a single survey.

2.1. Questionnaire Development

An online questionnaire was developed for both the baseline and post-DNBS surveys, using the online survey platform Limesurvey. The questionnaires contained 17 (baseline) and 10–16 (post-DNBS, depending on survey routing) questions, each survey having an estimated completion time of 5–10 min. Most questions utilised multiple-choice, ‘check all that apply’ or five-point response scale formats (e.g., ‘completely disagree’ to ‘completely agree’), while a few questions used other formats such as ranking (motivations) and semantic differentials (e.g., attitudes towards bees).

The baseline survey included sections on motivations for participating, demographics (age, gender, education level, employment, and province of residence), and respondents’ further engagement with green volunteering (information channels, involvement with the organisers of tNB, and further involvement in biodiversity monitoring and other nature-oriented volunteer work). These questions drew from previous studies on Dutch biodiversity citizen scientists [39] and green volunteers [40].

The post-DNBS questionnaire covered participation in the DNBS (e.g., location and company), evaluation of the project, and self-reported impacts of participation. Since our questions on learning outcomes were limited to self-reports, we included two different types of self-report questions to achieve a degree of triangulation. These included scale-based attitudinal measures, repeated before and after participation (an implicit way to assess change based on self-report), and questions where respondents were themselves asked explicitly to assess the level of change in their attitudes, knowledge, and behavioural intentions. For the latter, we partly drew on items used by the authors of [24,25]. Regarding attitudinal measures, we drew on existing literature to include five matching sets of statements in both surveys, in order to compare responses between both measurement moments. These five sets of statements covered the following constructs (for full item lists, see Appendix A):

• Attitudes towards bees: We developed seven semantic differential items on attitudes towards bees, with poles such as “dangerous ↔ not dangerous” or “rare ↔ common”. Use of this response format was inspired by the work of Fischer and van der Wal [41] on perceptions of puffins and tree mallow. Respondents could indicate their answer on a seven-point scale between each set of poles, with the middle option representing “neutral”;
• Nature bonding: Linked to the project goals regarding attitudes towards bees and biodiversity, this set of six statements aimed to measure a sense of attachment to local bees and nature. We selected three items from Raymond et al. [42] and included each statement twice, referring to either local nature or bees (e.g., “I would be saddened if there would be a loss of nature/bees in my environment”). Like the next three sets of items, responses were measured using a five-point Likert scale running from “completely disagree” to “completely agree”;
• Nature relatedness: In addition to the more specific construct of attachment to nature, we also included a measure of general nature relatedness [43]. We opted for the shortened NR-6 scale [44] which consists of six items on awareness of, connection to and interest in nature (e.g., “I always think about how my actions affect the environment”);
• Attitudes towards citizen science: Since the DNBS aimed to improve scientific literacy and stimulate active engagement in science, we included statements tapping into respondents’ attitudes towards citizen participation in scientific research. We developed seven items (e.g., “scientists don’t trust the skills of citizens to do research”), some based on [30];
• Attitudes towards citizenship: Finally, since citizen science is a form of active engagement in volunteering, we included six items measuring components of active citizenship attitudes (e.g., “It is important for me to contribute to my community”). We largely used the same scale as included in a previous survey of green volunteers in the Netherlands [45], which in turn drew on work in civic engagement [46,47] and was adapted to the research context.

2.2. Data Collection

Anyone interested to participate in the DNBS was able to enrol on the tNB website, although this was not a requirement. Two weeks before the DNBS, all enrolled DNBS participants up to that point who had also indicated an interest in project emails (N = 739) received an email invitation explaining the goal of the baseline study and the survey link. In addition, an invitation with the link was placed in the project newsletter sent to the same group and to other interested individuals who had subscribed to it. Finally, survey invitations were placed on social media channels such as the Facebook page of tNB. A week later, the tNB team emailed a reminder to the same 739 enrolled participants and once again included it in the newsletter. When we closed the baseline survey just before the start of the DNBS, 373 respondents had completed it.

Following institutional proceedings at the time of carrying out the empirical work, considering the design of the research and the target population no formal ethical approval was required. In terms of informed consent, as noted above the initial survey was distributed by the DNBS project team; the authors thus never had access to respondents’ contact information. To obtain informed consent for the follow-up survey, at the end of the initial survey we informed respondents about the follow-up study and asked them to share their email address if they were interested in participating; 327 respondents (87.7%) opted to do so. We sent them an invitation for the follow-up survey three weeks after the DNBS, with a reminder for non-respondents ten days later. When the data collection phase ended, 208 respondents (63.6%) had completed the second survey.

2.3. Analysis

Data were uploaded into SPSS Statistics version 21 for cleaning and analysis. For four of the five sets of attitudinal statements, we used reliability analysis (Cronbach’s α) to check whether they formed coherent and reliable combined scales. We carried out this analysis on the items regarding nature bonding, nature relatedness, attitudes towards citizenship, and attitudes towards citizen science; the items on attitudes towards bees were not intended to form a combined scale. We thus conducted eight reliability analyses, for each of the four sets of items at both measurement moments.

Results of these reliability analysis showed that nature bonding (pre-DNBS α = 0.663, post-DNBS α = 0.833) and nature relatedness (pre-DNBS α = 0.735, post-DNBS α = 0.784) formed reliable scales at both measurement moments, and the citizenship scale did so after dropping one item (pre-DNBS α = 0.716, post-DNBS α = 0.709). The items on attitudes towards citizen science, however, did not form a reliable scale at either measurement moment (pre-DNBS α = 0.351, post-DNBS α = 0.370). We thus calculated mean scores for nature bonding (six items), nature relatedness (six items), and attitudes towards citizenship (five items) for both measurement moments. We then checked these variables for normality using histograms, boxplots, Q–Q plots, and descriptive statistics on skewness and kurtosis. These analyses indicated that scores on nature bonding were strongly negatively skewed. As such, to check for demographic differences in scale scores and assess differences between the two measurement moments, we utilized parametric tests for nature relatedness and attitudes towards citizenship (paired-samples t-test, independent-samples t-test, Pearson’s correlation), while for nature bonding non-parametric alternatives were used (Wilcoxon signed-rank test, Mann–Whitney U test, Spearman’s correlation).

Finally, we elicited data on motivations using a ranking scale. We scored each of the eight motivation items (presented to each respondent in a randomly ordered list) based on how high they were ranked by the respondents; motivations not selected by the respondents were given the score of zero. This then allowed the calculation of combined rank scores for each motivation across all respondents.

3. Results

Section 3.1 will present findings related to the first research question, respondents’ background and motivations. Section 3.2 focusses on the second research question: DNBS participation and evaluation. In Section 3.3, we present our findings on the learning outcomes from DNBS participation.

3.1. Baseline Study

In contrast to earlier studies on biodiversity citizen scientists and green volunteers in the Netherlands [39,40], a majority of the respondents are female (

Table 1. Sociodemographic characteristics of the baseline study respondents.

Age a
Mean (SD)	53.2 (12.6)
Gender b
Female	60.6%
Male	39.4%
Higher education completed	65.2%
Paid position in nature/environment sector c
Yes	24.7%
No	45.5%
Unemployed/Retired	29.8%

^a n = 348, ^b n = 355, ^c n = 356.

). Respondents are of a relatively high age, with ages ranging between 15 and 83 years and the largest age group being those between 56 and 60 years of age (comprising 16.1% of all respondents). An independent-samples t-test also shows that the mean age of male respondents (57.0 years) is significantly higher than that of female respondents (51.4 years) (t(303) = −3.96, p < 0.01). In terms of education level, Table 1 shows that more than half of the respondents have obtained a higher vocational or academic degree; this rate is more than twice the Dutch average (29.4% in 2014; [48]).

Several questions tapped into the degree to which respondents are engaged with nature and the environment in their professional and personal lives. Table 1 indicates that almost one in four respondents work in the nature or environmental sector. In addition, 57.4% report that they donate to, are active in, or are member of at least one of the organising partners of tNB. These four organisations were also the main channel through which people got engaged with the DNBS (61.1%).

Furthermore, we asked participants if they participate in further nature-related volunteering. More than half (56.0%) have participated in the yearly national garden bird count, and 28.2% in the yearly national butterfly survey. In fact, 12.3% of respondents submit biodiversity data they collect to an (online) repository at least once a month. Data on respondents’ involvement in other forms of green volunteering (Figure 1) further confirm a broader engagement with nature: for instance, many respondents take part in greenspace restoration or maintenance (28.7%) and organising or leading excursions or education (24.7%).

Results of the motivation ranking (Figure 2) demonstrate that conservation concerns arise as most important for the respondents, which corresponds with the emphasis of tNB on halting the decline of wild bees. Respondents also prominently rank their connection to nature and learning about bees as motivating factors. Scientific contributions, spending time outdoors and learning about nature more generally are emphasised somewhat less, while the pursuit of a diversity of species, or the pleasure of doing something you are good at, are given the least emphasis.

3.2. Follow-Up Study

As described in Section 2, 208 respondents of the initial survey also completed the follow-up survey. To check whether the decision to participate in this second survey was related to demographic characteristics (which might introduce biases in the comparison), we compared age, gender, and educational composition of both study samples. The respondents of the follow-up survey displayed a highly similar profile on these characteristics (mean age 54.2 years, 60.9% female, 65.8% completed higher education) compared to the baseline sample as discussed previously; as such, we felt confident in ruling out demographic response biases in responding to the second survey.

As illustrated in

Table 2. DNBS participation categories.

	n	%
Participants (n = 155)
Counted and submitted observation data	135	64.9%
Counted, did not submit observation data	20	9.6%
Non-participants (n = 53)
Started counting, but did not finish	12	5.8%
Enrolled, but did not participate in the DNBS	41	19.7%

, the 208 respondents of the follow-up survey were categorised as DNBS participants or non-participants. The participants comprised all respondents who completed the full 30 min of counting wild bees, with most also submitting their collected data. The non-participants either ended up not participating at all (with commonly reported reasons being other commitments or lack of time) or started counting but quit prematurely (difficulties in recognising wild bees, or none to observe, were reported as the most common causes).

We made use of a branching survey structure; non-participants were only presented with the attitude scales (discussed in the next section) and two evaluation questions, while the participants were also presented with questions on the way they participated, two questions on perceived impacts of participation, and further evaluation questions. Here we focus on the latter set of questions among the 155 participants of the DNBS.

In terms of location and company, the vast majority (86.5%) counted wild bees in their own garden, with some others opting for communal gardens or public green spaces. Participation was largely a solitary activity (72.3%), although 23.9% of participants did so together with family members.

Participants were generally highly positive about the project. Using a set of four seven-point semantic differential scales (Figure 3), we found that almost all participants found the DNBS interesting (98.1%) and fun (94.2%), with the majority even choosing the far end of the scale. Similar results were found for the informative nature of participating, although results were more mixed. The question of whether participating in the DNBS was difficult or easy led to the most divided response: there was a relatively large group who found it neither easy nor difficult (14.8%), and the group who felt more inclined to consider it difficult is larger (47.7%) than the group who found it easier (37.4%).

Our final question, in which we asked both participants and non-participants whether they would like to take part in next year’s edition of the DNBS, also reflects this positive evaluation. The overwhelming majority (91.7%) indicated they aimed to participate next year; this included 97.4% of the participants (the rest of whom said “maybe”), and 75.5% of non-participants.

3.3. Impacts of Participation

As shown in Figure 4 and Figure 5, the 155 participants in the DNBS reported impressive learning outcomes as a result of their participation. Figure 4 shows that between 71.0% and 81.3% of respondents agreed or completely agreed with the three statements on increased knowledge and awareness of bees, as well as frequency of talking about bees with others. More than half (53.5%) also reported development of a stronger appreciation of bees. We found lower scores for gains in general appreciation of nature, understanding of pollination, and confidence in reporting observations, and we received the most mixed response regarding an increased feeling of comfort around bees or other insects: while 34.2% agreed or completely agreed, 20.0% disagreed, and almost half (45.8%) gave a neutral response. Overall, based on the self-reported perceived changes as a result of the DNBS, it appears that participation was highly impactful for participants, especially in terms of knowledge about and awareness of bees.

Figure 5 shows that participation in the DNBS also had a significant impact in terms of behavioural intentions. The majority of participants indicate an increased interest in making their garden more bee-friendly (e.g., diversifying plants; 61.9%), creating nesting areas (56.8%), and taking part in other actions to protect bees (54.8%). In addition, more than a third reported a stronger intention to talk with other people about bees (49.0%), take part in actions to protect local nature (42.6%), plant more native plants in their garden (41.3%), and learn more about bees and pollination (36.1%). Only the intention to reduce pesticide use was reported by a much smaller group (21.3%).

Taking the findings of Figure 4 and Figure 5 together, it appears that the DNBS accomplished its goals of increasing knowledge about bees, stimulating positive attitudes towards bees, and increasing intentions to engage in bee-friendly behaviours. However, we also presented our respondents in both surveys with five sets of statements on attitudes towards bees, nature, citizen science, and citizenship. These results are presented next.

Figure 6 shows the results of the semantic differential items on attitudes towards bees among all 155 participants in the DNBS, both before (left) and after (right) participation. A large majority of participants perceive bees as beautiful, precious, not dangerous, unique, and vulnerable to some degree; a majority considers them cute rather than scary (although with a considerably larger number of neutral responses), and the most mixed responses could be found on the perception of bees as either rare or abundant. As quickly becomes clear from Figure 6, these response patterns are highly similar between both measurement moments. In fact, if we group all yellow (checkerboard), grey (solid), and green (diagonal) responses in Figure 6 and compare the results between before and after the DNBS, the greatest shift is only 7.1% (the percentage of respondents who perceive bees as rare to some degree increased from 34.8% to 41.9%). Overall, however, Figure 6 provides little basis on which to claim that a significant shift in attitudes towards bees took place because of participation in the DNBS.

We found similar patterns when analysing the results of the scales on nature bonding, nature relatedness, attitudes towards citizen science, and attitudes towards citizenship (Appendix A). At both measurement moments, respondents’ scores on the nature bonding scale were overwhelmingly distributed towards (complete) agreement with the items. Responses to the nature relatedness scale were somewhat less skewed, though few respondents disagreed with any of the items; the major exception was the item “I feel spiritually connected to nature”, which evoked a greater percentage of neutral responses and disagreement.

The scale on attitudes towards citizenship showed a similar pattern of high levels of agreement with the items. Finally, the data on attitudes towards citizen science indicate somewhat more varied responses. While a large majority (fully) agrees that it is important that citizens are more actively involved in scientific studies and that results need to be shared with participants, there was slightly less agreement on whether scientists trust the ability of citizens to take part in science. Questions on data quality also evokes varied responses; for instance, while almost half our respondents (fully) agreed that scientists make fewer mistakes than citizens in conducting research, fewer respondents felt that data collected by citizens are thus also less reliable.

For further analysis, we first carried out demographic analyses; as noted in Section 2.3, we mostly used parametric tests for nature relatedness and attitudes towards citizenship (independent-samples t-test for gender, Pearson’s correlation for age, while using the nonparametric Spearman’s correlation for education). For investigating possible demographic differences in nature bonding scores, we used nonparametric tests (Mann–Whitney U test, Spearman’s correlation). These tests revealed no significant relationships between nature relatedness, nature bonding, and attitudes towards citizenship scale scores and respondents’ age, gender, or education. In addition, we used paired-samples t-tests (for nature relatedness and attitudes towards citizenship) and Wilcoxon signed-rank test (for nature bonding) to investigate if significant differences in scale scores could be detected between both measurement times. These tests found no significant changes post-DNBS in levels of nature bonding (T = 1409, p = 0.392), or in mean scores for nature relatedness (t(154) = 1.39, p = 0.166) or attitudes towards citizenship (t(154) = 0.42, p = 0.67).

4. Discussion

Our results show that survey participants positively evaluated their participation in the first DNBS, and overwhelmingly intend to take part again (91.7%). However, in terms of learning outcomes, a rather ambiguous picture emerges. Participants themselves directly perceived significant impacts of DNBS participation on their awareness of and knowledge about bees, appreciation of bees, and behavioural intentions such as adapting their garden and talking about bees with other people. However, few impacts of participation could be detected in responses to scale-based measurements of attitudes towards bees, nature, citizen science, and citizenship. In this section, we discuss three factors that may help explain these results: the nature of the project, the profile of the participants, and methodological aspects of our study.

Firstly, the DNBS is a yearly event consisting of participants counting wild bees for 30 min and submitting the data. More transformative learning outcomes, such as shifts in underlying attitudes towards nature, science, or citizenship, would likely arise more slowly than increased knowledge and awareness of bees, and a one-off event lasting 30 min might not realistically be able to bring about this sort of impact [49] (p. 759). In addition, participants’ contribution was limited to collecting and reporting observational data; the DNBS did not involve citizens in formulating research questions or analysing the data. Several authors have argued that deeper attitudinal shifts, especially in terms of attitudes towards science, would require involvement in more phases of the research process (e.g., [50] p. 183). However, Lawrence [51] cautions not to be too quick to discount more transformative learning in contributory projects, as the very act of data collection in nature can also be deeply moving for participants. Bonney et al. ([31] p. 11) add that “learning does not just ‘happen’”, and that deeper learning especially would require incorporating explicit reflection or learning aids into citizen science participation [52,53].

Secondly, we need to take the profile of our participants into account. Our results indicate that respondents were generally of a relatively high age and education level, that many had a pre-existing affinity with nature through work or volunteering, and that conservation concerns were the most important motivation for participating. Survey response biases and the representativeness of our sample should also be taken into account, yet our data seem to indicate that participants entered the project with pre-existing positive attitudes towards nature, science, citizen science, and citizenship. Our scale results reinforce this suggestion, especially regarding attitudes towards nature and citizenship. This is not a situation exclusive to our case study, and is likely common among nature-oriented citizen science projects; as noted by several other authors (e.g., [31] p. 6; [50] p. 177), such pre-existing positive attitudes also help explain why shifts in these attitudes are difficult to measure. This phenomenon leads to a so-called ceiling effect: changes in a response scale become difficult to measure if respondents already start out at an extreme end of a scale.

One methodological approach to mitigate this effect could be to use a more refined response scale: our study generally employed five-point response scales, so using six-point or seven-point scales could allow for more differentiated responses. However, Crall et al. ([49] p. 759) note that more nuanced and specific statements might also be necessary. Several scales used in this study, including the NR-6 and nature bonding scales, are intended for surveying the general population; for participants in nature-oriented volunteering projects, statements probing more specific attitudes or viewpoints could aid in more successfully differentiating respondents.

Finally, our results invite reflection on the exclusive use of self-report questions for evaluation studies [33] (p. 13). Our first type of self-report measure, self-reported changes in perspective, may be especially vulnerable to issues of social desirability as respondents may feel pressure to report those outcomes they expect the researcher to be looking for. However, several other authors note that scale-based self-report questions may not accurately detect certain learning outcomes either. Chase and Levine ([54], p. 7) argue that even if existing attitudes or behaviours do not significantly shift as a result of citizen science participation, reinforcement of existing attitudes through supportive exchanges with fellow volunteers or project staff is also of great value. Similarly, the quantitative study by Lynch et al. [55] found no statistically significant learning outcomes of participation in entomological research, yet in interviews their respondents did report more positive attitudes towards nature and insects and an increased sense of self-efficacy.

These examples and our own findings highlight that, while triangulation between different types of self-report questions can tease out interesting differences, the use of different question formats or mixed methods designs in evaluation studies holds great potential for understanding important nuances. Adding a qualitative component (e.g., interviews or focus groups with participants) to quantitative evaluations would be one way to improve our understanding of the diverse learning outcomes in citizen science, and to gain insight into both magnitude and meaning of change ([56] p. 84). Other qualitative methods may also shed new insights, such as the use of narrative approaches like research diaries or narrative interviews [57]. In terms of quantitative studies, even more sophisticated analytical designs such as structural equation modelling would allow for further exploration of factors influencing learning outcomes. In addition, while we did use a repeated measures design, a truly longitudinal approach that assesses the sustainability of learning impacts after a longer period would also be an interesting avenue for further research where possible.

5. Conclusions

Our evaluation of the first edition of the DNBS indicated that our respondents were highly positive about the project. We also found they were strongly motivated by conservation concerns and learning about bees, and many were more broadly involved in green volunteering, suggesting a pre-existing affinity for and engagement with nature. In terms of learning outcomes, our respondents reported significant gains in knowledge and appreciation of, attitudes towards, and intention to support wild bees. However, scale-based measures indicated no significant shifts in attitudes towards bees, nature, citizen science, or citizenship. We suggested several explanations for this finding, including the brief and contributory nature of the DNBS, and the pre-existing positive attitudes of our respondents.

Our research design incorporated several recommendations made in recent reviews in the citizen science evaluation literature: linking project aims and evaluation, focussing on more than content knowledge, reusing existing scale items where possible and available, and committing to reporting possible null results. We recommend the further application of scale items used in previous studies (including from this study), their fine-tuning in terms of content and response scales towards use among highly engaged groups of participants, and the adoption of diverse methods in evaluation studies (including mixed-methods designs). We thus hope this study inspires further research on the evaluation of learning outcomes in citizen science.

Citizen science projects with both scientific and outreach objectives are certainly capable of doing both, but will likely face tensions in terms of priorities or resources. As this study has demonstrated, any citizen science project that aims to achieve both objectives ought to invest in a detailed evaluation of its learning outcomes, both to check whether it is achieving its ambitions, and to improve our understanding of the potential for citizen science to stimulate new insights and inspiration among its participants.