Coordinator Perceptions When Assessing the Impact of Citizen Science towards Sustainable Development Goals

Abstract

Tracking progress towards the United Nations’ Sustainable Development Goals (SDGs) requires high-quality, timely, and accessible data, often in areas where data are rarely available. Problems exist due to socioeconomic variations between countries and the qualitative nature of certain indicators in their definition. Citizen science has the potential to contribute to several SDGs. However, whilst citizen science’s potential to contribute towards SDGs is well documented, limitations exist when measuring the impact that citizen science has made toward SDG progress. To better understand the issues and prospective solutions surrounding impact assessment towards SDG progress, this work presents the outcomes of semi-structured interviews with citizen science project coordinators. They reveal the complex nature of impact assessment within a citizen science context. Coordinators demonstrate greater confidence when the project is easier to relate to the SDGs, and the project methodology can objectively measure indicators. Issues exist, however, when considering SDGs with a broader, global context, those more difficult to link to project goals and when the project’s impact on them happens at timescales beyond the funding period. If the full potential of citizen-science contributions to the SDGs is to be realised, approaches are needed to fully consider practitioners’ needs and motivations.

1. Introduction

As part of the Rio+20 summit held in Brazil in 2012, the United Nations (UN) committed to create a set of Sustainable Development Goals (SDGs), designed to be a follow-up to the Millennium Development Goals that ended in 2015 [1]. In 2017’s UN summit on sustainable development, the ”Transforming our world: the 2030 Agenda for sustainable development” framework was released [2] that included 17 SDGs comprising 169 targets addressing a range of global challenges including environmental degradation, prosperity, poverty, inequality, climate, and peace and justice. A monitoring framework was created to track progress towards each SDG target, resulting in the current list of 231 SDG indicators [3]. Successfully tracking progress towards SDGs requires high-quality, timely, and accessible data, often in areas where few data are currently available. In appreciation of this limitation, the UN framework separates the indicators into two tiers, depending on the existence of agreed methodologies and data availability [4]. Current baselines and indexes developed for the measurement of SDG progress rely on data from formal international and national bodies, resulting in limitations due to socioeconomic variations between countries and the qualitative nature of certain indicators in their definition [5].

In response to such limitations, there has been a recent movement towards a bottom-up approach, focused on capacity building rather than the development of a global framework [6]. This has allowed for new initiatives and partnerships to develop, involving data collected by less-traditional methods such as remote sensing, mobile technologies, and social media [7]. Citizen science is one such method, with the potential to make contributions both in terms of data and regarding public engagement, generating the societal changes needed to make SDGs attainable [8,9]. Citizen science, defined as work undertaken by civic educators and scientists together with citizen communities to advance science, foster a broad scientific mentality, and/or encourage democratic engagement, which allows people in society to join the debate about complex modern problems [10], can contribute to the UN’s ethos that sustainability should be all-inclusive with all contributions valid, through inclusive participation, partnerships with different stakeholders, and education initiatives [11]. Additionally, citizen science can inform data processes, including defining targets and metrics, monitoring progress, and implementing action [12]. In an attempt to formalise the potential links between citizen science and SDGs, recent research has considered the possible contributions that citizen science projects can make to the SDG indicator framework [6]. Such approaches consider not only current contributions to specific SDGs and their indicators, but also the potential that future citizen-science initiatives could have if the SDGs were considered in their project design process [13].

Consequently, citizen science has been found to have the potential to contribute to several SDGs, most prevalently: SDG 15, life on land [14]; SDG 11, sustainable cities and communities [15]; SDG 3, good health and wellbeing [16]; and SDG 6, clean water and sanitation [17]. Furthermore, when combining citizen science methods with other data collection techniques such as remote sensing, the potential contribution can strengthen. For instance, studying visualisations of green spaces, use of natural resources, emissions, and other metrics combined with census data such as crime statistics can support communities in taking action towards SDG 4, quality education, and SDG 12, responsible consumption and production [18]. By combining satellite data with on-the-ground measurements, citizen science can contribute towards SDG 13, climate action [19], and by monitoring marine pollution and its evolution, SDG 14, life below water [20]. However, whilst the links between SDGs and their associated indicators and the potential of citizen science to contribute towards them are well documented, limitations still exist regarding the measurement of impact citizen science initiatives have made regarding SDG progress. Whilst citizen science can fill some of the data gaps that exist using national-level statistics alone [21], other issues exist regarding project teams’ expertise and capacity to measure different impacts towards SDGs, and the motivation to do so.

Measuring the Impact of Citizen Science

Citizen science defines a wide range of activities and practises, stemming from multiple scientific disciplines, from the natural sciences to the social sciences and humanities [22], and is growing exponentially [23]. Perhaps unsurprisingly, the methods by which the impact of a citizen science project can be measured have also developed, evolving over the past five decades from thematically focused, isolated undertakings to more comprehensive, integrated assessments [24]. For example, the early environmental impact assessments (1960s/1970s) were almost exclusively focused on the bio-physical environment, whilst those conducted during the late 2000s included human health and safety, heritage assets, and historical and cultural sites as well as livelihoods, lifestyle, and wellbeing of those living in affected areas [25].

Impact assessment is by no means unique to citizen science; indeed, there are multiple examples of general impact assessment frameworks that have been used by both citizen science and non-citizen-science projects, such as Metrics and indicators of Responsible Research and Innovation (MoRRI [26]) and IRIS+ [27]. Typically, these general impact assessment frameworks fall within one of three clusters: accounting (single purpose) frameworks, such as the Magenta Book (Central (U.K.) Government guidance on evaluation); frameworks focused on generating scientific knowledge and insights for particular audiences (e.g., policy makers), such as MoRRI indicators for selected impact domains; and frameworks for managing towards impact, including IRIS+ and the Theory of Change [24]. Citizen science impact assessment approaches do not, at first glance, appear to fall into such neat clusters. Indeed, the diversity of citizen science research topics, varying intervention designs, and complexity of outcomes render it difficult to evaluate participatory research projects [28]. Evaluation of knowledge exchange is often an afterthought [29], or, if impacts and outcomes are measured, their attribution to citizen science projects is often not adequately documented [30].

Nevertheless, many citizen science impact assessments do fall into clusters, themes, or “domains”. MICS: Measuring the Impact of Citizen Science (www.mics.tools (accessed on 22 February 2021)), an EU Horizon 2020 funded project that this research forms part of, considers the impacts of citizen science on five domains: environment, economy, society, governance, and science and technology. A recent systematic review of relevant literature about impact assessment in the field of citizen science, consisting of 77 publications, revealed that most publications only assess their impact in one or two domains (32 and 19, respectively)—only two out of the 77 papers reviewed referred to all five domains—with the vast majority focusing on the society domain and very few investigating the economy domain [31]. Impact is also assessed at different depths, with almost all papers providing insights at a thematic level, but few delving to the indicator level. For example, “agricultural productivity outcomes” was a theme within the environmental domain, broken down to include “soil fertility” and “water availability” indicators within this theme [32]. However, in the same publication, “individual and societal outcomes” was a theme within the societal domain that was not broken down to the indicator level. In other societal examples, “individual learning” was discussed in the context of “beliefs, attitudes and values” indicators [33], and “value creation” was dissected into “change in knowledge and skills” [34]. Consequently, there is a lack of standardised approach for assessing impact across different domains.

In terms of methodology, a mixed-methods approach—utilising both qualitative and quantitative measures—was the most common, with a variety of methods, including observations, checklists, interviews, and questionnaires, gathered from multiple stakeholders [31].

Despite multiple attempts across the literature, there remains no singular formal process by which to assess a citizen science project’s impact and to accurately compare that impact with other citizen science projects, traditional science, or the status quo. As such, there has been a call for the citizen science impact assessment field to reflect on its own approaches [35]. Therefore, issues also exist when measuring the impact of citizen science in terms of the SDG framework, as is the case with current tracking processes. In response, research has started to consider how best to link citizen science initiatives with SDGs, including the MICS project (developing metrics and instruments to evaluate citizen science impacts). In doing so, MICS seeks to provide a citizen science impact assessment framework that is a flexible yet standardised methodology which individual citizen science projects can use, based on their unique resources, yet generate comparable results across projects [24].

As part of this process, this paper presents a first appraisal of the current state of play regarding citizen science, its impact, and the relationship with the SDG framework. Through semi-structured interviews, the perceptions of citizen-science project coordinators are explored with respect to measuring the impact of projects, the positives and negatives of their approaches, and how they relate to differing SDGs and their indicators. The interviews reveal the different domains and SDGs that such impacts relate to; if existing mixed-methods approaches to assess impact are suitable; and the degree to which such assessments achieve this aim, through the eyes of those most involved.

2. Materials and Methods

2.1. Methodological Approach

The purpose of this study was to gain a better understanding of the processes project coordinators go through to assess the impact of their citizen science project, any issues that exist, and how they relate to the SDG framework. This involved a grounded and inductive approach, discovering and constructing theories from the data, obtained and analysed using comparative analysis [36]. In order to do this, a semi-structured interview process was followed. Semi-structured interviews are a verbal exchange, where whilst certain topics and questions are pre-determined in the form of an interview protocol, they unfold in a conversational manner that allows participants to raise and explore issues they feel are important [37]. As previously mentioned, this work was conducted as part of the MICS project; as such, the aims of the interviews were two-fold: to obtain insights into citizen science coordinators’ opinions, processes, and techniques when assessing the impact of their project and to inform the MICS project and any associated tools developed to improve the impact assessment landscape.

In order to analyse the data resulting from the interview process, a thematic analysis approach was taken. The approach involves looking across an entire data set to identify repeated patterns of meaning, allowing a data set to be described in rich detail through the identification and reporting of themes [38]. The analysis followed a two-stage process. Firstly, the interview responses were coded at a higher level as either positive, negative, or general in relation to the feasibility and effectiveness of impact assessment methods for citizen science. The use of positive and negative coding is standard practice with thematic analysis [39], and in this case was used to identify potential barriers to assessing the impact of a citizen science project, and also to highlight the benefits in doing so—both of which can inform future frameworks and tools developed to assist the impact assessment of citizen science towards the SDG framework. Secondly, the responses were coded in more detail in terms of which SDGs they related to. This was to identify if there were any particular issues or advantages related to specific SDGs. As a validation step, the coding was performed by two independent authors and then compared by a third author to assess consistency. The agreement between the authors was ~94%, above the 80% level considered acceptable for such an approach [40]. Additionally, the general scientific goals of each project were also assessed and linked to appropriate SDGs.

2.2. Interview Structure

The interviews were arranged into four broad, pre-determined sections to ensure the key points were covered and procedural requirements met. A full explanation of the interview protocol, procedure, and questions asked can be found in Document S1 (see Supplementary Materials). The sections are as follows:

• Welcome and introduction (~5 min): The interviewer introduces the purpose of the interview, its relation to the MICS project, and the approximate time it should take. Other procedural issues are also confirmed, including confirming consent has been given, the process to withdraw, and how the data will be stored and shared.
• Project-related questions (~30 min): Questions relating to the interviewees’ projects and their specific impact. Topics covered include the terminology used regarding impact, outputs, and outcomes; the purpose for measuring impact; assessment methods; indicators used to quantify impact; challenges associated with measuring impact; timing of impact measurement; prospective audience of their assessment; whether barriers exist for measuring impact; and consequences of not being able to measure impact in a meaningful way.
• General impact questions (~20 min): Questions relating to the interviewees’ more general awareness of impact and its assessment. Topics covered include awareness of existing impact measurement methodologies; reasons for agreeing to discuss impact and possible tools of measurement; possible solutions to issues regarding measuring impact; situations in which impact assessment is helpful; importance of different tools and support frameworks when measuring impact; impact as change—what relevant differences should be measured; comparison of different projects’ impact; and qualitative and quantitative representations of impact measures.
• Closing (~5 min): The interviewer performs the debrief, informing the interviewee about the next steps regarding data analysis and the MICS project. At this point, participants are invited to be informed about the project in the future, with the opportunity to give feedback regarding any impact assessment tools developed.

2.3. Participants and Projects

Eleven project coordinators were invited to take part in the interview process (six identifying as female, five as male), recruited through an open call at various citizen-science-focused meetings, conferences, and forums [41]. As such, participation was designed to be “opt-in” rather than “opt-out” with those expressing an interest contacted to take part. Project coordinators are effectively the principal investigators (PIs) of a citizen-science project, and are scientific professionals employed to do the research rather than the citizen scientists themselves. The 11 recruited are based throughout Europe, and the projects themselves operate worldwide. The projects they represent cover a range of academic disciplines including astronomy, biodiversity, water quality, pollution, the environment, and health. The length that each project has been active ranges between 1 and 11 years (both the mean and median project length is four years), with a range of between 10 to 15,000 participants taking part (mean of 2900, median of 2000 participants). Regarding the projects represented, eight are currently still active, whilst three no longer collect data. Although the project coordinators were not compensated for their time in a monetary way, they were offered the opportunity to remain informed on the progress of the MICS project and give feedback regarding any impact assessment tools developed.

2.4. Procedure

After accepting the invitation to participate in the interview process, participants received a consent form via email to complete and return before the interview date. All interviews took place online, and the answers provided were recorded in two ways. Firstly, if resources allowed, two researchers took part, one to act as interviewer and one to record answers and take notes. Secondly, if resources did not allow for this, or if the interviewee found the sound of typing notes distracting, the interview was recorded with permission given by the participant. At the start of each interview, general details were recorded regarding timings and location, the name of the participant, and the citizen science project(s) they were associated with. The main topics were then discussed as outlined previously, with prompts and examples given by the interviewer if the conversational flow stuttered at any point (see Document S1). After the interview, notes and recordings were transcribed on the same day to ensure a fair representation of the participants’ thoughts and views. Earthwatch Europe’s ethical procedure approved the study aims, methods, and procedures, including in terms of guaranteeing the anonymisation of data and prior informed consent of the participants.

3. Results

This section presents the results of the interviews, first in a summarised form looking at high-level thematic analysis of the participants’ views and concerns regarding measuring impact and SDGs (Section 3.1), and then in more detail, highlighting specific comments and issues to add context and a deeper understanding (Section 3.2).

3.1. Summary Statistics

Regarding the citizen science initiatives that the coordinators (the interviewees) are involved with,

Table 1. Potential contributions of citizen science projects towards Sustainable Development Goals (SDGs).

SDG	No. of Projects
3: Ensuring healthy lives and promote wellbeing for all at all ages	3
4: Inclusive and equitable quality education and promote lifelong learning opportunities for all	2
6: Ensuring availability and sustainable management of water and sanitation for all	4
9: Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation	3
11: Make cities and human settlements inclusive, safe, resilient and sustainable	4
12: Ensure sustainable consumption and production patterns	1
13: Take urgent action to combat climate change and its impacts	1
14: Conserve and sustainably use the oceans, seas and marine resources for sustainable development	2
15: Protect, restore and promote sustainable use of terrestrial ecosystems, sustainably manage forests, combat desertification, and halt and reverse land degradation and halt biodiversity loss	1

summarises the SDGs that the related projects have contributed towards (or have the potential to contribute to).

In total, the citizen science projects discussed in the interviews have the potential to contribute towards nine different SDGs, with SDGs 3, 6, 9, and 11 the most represented (Table 1). Other SDGs that could potentially be influenced are 4, 12, 13, 14, and 15. Considering each interview and its content,

Table 2. Interview responses by tone (positive, negative, or general).

Participant No.	No. of Positive Comments (%)	No. of Negative Comments (%)	No. of General Comments (%)
1	12 (21.8%)	10 (18.2%)	33 (60.0%)
2	11 (24.4%)	12 (26.7%)	22 (48.9%)
3	8 (23.5%)	7 (20.6%)	19 (55.9%)
4	13 (33.3%)	7 (17.9%)	19 (48.7%)
5	6 (12.8%)	12 (25.5%)	29 (61.7%)
6	6 (17.6%)	8 (23.5%)	20 (58.8%)
7	0 (0%)	7 (25.0%)	21 (75.0%)
8	7 (18.9%)	9 (24.3%)	21 (56.8%)
9	2 (8.0%)	7 (28.0%)	16 (64.0%)
10	6 (22.2%)	7 (25.9%)	14 (51.9%)
11	9 (34.6%)	8 (30.8%)	9 (34.6%)
Total	80 (20.2%)	94 (23.7%)	223 (56.1%)

summarises the first stage of the thematic analysis, defining coordinators’ comments as either positive, negative, or general with regards to the feasibility and effectiveness of current impact assessment methods for citizen science projects.

The majority of the comments in the interviews were of a general nature, with no strong positive or negative tone or connotations (ranging from ~48% to 75% of the total responses); see Table 2. This reflects the semi-structured interview approach, which allows a conversational, natural discussion rather than a formal question-and-response approach. The number of positive comments (in the context of the feasibility and effectiveness of impact assessment for citizen science) ranged from none at all (participant 7) to over a third of all responses (participant 11). In terms of negative comments, this ranged from ~18% (participant 4) to ~31% (participant 11). Overall, the average proportion of positive comments (mean 20%, median 22%) made by each participant was lower than that of negative comments (mean 24%, median 25%).

Finally,

Table 3. Interview responses, grouped by SDGs that projects contribute towards (difference column denotes the percentage difference between positive and negative comments; negative values indicate negative comments were dominant in the interview and vice versa).

SDG	No. of Positive Comments (%)	No. of Negative Comments (%)	Difference between Negative and Positive Comments (%)	No. of General Comments (%)
3	13 (14.3%)	28 (30.8%)	−16.5%	50 (54.9%)
4	8 (13.6%)	15 (25.4%)	−11.9%	36 (61.0%)
6	25 (23.6%)	26 (24.5%)	−0.9%	55 (51.9%)
9	21 (17.9%)	26 (22.2%)	−4.3%	70 (59.8%)
11	26 (17.2%)	35 (23.2%)	−6.0%	90 (59.6%)
12	7 (18.9%)	9 (24.3%)	−5.4%	21 (56.8%)
13	7 (18.9%)	9 (24.3%)	−5.4%	21 (56.8%)
14	15 (21.1%)	16 (22.5%)	−1.4%	40 (56.3%)
15	13 (33.3%)	7 (17.9%)	15.4%	19 (48.7%)
Total	135 (19.1%)	171 (24.2%)	−5.1%	402 (56.7%)

combines Table 1 and Table 2 by considering the number of positive, negative, and general comments grouped by the SDGs the projects’ impacts could be measured against.

As with Table 2, the majority of comments were of a general nature when grouped by SDG, with no strong positive or negative tone or connotations present (ranging from ~49% to 61% of the total responses). When considering positive and negative comments, however, differences can be seen depending on the SDG group. The positive/negative split shows that for SDGs 3 (healthy lives and wellbeing) and 4 (quality education), there is a greater proportion of negative comments compared to positive (16.5% and 11.9%, respectively). In contrast, SDG 15 (ecosystems, forests, and desertification) is the only SDG with a higher percentage of positive than negative comments (15.4%). In addition, SDG 15 also had the highest percentage of positive comments (33.3%) of all SDGs (Table 2).

3.2. Analysis of Interview Responses Related to Specific SDGs

Whilst the first stage of the thematic analysis has provided an overview of interview responses and their relationship with the SDGs associated with project goals, the second stage takes a deeper look at interviewees’ responses, their wording and meaning, and direct references to specific SDGs.

Table 4. Example positive and negative interview responses related to SDGs 3 and 4.

SDG	Positive Comments	Negative Comments
3 (healthy lives and wellbeing)	“Qualitative methods produce a greater richness of information, take more time to interpret so better for internal use. Help to capture the unexpected impacts…”	“Assessment indicators are so well defined, we aren’t able to use them without some adaptation, and we do not know enough about other examples of impact assessment…” “Timescale of impact doesn’t match timescale of project. The project is only three years, so how do we show and measure results in such a short term?” “Tricky to find a control group or to know what to measure at the start of the project.”
4 (quality education)	“For projects with an engagement focus, there might not be that much data collected, but there might be a big increase in media attention.”	“Didn’t perform a good measure of impact, just a small-scale evaluation of the project. It’s always hard to measure the long-term effect of the project”. “…There’s no guarantee that projects will want to measure impact once the project is over, but this is when most impacts manifest themselves.” “(The) number of citizens engaged is a shallow indicator which doesn’t really demonstrate impact.”

shows representative positive and negative responses related to SDGs 3 (healthy lives and wellbeing) and 4 (quality education), those where negative responses outweighed positive by the greatest margin.

As can be seen from Table 4, a number of issues are raised related to the adaptability of existing methods, the timing of impact assessment, and the differences between quantitative and qualitative approaches. Similarly,

Table 5. Example positive and negative interview responses related to SDGs 15, 6, and 14.

SDG	Positive Comments	Negative Comments
15 (ecosystems, forests, and desertification)	“…Our method is good; it is objective and rigorous and applied consistently.” “(Impact) can be measured through publications, serving as a benchmark for good quality citizen science.” “Our data is used by multiple funders, funders we want to fund us again.”	“A lot of environmental/physical scientists find impact assessment difficult/not intuitive.”
6 (sustainable management of water)	“All citizen scientists are interested in improving the health of our waterways.” “The council, water providers, and environmental regulators have a keen interest in the outputs of the citizen science activities and how they can be used.”	“It’s very project-specific. It is hard to compare against other projects as they are all so different.” “…Risk realising project was insignificant or didn’t really change anything.”
14 (sustainability of oceans, seas, and marine resources)	“14.1.1b (Index of plastic debris density) would be the single greatest impact that could be achieved through this project.”	“Long term impacts won’t be measurable/visible during the funded period of the project, so it’s impossible to report this back to the funder.”

shows representative positive and negative responses related to SDG 15 (ecosystems, forests, and desertification), where positive responses outweighed negative quantitatively, and SDGs 6 (sustainable management of water) and 14 (sustainability of oceans, seas, and marine resources), where the percentage of negative and positive responses were closest to parity.

Again, several issues are raised related to consistency and objectiveness, stakeholders, timescales, and funders. Finally,

Table 6. Example positive and negative interview responses related to SDGs 9, 11, 12, and 13.

SDG	Positive Comments	Negative Comments
9 (industry, innovation, and infrastructure)	“Comparison with traditional science might add credibility to the citizen science community.”	“I would be cautious as to how impact assessment tools would be used—I don’t think [it] is how innovation should work. Becoming too tool-driven will limit creativity.”
11 (sustainable cities and communities)	“Qualitative methods produce a greater richness of information, take more time to interpret so better for internal use. Help to capture the unexpected impacts.”	“…You do not consider the unexpected impacts, people act to comply with the measures, and they become the target rather than doing the best they can.”
12 (responsible consumption and production)	“Project funded by the water authority, and will have direct influence on consumption and production processes.”	“Stakeholders weren’t overly interested and didn’t really do anything with the outputs citizen scientists produced.”
13 (climate action)	“It would be really interesting to talk about common goals and desired outcomes. Helps to advance the field of citizen science.”	“…Not interested enough in the results of an in-depth study with small sample size to justify effort spent.”

shows representative positive and negative responses related to the remaining SDGs (9, 11, 12, and 13), where the percentage of negative responses outweighs positive by between ~4% and 6%.

The issues discussed regarding these SDGs are related to creativity, unexpected impacts, funders, stakeholders, and the scope of the impact. In the following discussion, the issues raised in both a positive and negative tone are unpacked in an attempt to better understand the reasoning behind them and differences that exist when measuring citizen science impact towards different SDGs.

4. Discussion

Through semi-structured interviews and a thematic analysis of the responses collected, this work presents a first, simplified attempt at understanding the perceptions of citizen-science project coordinators when measuring the impact of their project towards SDGs. This process revealed several issues that relate in differing ways to specific SDGs and the methods used to assess them.

4.1. Overview of Project Goals, Their Impact on SDGs, and the Nature of Coordinator Attitudes

Considering the contribution of citizen science projects and their goals towards SDGs at a broad level, the projects related to the coordinators interviewed in this study follow a similar pattern as already surmised in previous literature [6]. SDGs 11 (sustainable cities and communities), 3 (healthy lives and wellbeing), 6 (sustainable management of water), and 15 (ecosystems, forests, and desertification) are all represented, and with the combination of other data analysis techniques so are SDGs 4 (quality education), 12 (responsible consumption and production), 13 (climate action), and 14 (sustainability of oceans, seas, and marine resources). Finally, through creating innovative solutions to scientific problems, SDG 9 (industry, innovation, and infrastructure) is also represented, a view that has been posited by previous literature as a potential citizen science contribution to the SDG framework [13].

Whilst the SDGs represented by the coordinators that took part in the interview process are perhaps as expected, key differences between the SDGs become apparent when considering the tone of interview responses towards them. SDGs 3 (healthy lives and wellbeing) and 4 (quality education) had the greatest proportion of negative comments, whilst SDG 15 (ecosystems, forests, and desertification) had the least (Table 3). It could be argued that this is at least in part due to how directly the SDG theme links with science. Whilst sustainable ecosystems, forest management, and desertification can easily be connected with the aims of ecological citizen-science projects, as suggested in the characteristics of citizen science defined by the EU-Citizen.Science project, the effects of citizen-science practices on education, learning, and wellbeing are arguably a by-product of citizen science projects [42]. This position is supported by the fact that the next two SDGs that have a greater proportion of positive responses (achieving near parity with negative) are SDGs 6 (sustainable management of water) and 14 (sustainability of oceans, seas, and marine resources), themes that again can be more directly linked to the aims of the citizen science projects interviewed.

4.2. Objective Measures Versus Subjective Judgements

Through the second stage of thematic analysis carried out on the interview responses, key themes emerged regarding coordinators’ attitudes of measuring the impact of their project in relation to the SDG framework. One such theme relates to the method of assessment, and whether it can be measured objectively or as a qualitative judgement. For instance, regarding SDG 15 (ecosystems, forests, and desertification), one coordinator pointed towards the advantage of a method that is “objective, rigorous and applied consistently”; another regarding SDG 14 (sustainability of oceans, seas, and marine resources) mentioned that the creation of a specific index “14.1.1b (plastic debris density) would be the single greatest impact achieved by the project”. Such comments highlight the advantage of SDG indicators that address specific scientific goals when considering the impact of a project [15,16,17]. Regarding SDG 15, publications were mentioned as a good method of measuring impact and demonstrating good-quality citizen science is being conducted (Table 5). However, regarding SDG 4 (quality education), it was noted that objective measures such as the number of citizen scientists involved are shallow measures of engagement.

In a similar vein, responses regarding SDGs 3 (healthy lives and wellbeing) and 11 (sustainable cities and communities) point towards the advantages of qualitative methods that produce a “greater richness of information” and can help to “capture unexpected impacts”. However, issues exist regarding such approaches, with coordinators finding existing protocols too well defined, having to adapt measures to their use-case as they lack awareness of “other examples of impact assessment”. It was also pointed out that citizen science practitioners and the science teams involved often lack the expertise or experience in dealing with qualitative data, finding such approaches to impact assessment “difficult” to implement and “not intuitive”. In relation to SDG 11, the danger of a pre-occupation with impact assessment methodologies that go beyond the quantitative measures of the science conducted was raised. The projects’ focus could shift to compliance with such measures, and they become the target rather than achieving the best scientific outcomes.

The general perceptions of citizen science coordinators, revealed by the interview process, lean towards a preference for objective, quantitative methods of measuring impact. Although there is an acknowledgement of the benefits of a qualitative approach, and of using established impact-assessment methodologies, there exists a lack of confidence that the expertise and knowledge exist within project teams to fully exploit such practices. The issues associated with qualitative measures regarding SDG progress are already known [5], and coordinator responses reinforce the need for tools and guidance where expertise is lacking.

4.3. Long-Term Versus Short-Term Impact

A second theme revealed by interview responses focuses on how time-sensitive SDGs are, and if any contribution towards them can be measured in a short or long-term manner. For instance, responses showed a belief that SDGs 3 (healthy lives and wellbeing) and 4 (quality education) required a longer time frame in order to measure any contribution and highlighted several issues with this. This perhaps highlights the issue of a lack of a standardised approach [33,34] when dealing with longer-term impacts. A number of comments were related to funding periods and how the timescale of the project does not match the timescale of potential impact. Linked to this issue, comments revealed the “difficulty of measuring the long-term effect of the project”. To tackle this problem, one coordinator described the use of a “small-scale evaluation of the project” to provide a snapshot of potential impact, but subsequently went on to describe the fact that the process “didn’t perform a good measure of impact”.

The consideration of long and short-term impacts and their measurement also provided an insight into the importance of motivation when assessing the contribution citizen science can make towards SDGs. Responses revealed that there is “no guarantee that projects will want to measure impact” once they’re over, even though this can be the time when “most impacts manifest themselves”. Regarding SDG 14 (sustainability of oceans, seas, and marine resources), one coordinator linked their motivation (or lack of it) to a prospective audience, noting that long-term impacts will not be visible during the funding period, so “it’s impossible to report this back to the funder”.

Although not specifically referencing long or short-term timescales, positive comments regarding SDGs 15 (ecosystems, forests, and desertification) and 6 (sustainable management of water) tend towards a preference for measures that can be evidenced in a timelier manner. Regarding SDG 15, one coordinator referred to their “data being used by multiple funders” as a potential measure of impact, funders that they would want funding from in the future. Similarly, regarding SDG 6, responses highlighted the active involvement of prospective stakeholders, who have “a keen interest in the outputs of citizen science activities and how they can be used”.

Attitudes shared by project coordinators during this process have shown the importance of short and long-term impacts, and the potential audiences of their assessment. Motivation to consider the links between citizen-science activities and their impact on SDGs seems to be driven by whether such assessment can be measured during the funded timeframe, the importance that stakeholders can be engaged by the process [11], and the potential funding opportunities that can be exploited.

4.4. Project Scope and Comparisons

The final theme that was highlighted by the interview process considers the scope of a project, specifically in relation to the potential scope of SDGs, and comparisons that can be made either between citizen science projects or with other traditional scientific methods. Comments concerning SDG 13 (climate action) sum up the issue, with positive responses referring to the advantages of “discussing common goals and desired outcomes” as a way to “advance the field of citizen science”. In opposition, negative responses show a lack of interest in conducting in-depth studies with a small sample size due to the effort required. This highlights an issue regarding the potential impact on the SDGs with a broad, almost global scope. For instance, SDG 13 is concerned with combating climate change and its impacts. Such a mission has worldwide interest and exposure, with many solutions that can contribute towards it. Therefore, it is not difficult to envisage that many citizen projects can contribute to this goal, garnering the interest of a range of different stakeholders. However, the task of measuring an individual project’s impact towards this goal is difficult, as one coordinator explains: “you risk realising the project was insignificant or didn’t really change anything”. As previously discussed [21], citizen science endeavours are often used to address gaps in data created through other methods, with which they are often combined. As such, it can be difficult to separate the contribution of citizen science from the many solutions involved.

Attitudes towards the comparison of their project with other solutions also varied and were tied to scope. On a positive note, there was an appreciation shown that a comparison to more traditional scientific methods could add “credibility to the citizen science community” by showing that results are as robust whilst providing a greater coverage either spatially or in terms of data quantity. However, when comparing citizen science projects against one another, it is not as straightforward. Coordinators believe that their projects are unique and very specific in design, making it “hard to compare against other projects” in a meaningful way. This is perhaps not unusual in academic and scientific research, with individuals feeling a sense of ownership of their work and therefore being protective of it.

Coordinators’ responses to the interview process show the importance that their projects are judged fairly, both when considering their scientific impact and when comparing them to alternative solutions. Whilst there is acknowledgement that measuring a project’s impact can have benefits in engaging potential stakeholders and legitimising their approach, there is also a fear that their efforts are misrepresented or not taken in the right context.

5. Conclusions

The points raised and conclusions derived through this work should be considered in the context of it being a first approach, using a relatively small sample of project coordinators. However, although this is the case, the interviewees involved are representative of a range of different projects in terms of size, age, and discipline, and a number of crosscutting themes are apparent.

The perceptions and views of citizen science project coordinators that were collated through the semi-structured interview process reveal a complex picture when considering impact and its relation to SDGs. Perhaps unsurprisingly, a more positive attitude exists when the contribution towards SDGs is measurable objectively and directly by the associated SDG indicators connected to the scientific aims of the project. When the methods used to measure impact veer towards a qualitative and less standardised approach, whilst coordinators acknowledge the benefits of the richness of the data produced, they also revealed concerns regarding their limited expertise and experience when dealing with such methods. The issue of time was also prevalent, where SDGs associated with education and wellbeing are seen as long-term impacts, which often are realised beyond the funding periods of projects that have no provision for future impact evaluation. This gave rise to a potential lack of motivation, with coordinators perhaps unwilling to commit time to evaluating impact when there is no funding available or funder to report back to. Finally, the issues of scope and comparison revealed that whilst coordinators can understand the benefits of comparisons to other, traditional scientific techniques to legitimise their approach, concerns were raised regarding comparisons within the citizen science domain. The perceived scope of SDGs influenced coordinators’ confidence of their potential impact, with broader, global themes such as climate seen as more challenging to meaningfully contribute towards.

The range of differing concerns and issues revealed by citizen science coordinators, and their variability across different SDGs, has consequences for any solutions that are derived to assist in the process of measuring impact and contribution to the SDG framework. The link between citizen science and its potential towards the SDGs is now out in the open, with journal special-issues like this and dedicated conferences such as the Citizen Science SDG Conference [43] likely to be repeated. In order to realise this, potential future frameworks and tools, including the MICS project that this research is part of, will need to consider the different ways impact is measured, and how to ensure they are accessible to users who do not necessarily have past expertise or experience in their use. The motivations of coordinators will need to be considered, with feedback designed to appeal to funders and stakeholders where appropriate. Appreciation will need to be shown to funding timelines, and thought given on how best to represent or predict long-term impact from smaller sample evaluations. The broad scope and global nature of certain SDGs can also result in a lack of connection to certain projects, and they will perhaps need to be reimagined or represented in a way that is easier for coordinators to link to their project and its aims. Ultimately, future frameworks supporting the impact assessment of citizen science towards the SDGs will have to acknowledge existing assessment techniques, whilst building upon them. It is clear that citizen science has the potential to contribute towards a wide range of SDGs, and therefore existing impact assessment methods that involve only a few domains will need to be built upon, to ensure the full potential impact of a citizen science project is realised. Whilst the mixed-methods approach used previously might well be able to best represent the different types of impact possible, future solutions need to be aware that such methods are not intuitive to all citizen science coordinators, and therefore support needs to be provided. Citizen science also comes in a range of guises, and impact assessment frameworks need to be able to take into account the breadth of potential impact, and whether it contributes towards broad SDG descriptions or specific indicators.

The interviews carried out in this study are a first, inductive attempt to better understand the perceptions of citizen-science coordinators when it comes to projects’ impact and contribution to the SDG framework. These perceptions confirm previous knowledge regarding the potential influence of citizen science toward achieving the SDGs, but highlights that, in many cases, this relationship has yet to be formally realised. Several themes and issues are revealed that demonstrate the linking of citizen science to SDGs; however, measuring their contribution is not a straightforward process. It is clear that there is space for further research to consider these issues with greater depth and across a larger sample of citizen science stakeholders. Such research could potentially take a more deductive approach, forming hypotheses based on the arguments of this work. If the full potential of citizen science as a contributor to the SDG framework is to be realised, impact assessment approaches need to be designed that fully take into account the coordinators’ point of view, thus ensuring that the problems associated with measuring the impact of traditional science practices are not repeated.