Next Article in Journal
The Opportunity Cost of Labor for Valuing Mangrove Restoration in Mahakam Delta, Indonesia
Previous Article in Journal
Social and Solidarity Economy, Sustainable Development Goals, and Community Development: The Mission of Adult Education & Training
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Indicators for Responsible Research and Innovation: A Methodological Proposal for Context-Based Weighting

INGENIO (CSIC-UPV) Universitat Politècnica de València, Cámino de Vera s/n, València 46022, Spain
*
Author to whom correspondence should be addressed.
Sustainability 2017, 9(12), 2168; https://doi.org/10.3390/su9122168
Submission received: 17 October 2017 / Revised: 12 November 2017 / Accepted: 20 November 2017 / Published: 24 November 2017

Abstract

:
In the last decade, the term Responsible Research and Innovation (RRI) has rapidly attracted the attention of policy-makers and researchers of Europe, mainly due to its promotion by the European Commission (EC). The concretion of this framework of RRI has been articulated by the EC around six key areas: governance, public engagement, gender equality, science education, open access, and open science and ethics. The indicators to measure these dimensions have been proposed recently. In our opinion the set of indicators available so far has two weaknesses: a lack of context-based indicators and a need for hierarchical ordering. Our aim is to provide tools for policy- and decision-makers that might need to identify the more important indicators in a specific context. In this work, we explored how the multicriteria analysis technique Analytical Hierarchical Process (AHP) can be used to prioritize indicators for RRI by involving experts in the specific context. The AHP method allowed weighting indicators according to experts in the different areas and producing four different options to select indicators. The method of AHP can be an appropriated instrument to select the most suitable indicators for RRI policies and initiatives.

1. Introduction

Responsible Research and Innovation (RRI) has attracted notable interest of the scientific community in Europe. The inclusion of this concept in the European scientific policies since 2010 has been an indisputable driving force. In 2014, the European Commission (EC) defined Responsible Research and Innovation (RRI) as a practice in which “societal actors work together during the whole research and innovation process in order to better align both the process and its outcomes with the values, needs, and expectations of European society” [1]. RRI brings together a body of theoretical concerns that arose since the 18th century [2] and crystallized during the 19th century [3] around the expected and unexpected impacts of research and innovation, technology assessment, and regulation of science. Increased technological capacity has evolved in parallel with the interest in the consequences and the impact of such research activity. However, using the acronym RRI to channel these concerns is much more recent [4]. The quick and mainstreamed irruption of the concept RRI in the European scientific agenda caused a reaction from the scientific community, which reflects on how it might be articulated and put in practice, how to measure efforts in this context, and how to integrate the efforts made in other areas of knowledge. The purpose of this article is to present an exploratory study on how the multicriteria analysis technique Analytical Hierarchical Process (AHP) [5] could be used to weight and select by relevant actors the indicators proposed so far [6] to measure the processes, results, and outcomes of the RRI policies and initiatives in specific contexts. AHP is a well-known technique based on assessing criteria by means of pairwise comparisons. This technique enables taking into account several variables in accordance with the multidimensional structure of the object of analysis, making judgments based on paired comparisons and obtaining one-dimensional AHP weights that represent the relative importance of the indicators. A detailed description of this technique is presented in Section 2.3 of this article.

1.1. Responsible Research and Innovation in Europe

Proponents of the concept RRI have been mainly organizations funding public research programmes. In this regard, the European Commission (EC) has played a key role due to its influence among the agents of research in Europe.
The current organization of RRI proposed in Horizon 2020 program is the culmination of a process which began in 2001 [1]. The action plan “Science and Society” of the Sixth Framework Program for Research and Innovation of the European Commission (2002–2006) defined a common strategy that would allow a better connection between science and the public. The inclusion of that consideration moved to the next action plan, “Science in Society”, in the Seventh Framework Program for Research and Innovation (2007–2013), also of the European Commission. “Science in Society” had as objectives to accelerate participation and promote two-way dialogue between the scientific community and civil society. The interest to improve relations between science and society culminated with the adoption of the concept RRI in Part V, “Science with and for Society”, of the Horizon 2020 funding program of the European Commission. This represented a milestone in the consolidation of RRI stands in the jargon of the policies of research and innovation in Europe.
During the last years, the connections between science and society, and efforts to organize innovation ecosystems in which the concerns of the general public were considered also attracted the interest of different groups of academics [7]. They often used the term Responsible Innovation (RI), with a special focus on the management and decision-making processes around expected and unexpected impacts of technology and innovation.
The commitment to actively engage stakeholders to improve the processes of decision-making and mutual learning in the field of research and innovation is identified as one of the main features in different definitions of the term RRI [8], and it is aligned with the discourse ethics formulated by the German philosophers Karl-Otto Apel [9] and Jüger Habermas [10]. The participation of stakeholders in these decision-making processes in science through rational dialogue would be necessary to legitimize the demands, interests, and expectations of society. In this regard, the European Commission [1] (p. 2) proposed a definition that adds references about the alignment of values among stakeholders and the research and innovation ecosystem that this process involves: “Responsible Research and Innovation means that societal actors work together during the whole research and innovation process in order to better align both the process and its outcomes with the values, needs, and expectations of European Society”.
Von Schomberg’s definition of RRI [2] (p. 19) combines the reference to dialogue among stakeholders groups with the shared responsibility of the different actors: “Responsible Research and Innovation is a transparent, interactive process by which societal actors and innovations become mutually responsive to each other with a view to the (ethical) acceptability, sustainability, and societal desirability of the innovation process and its marketable products (in order to allow a proper embedding of scientific and technological advances in our society)”.
The reference to the sustainability imperative proposed by the Brundtland Commission [11] is also mentioned in the definition of Stilgoe et al. [3] (p. 1570): “Responsible innovation means taking care of the future through collective stewardship of science and innovation in the present”, in line with the concept of responsibility formulated by Hans Jonas [12].
In a similar way, van den Hoven and Jacob [13] (p. 3) remarked the importance of active participation of stakeholders in considering future impacts of research and contrasting them with the ethical values as a key feature of RRI: “Responsible Research and Innovation (RRI) refers to the comprehensive approach of proceeding in research and innovation in ways that allow all stakeholders that are involved in the processes of research and innovation at an early stage (A) to obtain relevant knowledge on the consequences of the outcomes of their actions and on the range of options open to them and (B) to effectively evaluate both outcomes and options in terms of societal needs and moral values and (C) to use these considerations (under A and B) as functional requirements for design and development of new research, products, and services”.
To the active involvement of stakeholders in all phases of the research and innovation process, a set of dimensions has been added with the purpose of establishing a framework to construct responsible ecosystems of innovation. These dimensions, identified by researchers such as Stilgoe et al. [3,4] using the term Responsible Innovation rather than RRI, are relevant to the development of the theoretical framework around RRI and RI in the future and include: anticipation, reflexivity, inclusion, and responsiveness.
As far as both concepts, RRI and RI, are developed, different combinations of levels or responsibility can be identified. On the one side, the concept of RRI combines thematic areas of the intrinsic personal responsibility of scientists and collective responsibilities of research and innovation organizations. On the other hand, the framework for RI emphasizes the features that a responsible ecosystem should have to assure correct technology assessment and decision-making processes.
For the purpose of our study, focused on the indicators proposed to measure RRI initiatives, the approach of the European Commission has been used, which is structured around thematic elements or areas of key importance in the articulation of RRI. The importance of analysing those key areas lies on the fact that they define the components of all the proposals of indicators available so far to measure RRI initiatives and policies. In this sense, the European Commission has a central role in the promotion of Responsible Research and Innovation and is responsible for articulating a framework for its promotion in European scientific policies. This newly created framework was structured by the EC around six key areas [1,14]:
  • Public engagement, which refers to the “engagement of all the societal actors—researches, industry, policy-makers, and civil society and their joint participation in the research and innovation process” [1,14].
  • Gender equality, which means that “all actors—women and men—are on board” in the public engagement activities [1,14].
  • Science education, which implies the enhancement “the current education process to better equip future researchers and other societal actors with the necessary knowledge and tools to fully participate and take responsibility in the research and innovation processes” [1,14].
  • Open Access, which means “giving free online access to the results of publicly-funded research (publication and data)” [1,14].
  • Ethics, which highlights that “in order to adequately respond to societal challenges, research and innovation must respect fundamental rights and the highest ethical standards” [1,14].
  • Governance, an umbrella key area that remarks that policy-makers “have a responsibility to prevent harmful or unethical developments in research and innovation” [1,14].
The proposal by the European Commission of these six key dimensions has led to the use of this categorization in the research projects and tenders funded by this supra-national administration. Following the argumentation about the current differences between RRI and RI, it can be identified that, apart from the key areas of public engagement and governance, the articulation of the other four areas of the EC is much more aligned with the so-called ethics of causality. In other words, some of the key areas are highly connected with the core role of science, developing knowledge about the world, and not as much with the ethics of dialogue. In fact, the content of those four areas refers to the expectation of responsible procedures and behaviour by scientists such as honest reporting of data, sharing important results, serving as peer reviews, and training the next generation of scientists properly [15].
The inclusion of these six areas in RRI, and no others, have been timidly discussed. Van den Hoven and Jacob raised attention to the fact that “many societal needs around the globe have not yet received a lot of attention from researchers, companies, and governments, despite the enormous potential for innovative solutions which accommodate widely shared public values” [13] (p. 15). Some authors [6] have proposed to increase the number of key areas of the EC framework, providing two new ones, social justice and environmental sustainability. These new areas could integrate the unattended fields mentioned by van den Hoven and Jacob. Additionally, the authors of this work consider that the area of gender equality could also include other factors of diversity such as race, age, and disability.
Recently, Stahl et al. [16] defined a Maturity Model of RRI to integrate the dimensions proposed by Stilgoe et al. [3,4] with the key areas proposed by the European Commission [1,14] around categories and components of the term.
The discussion about what areas, dimensions, and categories should articulate the framework of RRI is certainly incipient, and a passionate intellectual debate on this subject will probably take place in the next years. This debate will be of vital importance, as it will determine the content of the proposals for the monitoring of initiatives, projects, and policies of RRI. The next section will explore the state of the art in that process of monitoring RRI in Europe.

1.2. Indicators for Monitoring Responsible Research and Innovation

The commitment of the European Commission to promoting the RRI approach has resulted in the inclusion of specific calls for projects of research and coordination under this topic in Horizon 2020, concretely under the umbrella of Part V, “Science with and for Society Work Programmes”. In parallel, the European Commission is making efforts to promote the consideration of RRI as an inter-sectorial issue in the other parts of the Horizon 2020 programme not specifically dedicated to this issue.
The European Commission has also reinforced the commitment to the monitoring of the RRI approach by promoting definitions of indicators to measure initiatives and policies on RRI. For this purpose, it has funded two studies with the objective of evaluating and monitoring the performance of the different scientific actors in each of the six key dimensions previously mentioned.
The first initiative took place in 2015 with the appointment of a group of experts with the objective of establishing indicators for monitoring the impact of RRI. The result of this first action was a report published in June 2015 that provided a list of 82 indicators for the six areas proposed by the European Commission [6]. Those indicators were organized in three categories: process, outcomes, and perception, in order to include indicators referring to the action taken within the Research and Innovation sector (processes and outcomes) and to the perception of those processes and indicators by other actors and society in general [6]. The report also proposed the inclusion of two new areas: (environmental) sustainability and social justice. The group of experts decided to include these two new areas to introduce in their work the importance of addressing the question about to what extend does a research field, a research programme, or an RRI initiative contribute to sustainable growth, in the case of (environmental) sustainability and to monitor the impact of research and its effect on social justice and inclusion, in the case of social justice. In reference to these new two areas, the authors did not propose a detailed list of indicators but instead proposed ideas and suggestions about how they could be defined.
This first list of indicators is relevant to the study of monitoring RRI, as it is the first effort made to operatize a system of monitoring and reporting. Even though, it cannot be considered a definitive proposal, as in 2013 the European Commission launched a tender with a double objective: “select a set of quantitative and qualitative indicators and metrics and develop a methodology and the related tools to collect and analyse data in order to monitor evolution of RRI dimensions and benefits over time” [17] (p. 16). The results of this tender will not be available until mid-2017, which reinforces the idea of Iatridis and Schroeder [18], that metrics and tools for RRI are still preliminary or in the phase of development.
The potential of applying standards, initiatives, and principles already existing for the monitoring of other disciplines as Corporate Social Responsibility has also been explored [18]. In a similar way, Wickson and Carew [8] explored the development of indicators and quality criteria through the integration of knowledge of other disciplines such as Corporate Social Responsibility, Public Value Failure Mapping, and Multi-Criteria Decision Analysis.
Under this context, wherein the authors of this study foresee a forthcoming interest to define the areas which compose the RRI approach and the respective indicators of monitoring, this study aims to apply a methodology that will facilitate the adaptation of extended lists of indicators to the reality and requirements of specific contexts.
In this work, an explorative application of the AHP method was designed and applied to assess the importance of the indicators for six areas of RRI based on the available list of indicators. It relies on the opinion of experts in each of the six key areas of RRI proposed by the European Commission who provided different weights for the different indicators under each category (process, outcome, and perception) of the six areas. These weights are later used to propose different options to reduce a large set of indicators to a smaller one according to the experts’ opinions. These smaller sets of indicators allow decision-makers to identify where efforts should be made to measure in a specific context the more relevant information of RRI performance.
Our work is based on the three hypotheses:
  • Indicators to monitor and evaluate RRI initiatives and policies might not have the same relevance in different contexts.
  • Indicators to monitor and evaluate RRI initiatives and policies can be prioritized.
  • The AHP methodology might be a useful tool to propose reduced and context-based sets of indicators to specific contexts.
The availability of a methodology to select reduced and context-based sets of indicators of RRI initiatives and policies can be of great value for policy-makers, funding organizations, and any other decision-makers who have to decide what indicators might be used to evaluate RRI policies and initiatives in a specific context.
The remaining paper is as follows: in Section 2 the methodology for the profiling of stakeholders and the prioritization of indicators is presented, and in Section 3 the application of the proposed methodology and a detailed case study is presented with a broad description of the obtained results. Finally, in Section 4 the authors discuss the results and the main conclusions of the work.

2. Materials and Methods

The proposed methodology requires the participation of two types of agents for the adequate implementation of the working sessions, (i) the facilitators of the prioritization process and (ii) a panel of experts in the six key dimensions of RRI (public engagement, gender equality, science education, open access, ethics, and governance). In this work, the facilitators of the process (authors of the article) have reviewed the available list of indicators to measure RRI policies and prepared the materials to carry out the prioritization. They have chosen the proper list of experts for each RRI key dimension who also had a good level of knowledge of the research and/or innovation ecosystem in Spain. The facilitators have guided the experts all along the process of weighting the evaluation criteria. With these weights, the facilitators have finally ranked the indicators and proposed different options for reducing the initial broad set of indicators into a smaller one. A detailed explanation and the possibilities for its application are presented in Section 3.
The selection of Spain as a case study is based on the reason that the authors of this article have received funds from the Spanish National Ministry of Economy, Industry, and Competitiveness to analyse how the indicators proposed to measure RRI at the European level can be adapted to the reality of the Spanish Research and Innovation system. Spain, as a member of the European Union, is affected by the Research and Innovation European policies, and the methodology applied in this study can be used as an example to develop further adaptations of indicators to other regional research and innovation ecosystems as well as adaptations in other European countries. The discussion of how to identify the most appropriate indicators in different ecosystems might be of high relevance to enrich the debate around RRI indicators with a bottom-up approach.

2.1. Regarding the Experts

This work included six groups of experts, one per each dimension of RRI. The creation of a unique pool of experts to measure all the list of indicators was initially considered but discarded. The different areas of RRI have their own problems, as well as different regulatory tracks and grades of implementation in Spain. For instance, there is a large corpus of both legislation and monitoring frameworks in the field of gender equality and ethics in Spain. Working with areas with different grades of implementation and monitoring frameworks led the authors to consider that it was much more important to have reduced groups of experts with a solid knowledge of the area rather than generalists that might not understand the specific technical problems of each of the six areas. Therefore, experts should fulfil two requisites: (i) having a solid academic or professional background in the respective area and, (ii) having knowledge and professional experience in the Spanish research and/or innovation ecosystem.

2.2. Regarding the List of Indicators

This study considered the list of indicators provided by the EU report “Indicators for promoting and monitoring Responsible Research” [6]. The set included 82 indicators for the six key areas of RRI (governance, public engagement, gender equality, science education, open access/open science, and ethics) and several suggestions for defining indicators under the two new proposed areas (sustainability and social justice/inclusion). In our study, it was decided to use for prioritization the indicators directly defined in the report. Therefore, the suggestions given to define indicators for the categories of sustainability and social Justice/inclusion were not used.
As identified by the authors of the aforementioned report, the set of criteria is diverse and heterogeneous, and there is also overlap between them. Moreover, some (in particular ethics, sustainability, and social justice/inclusion) may be thought of as being more overarching and encompassing than certain others (public engagement, science education, and open access).

2.3. Regarding Weighting of Criteria

In this study, the AHP method was used for weighting the indicators. The Analytic Hierarchy Process (AHP) proposed by Saaty is a measurement theory of intangible criteria (Saaty 1980). AHP is based on the fact that the inherent complexity of a multiple criteria problem can be solved through the construction of hierarchic structures consisting of a goal, criteria, and alternatives. In each hierarchical level, paired comparisons are made with judgments using numerical values taken from the AHP absolute fundamental scale of 1–9. These comparisons lead to dominance matrices from which ratio scales are derived in the form of principal eigenvectors. These matrices are positive and reciprocal (aij = 1/aji). The synthesis of AHP combines multidimensional scales of measurement into a single one-dimensional scale of priorities.
The method is one of the most extended multicriteria techniques, and it adapts very well to the hierarchy of criteria proposed by the Expert Group on Policy Indicators for Responsible Innovation of the EU. It also has the additional advantage of being easy to explain to the experts that have to assess the different criteria in a simple and systematic way. More details on the AHP can be found in Saaty [5], Saaty and Peniwati [19], and García-Melón et al. [20].

3. Results

This section presents the results of the work. In order to facilitate the understanding, a description of the steps undertaken to measure the indicators of each of the six areas is presented and complemented with a detailed description of the tools used and the results obtained of the area that included more complexity in the work, that is, gender.

3.1. Step 1: List of Indicators

For the selection of indicators to be weighted, the only available proposal of indicators of RRI at the moment the study was designed was used as a reference. This list was included in the report “Indicators for promoting and monitoring Responsible Research” of the Expert Group on Policy Indicators for Responsible Innovation (2015).
The 82 indicators proposed were organized in three categories for nearly all of the RRI areas: process, outcomes, and perception. To facilitate the elaboration of the materials used for weighting, the indicators were codified according to the area (and three sub-areas in the case of public engagement) and category they belong to (see Table 1 for the summary of the list of indicators, Appendix A for the full list of indicators and their codes, and Table 2 for the codification of the indicators of gender). During these preparatory activities, the authors considered that two of the indicators under the category process (codes: GE_PR2.1 and GE_PR2.2) of the dimension of gender should be classified as hierarchically dependent of another indicator (code: GE_PR2).

3.2. Step 2: Identify Key Expets

As stated above, six groups of experts were identified, one per RRI area. For our work, two experts per area were selected. In the selection of the experts, it was taken into account their level of expertise in each of the areas, their knowledge of the current research and/or innovation ecosystems in Spain, and their willingness and availability to participate in this study. Moreover, some other personal average data such as gender was also considered.
A description of participant experts is given in Table 3. For some of them it has not been possible to give more details about their names or companies, due to confidential reasons. In brackets, the gender of the expert is shown: male or female.

3.3. Step 3: Weigh the Indicators

For the weighting of the indicators, the AHP method was used. AHP requires a hierarchical model of criteria (see Figure 1 for the category “Process” of the area “Gender”) to pairwise compare all the criteria (indicators) and to obtain a final weight for them [19]. A questionnaire was designed for each area for this purpose. The questionnaires were conducted through a personal interview with each of the 12 experts. Interviews were carried out either with face-to-face meetings or by videoconference, depending on the interviewee’s preferences and availability. First, a set of instructions was presented to explain which comparisons were to be made according to the hierarchical structure proposed and the nine-point Saaty’s scale. Last, the surveys were processed using software Superdecisions © v. 2.4.0. (Creative Decision, Pittsburgh, PA, USA). Weights or relative importance for each indicator and for each expert were derived. A sample of the questionnaire for the dimension “Gender” and the category “Process” is shown in Table 4.
All interviews were carried out personally; on the one hand because experts had to understand the research aims, the AHP method, and the AHP questionnaires, and on the other hand because all comments and other valuable information experts could give were to be gathered for the research. Interviews lasted around 45 min; the first stage was devoted to the research aims, the method, and the questionnaire. The second stage was devoted to answering the questions (comparisons) and the AHP facilitators did not interfere in this stage.
Every expert obtained a different set of weights, according to his/her preferences. In order to obtain the global weighting, the aggregation of the results of the individual priorities by means of geometric mean was used as suggested by Saaty and Peniwati (2008).
The detailed results for the area “Gender” are presented in Table 5, and the abbreviations for all the areas are shown in Figure 2, Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9 (for the detailed results of all areas see Appendix B).

3.4. Step 4: Reduced Set of Indicators

The prioritization of the indicators presented in the previous section allows ordering a long set of indicators by the importance given by a group of experts under each category and for each area analysed. A list with these characteristics can be a starting point to identify what indicators should be prioritized in a context where, for different reasons, it might not be possible to collect data for all the indicators. With this tool, policy- and decision-makers can identify what sort of information should be collected to monitor RRI initiatives and policies, taking into account the most relevant indicators according to a set of experts or other stakeholders in the area.
Policy- and decision-makers could use different approaches to identify the minimum indicators to be considered when monitoring and evaluating an initiative or policy. One option would be applying the criteria “Best in class”; selecting the more relevant indicator under each area and category. Following with the case of gender, the reduced list of indicators imply considering five indicators from the original list of 16 indicators (see Table 6); one in the categories of outcome and perception and three in the category of process, as the one with higher normalized geometric mean has two hierarchical dependent indicators.
A second option would be selecting a cut-off percentage and selecting just those indicators starting from the first positions for which normalized geometric means (NGM) sum up the cut-off percentage. This could be applied using the Pareto rule and, supposing that selecting 20% of the indicators could allow monitoring 80% of the phenomena, that will imply selecting the first indicators for which the normalized geometric means add up to 20%. In this study, the authors opted to use 50% instead of 20% according to the Pareto rule, as if this rule was applied, the result would be completely the same as the “Best in class” option. As our aim was to show different possibilities to accommodate the concrete needs of policy- and decision-makers, using 50% showed difference in the final sets obtained.
The application of the “50% NGM” resulted in the selection of the indicators listed in Table 7 for the area “Gender”.
A third option would be considering a percentage of the total indicators to be selected. For instance, at least half of the most relevant indicators would be selected in order to reduce the amount of data to be gathered, but still collect 50% of the more relevant indicators. By applying the rule “50% of indicators”, the results of the reduced set for the dimension “Gender” implies the reduction of the initial 16 indicators into nine, four of them from the category process, another for the category outcome, and one for the category perception, as presented in Table 8.
Finally, a fourth option would be to select the most relevant indicators not to be discarded in case of a need to reduce a long set of indicators, by selecting those indicators with normalized geometric means higher than the difference between the highest and the lowest normalized geometric mean in the category. The application of this criteria implies the selection of six indicators from the initial list of 16 indicators in the dimension “Gender” (see Table 9). In Table 9, both the selected and not selected indicators are included. The aim of this is to facilitate the understanding of the reader, as the difference between the highest and the lowest normalized geometric means is calculated from the NGM of both selected (the highest NGM) and not selected (the lowest NGM) indicators.
The reduced set of indicators for all of the areas, which can be consulted in Appendix B, differ in number depending on the criteria of the selection process used. So, for criteria “Best in class” the resulting indicators number 24, for criteria “50% NGM” there are 38, for criteria “50% indicators” there are 45, and finally for criteria “NGM highest than the difference between the highest and lowest NGM” there are 37.

4. Discussion

The way in which science and innovation will be aligned with societal concerns and values will be an increasing topic of discussion in the coming years. The proposals to operatize a framework around key areas, as put forth by the European Commission regarding RRI, or around dimensions, as proposed in the framework for RI, still need reflection and broader debate and discussion to be implemented in European and national science and innovation ecosystems. Understanding how to approach science in society and establish a continuum dialogue between scientists and the general public is of key importance both for the implicit responsibility that accompanies the enormous power of science and innovation in transforming the living conditions of society and for a justice imperative.
This exploratory study demonstrated our initial hypothesis, since the proposed methodology is suitable to weight indicators by a group of experts. As a result, we obtained different proposals of reduced sets of weighted indicators. The availability of prioritized sets of indicators can be a useful candidate for a policy-making tool to aid decisions on data gathering for monitoring cross-cutting RRI initiatives in a specific context.
Building a fair society where people can live the life they have reasons to value [21] requires that the scientific praxis and science and innovation incorporate the principles of civic ethics [22] in its functioning. Our research shows an approach to incorporating the principles of civil ethics in the process of defining how to select specific indicators to assess RRI performance in a specific context to orientate public scientific policies.
Therefore, our aim with this study was not to offer a definitive prioritized list of indicators of RRI for the Spanish research and innovation ecosystem, but to explore the potential of the methodology to advance towards that broader objective by involving relevant actors in the process of selection of indicators.
With this study, we explored how prioritization could be achieved in case that extensive list of indicators could not be used to measure policies due to lack of sources to gather information or due to the will to adapt extensive sets to specific contexts.
In order to produce reduced sets of indicators, four approaches were proposed to numerically reduce the number of indicators for each key area of RRI. The application of each of the four methods used presents different advantages and limitations.
The method “Best in class” allows selecting a short number of indicators and identifying those that are more relevant. However, this advantage is also its main limitation, as it results in a drastic reduction of the number of indicators. Furthermore, there is a relevant dependency of the resulting set of indicators to the inclusion of categories. In the application of all of the criteria, if categories had not been included and all the indicators in each key area had been compared among them, the resulting set of indicators would have been different; this fact makes a very clear impact in the method “Best in class”. Therefore, for the use of this method in future studies, a previous profound analysis of the importance of categorizing indicators or not should be conducted.
The approach based on a cut-off percentage that allows selecting the indicators in the first positions summing up the cut-off percentage, in this study the “50% NGM” method, facilitates the identification of a reduced set of indicators, while at the same time respecting the priorities given in the prioritization process by the experts.
The approach based on reducing the indicators to a concrete number of indicators, in our case to half of them, facilitates establishment of the final number of indicators by selecting the percentage of reduction expected. The main limitation of the “50% of indicators” method is that the amount of indicators resulting under each area or category responds only to a numerical value and does not consider the amount of importance given by the experts. The relevance of this method is difficult to justify, as the requirement of reducing indicators in a specific context might be linked to limitation in the resources to collect and analyse data for large lists of indicators. It seems too aleatory if no reason for limiting the amount of indicators is provided. To address this limitation, it would be interesting to combine this criteria with the inclusion of the costs of measuring indicators. The maximum budget available for evaluating and monitoring RRI policies could be a criterion used in order to select the cut-off percentage in this approach.
The fourth and last method proposed has a main advantage as it does not limit the number of indicators a priori, but the result depends on the differences between the highest and lowest normalized geometrical means and the position of each indicator in relation to that difference.
The application of these four methods showed that reductions can be made in different ways, and there are some limitations in the methodology used in this study that will have to be overcome to propose a concrete set of indicators by using this or other methods.
Firstly, it should be noted that working with a different expert team could lead to some changes in the ranking, and that the number of experts participating in the study should be increased in future studies. Additionally, the inclusion of other techniques to promote consensus could be used, such as a second round of weighting by using the Delphi method, as has already been conducted in previous studies [20].
Additionally, a definitive list of context-based set of indicators would require the integration of the forthcoming sets of indicators that are under development. In the coming months, the results from the tender launched in 2013 by the European Commission to identify quantitative and qualitative indicators and metrics to monitor the evolution of RRI dimensions will be available. Therefore, any attempt to provide context-based sets will have to integrate the proposal made in that study.
It would also be necessary to critically review the potential indicators under each dimension considering the current regulation in the specific country or region where the indicators might be applied. In this regard, the Spanish Organic Law 3/2007 of 22 March 2007 for equality among women and men already imposes some reporting requirements to different agents in the research and innovation ecosystem. Not considering those requirements would be a strategical failure to promote gender in science in the Spanish national context.
In a similar way, understanding the current efforts of research and innovation actors reporting their performance on Corporate Social Responsibility and Responsible University policies would be key to assuring the commitment in this process of companies and universities already monitoring and reporting areas of RRI. In this line, it would be also beneficial for future studies to include in the panel of experts professionals responsible for research and innovation in companies. This will be key to help to introduce into the debate companies developing research and innovation and assuring the success of the proposed indicators both to public and private developers of science and innovation.
Additionally, a profound reflection on how to measure RRI initiatives and policies among experts from the different dimensions would be necessary to be able to include new indicators to measure relevant goals of each of the key areas responding to the specific state of the art in the country. The authors propose to arrange experts focus groups and the use of participatory methods to define in a specific context the complete list of indicators that should be weighted as a first set in the process to provide context-based indicators for any science and innovation system.
In conclusion, the design of a study to select the more suitable and urgent indicators to measure RRI performance in a territory should include previous work conducted to identify new indicators that complement the proposals made so far. During the development of this study, the authors considered the suitability of the indicators proposed under a European perspective for use in national contexts without any type of adjustment. From our point of view, future study should respond to so far non-explored questions.
Firstly, it would be relevant to explore whether the list of indicators used in this exploratory study includes all the possible and relevant indicators to measure the six dimensions proposed by the European Commission in the Spanish research and innovation ecosystem. It is also necessary to reflect on the inclusion of those widely accepted in our territory and included in the Global Reporting Initiative, the Equality Plans, or the Responsible University Plans. Additionally, future studies should explore whether the set of indicators proposed has the same importance for different research and innovation contexts such as geographic areas (countries and regions) or areas of knowledge (Humanities and Social Sciences, Biology and Biomedicine, Natural Resources and Agricultural Sciences, etc.). Finally, it should be explored whether the all the research and innovation agents should inform the whole set of indicators, or if would it is necessary to establish specific sets of indicators for reporting the performance of different types of agents such as funding agencies, universities, departments and institutes, or private organizations that conduct research and innovation.

Acknowledgments

This research work was realized under the project “Propuesta de indicadores para impulsar el diseño de una política orientada al desarrollo de Investigación e Innovación Responsable en España” (INPERRI). This project is funded by the Spanish National Ministry of Economy, Industry, and Competitiveness (reference: CSO2016-76828-R). The funds received under this project covered the cost to publish in open access.

Author Contributions

Irene Monsonís Payá, Mónica García Melón, and Félix Lozano conceived and designed the experiments; Irene Monsonís-Payá performed the experiments; Irene Monsonís Payá and Mónica García Melón analysed the data; Félix Lozano contributed the literature review; Irene Monsonís Payá wrote the article.

Conflicts of Interest

The authors declare no conflict of interest. The founding sponsors had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, and in the decision to publish the results.

Appendix A

Table A1. Full list of indicators.
Table A1. Full list of indicators.
Dimension/Sub-DimensionCategoryCodeIndicator
GovernanceProcessGO1_PR1Identification of formal and informal networks that promote RRI, at both the national and EU level
GO1_PR2Activities (number) of funders to promote RRI
OutcomeGO2_OU1For each of these networks (PR1.) number of RRI debates
GO2_OU2For each of these networks (PR1.) number of RRI protocols
GO2_OU3For each of these networks (PR1.) number of RRI policies
GO2_OU4For each of these networks (PR1.) of RRI agreements
GO2_OU5Number of funding mechanisms to support RRI activities
GO2_OU6Number of euros invested in RRI projects
PerceptionGO3_PE1Involvement of the wider public in RRI debates, measured for example through social media
GO3_PE2Involvement of the wider public in RRI policy, the development of policy, protocols, agreements
GO3_PE3Number of references in applications to RRI
GO3_PE4Number of collaborative RRI projects
Public engagement/Policies, regulations, and frameworksProcessPE_PRF1_PR1Formal commitment
OutcomePE_PRF2_OU1Public engagement funding percentage from R&I
PE_PRF2_OU2Public influence on research agendas
PE_PRF2_OU3Share of public engagement in R & I projects based on consultation, deliberation or collaboration
PerceptionPE_PRF3_PE1Public expectations of involvement
PE_PRF3_PE2Researcher’s openness to pursue PE
PE_PRF3_PE3Interest of publics
Public engagement—Event and initiative making; attention creationProcessPE_EV1_PR1Science events and cycles
PE_EV1_PR2Referenda and Danish-model activities. Organised debates
PE_EV1_PR3Museums/Science centers informal settings
PE_EV1_PR4Citizen science initiatives
PE_EV1_PR5Crowd funded science and technology development
OutcomePE_EV2_OU1Media coverage
PE_EV2_OU2Social media/web 2.0 attention
PE_EV2_OU3Museum visits and impacts (on visitors, stakeholders, local communities)
PE_EV2_OU4Civil society organization activities and impacts
PerceptionPE_EV3_PE1Engagement activities (ladder)
PE_EV3_PE2Interest in science
PE_EV3_PE3Issue discrimination
PE_EV3_PE4Image of an atmosphere of science culture
Public engagement/Competence buildingProcessPE_CB1_PR1Training of communicators
PE_CB1_PR2Training of scientists/engineers
PE_CB1_PR3Mediators
PE_CB1_PR4Grass roots
OutcomePE_CB2_OU1PR staffing
PE_CB2_OU2Social Scientists collaboration
PE_CB2_OU3In house/outsourced consultancies
PE_CB2_OU4The state of science journalism
PerceptionPE_CB3_PE1Knowledge beliefs
PE_CB3_PE2Trust, confidence
PE_CB3_PE3Attitudes (utilitarian expectations, fundamental orientations)
Gender equalityProcessGE1_PR1% of Member State funding programmes explicitly including gender requirements
GE1_PR2% of research institutions (including universities) that (a) have gender equality plans and (b) provide documentation of their implementation
GE1_PR2.1% of research institutions that document specific actions that minimize/reduce barriers in work environment that disadvantage one sex (e.g., flexibility of working hours)
GE1_PR2.2% of research institutions that document specific actions aiming to change aspects of their organizational culture that reinforce gender bias
GE1_PR3% of research institutions that provide training/support for researchers in regard to the inclusion of gender dimensions in the content of research
GE1_PR4% of schools (primary and secondary) that have programmes promoting gender equality issues in regard to career choices
OutcomeGE2_OU1% of women in advisory committees
GE2_OU2% of women in expert groups
GE2_OU3% of women in proposal evaluation panels
GE2_OU4% of women in projects throughout the whole life cycle (in full-time equivalent)
GE2_OU5% of women that are principal investigators on a project
GE2_OU6% of women that are first authors on research papers
GE2_OU7% of research projects including gender analysis/gender dimensions in the content of the research
GE2_OU8% of women taking part in research mobility programmes
PerceptionGE3_PE1Perception of gender roles in science amongst young people and their parents, e.g., percentage of young people who believe that science careers are equally suitable for both women and men; percentage of parents who believe their children (daughters) will have equal opportunities to pursue a career in STEM
GE3_PE2Perception of people working in the area of R&I in regard to gender equality, e.g., percentage of women in R & I, who believe they have equal opportunities to pursue their careers in R & D in comparison to men
Science educationProcessSE1_PR1The inclusion of an initiative or requirement for RRI-related training in a research strategy/call/work programme, etc.) (yes/no; %)
SE1_PR2Capacity building for RRI-related training (existence; % of funds allocated)
OutcomeSE2_OU1Presence of RRI descriptors in the qualification frameworks for lower and higher education (EU and national level)
SE2_OU2Presence of RRI education/training (education institution/research disciplines)
SE2_OU3Presence of encouragement of RRI education/training by R & D projects (e.g., in an integrated ELSA model)
SE2_OU4Percentage of research projects with at least one educational resource delivered
SE2_OU5Percentage of research projects involving STEM teachers or students
SE2_OU6Number of projects registered in the Scientix collaboration
Open access/Open scienceProcessOSA1_PR1Documentation of open science policies
OSA1_PR2Documentation of institutional mechanisms for promoting open science
OSA1_PR3Documentation of mechanisms for learning from open science experience
OSA1_PR4Inclusion of open science measures in research policies and calls for proposals
OutcomeOSA2_OU1Percentage of research projects with a virtual environment that is updated and actively used with a threshold frequency (to be defined)
OSA2_OU2Percentage of data repositories that include explanation and commentary to facilitate use
OSA2_OU3Percentage of research projects with daily laboratory notebooks online
OSA2_OU4Percentage of research projects that report real added value by an open science mechanism (for themselves and/or other actors)
PerceptionOSA3_PE1The extent to which members of the public have visited such environments and found them useful
EthicsProcessETH1_PR1Mechanisms for multi-stakeholder/transdisciplinary processes of appraisal of ethical acceptability (best practices)
ETH1_PR2Documented ELSI/ELSA project component for ethical acceptability (best practices)
ETH1_PR3Documentation regarding normative tensions related to research integrity policies and actions
ETH1_PR4Formal and actual scope of ethics review/IRB clearance
OutcomeETH2_OU1Documented change in R & I priorities attributable to appraisal of ethical acceptability
ETH2_OU2Percentage of research proposals for which ethics review/IRB clearance process requires substantive changes in grant application or second ethics assessment

Appendix B

Appendix B presents the full list of indicators ordered by the weights obtained. It also includes the reduced sets of indicators for each area resulting from the application of the criteria presented in Section 3: “Best in class”, “50% NGM”, “50% indicators” and “NGM higher than the difference between the highest and the lowest NGM”. The final table in Appendix B shows the total number of indicators that would be selected after the application of the before-mentioned criteria.
Table A2. Normalized Geometric Mean and reduced sets of indicators in the area “Governance”.
Table A2. Normalized Geometric Mean and reduced sets of indicators in the area “Governance”.
Code of the IndicatorNormalized Geometric MeanReduced Set of Indicators “Best in Class”Reduced Set of Indicators “50% NGM”Reduced Set of Indicators “50% Indicators”Reduced Set of Indicators “NGM Higher Than the Difference between the Highest and the Lowest NGM”Difference between the Highest and Lowest Normalized Geometric Mean
Category: Process 26.79
GO1_PR163.40XXXX
GO1_PR236.60
Category: Outcome 15.93
GO2_OU325.95XXXX
GO2_OU618.89 XXX
GO2_OU517.58 XXX
GO2_OU215.83
GO2_OU411.74
GO2_OU110.02
Category: Perception 10.15
GO3_PE430.12XXXX
GO3_PE229.25 XXX
GO3_PE320.66 X
GO3_PE119.97
Total indicators selected in the reduced sets in the area 3667
Table A3. Normalized Geometric Mean and reduced sets of indicators in the area “Public Engagement”, sub-area “Policy, Regulations, and Framework”.
Table A3. Normalized Geometric Mean and reduced sets of indicators in the area “Public Engagement”, sub-area “Policy, Regulations, and Framework”.
Code of the IndicatorNormalized Geometric MeanReduced Set of Indicators “Best in Class”Reduced Set of Indicators “50% NGM”Reduced Set of Indicators “50% Indicators”Reduced Set of Indicators “Higher Than the Difference between the Highest and the Lowest Normalized Geometric Mean”Difference between the Highest and Lowest Normalized Geometric Mean
Category: Process
PE_PRF1_PR1 XXXX
Category: Outcome
PE_PRF2_OU259.44XXXX41.01
PE_PRF2_OU122.13 XX
PE_PRF2_OU318.43
Category: Perception
PE_PRF3_PE266.22XXXX51.58
PE_PRF3_PE319.14 XX
PE_PRF3_PE114.64
Total indicators selected in the reduced sets in the area 3355
Table A4. Normalized Geometric Mean and reduced sets of indicators in the area “Public Engagement”, sub-area “Event and initiative making/attention”.
Table A4. Normalized Geometric Mean and reduced sets of indicators in the area “Public Engagement”, sub-area “Event and initiative making/attention”.
Code of the IndicatorNormalized Geometric MeanReduced Set of Indicators “Best in Class”Reduced Set of Indicators “50% NGM”Reduced Set of Indicators “50% Indicators”Reduced Set of Indicators “Higher Than the Difference between the Highest and the Lowest Normalized Geometric Mean”Difference between the Highest and Lowest Normalized Geometric Mean
Category: Process 40.59
PE_EV1_PR246.35XXXX
PE_EV1_PR425.90 XX
PE_EV1_PR315.21 X
PE_EV1_PR16.76
PE_EV1_PR55.76
Category: Outcome 21.47
PE_EV2_OU134.54XXXX
PE_EV2_OU331.27 XXX
PE_EV2_OU421.13
PE_EV2_OU213.07
Category: Perception 26.22
PE_EV3_PE136.70XXXX
PE_EV3_PE428.26 XXX
PE_EV3_PE224.57
PE_EV3_PE310.47
Total indicators selected in the reduced sets in the area 3675
Table A5. Normalized Geometric Mean and reduced sets of indicators in the area “Public engagement”, sub-area “Capacity building”.
Table A5. Normalized Geometric Mean and reduced sets of indicators in the area “Public engagement”, sub-area “Capacity building”.
Code of the IndicatorNormalized Geometric MeanReduced Set of Indicators “Best in Class”Reduced Set of Indicators “50% NGM”Reduced Set of Indicators “50% Indicators”Reduced Set of Indicators “Higher Than the Difference between the Highest and the Lowest Normalized Geometric Mean”Difference between the Highest and Lowest Normalized Geometric Mean
Category: Process 23.37
PE_CB1_PR138.11XXXX
PE_CB1_PR223.58 XXX
PE_CB1_PR423.58 X
PE_CB1_PR314.74
Category: Outcome 27.29
PE_CB2_OU238.31XXXX
PE_CB2_OU327.18 XX
PE_CB2_OU423.50
PE_CB2_OU111.01
Category: Perception 56.44
PE_CB3_PE367.72XXXX
PE_CB3_PE221.00 X
PE_CB3_PE111.28
Total indicators selected in the reduced sets in the area 3565
Table A6. Normalized Geometric Mean and reduced sets of indicators in the area “Gender”.
Table A6. Normalized Geometric Mean and reduced sets of indicators in the area “Gender”.
Code of the IndicatorNormalized Geometric MeanReduced Set of Indicators “Best in Class”Reduced Set of Indicators “50% NGM”Reduced Set of Indicators “50% Indicators”Reduced Set of Indicators “Higher Than the Difference between the Highest and the Lowest Normalized Geometric Mean”Difference between the Highest and Lowest Normalized Geometric Mean
Category: Process (first hierarchical level) 34.84
GE1_PR239.48XXXX
GE1_PR133.56 XX
GE1_PR322.32
GE1_PR44.64
Category: Process (second hierarchical level)
GE1_PR2.275.00XXXX
GE1_PR2.125.00XXXX
Category: Outcome 17.34
GE2_OU623.11XXXX
GE2_OU521.56 XXX
GE2_OU715.04 XX
GE2_OU211.06 X
GE2_OU39.06
GE2_OU17.81
GE2_OU86.59
GE2_OU45.77
Category: Perception 64.17
GE3_PE282.09XXXX
GE3_PE117.91
Total indicators selected in the reduced sets in the area 5896
Table A7. Normalized Geometric Mean and reduced sets of indicators in the area “Science education”.
Table A7. Normalized Geometric Mean and reduced sets of indicators in the area “Science education”.
Code of the IndicatorNormalized Geometric MeanReduced Set of Indicators “Best in Class”Reduced Set of Indicators “50% NGM”Reduced Set of Indicators “50% Indicators”Reduced Set of Indicators “Higher Than the Difference between the Highest and the Lowest Normalized Geometric Mean”Difference between the Highest and Lowest Normalized Geometric Mean
Category: Process 58.96
SE1_PR279.48XXXX
SE1_PR120.52
Category: Outcome 20.66
SE2_OU525.77XXXX
SE2_OU220.84 XXX
SE2_OU319.79 XX
SE2_OU419.97
SE2_OU18.52
SE2_OU65.11
Total indicators selected in the reduced sets in the area 2443
Table A8. Normalized Geometric Mean and reduced sets of indicators in the area “Open Science/Open access”.
Table A8. Normalized Geometric Mean and reduced sets of indicators in the area “Open Science/Open access”.
Code of the IndicatorNormalized Geometric MeanReduced Set of Indicators “Best in Class”Reduced Set of Indicators “50% NGM”Reduced Set of Indicators “50% Indicators”Reduced Set of Indicators “Higher Than the Difference between the Highest and the Lowest Normalized Geometric Mean”Difference between the Highest and Lowest Normalized Geometric Mean
Category: Process 52.95
OSA1_PR459.25XXXX
OSA1_PR320.93 X
OSA1_PR213.51
OSA1_PR16.31
Category: Outcome
OSA2_OU467.29XXXX61.79
OSA2_OU220.14 X
OSA2_OU37.08
OSA2_OU15.50
Category: Perception
OSA3_PE1 XXXX
Total indicators selected in the reduced sets in the area 3353
Table A9. Normalized Geometric Mean and reduced sets of indicators in the area “Ethics”.
Table A9. Normalized Geometric Mean and reduced sets of indicators in the area “Ethics”.
Code of the IndicatorNormalized Geometric MeanReduced Set of Indicators “Best in Class”Reduced Set of Indicators “50% NGM”Reduced Set of Indicators “50% Indicators”Reduced Set of Indicators “Higher Than the Difference between the Highest and the Lowest Normalized Geometric Mean”Difference between the Highest and Lowest Normalized Geometric Mean
Category: Process 34.61
ETH1_PR140.62XXXX
ETH1_PR236.91 XXX
ETH1_PR316.46
ETH1_PR6.01
Category: Outcome 58.96
ETH2_OU179.48XXXX
ETH2_OU220.52
Total indicators selected in the reduced sets in the area 2333
Table A10. Total indicators selected for all the areas in the reduced sets.
Table A10. Total indicators selected for all the areas in the reduced sets.
Reduced Set of Indicators “Best in Class”Reduced Set of Indicators “50% NGM”Reduced Set of Indicators “50% Indicators”Reduced Set of Indicators “Higher Than the Difference between the Highest and the Lowest Normalized Geometric Mean”
Total indicators selected in the reduced sets for all the areas24384537

References

  1. European Commission. Responsible Research and Innovation: Europe’s Ability to Respond to Societal Challenges; European Union: Brussels, Belgium, 2012. [Google Scholar]
  2. Von Schomberg, R. A Vision of Responsible Research and Innovation. In Responsible Innovation; Owen, R., Heintz, M., Bessant, J., Eds.; John Wiley: London, UK, 2013; pp. 51–74. [Google Scholar]
  3. Stilgoe, J.; Owen, R.; Macnaghten, P. Developing a framework for responsible innovation. Res. Policy 2013, 42, 1568–1580. [Google Scholar] [CrossRef] [Green Version]
  4. Owen, R.; Stilgoe, J.; Macnaghten, P.; Gorman, M.; Fisher, E.; Guston, D. A Framework for Responsible Innovation. In Responsible Innovation; John Wiley & Sons, Ltd.: Chichester, UK, 2013; pp. 27–50. [Google Scholar]
  5. Saaty, T.L. The Analytic Hierarchy Process; RWS Publications: Pittsburgh, PA, USA, 1980. [Google Scholar]
  6. Expert Group on Policy Indicators for Responsible Research and Innovation. Indicators for Promoting and Monitoring Responsible Research and Innovation; Report from the Expert Group on Policy Indicators; Publication Office of the European Union: Luxembourg, 2015. [Google Scholar]
  7. Owen, R.; Macnaghten, P.; Stilgoe, J. Responsible research and innovation: From science in society to science for society, with society. Sci. Public Policy 2012, 39, 751–760. [Google Scholar] [CrossRef]
  8. Wickson, F.; Carew, A. Quality criteria and indicators for responsible research & innovation: Learning from transdisciplinarity. J. Responsib. Innov. 2014, 1, 254–273. [Google Scholar]
  9. Apel, K.-O. Diskurs und Verantwortung; Suhrkamp: Frankfurt, Germany, 1988. [Google Scholar]
  10. Habermas, J. Faktizität un Geltung; Suhrkamp: Frankfurt, Germany, 1992. [Google Scholar]
  11. World Commission on Environment and Development, United Nations. Our Common Future; Oxford University Press: Oxford, UK, 1987. [Google Scholar]
  12. Jonas, H. Das Prinzip Verantwortung; Suhrkamp: Frankfurt, Germany, 1979. [Google Scholar]
  13. Van den Hoven, J.; Jacob, K. Options for Strengthening Responsible Research and Innovation; European Union: Brussels, Belgium, 2013. [Google Scholar]
  14. European Commission. Responsible Research and Innovation: Europe’s Ability to Respond to Societal Challenges; European Union: Brussels, Belgium, 2014. [Google Scholar]
  15. Douglas, H.E. The moral responsibilities of scientists: (Tensions between autonomy and responsibility). Am. Philos. Q. 2003, 40, 59–68. [Google Scholar]
  16. Stahl, B.; Obach, M.; Yaghmaei, E.; Ikonen, V.; Chatfield, K.; Brem, A. The Responsible Research and Innovation (RRI) Maturity Model: Linking Theory and Practice. Sustainability 2017, 9, 1036. [Google Scholar] [CrossRef]
  17. European Commission. Tender Specifications for the Study on Monitoring the Evolution of Responsible Research and Innovation. Available online: https://infoeuropa.eurocid.pt/files/database/000056001-000057000/000056403_2.pdf (accessed on 22 November 2017).
  18. Iatridis, K.; Schroeder, D. Responsible Research and Innovation in Industry: The Case for Corporate Responsibility Tools; Springer: Cham, Switzerland, 2016. [Google Scholar]
  19. Saaty, T.L.; Peniwati, K. Group Decision Making: Drawing out and Reconciling Differences; RWS Publications: Pittsburgh, PA, USA, 2008. [Google Scholar]
  20. García-Melón, M.; Gómez-Navarro, T.; Acuña-Dutra, S. A combined ANP-delphi approach to evaluate sustainable tourism. Environ. Impact Assess. Rev. 2012, 34, 41–50. [Google Scholar] [CrossRef]
  21. Sen, A. Development as Freedom; Radom House: New York, NY, USA, 1999. [Google Scholar]
  22. Cortina, A. Ciudadanos del Mundo: Hacia una Teoría de la Ciudadanía; Alianza Editorial: Madrid, Spain, 1997. [Google Scholar]
Figure 1. Hierarchy for criteria (indicators) for the dimension “Gender”, category “Process”.
Figure 1. Hierarchy for criteria (indicators) for the dimension “Gender”, category “Process”.
Sustainability 09 02168 g001
Figure 2. Prioritization of indicators of Governance.
Figure 2. Prioritization of indicators of Governance.
Sustainability 09 02168 g002
Figure 3. Prioritization of indicators of Public Engagement in the sub-dimension of Policies, Regulations, and Framework.
Figure 3. Prioritization of indicators of Public Engagement in the sub-dimension of Policies, Regulations, and Framework.
Sustainability 09 02168 g003
Figure 4. Prioritization of indicators of Public Engagement in the sub-dimension of Events and Initiative making/Attention.
Figure 4. Prioritization of indicators of Public Engagement in the sub-dimension of Events and Initiative making/Attention.
Sustainability 09 02168 g004
Figure 5. Prioritization of indicators of Public Engagement in the sub-dimension of Competence Building.
Figure 5. Prioritization of indicators of Public Engagement in the sub-dimension of Competence Building.
Sustainability 09 02168 g005
Figure 6. Prioritization of indicators of Gender equality.
Figure 6. Prioritization of indicators of Gender equality.
Sustainability 09 02168 g006
Figure 7. Prioritization of indicators of Science education.
Figure 7. Prioritization of indicators of Science education.
Sustainability 09 02168 g007
Figure 8. Prioritization of indicators of Open Science/Open Access.
Figure 8. Prioritization of indicators of Open Science/Open Access.
Sustainability 09 02168 g008
Figure 9. Prioritization of indicators of Ethics.
Figure 9. Prioritization of indicators of Ethics.
Sustainability 09 02168 g009
Table 1. Summary of the list of indicators.
Table 1. Summary of the list of indicators.
AreaCategoryNumber of Indicators
GovernanceProcess2
Outcome6
Perception4
Public engagement/Policies, regulations and frameworksProcess1
Outcome3
Perception3
Public engagement/Event and initiative making; attention creationProcess5
Outcome4
Perception4
Public engagement/Competence buildingProcess4
Outcome4
Perception3
Gender equalityProcess6
Outcome8
Perception2
Science educationProcess2
Outcome6
Open access/Open scienceProcess4
Outcome4
Perception1
EthicsProcess4
Outcome2
Total 82
Table 2. List of indicators and codes for the dimension Gender.
Table 2. List of indicators and codes for the dimension Gender.
AreaCategoryCodeIndicator
Gender equalityProcessGE1_PR1% of Member State funding programs explicitly including gender requirements
GE1_PR2% of research institutions (including universities) that (a) have gender equality plans and (b) provide documentation of their implementation
GE1_PR2.1% of research institutions that document specific actions that minimize/reduce barriers in work environment that disadvantage one sex (e.g., flexibility of working hours)
GE1_PR2.2% of research institutions that document specific actions aiming to change aspects of their organizational culture that reinforce gender bias
GE1_PR3% of research institutions that provide training/support for researchers in regard to the inclusion of gender dimensions in the content of research
GE1_PR4% of schools (primary and secondary) that have programs promoting gender equality issues in regard to career choices
OutcomeGE2_OU1% of women in advisory committees
GE2_OU2% of women in expert groups
GE2_OU3% of women in proposal evaluation panels
GE2_OU4% of women in projects throughout the whole lifecycle (in full-time equivalent)
GE2_OU5% of women that are principal investigators on a project
GE2_OU6% of women that are first authors on research papers
GE2_OU7% of research projects including gender analysis/gender dimensions in the content of the research
GE2_OU8% of women taking part in research mobility programs
PerceptionGE3_PE1Perception of gender roles in science amongst young people and their parents, e.g., percentage of young people who believe that science careers are equally suitable for both women and men; percentage of parents who believe their children (daughters) will have equal opportunities to pursue a career in STEM 1
GE3_PE2Perception of people working in the area of R & I 2 in regard to gender equality, e.g., percentage of women in R & I, who believe they have equal opportunities to pursue their careers in R & I in comparison to men
1 STEM is the acronym of Science, Technology, Engineering and Mathematics; 2 R&I is the acronym of Research and Innovation.
Table 3. List of interviewed experts.
Table 3. List of interviewed experts.
DimensionExperts Interviewed
Public engagementOne project manager responsible for public engagement policies in a public research institution (F 1)
One academic with experience in public engagement in different research and innovation public funded projects (F)
Gender equalityOne academic in the field of gender and public policies (F)
One representative of a public research and innovation funding body (M 2)
Science educationOne academic in the field of science in society, expert knowledge and social engagement, science and values, and science communication and education (F)
One academic in the field of Science Education (F)
Open accessOne academic in the field of open access and open science (F)
One representative of a public research and innovation funding body (M)
EthicsTwo academics in the field of research ethics (M)
GovernanceOne academic in the field of governance (M)
One academic in the field of regulatory science and governance (F)
1 F refers to female; 2 M refers to male.
Table 4. Sample of the AHP 1 questionnaire for gender indicators in the category “Process”.
Table 4. Sample of the AHP 1 questionnaire for gender indicators in the category “Process”.
From Your Point of View, Which Indicator Is More Important and to What Extent
Can It Be Used to Evaluate and Monitor RRI Initiatives in Spain Regarding Gender
in Research and Innovation Processes?
GE1_PR1. Explicit gender requirements in funding programmes975313579GE1_PR2. Implementation of equality plans in research institutions
GE1_PR1. Explicit gender requirements in funding programmes975313579GE1_PR3. Provision of training and support for researchers on gender analysis in research
GE1_PR1. Explicit gender requirements in funding programmes975313579GE1_PR4. Schools that have programmes promoting gender equality issues on regard to career choices
GE1_PR2. Implementation of equality plans in research institutions975313579GE1_PR3. Provision of training and support for researchers on gender analysis in research
GE1_PR2. Implementation of equality plans in research institutions975313579GE1_PR4. Schools that have programmes promoting gender equality issues on regard to career choices
GE1_PR3. Provision of training and support for researchers on gender analysis in research975313579GE1_PR4. Schools that have programmes promoting gender equality issues on regard to career choices
1 AHP refers to Analytical Hierarchical Process.
Table 5. Normalized geometric mean of the weights obtained in the dimension “Gender”.
Table 5. Normalized geometric mean of the weights obtained in the dimension “Gender”.
Code and Name of the IndicatorGlobal Weight (% within Category)
Category: Process (first hierarchical level)
GE1_PR2. Implementation of equality plans in research institutions39.48
GE1_PR1. Explicit gender requirements in funding programmes33.56
GE1_PR3. Provision of training and support for researchers on gender analysis in research22.32
GE1_PR4. Schools that have programmes promoting gender equality issues in regard to career choices4.64
Category: Process (second hierarchical level)
GE1_PR2.2. Specific actions to change organizational culture reinforcing gender bias in research institutions75.00
GE1_PR2.1. Specific actions to minimize sex barriers in work environment that disadvantage one sex in research institutions25.00
Category: Outcome
GE2_OU6. Women that are first authors in papers23.11
GE2_OU5. Women that are principal researchers in projects21.56
GE2_OU7. Research projects including gender analysis in the content of research15.04
GE2_OU2. Women in expert groups11.06
GE2_OU3. Women in evaluation panels9.06
GE2_OU1. Women in advisory committees7.81
GE2_OU8. Women taking part in research mobility programmes6.59
GE2_OU4. Women in projects throughout the whole life cycle5.77
Category: Perception
GE3_PE2. Perception of people working in R & I in regard to gender equality82.09
GE3_PE1. Perception of gender roles in science amongst young people and their parents17.91
Table 6. Reduced set of indicators in the area “Gender” applying the criteria “Best in class”.
Table 6. Reduced set of indicators in the area “Gender” applying the criteria “Best in class”.
Code of the IndicatorNormalized Geometric Mean
Category: Process (first hierarchical level)
GE1_PR239.48
Category: Process (second hierarchical level)
GE1_PR2.275.00
GE1_PR2.125.00
Category: Outcome
GE2_OU623.11
Category: Perception
GE3_PE282.09
Total indicators in the reduced set5
Table 7. Reduced set of indicators in the area “Gender” applying the criteria “50% NGM”.
Table 7. Reduced set of indicators in the area “Gender” applying the criteria “50% NGM”.
Code of the IndicatorNormalized Geometric Mean
Category: Process (first hierarchical level)
GE1_PR239.48
GE1_PR133.56
Category: Process (second hierarchical level)
GE1_PR2.275.00
GE1_PR2.125.00
Category: Outcome
GE2_OU623.11
GE2_OU521.56
GE2_OU715.04
Category: Perception
GE3_PE282.09
Total indicators in the reduced set8
Table 8. Reduced set of indicators in the area “Gender” applying the criteria “50% of indicators”.
Table 8. Reduced set of indicators in the area “Gender” applying the criteria “50% of indicators”.
Code of the IndicatorNormalized Geometric Mean
Category: Process (first hierarchical level)
GE1_PR239.48
GE1_PR133.56
Category: Process (second hierarchical level)
GE1_PR2.275.00
GE1_PR2.125.00
Category: Outcome
GE2_OU623.11
GE2_OU521.56
GE2_OU715.04
GE2_OU211.06
Category: Perception
GE3_PE282.09
Total indicators in the reduced set9
Table 9. Reduced set of indicators in the area “Gender” applying the criteria “NGM higher than the difference between the highest and the lowest NGM”.
Table 9. Reduced set of indicators in the area “Gender” applying the criteria “NGM higher than the difference between the highest and the lowest NGM”.
Code of the IndicatorNormalized Geometric MeanDifference between the Highest and Lowest NGM
Category: Process (first hierarchical level) 34.84
Included in the reduced set
GE1_PR239.48
Not included in the reduced set
GE1_PR133.56
GE1_PR322.32
GE1_PR44.64
Category: Process (second hierarchical level)
Included in the reduced set
GE1_PR2.275.00
GE1_PR2.125.00
Category: Outcome 17.34
Included in the reduced set
GE2_OU623.11
GE2_OU521.56
Not included in the reduced set
GE2_OU715.04
GE2_OU211.06
GE2_OU39.06
GE2_OU17.81
GE2_OU86.59
GE2_OU45.77
Category: Perception 64.17
Included in the reduced set
GE3_PE282.09
Not included in the reduced set
GE3_PE117.91
Total indicators in the reduced set6

Share and Cite

MDPI and ACS Style

Monsonís-Payá, I.; García-Melón, M.; Lozano, J.-F. Indicators for Responsible Research and Innovation: A Methodological Proposal for Context-Based Weighting. Sustainability 2017, 9, 2168. https://doi.org/10.3390/su9122168

AMA Style

Monsonís-Payá I, García-Melón M, Lozano J-F. Indicators for Responsible Research and Innovation: A Methodological Proposal for Context-Based Weighting. Sustainability. 2017; 9(12):2168. https://doi.org/10.3390/su9122168

Chicago/Turabian Style

Monsonís-Payá, Irene, Mónica García-Melón, and José-Félix Lozano. 2017. "Indicators for Responsible Research and Innovation: A Methodological Proposal for Context-Based Weighting" Sustainability 9, no. 12: 2168. https://doi.org/10.3390/su9122168

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop