Evaluating Research Partnerships through ISO 56003 Guidelines, RRI Concepts, and Ex Post Facto Cases

Abstract

This article aims to evaluate improvements in opportunities for research partnerships focused on innovation, considering the ISO 56003 guidelines, Responsible Research and Innovation (RRI) concepts, and the experiences of the authors of this article, who are researchers in various areas (ex post facto). The results presented arise from structured debates and discussions. As a result of the analysis and reflections carried out, some improvement opportunities in most of the partnerships established among research groups can be visualised. Partners in a research group often debate aspects related to technical knowledge associated with the research and contractual procedures in detail; however, they do not equally emphasise other important partnership management aspects, such as the analysis of the research impacts on all stakeholders, the definition and adoption of a governance policy to better conduct the interaction among members, and other actions that can enhance the partnerships. This article can be characterised as a synthesis of lessons learned, in addition to a presentation of guidelines mentioned in the literature. This knowledge can be useful to other researchers in planning new partnerships or evaluating existing ones. For deciding whether to enter into a partnership or withdraw from it, the information presented here is of great value.

1. Introduction

Continuous scientific and technological development is a fundamental requirement for social evolution. In the Sustainable Development Goal 9 (SDG 9), the United Nations [1] highlights the importance of innovation and scientific research to reach sustainable development. If properly managed, the role played by science, technology, and innovation for knowledge generation and transmission can be important for humanity in several aspects [2,3].

Seturidze and Topuria [4] and De la Vega Hernández and Barcellos de Paula [5] corroborate this statement, and emphasise that the COVID-19 pandemic even more strongly evidenced the importance of science, technology, and innovation for finding solutions to society’s problems.

Although the contributions of science to social evolution have already shown their value throughout history [4,5,6,7,8,9], disinformation movements still require science to continually prove its importance in solving contemporary problems [10]. The need is evident, therefore, to reaffirm scientific and technological progress as fundamental instruments for societal evolution crucial for improving people’s quality of life, reducing inequalities, and exchanging knowledge among different individuals, contributing to achieving the targets of the 17 Sustainable Development Goals.

Scientific and technological progress can be materialised and enhanced by establishing partnerships among different research groups [11,12]. Partnerships between industry and universities can also benefit technological progress [13]. Focusing on research group partnerships, these involve establishing cooperation networks for the creation of new knowledge. These groups demand intense communication and coordination processes and involve governance practices to define procedures to transmit experiences, knowledge, and research results among members [4,14,15].

An important instrument that can assist partnerships among research groups is the ISO 56003 standard. This document considers that partnerships “create value resulting from joint work” [16]. Launched in 2019, it is an international standard that aims to present tools, methods, and guidelines to enhance partnerships for innovation. It is applicable for any type of partnership, collaboration, or organisation [16]. The ISO 56003 guidelines are plausible to be adopted in relationships among research groups, since these groups can be considered small organisational structures. The mentioned standard also recommends discussing the consequences of the desired partnership and the resulting impacts for all stakeholders (16). In this context, the concept of Responsible Research and Innovation (RRI) emerges. Increasingly, all research groups must analyse the impacts for stakeholders, whether the impacts are negative or positive, and critically analyse how their studies can contribute to sustainable development. It is an act of reflection necessary to enhance the contributions of the research in all dimensions [17,18,19].

When analysing the academic literature, we find a lack of cases and reflections provided by researchers on how research partnerships have been developed considering the mentioned aspects, exhibiting a gap to be explored (research gap). The main objective of this article is to present reflections on research partnerships developed by groups focused on innovation, considering the ISO 56003 guidelines, RRI concepts and the experiences of the authors who prepared this text.

In addition to this introduction, this article has four more sections. Section 2 presents the theoretical background to support the reflections; Section 3 is dedicated to the methodological procedures carried out; Section 4 presents the results and discussion and, finally, Section 5 presents the conclusions and final considerations.

2. Theoretical Background

Partnerships can enhance the results in different organisations [20]. In Caloghirou et al. [21], for example, the authors argue that companies use partnerships in research and development processes to share uncertainties and risks. When focused on the Sustainable Development Goals, the partnerships are characterised as essential elements to the achievement of these goals, enabling the stakeholders to maximise the impact of their actions [20].

The Partnering Initiative and the UN DESA [20] present the lifecycle of partnering, showing the relevance of defining a scope and determining the relevant partners, the need for the proper management and revision of the partnership, as well as evaluation of whether or not to continue the partnership. The same authors emphasize the need and opportunity of partnerships to create value (additional value) for the partnership and for the partners involved to justify the continuity of the partnership. Focusing on the targets of the SDG 17, the UN [22] highlights the need for developed nations to support developing countries, increasing the capacity of the developing nations to enhance their economies, and their capacity to become socially and environmentally sustainable, as well as to measure this sustainability. Public-private partnerships are also pointed out as a target for enhancing the infrastructure of these countries. Regarding the sharing of knowledge, the need for exchanging information on technology, science, and innovation is emphasized as an important partnership target to achieve the SDG 17 [22]. In Oliveira-Duarte [23], the authors corroborate this perspective and highlight the contributions that multi-stakeholder partnerships can make to this knowledge sharing, if properly prepared to participate of it. Research groups can also be benefited by these partnerships.

An important instrument to guide partnerships is the ISO 56003 standard. It covers all phases associated with a partnership, from initial debates (for example, when a team debates whether or not they want to establish a partnership), to final procedures, to the closure of activities [16]. In terms of structure, the standard is composed of eight sections.

Most of the content is presented in Section 4. Section 4 emphasises the structure of partnerships for innovation, followed by Section 5, which presents essential points to be considered when deciding whether or not to establish a partnership. Section 6 details the selection of partners; Section 7 establishes the alignment among them and, finally, Section 8 presents guidelines to define governance rules and policies (interaction among partners) [16]. The subsections from A to E present tools and criteria useful for building synergistic and lasting partnerships for innovation [16].

An essential aspect to consider when forming partnerships is the evaluation system. Several authors pointed out that the evaluation system is an essential element in improving the management process [24,25,26,27]. Focusing on the partnership for innovation, the ISO 56003 standard [16] recommends adopting a system to assess the partnership’s performance by periodical analysis, identify opportunities for improvement and disseminating lessons learned. The levels of motivation, engagement, and productivity among partnerships can vary over time [20]; therefore, periodic evaluations are important tasks to be performed.

Despite being published fourteen years ago, the importance of the study carried out by Fielden et al. [28] for managing research partnerships must be highlighted. The authors reported a management model in which academic and community leaders collaborated. The authors discussed the complexity of this relationship, how they tried to seek collaboration in the context of ongoing research, and how they continually improved the partnership. Similar to the present article, the research of Fielden et al. [28] also contributed to the process of planning and evaluating research partnerships.

The ISO 56003 recommends debating the consequences and impacts of the research for all stakeholders [16], as mentioned in the introduction. In this context, the concept of Responsible Research and Innovation (RRI) emerges [29,30,31,32]. Thapa et al. [7] mentioned that science and innovation have as their central objectives socioeconomic transformation and social development. In this sense, the RRI guidelines lead the participants in an innovative process to reflect on the implications and consequences of their actions. In their paper, Callegari and Mikhailova [6] corroborated the above-mentioned point of view.

Despite the strong emphasis on RRI, the theme has been debated for some years in different spheres. In the 20th century, for example, the European Union established the European Group on Ethics in Sciences and New Technologies to provide recommendations and establish a better dialogue between science and civil society. Another relevant milestone occurred in 2007, when the European Commission created the Science in Society (SiS) organization to intensify these dialogues. It was precisely within the SiS scope that the RRI concept was developed [33].

RRI disseminates a conception of sustainable and inclusive research and innovation. It involves establishing partnerships among different social actors (such as researchers, public policymakers, citizens, companies, and third sector organisations, among others) [34]. Stilgoe et al. [17] cited that RRI is based on reflexivity, anticipation, inclusion, and responsiveness, having diversity at its core. In fact, diversity nourishes RRI and leads it to true results. Owen et al. [18] mentioned that RRI encourages a constant review and evaluation of the “responsibility” concept by scientists and other actors participating in a partnership.

The European Commission mentions that scientific research must involve society and be addressed to society, making clear its direct relationship with sustainable development [35]. When evaluating the literature on RRI, it was observed that it is considered in many different contexts and disciplines [18,19]. Despite small peculiarities, it was possible to verify the existence of a consensus concerning its alignment with social, environmental, economic, and ethical needs and the engagement and involvement of different stakeholders, such as in areas of information and communication technology [36], and in the private sector [37,38], among others.

Callegari and Mikhailova [6], Giovanetti et al. [32], and Bauer et al. [39] recognised the complexity existing in the concept of RRI. In their study, Bauer et al. [39] pointed out a series of challenges for social engagement in RRI. Examples of these challenges are “lack of capacity to participate, lack of tangible influence, missing links for decision making, lack of debate and interests, lack knowledge, among other aspects” [39]. The complexity mentioned, however, reinforces its potential to engage professionals involved in research partnerships for innovation to mitigate social problems and minimise risks involved in the process [6]. As mentioned, partnerships among organisations can enhance the achievement of the SDGs [20].

It is essential to remember that multidisciplinary actions characterise the solutions for the problems of sustainable development and, in this sense, partnerships also enhance research actions. In general, when deciding to establish a partnership, groups or organisations are looking for the knowledge they do not have that can be provided by a partner, as pointed out by ISO 56003 [16]; in the context of sustainability, this is essential for achieving better results.

3. Methodological Procedures

The development of this article was conducted through four steps, presented in Figure 1, and subsequently debated.

At first, a careful analysis of the ISO 56003 standard was carried out and, based on this analysis, a list of best practices for partnerships was structured (step 2 was used to guide the debates in the next stage). It should also be noted that the best practices content was complemented through the RRI concepts presented in Section 2 (Theoretical Background). The final structure used to guide the reflections is presented in

Table 1. The list of best practices for partnerships used to guide the debates in this study (Source: Adapted from [4,6,7,14,15,16,17,34,36]).

#	Best Practices for Partnerships for Innovation
1	Assess competence gaps in group members regarding technical knowledge for conducting the research.
2	Assess other competence gaps in group members, including skills for managing responsible research.
3	Discuss the best approach to fill in the competence gaps (internally or through partnerships).
4	Benchmark successful research partnerships with responsible innovation.
5	Adopt an open-minded approach before prematurely choosing closest partners or those from the network.
6	Consider previous experiences of potential partners regarding technical perspectives.
7	Consider previous experiences of potential partners regarding responsible research management.
8	Align the objectives of research partners to reach a common scope of understanding.
9	Analyse stakeholders’ requirements in advance, adjusting the scope of research partnerships, if appropriate.
10	Analyse the alignment of the scope established for research partnership with United Nations Sustainable Development Goals.
11	Discuss with partners the aspects that can influence the achievement of research results and the Critical Success Factors for these results.
12	Correctly define confidentiality agreements and manners for disclosing generated results.
13	Create a structure to govern the interaction among partners, including knowledge dissemination and responsible research innovation best practices.
14	Conduct a continuous review on the understanding of members regarding partnership goals, members’ roles, and responsibilities assigned.
15	Periodically carry out a SWOT analysis to identify improvement opportunities associated with the research partnership established and to define action plans.
16	Implement a periodic process to measure and analyse the performance of the established research partnership.
17	Disseminate and debate lessons learned with partnership members for use in future research, including information on innovation and responsible research.

.

As mentioned, the content of Table 1 was used to promote a debate among the authors of this article, who each presented their perceptions considering their professional experiences as researchers. Each practice was classified as “Best practice adopted” or “Not always adopted.” The experiences of the authors of this article as researchers are considered here as cases, in an ex post facto analysis. According to Salkind [40], in this kind of study, the researcher analyses the cases after their occurrence, without interfering with the facts. For this research, the authors of this manuscript have different research backgrounds, with experience in different areas and countries. Their previous knowledge was used as input in the above-mentioned debate.

There are examples of ex post facto research in various areas of knowledge in the literature, as follows. In Goodman-Scott et al. [41], the authors perform an ex post facto analysis on the implementation of Positive Behavioral Interventions and Supports in schools of the United States. In Bellini et al. [42], a combination of ex post facto and case study was used to analyse an Enterprise Resource Planning implementation project. In Rodriguez Aramendiz and Gallardo Córdova [43], the ex post facto approach was used to evaluate the decisions made during the evaluation of virtual classes. Finally, Coral et al. [44] performed an ex post facto study to assess foreign language learning in Catalonian multilingual schools.

4. Results and Discussion

Knowledge generation and innovation are elements resulting from interactive and cumulative processes—interactive because it is through inquiries, reflections, and tensioning of existing knowledge that new knowledge flows are generated. It is cumulative because all knowledge, scientific, and innovative processes are generated from their constant accumulation over time. These two elements are enhanced when research groups establish partnerships. The approximation among research groups occurs based on scientific interest, mutual exchange, and bonds of trust. The maintenance of these aspects needs to be constantly encouraged and cultivated.

To achieve this, research governance and the interaction among research groups are fundamental requirements. In addition, it is equally essential that research groups consider that science, technology, and innovation (STI) are not ends in themselves; instead, they are the means for socioeconomic transformation and the development of humanity, as mentioned by Thapa et al. in their study [7].

Based on RRI concepts, which establish that responsible research must include anticipation, reflexivity, inclusion, and responsibility [45]; the ISO 56003 guidelines, which present tools and methods to enhance partnerships among groups; and the experiences of the authors of this article, the results are presented in

Table 2. Results obtained from debates performed.

#	Best Practices for Partnerships for Innovation (Source: See Table 1)	Results
1	Assess competence gaps in group members regarding technical knowledge for conducting the research.	Best practice adopted
2	Assess other competence gaps in group members, including skills for managing responsible research.	Not always adopted
3	Discuss the best approach to fill in the competence gaps (internally or through partnerships).	Best practice adopted
4	Benchmark successful research partnerships with responsible innovation.	Not always adopted
5	Adopt an open-minded approach before prematurely choose closest partners or from the network.	Not always adopted
6	Consider previous experiences of potential partners regarding technical perspective.	Best practice adopted
7	Consider previous experiences of potential partners regarding responsible research management.	Not always adopted
8	Align the objectives of research partners to reach a common scope of understanding.	Best practice adopted
9	Analyse stakeholders’ requirements in advance, adjusting the scope of the research partnership, if appropriate.	Not always adopted
10	Analyse the alignment of the scope established for research partnership with United Nations Sustainable Development Goals.	Not always adopted
11	Discuss with partners the aspects that can influence the achievement of research results and the Critical Success Factors for these results.	Not always adopted
12	Correctly define confidentiality agreements and manners for disclosing generated results.	Best practice adopted
13	Create a structure to govern the interaction among partners, including knowledge dissemination and responsible research innovation best practices.	Not always adopted
14	Conduct a continuous review on the understanding of members regarding partnership goals, members’ roles, and responsibilities assigned.	Not always adopted
15	Periodically carry out a SWOT analysis to identify improvements opportunities associated with the research partnership established and define action plans.	Not always adopted
16	Implement a periodic process to measure and analyse the performance of the established research partnership.	Not always adopted
17	Disseminate and debate lessons learned with partnership members for use in future research, including information on innovation and responsible research.	Not always adopted

.

From the seventeen practices mentioned, five of them were evaluated as always adopted in established partnerships. It can be inferred that they are naturally adopted because they are intrinsically related to the processes of research projects, including the definition of scope, frameworks, gaps, and possible risks from a technical perspective, expected results, and confidentiality procedures. These aspects are only expanded to the research partnership.

The other twelve practices presented in Table 2, related to the ISO 56003 guidelines and RRI concepts, are not always adopted. Analysing their content, it is possible to verify that they refer to people management, strategic management of the partnership, the management of results, and continuous improvement. Regarding people management, it is not only necessary to map technical competencies, but it is also essential to analyse motivational aspects, to carry out a continuous review of the alignment and understanding of the roles and responsibilities attributed to each partnership member, and to promote systematic discussions about factors that can influence the results. The Partnering Initiative and the UN DESA [20] mention that the levels of motivation, engagement, and productivity between partnerships can vary throughout the length of the partnership, and periodic evaluations are essential.

Focusing on the strategic management of the partnership, it is first essential to analyse the scope of the partnership and the scope of the research in order to debate possible impacts for all stakeholders, considering the United Nations Sustainable Development Goals. As mentioned by the European Commission, scientific research must be developed considering the society and be addressed to the society [35]. During the partnership development, teams need to carry out a systematic SWOT analysis, mainly to identify weaknesses and threats in the partnership and to define action plans. Partnership planning is critical for success and for achieving effective results [20].

The implementation of a process to measure and analyse the performance of the relationship among groups can also provide valuable information for strategic management. Benchmarking actions can contribute to expanding the vision of the group about possibilities, as well as sharing experiences with other research groups. A governance policy and correlated tools are essential elements to define the interaction among partners, including knowledge dissemination and responsible research innovation best practices. This is aligned with ISO 56003 [16] and ERA Learn [35] statements.

Finally, some practices are associated with results management and continuous improvement. The dissemination of lessons learned by partnership members that can be used in future research is an example of this. As seen by the results, the authors of this article envision many opportunities for improvement in the relations of partnerships among research groups considering the ISO 56003 guidelines and RRI concepts.

This article gathered experiences from researchers who participate in research partnerships in different areas. These experiences were considered ex post facto cases, as the studies can be considered useful for other researchers. When carrying out the evaluation of ex post facto cases, important lessons were learned: (1) When evaluating the performance of research partnerships focused on innovation, always consider the ISO 56003 guidelines and RRI concepts, since the theories are presented synergistically. Table 1 (presented in this article) is characterised as an excellent point of departure for future analysis. (2) This performance evaluation should not be conducted individually; all the researchers should be included to debate the strengths and weaknesses of the partnership under evaluation, through the use of a focus group, for example.

5. Conclusions and Final Considerations

This article aimed to study the possibilities for improvement in research partnerships developed by groups focused on innovation. The guidelines of the ISO 56003 standard, RRI concepts, and the authors’ experiences (ex post facto cases) were used to establish the main elements that should be improved. By analysing the best practices in partnerships for innovation, it was possible to verify some notorious points. First, science must increasingly be oriented in a broad sense to contemplate the interests of all stakeholders and promote social development. Increasingly, scientific research is characterized as an essential element for achieving sustainable development. Second, partnerships for innovation have a synergistic action; they can greatly enhance the achievement of results and be stimulated to the maximum. The ISO 56003 guidelines and RRI concepts are valuable to help reach the mentioned goals. The main theoretical implication of these findings is the need for more literature incorporating management guidelines for research projects. Besides guidelines, the sharing of successful experiences about research partnerships is also required in the literature to help other researchers conduct their projects effectively and efficiently.

As a result of the debates and reflections presented in the previous section, improvements opportunities for partnerships established to conduct research can be visualised. In general, research group partners adequately debate aspects related to technical knowledge required to conduct studies and necessary contractual procedures; however, issues that could improve partnership management are not always adopted, such as performance benchmarking of successful partnerships, debating the impacts of research for all stakeholders, defining a governance policy to provide better interaction among group members, continuously reviewing roles, assigning responsibilities, and addressing perceptions regarding partnership progress, among other actions, that could enhance the relationship of the groups. Thus, as a practical application of this study, the need for research groups to develop more structured planning for projects, even when they do not need to provide detailed reports about the projects, is suggested. The motivational and strategic aspects of partnerships stood out as items rarely addressed by research teams; these aspects are essential to reach projects goals and to be able to replicate actions that result in successful experiences.

Part of the information presented in this study was derived from the ex post facto cases and, in this sense, this aspect can be understood as a limitation of the study. However, it should be noted that the authors are experienced researchers from various areas. As a suggestion for future studies, it is recommended to carry out case studies within research group partnerships that can be considered examples the adoption of best practices.