The Challenge of Transdisciplinary Research: A Case Study of Learning by Evaluation for Sustainable Transport Infrastructures

Abstract

While transdisciplinary (TD) research is desired in order to solve real world sustainability issues, this may be challenging for both academic and non-academic participants. Supporting learning through evaluation, we analyzed a project aiming at sustainable transport infrastructures. After developing a TD research framework as a benchmark, two external independent evaluators interviewed all project researchers, representatives for end-users, and donors. The evaluators compared results with the framework, and evaluators and participants critically reflected on the results together. There were three inconsistencies relative to the framework: (1) limited understanding of TD research among project management, end-users, and most of the researchers; (2) no structured learning process among end-users; instead, they expressed very diverse opinions about what they expected from the project; (3) project leaders had limited understanding of the special challenges of TD research, did not fully understand the status of the project’s social system, and thus did not act as facilitators of the required collaborative learning process. Non-academic participants saw themselves as customers and not as partners in the knowledge production process. We conclude that TD problem-solving research requires much time and needs facilitation and training. A preparatory phase with a lower level of funding would be helpful in preparing for TD processes.

1. Introduction

Sustainable development (SD) as a societal process towards sustainability of landscapes in a non-disciplinary reality is a major challenge for researchers, planners, and land managers [1]. Planning processes involve both the social system, with its actors and stakeholders from multiple sectors and levels [2,3], and sustainability consequences in a geographical area with biophysical, anthropogenic, and perceived dimensions [4,5]. Planning with the aim to realize sustainability policy should be based on evidence-based knowledge [6,7,8] and planners that use a learning approach as reflective practitioners [9,10,11]. However, it is not easy for planners to use an evidence-based approach because both academic research and researchers are disciplinary and thus often do not address real world problems in a holistic manner [12]. Society has for a long time recognized this issue and adapted research policy to favor problem-solving transdisciplinary (TD) research, e.g., [13,14]. Research foundations and funding programs have adapted accordingly and often request projects with a mix of competences from several disciplines and end-users [14,15,16,17,18,19,20,21].

TD research applied to solving sustainability issues in landscapes as integrated social and ecological systems requires that (1) different research disciplines together build a framework for the research based on input from non-academic stakeholders and (2) the research is done in collaboration between academic and non-academic stakeholders to address real world sustainability challenges, e.g., [5,22]. This means that also disciplinary research results are important building blocks for a TD research process. TD research can thus be seen as an effort to develop evidence-based solutions to sustainability challenges together with stakeholders [23,24,25]. There are many terms describing TD research that are useful to address real world problems, including, but not limited to, integrative, interactive, action, participatory, interdisciplinary, and applied research, e.g., [22]. The interest in TD research is not new. As early as 1944, Brozek and Keys [26] proposed three steps to make research more useful to society: (1) getting acquainted with the problems and methods of the neighboring fields, (2) addressing the “science of science” which would provide necessary philosophical and epistemological perspectives, and (3) developing the social skills required for stimulating effective cooperation among disciplines. Due to academia’s limited capacity to solve real world problems, the traditional monopoly of academia in knowledge production has been questioned [27,28]. This is because traditional academic research often has a main focus on “increments to knowledge”, measured quantitatively as the number of scientific articles and citations, e.g., [13,29]. A key benefit of TD research is to support the sustainability of social–ecological systems, hence the emergence of sustainability science [25,30,31].

There are indeed many barriers to TD research. For example, Angelstam et al. [5] identified the following four: the amount of traditional disciplinary formal and informal control, adaptation of project applications to fill the TD research agenda, stakeholder participation, and functional team building/development based on self-reflection and experienced leadership. Axelsson [22] reviewed the literature about TD research and identified 15 barriers including lack of a generally accepted definition and associated terminology; limited capacity to collaborate within academia and with non-academic stakeholders; absence of a reward system for researchers that supports integrative research; language and cultural barriers among different research disciplines, between academic and non-academic stakeholders, and between different non-academic stakeholders, e.g., [32]; insufficient project periods because integrative research takes longer; few neutral societal platforms that could host integrative research, e.g., [33]; failure to recognize the need for additional competencies such as communicators, facilitators, and knowledge brokers as complements to researchers. Polk [34] summarizes the origins of barriers in TD research to institutional, organizational, and cognitive differences in the participating organizations.

Nevertheless, many projects aim to perform TD research. However, reviews and assessments about if and how these projects manage to integrate different research disciplines and non-academic actors to provide solutions, improvements, or new ways to handle sustainability challenges are not common [35,36]. This is despite the need for learning through evaluations to improve research project success [37,38,39,40] and to contribute to the body of knowledge on different approaches to make TD research more socially robust, e.g., [41]. Studies about human health, e.g., [42,43,44], and the environment provide examples [45,46].

Management of complex TD research projects benefits from learning through ongoing evaluations [37]. This means that an external reviewer follows the process, collects data, and questions actions, strategies, and results to ensure that the project is steered towards its agreed goals [37]. This is especially true of large projects that aim to contribute to long-term sustainability [47]. Since TD research project results cannot be measured by the number of scientific publications only, a learning evaluation that includes academic and non-academic actors and the funding agency is important [47]. This process can pro-actively identify gaps and solutions, instead of just identifying failures and failing actors [41].

This study focuses on what can be learned from a comprehensive and reflective analysis of a particular research project towards sustainable transport infrastructure as a case study. The Swedish research council for strategic environmental research (Mistra) aims for top scientific quality research, which is put to practical use in companies, public administration, and NGOs [48,49]. The TransportMistra research program began in 2006 with the first out of two planned 3-year phases aimed at contributing to more sustainable transport road and railroad systems. INCLUDE was one of three projects in TransportMistra and focused on how to include ecological and cultural values, e.g., [50,51], in planning for sustainable transport infrastructures. After the first of two planned 3-year phases, the TransportMistra research program was terminated. According Mistra’s own evaluations [52,53], this was due to poor scientific quality and poor collaboration with end-users. As a response to the pre-mature discontinuation of the TransportMistra, the INCLUDE researchers initiated an evaluation of their own performance and potential contribution to the termination. As the project was complex and covered numerous disciplines, as well as requiring continuous collaboration with end-users, it was deemed to be a type of project that required a TD approach. In addition, the application for funding described an inter- and transdisciplinary research project. We therefore used a TD framework to evaluate the project’s development and outcomes.

The aim of this study is to evaluate the extent to which the research project INCLUDE met the target of being a TD knowledge production process among researchers and stakeholders as a base for discussing the challenges and opportunities for working with transdisciplinary research in general. The research process in this study included the following steps: (1) defining TD research, (2) mapping of the chronology of the TransportMistra research program and the INCLUDE project development and its social system, (3) development of a normative model for TD research, and (4) an assessment of the INCLUDE research project in relation to the model. Finally, based on our assessment, we provide recommendations for succeeding with TD research.

2. Materials and Methods

2.1. Overview of Approach

The evaluation procedure was initiated and started during two INCLUDE workshops about communication (April and December 2008), where the idea to assess the INCLUDE project was formed, and an outline of this study and of the research process was produced by the workshop participants. Two external researchers (R.A. and M.L.) were asked to lead the auditing work to secure an un-biased, neutral assessment. Both these evaluators each had >20 years’ experience of environmental auditing and facilitation of learning processes, both in Sweden and internationally. Except for the participation in two workshops to introduce this study, and performing the study itself, the two external evaluators were never involved in the INCLUDE project. The two external researchers did the literature review and all interviews, while all co-authors contributed to the development of a TD framework as a normative model for comparative analysis and the interview guide. The two external evaluators were given full freedom to evaluate the research project in relation to the normative model. These two researchers collected all the data and did all the initial analyses. Subsequently, the evaluators and the researchers together critically reflected on both the TD research process and individual researchers’ contributions. One criterion when choosing the two external researchers was that they were independent enough to tell the story as they perceived it in relation to the normative model and based on the empirical data, which cannot easily be affected by the will of the researchers to be viewed as having performed well.

Following Flood and Romm [54], multiple sets of methods and data sources were used to collect data: (1) a literature review about integrative knowledge production and research to define TD research and to develop a normative model for TD research [22], (2) qualitative interviews [55,56] with academic and non-academic partners, as well as donors of INCLUDE, and (3) collaborative critical reflections on the results together with the INCLUDE researchers.

The literature review on integrative and TD research was carried out using several academic internet databases and web search engines. As the point of departure for the literature review, we used the following search words: integrative research, interactive research, transdisciplinary, interdisciplinary, knowledge production, collaborative learning, and integrated natural resource management.

Following Roux et al.’s [57] advice on reflecting the aspirations of funders, providers, and users of research, the two external researchers interviewed the INCLUDE project management team (2 persons), all INCLUDE researchers and research assistants (12 persons), TransportMistra board members (2 persons), end-users (5 persons), and Mistra staff (2 persons). The interviews were qualitative and open-ended to give the informants full freedom to express their impressions of the research project [55,58]. All interviews were conducted during 2009 and 2010 and lasted between 30 and 105 min, with an average of just above an hour. We used the same interview guide for all informants. The interviews were recorded and later transcribed by a professional consultant, resulting in approximately 1500 A4 pages of transcribed qualitative data.

By working iteratively on the manuscript, collaborative critical reflection was encouraged, e.g., [59]. When reflecting on the results, the analysis of the INCLUDE research project emphasized the qualitative approach based on comparison with the normative benchmark model, which might be described as auditing, e.g., [40,60]. It was a demanding and subtle work to develop a narrative based on interviews, e.g., [61], and then compare these with the different steps of the normative model. To enhance the opportunities for evaluators, researchers, and other informants to talk freely about each other, we agreed beforehand to allow only the evaluators to read the transcripts. We also limited the use of quotations as much as possible to protect the informants’ anonymity and decrease the risk of further conflicts. Besides the above, while quotes are commonly used as an instrument for the documentation of evidence, “the credibility of this is not sufficient” [62] (p. 475). Flick [63] used the term selective plausibilization to capture the risk that only passages are chosen that are believed by the observer to capture the key characteristics of the case. During the interviews, information about several conflicts during the research program surfaced, and because of this, the team of authors decided to wait some years before publishing the study.

2.2. Analytic Framework

A normative model describing the steps of a TD research process was developed as a benchmark for comparison with the performance of the INCLUDE project. The statutes of the research council Mistra defined the kind of research that was supposed to take place in a Mistra program. Using this as the point of departure, the evaluators reviewed the relevant literature. For the normative model, we used and adapted the works by different scholars on definitions of integrative and collaborative approaches to TD research (

Table 1. The gradient from disciplinary to transdisciplinary research. Terms and their meanings (adapted from [78]). By different disciplines, we mean human, social, and natural sciences as well as different disciplines within the main scientific disciplines. Policy cycle refers to the continuously ongoing process of (1) policy formulation, (2) governance to find acceptable ways to management to reach the aim of the policy, (3) actual management, and (4) evaluation of the process and outcome.

Term	Explanation/Properties
Disciplinary research	- Only one academic discipline represented - Disciplinary aim of research project - No exchange or cooperation with other academic disciplines - Development of disciplinary knowledge and theory
Multidisciplinary research	- Two or more academic disciplines - Work from disciplinary perspectives with a common theme - Loose cooperation between researchers from different disciplines - Development of disciplinary knowledge and theory
Interdisciplinary research	- Two or more integrated academic disciplines - The development of a common scientific framework and goal among participants from different academic disciplines and for the whole research project - Development of integrated scientific knowledge and theory
Participatory research	- Academic and non-academic actors - Exchange of knowledge and research results as information or dialogue between researchers and non-academic actors - Disciplinary or multidisciplinary - Ending with the development of disciplinary theories and knowledge
Participatory knowledge production	- Academic and non-academic actors - Exchange of knowledge and research results as information or dialogue between researchers and non-academic actors - Disciplinary or multidisciplinary - Policy cycle, implementation of results as an effort to improve, handle, or solve real world problems, and development of disciplinary theory
Transdisciplinary research	- Multiple academic and non-academic actors - Development of a common framework and goal among all actors - Integration of academic and non-academic actors - Ending with the development of integrated scientific knowledge and theory - Policy cycle, implementation of results as an effort to improve, handle, or solve real world problems, and development of transdisciplinary theory.

) and different modes of knowledge production (

Table 2. Different modes of knowledge production in natural resource management, research, and education.

Actors Involved	No Integration	Some Integration	High Integration, True Partnership
Non-academic	- consulting - one actor or actors from one sector - reports or no written documentation	- local collaboration - development projects - actors from different sectors - reports or no written documentation	- collaborative learning - problem-based learning - representative actors - reports or no written documentation
Non-academic and academic	- consulting - customer-defined research - customer–producer relation - reports	- participatory research - participatory research collaboration - reports and scientific publications	- landscape approach - transdisciplinary research - transdisciplinary knowledge production for the issue - representative actors - reports and scientific publications
Academic	- disciplinary research - scientific publications	- multidisciplinary research - scientific publications	- interdisciplinary research - scientific publications

). The distinction between research and knowledge production is important to notice. We see this separation as crucial because, for research, the ultimate goal is always the development of scientific knowledge and theory, even though there are many researchers that wish for their research to also be used to solve real world problems. In comparison, knowledge production is a process taking place in society, aiming to solve real world problems or challenges. This implies that, for researchers participating in an integrative TD process, there is a need to participate in the entire process, from problem identification through the phases of knowledge production and learning until the problem is solved.

Inspiration for the normative model for TD research came also from the collaborative learning approach, which is a method aiming to establish good governance for sustainable management of natural resources [64], scholarly work on new modes of knowledge production and socially robust knowledge [27,28,65], and existing frameworks for inter- and TD research projects, as well as for the evaluation of such projects, e.g., [64,66,67,68]. Those researchers describe processes for the development of governance systems for conflict situations including a wide range of landscape approach concepts e.g., [69,70,71], such as model forest [72], biosphere reserve [73,74] and LEADER [75], the collaborative learning approach [63], research on integrative research processes [42,76,77,78,79,80], and learning through ongoing evaluation [37].

Daniels and Walker [63] describe five distinct phases in a collaborative learning process: (1) assessment—an evaluation of the context and the potential for collaboration takes place, (2) training—stakeholders build an appreciation for collaboration and learn some specific techniques of collaborative learning, (3) design—development of a context-specific strategy for involving stakeholders in a meaningful process, (4) implementation/facilitation—to conduct project activities and decision making, (5) evaluation—data gathering and reflection to learn from participating stakeholders with the aim to assess different approaches and their results in order to assist project adaptation and to learn for future projects.

Similarly, Tress et al. [78] emphasized five steps in order to achieve successful integrative interdisciplinary and TD processes. They emphasized the following: (1) preparing a plan that identifies the crucial importance of integration, the necessary steps to realize integration of the expected integrative outputs, and a clear time schedule; (2) planning for smaller rather than larger projects; (3) allowing additional time to develop a common language, a common aim, and common outputs; (4) arranging regular meetings and events to help project participants get to know one another, trust each other, and develop a common understanding of the research process; (5) planning realistic outputs that can be delivered on time and avoid setting expectations too high in order to please funding agencies and stakeholders [78].

The normative model that is used for the evaluation in this study includes eight steps (

Table 3. A normative model for transdisciplinary research. In a true transdisciplinary research project, the process continues with dissemination of results, implementation, a learning evaluation, and potential scaling up of the implementation. However, these steps were not a part of this evaluation.

Step	Name	Activities
1	Problem identification	Identify the problems and challenges; brief assessment of the problem context, including as many actors with different views as convenient.
2	Learn about the social system	Map the actors; learn about their knowledge, ability, and willingness to act in relation to the problem, collaboration, and previous experiences of integrative research. Identify both gaps and the potential for an integrative research project.
3	Partnership building	Integration and partnership building among academic actors, non-academic actors, and integration of the two groups; learn collaboration, start small and develop the skills step by step.
4	Planning the research	Planning for the implementation of the research project, write an application, and apply for funding. Is there a need for a matching end-user project, can end-users be paid for their participation, or can end-users participate as part of their normal jobs?
5	Develop a common framework	Development of a common framework for the research, including both researchers and end-users. Learn collaboratively about and compare different perspectives among researchers and end-users. Agree on the roles of different researchers and end-users in an integrative research project.
6	Integrative research	Implementation and facilitation of the research project, including both researchers and end-users.
7	Learning through ongoing evaluation	Continuous evaluation; analytical seminars with reflection; discussions and adaptation, including both researchers and end-users. This should be led by an external researcher who is not directly involved in the research project, and all researchers and end-users need to participate.
8	Peer-review publishing and effects on the ground	Led by researchers with participation from end-users; research results are published in peer-reviewed scientific journals as quality assurance of the research results. End-user participation ensures that also the peer-reviewed publications are transdisciplinary in their character. This is also an important part of making the tacit new knowledge implicit, generally available, and applied.

). These eight steps can thus be used as a guide when TD research programs and projects are planned.

3. Results

3.1. Step 1—Problem Identification

Before the Mistra call for a program focused on transport infrastructure, several of the subsequent INCLUDE researchers had identified gaps in relation to the integration of ecological and cultural dimensions in transport infrastructure. There had been one larger and several smaller previous research and consulting projects that were relevant to this program. The initial problem identification in the INCLUDE project was based on several years of related applied research and end-user needs. This early and promising integration attracted interest from four government agencies which provided parts of the funding for the INCLUDE project. The researchers identified very good potential for collaboration.

3.2. Step 2—Learn about the Social System

All researchers viewed themselves as interested in interdisciplinary and TD research, including collaboration with end-users. Most researchers even claimed that they had experience with TD research projects. They were also interested in learning about other scientific fields and developing TD research processes. It was thus clear that more or less all researchers had a positive attitude towards TD research. However, very few were aware of the differences between multidisciplinary, participatory, interdisciplinary, and TD research (see Table 1 and Table 2 for definitions and components of these research types). The early plan was to involve end-users, and even encourage their commitment by providing a large part of the funding themselves. In addition, some of the researchers had a history of collaboration with some end-users. The end-users were also active partners in the early planning of the project. Besides identifying research topics, the early planning was also done in order to identify researchers and end-users that could be a part of the research process.

We identified three main types of researchers involved in the project: (1) the consultants, i.e., researchers who worked more as consultants than as independent researchers, thus performing the studies that the end-users asked them to perform; (2) the curious, i.e., researchers who were open to exploring and learning more about TD research; (3) the independent, i.e., researchers who expressed that there should not be any involvement or interference from end-users as part of the research process. The latter group stressed the need to learn about the topics for research by interviewing the end-users and claimed that researchers need to be fully free to choose their research questions.

Moreover, on the end-user side, we distinguished three main categories, even if they were not as clear as for the researchers. There were (1) the buyers (of solutions), i.e., end-users who did not see themselves as partners in the knowledge production process but rather as passive purchasers of new knowledge. This was the dominant group and it consisted of two subgroups: (a) end-users who wanted to buy consulting-style research to solve problems they had identified and (b) end-users who just saw challenges connected to the integration of ecological and cultural dimensions in transport infrastructure. There were also (2) the open-minded end-users, i.e., a small group willing to participate in and learn about TD research and (3) the curious end-users—a small group that believed that research should be curiosity-driven and independent.

As part of this evaluation, we learned that the end-user representatives that participated in the INCLUDE project were more or less only environmental staff. However, the operational end-users, i.e., project managers for transport infrastructure projects, who are responsible for the final decisions related to ecological and cultural dimensions in transport infrastructure projects, were not included. Thus, there was an extra layer in the social system of the project between the research and reality. Their direct feedback on problems, usefulness, and the situation for a project manager would have been very useful for the research. Staff with environmental tasks naturally have the main ambition of making operations more environmentally friendly, while project managers have the main ambition of building transport infrastructures. There was no structured effort to learn more about the social system (

Table 4. Organization scheme of TransportMistra research program and the INCLUDE project. Information that was not available or irrelevant to this study has been marked NA (not available/applicable). The following abbreviations are used in the table: National Swedish Railroad Administration (NSRA), Swedish Road Administration (SRA), National Heritage Board (NHB), Swedish Environmental Protection Agency (SEPA), Swedish Biodiversity Centre (CBM).

Level in the Social System		Actors at Different Levels in Social System of the TransportMistra Research Program and the INCLUDE Project and Its Tasks A-E
1	End-users and external donors	NSRA		SRA			NHB	SEPA
2	Donor	Mistra
	Review	External reviewers
3	TransportMistra	TransportMistra program’s steering committee
	Administration and control	Trivector Traffic AB
4	Project	INCLUDE					IMPACT	INFORM
	Reference	International reference group					NA	NA
	Reference	National reference group					NA	NA
	Administration and control	CBM					NA	NA
	Management	INCLUDE management group					NA	NA
5	Project tasks	A	B	C	D	E	NA	NA
6	Individual researchers and PhD students	NA	NA	NA	NA	NA	NA	NA

) in the project including different levels from end-users and donors to individual research tasks and the researchers (see Table 4). After the successful problem identification and interest from four government agencies, there were limited attempts to build a better understanding of the social system related to the identified research problem. A major problem was that very little, if anything, was known about the researchers’ and end-users’ experience and capacity to perform TD research.

3.3. Step 3—Partnership Building

The project starting point was promising, with already existing relations among most of the researchers and with several of the end-users. However, in contrast with the normative model, this dimension of the project’s social system was not considered important by the INCLUDE project management and most of the researchers. It was considered something that should happen automatically when researchers and end-users meet during the project. At project initiation time, when partnership building is advised to take place according to the normative model for TD research, there were no clear partnership building efforts across all INCLUDE research tasks. The limited effort for partnership building among researchers and end-users was worsened by the traditional funding scheme, which was without a preparatory phase (Steps 1–3). During the interviews, Mistra staff claimed that this had been identified as a problem and that adequate ideas for new funding schemes were being elaborated.

The limited understanding of the need for partnership development resulted in only some vague efforts to build better relations and improved collaboration between researchers, among end-users, and among researchers and end-users during the research project. Compared to the normative model for TD research, these efforts occurred too late in the project and were too limited. TransportMistra and INCLUDE workshops were set up as small or large meetings with invited external researchers, reference groups, end-users, and INCLUDE researchers. Several informants said that the setting was more familiar to the researchers and that it might not have been optimal for integration with the end-users. There were in addition some special events arranged, with the main aim of learning from end-users. The largest effort, the main event in this field, planned to reach a larger and representative crowd, was cancelled due to a lack of interest among most of the researchers.

3.4. Step 4—Planning the Research

The planning of the research project was mainly achieved by development of the applications for funding to Mistra’s Sustainable Mobility call. This included both the first application from the INCLUDE team and the merging of the INCLUDE, IMPACT, and INFORM applications, as requested by Mistra and the four government agencies involved. Both the first application to Mistra and the second step with the merging of the three applications were developed by joint efforts among the researchers. During the early application pre-planning process in 2003/2004, there was no process to develop a common understanding of the problem or research issues together with participants from different research disciplines and end-users. Instead, the researchers that had a history of cooperation with the National Swedish Road Administration (NSRA) and the Swedish Rail Administration (SRA) used their prior understanding of the situation to identify the problems that needed to be solved. This resulted in a limited assessment of the problem and the use of a relatively narrow perspective. The successful problem identification and good knowledge about and contact with the end-user organizations formed the basis of a successful application.

There was no consideration of the possibility to develop a matching end-user project to allow the end-users to participate fully in the research project and to facilitate a learning process among the involved government agencies. No plan for a TD research process was developed. Instead, the plan was like a linear Gantt chart plan for a traditional disciplinary project, a list of deliveries and names of researchers responsible for the delivery.

3.5. Step 5—Development of a Common Platform

This step in the normative model comes after the writing of a successful application. In TD, a common research framework means that researchers from different disciplines and end-users should bring building blocks from their respective fields of expertise to build a framework that fits with the identified problem and is useful for the development of solutions or improvements. However, neither the project management nor non-academic stakeholders saw the need for the development of a common framework for the research.

The development of the INCLUDE project was based on an agreement about the field of research but lacked agreement among the researchers and end-users when it came to specific research questions within individual project tasks A, B, C, D, and E (Table 4), methodology, and how to work together. Since no facilitated integration took place at this point, it was not possible to develop a common framework that all project stakeholders accepted and agreed on. One problem here was the funding scheme. Even though Mistra’s aims are consistent with the ideas of TD research, which is complex and time-consuming, the INCLUDE project was short-term, with one or two evaluations. Eventually, the late merger with two other TransportMistra projects, IMPACT and INFORM (Table 4), meant that two new groups of players were introduced, and additional integration would have been required. No efforts to integrate the three applications and develop a common framework for all of them were observed. Instead, the three applications in reality led to three separate research projects with almost no interaction between them.

3.6. Step 6—Integrative Research

The IMPACT, INFORM, and INCLUDE projects formed the TransportMistra research program, which resulted in a very complex structure and organization (see Table 4). Organizational complexity is not optimal for a TD research process, which is quite complex in itself [78]. The very complex organization was not developed with the aim of high integration among researchers from different fields, neither among the TransportMistra projects nor among practitioners from different government agencies and among all TransportMistra stakeholders, both academic and non-academic. Thus, administrative tasks dominated integrative research tasks. Furthermore, at the TransportMistra program level, there were very few, if any, efforts made to improve the collaboration between the three projects. For instance, four workshops designed top-down were arranged without any special activities to facilitate TD research. At this level, the TransportMistra research program steering committee and representatives from the end-user organizations acted independently, as in a consulting or disciplinary research project, with limited efforts to improve TD collaboration.

At initiation, the TransportMistra research program was heavily delayed by the slow process to sign contracts with all involved research organizations of the individual researchers. The contracts were not signed until August 2006, which meant an eight-month delay. Already in the winter–spring of 2008 (January 31 to April 1), i.e., after only around 18 months with accessible funding, the research program was evaluated by Mistra. Even if the evaluators engaged were looking into both already existing and planned project outcomes, they found that the expectations of Mistra were not met.

During this phase, the poor preparations for a TD research process according to the normative model became even more evident. Cooperation among researchers and end-users at the levels of the TransportMistra research program and the INCLUDE project did not work well. Instead, many of the researchers mentioned that they felt that most meetings and correspondence dealt with administrative work instead of research. The perceived focus on and failure to find easy solutions to administrative issues resulted in a command and control kind of management that requested or demanded different kinds of information and reports with little or no discussion to support the research process. This took away the focus from the research and resulted in a poorly integrated research environment in the INCLUDE project.

The poor integration among and between the academic and non-academic stakeholders provided an unfavorable starting point for the TD research. The researchers behaved as previously, by taking their share of the funding and going home to conduct the research they were interested in. End-users gave very differing and confusing signals to the researchers. The collaboration with end-users and interactions with the reference group was neither a well-planned nor a facilitated collaborative learning process. Instead, the INCLUDE project management team expected that this would happen on its own, supported by the INCLUDE workshops. The development of a TD research process was hampered by poor communication with and mainly indirect input from end-users to the researchers via the INCLUDE project’s management team. As a solution, the researchers used old contacts, or developed new contacts with a limited group of end-users, in order to be able to perform the research. This resulted in a very diverse and thematic research process with research that can be considered multidisciplinary and participatory (Table 1).

Among these many sub-processes, at the lowest level, we identified some initiated TD research processes. The integration that occurred was mainly between researchers but with some influence also from end-users. Researchers at this level, i.e., the level of the INCLUDE project’s individual research project tasks A–E (bottom level in the social system; Table 4), felt that they could not influence the two higher levels, the TransportMistra research program and the INCLUDE project management team (see Table 4). However, within the INCLUDE project’s tasks A-E, the researchers were able to influence their own research process. This, together with some of the researchers gradually learning more about TD research and being interested in applying their new knowledge, were the main factors supporting the initiation of an integrative collaboration mainly between researchers, first within individual tasks and then, to some extent, among them. This was the result of bottom-up processes initiated by the researchers themselves. This led to a process among the INCLUDE researchers to improve collaboration and joint learning and aimed to develop a common research framework. The researchers held a series of workshops and meetings with the main focus on developing a common research framework. However, this process started too late, compared with the normative model used for this study. First, the researchers had to learn; when they noticed that the collaboration did not work well, they started to act. This was towards the latter part of the project, with the aim of making the second phase more successful in terms of integrative research and end-user participation. Despite being late, some of the researchers at the project level mentioned that this work by the research group had facilitated trust building and interest within the group. They thus saw the INCLUDE project as the start of a long-term collaboration among them.

The four government agencies contributed strongly to the failure of the research program by having and expressing inconsistent opinions to the INCLUDE researchers. TD research requires not only participation in meetings; it requires co-development and co-production with both academic and non-academic stakeholders. To align with the TD normative model, there would have been a need for a matching end-user project that initially could have secured participation in the project, development of collaboration, co-development, and co-production. The research project paid the researchers for their engagement while end-users, staff from the government agencies, had to squeeze their participation into an already full calendar.

When the end of the TransportMistra research program’s first planned 3-year period came closer, several negative evaluations by funders had reached the researchers, at the same time as some of them had actually started to learn more about TD research. As a result, some initial TD research processes were initiated. The main aim was to get the project on track for its second phase. Researchers described long and often hard discussions among colleagues from other disciplines during meetings and workshops that improved the understanding of one another’s scientific fields, thus paving the road for TD research. This process was perceived as very rewarding and interesting. Several very tense situations were described by the interviewees, in which different researchers and groups had different opinions and did not agree. Nevertheless, around half of the researchers described this process as quite successful, but they argued that more time and effort had been needed; the process started too late to reach a common framework for the research. For some of the others, the process was negative as they felt that their scientific field and contributions were not respected or appreciated, which led to conflict.

The INCLUDE project management team, which had the only official contact with end-users, believed that they understood and could interpret the users’ needs. When communicated to the INCLUDE researchers, the researchers did not fully understand whether the message was derived directly from the end-users and questioned the extent to which end-user views were communicated well. Most researchers said that they had preferred the direct involvement of the end-users in this process, thus allowing formal documentation of end-user needs, using quantitative and qualitative research methods that the management level was not familiar with. When, on a few occasions, the researchers met the end-users, it was clear that the researchers and end-users were not able to communicate efficiently. The researchers did not fully understand what the end-users described as their problems. Here, it also became clear that the lack of a matching, sufficiently funded end-user project with the aim to allow a learning process also among the end-users and to enable them to participate fully in the research process made the knowledge production process very difficult. It was also unclear and confusing for the researchers to hear different voices, even originating from the same end-user organization, instead of a unified voice. This resulted in the researchers using old contacts and personal contacts from projects, workshops, and meetings, instead of developing a common learning process.

An interesting observation is that, because the small seed of integrative research that had started to grow gradually among the researchers as the project time passed on the lowest level of individual project tasks A-E was unnoticed, or seen as a problem at the INCLUDE management level, it was not visible at all for MISTRA’s project evaluators. Even if it had been noticed, the INCLUDE project would likely have received a poor evaluation due to the inefficient scientific leadership as described by the MISTRA evaluators [52] and the fact that researchers and management were not collaborating as a team at the INCLUDE project level.

Another unclear issue was the role of the reference group for the INCLUDE project. A national reference group was established initially. Some funding was used to pay for accommodation and travel expenses when the reference group participated in INCLUDE project activities. However, the existence of the national reference group was not communicated to the researchers. During the first of two planned three-year program periods, when the INCLUDE management was changed, information about the reference group was lost and the collaboration with them decreased just to invitations to participate in workshops, i.e., they lost their status as reference group and were treated as stakeholders only. The efforts to establish an international reference group were even less successful. There were some invited international experts present at several workshops but, as it seems, the offer to cover expenses for travel and accommodation was insufficient to attract broader interest from abroad. Moreover, here, communication with the researchers was not good and only a few of them were aware of and involved in the development of the international reference group.

3.7. Step 7—Learning through Ongoing Evaluation

There was limited effort for constructive evaluations, reflections, and adaptations within the INCLUDE project. What could be perceived as an evaluation were the lists of planned outputs that were strictly followed up and where deliveries were accepted or not. There was no evaluation or reflections made with the aim to improve the ongoing research process and research environment at the INCLUDE project level.

Based on evaluations of the TransportMistra research program in 2008 [52,53] and evaluations of reports from the research program, Mistra concluded that the academic quality of the TransportMistra research program was insufficient. This was based mainly on the poor integration of researchers and end-users and the fact that the researchers were not able to show that their work would contribute to future practical applications to improve sustainability in the transport sector. Most researchers also agreed that no or few only directly applicable results or socially robust knowledge [28] came out of the INCLUDE project. The perception among the researchers was still that a lot of good quality research was produced that would be of key importance for the inclusion of natural and cultural landscape values when it eventually was published and depending on whether it was picked up by the end-users. It is clear that the researchers conducted and published a lot of research. The failure was not that the researchers were not carrying out high quality research. Instead, the failure was that too many of the researchers and the project management did little integrative research but instead conducted the kind of research that they had always done and thus did not meet the requirements of a TD research project (Table 1). An implication of the consultancy-oriented leadership of the INCLUDE project was that some of the TD research done in the projects was not sanctioned and supported by the INCLUDE project management team. This meant that potential for future applications was not communicated to the evaluators.

3.8. Step 8—Peer-Review Publishing and Effects on the Ground

Two research activities that have a much longer time span than a project itself can be identified. The first is peer-review publication. This process starts during the project, but peer-review publishing often takes time and might not be completed until after the integrative research project ends. In a TD research project aiming to contribute to the handling of or solutions to complex sustainability challenges, the synthesis of all building blocks of knowledge often takes even longer time. First, bits and pieces are researched and then the overall synthesis that matches the real world can be developed. The researchers focused less on research for real world problem-solving and rather on performing and publishing peer-review research in general.

The second step is the implementation of the research results. This starts as testing during the project and ends with subsequent up-scaling of implementation after the research project ends. This is in stark contrast to 3-year project cycles, often with half-time evaluations.

4. Discussion

4.1. Compliance with the Normative Model

An overall comparison of the INCLUDE research project with the normative model (

Table 5. Comparison between the normative model for transdisciplinary knowledge production (Table 3) and the performance of the INCLUDE project.

Step	Name	Activities/Result
1	Problem identification	- Initial assessment to assist the application - Assessment focused on the needs of end-users - Potential for a transdisciplinary knowledge production process was identified
2	Learn about the social system	- Identification of official actors - No or few other stakeholders - No learning about the actors’ knowledge, attitudes, and capacity for change
3	Partnership building	- No or little only partnership building
4	Planning the research	- Poor understanding of and no learning about integrative research - Good collaboration with end-users - No matching end-user project - No agreement among actors on gaps, questions, or problems - Plan as list of deliverables - No plan for a transdisciplinary research process - An application for funding was written
5	Development of a common framework	- No common research framework developed - No common framework developed with end-users - Some integration among researchers - Some integration with end-users
6	Integrative research	- Command and control style leadership - No facilitation of a collaborative learning process - Perceived focus on administration among researchers - Failed to develop of creative research and transdisciplinary environment
7	Learning through ongoing evaluation	- No integrated evaluation process - External reviews and criticism/conflict
8	Peer-review publishing and effects on the ground	- Large number of peer-review articles published - No holistic publication with synthesis of relevant project results as project was terminated early

) shows that, within the INCLUDE project as a whole, there was no widespread understanding of what a TD research process implies. The initial assessment step in the normative model by the researchers to assist the INCLUDE application was focused on the needs of the end-users and the need to acquire research funding to perform research. A potential for collaboration was identified, but this was more understood by project management as a need for discussion, some interaction, and information exchange to assist the researchers and thus not a part of a true TD research process. Since the term TD research was not fully understood or accepted, neither by end-users nor by any researchers, the potential for collaboration could not be translated into a detailed plan for a TD research process. Gaps that needed to be bridged for a successful collaborative process were not identified and documented by the researchers. The work with the prerequisites for the project was successful in identifying actors. The main end-users were included in the INCLUDE project, and a national reference group with participants from a very representative and influential group of actors was created. However, in relation to both the normative model used for this evaluation and Mistra’s evaluations of the INCLUDE project, integration and partnership building among participants in the project was limited. This was due to the limited understanding in the INCLUDE project of TD research and process, including the need for planning and facilitating co-operation between researchers and end-users, and thus without resources spent on it. In addition, there was no general initial project-level understanding of integrative research as something that the participants need to learn and do step by step. There was no matching end-user project that could act as a unified counterpart to the INCLUDE project. Instead, there was a double-command system consisting of the TransportMistra board and independent representatives from the end-user organizations. This made integration between researchers and end-users much more difficult. In addition, a common framework both among the researchers and with the end-users was never developed, and there was no matching end-user project.

Since it is not an easy process for the involved academic and non-academic actors to develop a common framework for the research, we propose a two-step model. First, a common framework could be created among the researchers and in parallel within the end-user group. When this first step has been successful, the two frameworks can be integrated. TD research is different from research on a common theme. This means that researchers likely will have to work with something outside their previous expertise. Of course, this does not exclude that some elements of the TD research process can be disciplinary. It is also clear that integrative research does not happen by accident. It requires careful facilitation of researchers and end-users [41].

When research is TD, it moves closer to non-academic knowledge production and consulting, even if peer-review publishing still is important. Many researchers feel threatened when their work comes close to the border between applied research and consulting, e.g., [41]. Properties such as demand-driven, professional practice, problem-solving, dependence, and acceptance/agreement on problems can be used to describe consultancy, and such expressions are also applicable for applied integrative research. The border between TD research and consulting thus needs to be explored. Results of consulting projects are rarely published because the customer does not pay for them. Instead, the results are owned by the customer and therefore the consultant often cannot use the results after their consulting work. Consulting that produces new knowledge, but which is not spread or published, is equivalent to the concept of tacit knowledge [81]. In contrast, research is subject to peer-reviewed publication, i.e., creating explicit knowledge [81]. However, this is not a unique property since results from consultancy work performed by researchers are also sometimes published in scientific journals. A key property of consultancy work is that it is the buyer who defines the problem and sometimes even the solution. By contrast, in transdisciplinary research, the researchers and practitioners define the problems as a collaborative learning process [22]. The researchers and end-users handle the research process and maintain the right “to bite the hands that feed them”.

4.2. Development through Evaluation—Pros and Cons of Our Approach

No complex activity or business can develop towards success without continuous reviews, evaluations, critical reflections, and the resulting learning [9,37,40,82]. However, evaluation through audits involves many challenges. It is not easy for humans to analyze and criticize themselves [83]. Some of the informants mentioned that they were aware of several other program and project evaluations and reports that were too honest to be accepted by researchers or peers. Others might be using self-evaluations because they are scared of external reviewers coming in, collecting data, and presenting a disadvantageous picture of their activities. Nevertheless, the involved researchers knew the INCLUDE project better than anyone else. It is not always easy for an external reviewer to get the “full picture” through interviews and other kinds of data collection. On the other hand, how can we be sure that involved researchers will present also failures? There is definitely a risk that the involved academic and non-academic participants will present everything as being “good”. Nevertheless, we are confident that the “development through evaluation” principle described by Tranquist [84] is a valuable contribution to learning TD research. This matches well Roux et al.’s [57] statement about the “… importance of early investment in processes, behaviours and relationships that foster social learning and the co-production of the knowledge and understanding that are required to ensure relevance…”. Finally, we want to stress the difference on one hand between auditing with focus on fostering social learning and knowledge production being the focus in this study and, on the other hand, the so-called audit culture [85] that focuses on quantitative measures of research impact. For example, use of bibliometrics is likely to miss or de-select against research innovation and creativity [86,87].

4.3. Solving Problems or Removing Symptoms?

To solve large and complex sustainability problems, such as the inclusion of ecological and cultural values in transport infrastructure, a wealth of different approaches and actors are needed. This requires a holistic view of the landscape, including both the ecological and the social system, e.g., [50,71]. The main issue is to understand approaches to solve different kinds of problems among multiple actors. Some solutions can be produced by disciplinary research, some require an interdisciplinary approach by a group of researchers from different disciplines, and TD research requires close collaboration among academic and non-academic actors, including decision-makers, to solve or handle real world sustainability challenges. TD could be described as a partnership of actors. A partnership of actors can be perceived in many different ways by the involved parties [88]. Nevertheless, to make the process successful and to make participation meaningful for all involved, there is a need to reach one of the higher levels of collaboration, as described by Arnstein’s [89] ladder of participation.

One problem that the INCLUDE project faced was the wide gradient among research traditions from conducting independent disciplinary research to being involved in TD research and ultimately also consulting without peer-review. Many of the researchers were careful to not become consultants, while some others thought that this was not a problem. TD research is in some ways superficially closer to consulting than disciplinary academic research. The main distinction is, however, that in consulting, the end-users know the problems well and define clearly what should be solved and, in some cases, even how. Another distinction is the need for peer-review publication in academic research, which is time-consuming, often appearing after the actual project is officially completed and usually not of interest to the typical consultant customer. This is in line with the organization of different kinds of research made by Svensson et al. [77]. They call consulting “research for”, which means that researchers assist end-users in their knowledge development process according to the conditions of the end-users. Research as consulting, as it could be termed, has some problems in the context of TD research. This type of research focuses on short-term knowledge needs. End-user organizations provide the questions and a framework for the work. If research is this strictly controlled and steered, critical reflections will most probably not take place, and theoretical development and contributions to the general scientific knowledge base will not be the end result.

In contrast, “research with” means that researchers and end-users can combine theoretical development, critical reflections, and high usefulness toward problem-solving. This is in line with the terms of TD research and knowledge production. Researchers and end-users thus learn and develop together. The aim is to integrate the two groups and develop equal relations and balance but also to emphasize the different roles of the researchers and the end-users in the knowledge producing process [77].

4.4. Challenges of Transdisciplinary Research

Many researchers and end-users are aware of and can appreciate the need for TD research, but few understand what it really means, as is illustrated by the project assessed in this study. The growing interest in TD concerning sustainability suffers from the lack of a common glossary, no focused communication platform, and no commonly shared research framework [30]. Tress et al. [78] claimed that TD research might work in small groups but is hard in large projects. For large and complex projects, we see the need to understand knowledge production as a process that needs to be actively facilitated as an iterated process including not only TD research but also disciplinary work [41]. With facilitation, we mean that there is a need to assist communication, to make sure everyone is listened to and taken into account and support the progress of the process. This requires either seniority in terms of previous, long-term practical experience in research program management or an independent, well suited facilitator.

For projects aiming at TD research, we strongly emphasize the need for comprehensive training at the beginning of the project implementation. This means that the research council Mistra, the project management team/facilitator, the researchers, the end-users, and an invited expert team learn about integrative research, starting from the participants’ own previous experiences [21]. Angelstam et al. [5] provides an example of this approach, which was based on 14 TD experiences. While all participants in any project have some past experience, we acknowledge that there seem to be very few experts in the field of TD research at most universities [41]. However, the interest in transitioning into more transdisciplinary research activities is increasing, particularly in the sustainability sphere [90]. Nevertheless, gaps need to be identified and bridged by a carefully facilitated collaborative learning process that would in addition be the beginning of the integration process. We feel that the largest challenges for the INCLUDE project to implement the idea of TD research were (1) the limited understanding and appreciation of TD research among the INCLUDE management team, most researchers and end-users, and (2) in society and at most of the participating universities’ different departments, as well as (3) difficulties in engaging sufficient time for TD research due to short-term funding. We view this as a general mismatch between the paradigms of traditional disciplinary research and TD research. However, the Mistra research council had indeed partly adapted to this need by issuing projects lasting for 3 + 3 years.

TD research has similarities with concepts like experiential and problem-based learning (PBL), which is an approach to learning in which students collaboratively solve problems and reflect on their experiences and search for new knowledge [1,91,92]. Some characteristics of PBL are that learning is driven by challenging, open-ended problems where students are encouraged to work in small collaborative groups and with teachers as supporting facilitators of the learning process. Thus, students are encouraged to be responsible for and organize the learning process with support from the facilitator. Given the multi-sector and multi-scale endeavor of SD based on natural resources, seminars and practical studies could be based on learning from experiences from and in concrete landscapes (social–ecological systems) as laboratories [40,93,94]. To invite actors and stakeholders to interact and discuss with students is crucial. The same applies to effective international development and educational projects concerning sustainability [95,96].

Hence, to achieve the ambitions of TD research, a matching end-user project for the integration of academic and non-academic actors is required [97]. Integrating project funding from different sources supplying both researchers and stakeholders is a necessary strategy to fulfil the TD research agenda. However, formal and informal disciplinary and administrative barriers can limit team building in spite of self-reflection and experience. In a small project, researchers and end-users can be integrated simultaneously. However, in a complex program with many actors, like TransportMistra and the INCLUDE project, there is a need for a step-wise approach with an overarching strategy and several tactical sub-processes. It can start with the large process, leading to several sub-processes, but should end with the overarching process, collecting and synthesizing a result that matches the research challenge at hand. The actual duration of a TD research and learning process is probably more than twice (5–6 y and with a preparatory phase) that of a traditionally funded research project (2–3 y or 3 + 3 y, without preparatory phase). Since the foundation for a successful TD research program is laid before the funding is secured in terms of the integration and partnership building, it could be an interesting approach to arrange the funding over several phases, resulting in a bell-shaped budget over time [98] (Figure 1). To carry out TD research, the differences from disciplinary research must be recognized by universities, researchers, end-users, and funding agencies, e.g., [21]. One key obstacle is the contradiction within the emergence of new public management in the academic world, whereby administrators can decide on the distribution of funds and thus rule how funding is spent, above the head of a principal investigator, who thus becomes deprived of the leadership capacity and power to steer e.g., [99].

Funding for an integrative research project most often follows the same model as a traditional disciplinary research project despite its different dynamics (Figure 1). Mistra has identified the problem with TD research processes being very different from traditional disciplinary research and thus might need a new adapted approach to funding. The informants from Mistra informed us about the need for a new system with planning grants. Similarly, the funding mechanism Biodiversa allows projects to begin with stakeholder engagement processes [21]. To do this, there is maybe even a need for research donors to include some of their own staff in the preparatory phases of TD research programs to ensure that they are on track when the research funding begins.

The need for TD research has been identified by many actors [15,16,19,20,100]. However, PhD education in Europe has still not yet adapted to this need [20]. The absolute majority of researchers are thus educated for disciplinary research only. The differences from traditional disciplinary research and the lack of integrative researchers also mean that competent evaluators often are hard to find. End-users and funding agencies should, to be sure, clearly indicate the nature of the research project and provide guidelines for evaluators. It was indicated by the Mistra management that the only existing TD PhD education they were aware of in Sweden was done as a part of a few recent successful Mistra research programs.

The style of leadership needed for integrative research is characterized by openness, transparency, and a strong emphasis on collaborative learning, team-building, and networking [63,77,78]. However, in addition, there is also a need for more formal steering and support from university management to ensure that the participants do not take their part of the funding without fulfilling the TD research agenda. Such a requirement is of particular importance in sustainability research, where transdisciplinarity is essential [101]. Tools like the open space method, which supports meetings where all participants are given the same meeting space, could be important [102]. One option might also be to meet the end-users in their home territory, i.e., consistent with a TD case study approach, being either thematic or geographical. It can also be important to develop or benefit from an existing neutral arena for knowledge production collaborative learning and to build respect and understanding among the stakeholders, e.g., [32,40].

4.5. Bridging Barriers and Platforms for TD Research

A key recurrent challenge in TD research is the limited understanding and prejudice among both researchers and practitioners about what TD research is and how it should be performed [103]. A structured TD social learning process occurs only when researchers and practitioners have built trust and can co-create a process of interaction between scientific knowledge and real world experiences [104]. This observation is in line with [105], who emphasized knowledge integration, quality criteria for assessment of the research process, and consideration of project partners’ different norms and values for a successful TD research process.

For a successful TD research process, there is therefore a need for skilled and experienced leadership and a collaborative approach to leadership [41], which was confirmed by [106]. Additionally, in a TD research process, researchers often need to leave the comfort zone of the academic ivory tower [107], and practitioners need to leave their traditional role as passive customers of research to become actively involved [108]. To steer a TD research project towards its goals, there is also a need for continuous evaluation and adaptation [109]. Angelstam et al. [40] analyzed 67 long-term socio-ecological research (LTSER) platforms aimed at transdisciplinarity and showed that this network is well designed by representing biogeographic regions, land use history, and democratic governance. This type of network for landscape approach initiatives is well suited to TD research aiming to solve real world sustainability challenges. For TD research to result in real world impact, [110] identified three factors, viz. (1) impact generation mechanisms, promoting systems solutions and transformation knowledge for more informed and equitable decision-making, (2) focus on social learning for collective action, and (3) development of reflective leadership. Finally, we encourage other researchers to conduct meta-analyses of multiple TD research assessments.

5. Conclusions

Realizing the visions of research councils and other donors to support TD research processes requires knowledge and understanding of these processes. Introductions to transdisciplinarity, integration, and facilitation are important. Firstly, all participants, both end-users and researchers, need a comprehensive introduction to knowledge production by TD research. Secondly, a key task is integration; all participants must feel needed and have the same opportunities to influence the research process, while, at the same time, differences between the roles of researchers and end-users must be understood. End-users must re-evaluate their role as customers and receivers of the research results since much more active participation is a requirement for TD research and knowledge production. It is also important that the right individuals from the end-user organizations are involved, especially the busy operational project managers from different kinds of transport infrastructure projects. All this will not happen by itself in a short-term project. Thirdly, facilitation and relevant steering of the process is crucial to create a productive TD research environment. We advocate that donors and funding agencies with an interest in TD research should advance their own knowledge and skills and request a plan for the TD research process as a part of the application for funding. Additionally, integration of research funding and funding needed for the end-users to participate in the process is crucial. Finally, it needs to be acknowledged that TD research is more costly and time-consuming than disciplinary research.