Setting the Direction for a Sustainable Future? A Critical Review of University-Enterprise Partnership Evaluation ^†

Abstract

University–enterprise partnerships foster great opportunities for sustainable development by enabling the translation of research into viable commercial solutions. However, how do we measure the relative “success” of such a partnership? This paper critically considers traditional evaluation methodologies to identify relative merits and shortfalls. Common themes include measuring against predominantly economic criteria and an over-reliance on global metrics at the expense of capturing regional impact. Potential consequences include undervaluation of local impact, disproportionate prioritization of economic impact, and the subsequent sidelining of environmental benefits. A place-based evaluative approach is proposed for future research to alleviate shortfalls.

1. Introduction

This paper recognises the importance of university-business collaboration (UBC) in fostering sustainable outcomes and specifically aims to shed light on a crucial yet frequently overlooked element of UBC: evaluation. Evaluation, when completed successfully, provides an opportunity for growth, reflection and strategic planning [1]. It captures issues and achievements that may otherwise have been overlooked and allows for proactive action to maximise the potential for sustainable development.

Looking to university-business collaboration (UBC) for sustainability ideas and projects is by no means a new concept. Its utility for sustainable development has been widely cited throughout the literature [2,3,4]. This has been echoed within the public sector, with the formation of the European University-Business Forum [5] and special mention within the United Nations Sustainable Development Goals [6]. Despite this, relatively little attention has been given to critical evaluation of UBC. Although the opportunities are significant for accelerating sustainable entrepreneurship, it is important to consider structural, cultural and social disparities which may act as barriers to enabling successful innovation.

One common critique of studies of social systems is that the target systems are often analysed in isolation of their external environment; as Cresswell [7] puts it, “the fish don’t talk about the water (p. 263)”. Through greater consideration of the place in which social phenomena are situated, there is reason to suggest that social elements shaped by place may be highlighted and shown in a new perspective [8]. In addition, such thinking may offer unique opportunities for questions of sustainability: through analysing how place influences social interactions, the manners in which social interactions shape place are equally highlighted [9].

Given the centrality of place in discussions of sustainability, this paper uses the lens of place to critically explore the ways we evaluate UBC for sustainability. Through a systematic review of the literature, the paper identifies three different approaches to evaluation across the literature, and points to the shortcomings of these approaches to evaluation. In order to overcome these challenges, the paper recommends a place-centred approach to evaluation which captures the often-overlooked elements of UBC: local impact, existing power relations and the less tangible social relations frequently neglected in reductionist evaluative approaches.

2. Materials and Methods

To investigate current practice relating to UBC evaluative research, a comprehensive systematic literature review was undertaken. This was the chosen methodology due to its demonstrable ability to reduce the risk of bias within the literary study, providing a robust and replicable means of study and representing a thorough coverage of the chosen area [10]. In particular, systematic reviews have been shown to add great value specifically to the field of university-business collaboration however significant efforts are required to ensure that the methodology is fit for purpose and used in an effective manner [11]. Following this guidance, clear research questions were identified:

• How are university-business collaboration projects for sustainability currently being evaluated?
• How is place considered within university-business collaboration evaluation for sustainability?
• How does evaluation of university-business collaboration projects contribute to their sustainability aims?

A title/abstract keyword search was completed highlighting three terms used to describe university-business partnerships: “co-creation”, “co-production” and “third-mission”. As university-business collaborative projects span a diverse range of topics, words stemming from “sustainab” (e.g., sustainability/sustainable/sustainably) were included to narrow the search to the most relevant projects. This resulted in the following keyword search:

TI/AB (“Co-Creation” or “Co-Production” or “Third-Mission” or “Third Mission” and “Sustainab*”)

This search was deployed for articles in the English language within Web of Science, SCOPUS, and Academic Search Ultimate. Following deletion of duplicates, 324 papers remained. A screening process followed whereby abstracts were read and articles which were not relevant to the study were removed. This resulted in 101 papers which were fully read and analysed. Within the review, we also included articles recommended by peers alongside relevant citations within articles.

3. Results

3.1. Loss of Complexities of Place through Reductionism and Compartmentalisation

Compartmentalisation remains a dominant feature of sustainable UBC evaluation frameworks, with elements of a project commonly organised into discrete areas (e.g., “levels”, “areas”, “topics”) typically in order to create an overall score or strategy incorporating various components of a project [12,13]. Compartmentalisation aims to deliver an all-encompassing metric for sustainability in order to provide easily comparable “benchmarks” for projects. Such proposed metrics include the Ecological Footprint, Urban Environmental Carrying Capacity and Index of Sustainable Economic Welfare (ISEW) [12,14].

Compartmentalist approaches can often be advantageous, allowing for delegation of key areas to their respective authorities and, in theory, ensuring that every area of the project is covered in a relatively equal depth [13]. Amidst academic institutions, with the creation of disciplinary boundaries, compartmentalisation can appear a logical option to break down complex social and technical structures into more digestible components [15].

Conversely, a reductionist approach whereby categorisation forms a major basis of a framework can result in little more than reorganisation of pre-known information, adding little value to the thinking process. Reductionist frameworks have a tendency to result in problem-shifting and displacement which requires new modes of strategic thinking to overcome [16]. Reductionism can also lead to overly prescriptive frameworks limiting its potential to be used across sectors, particularly lending itself to application across technical fields where reductionist thinking is the most common [17,18]. A process which focuses thinking on individual components may confer a tendency to attempt to control for details in absence of greater perspective, risking that resultant decisions be made in absence of wider context [17]. Crucially, compartmentalisation unavoidably confers weighting to various elements of a project, lending itself to the (subconscious or otherwise) prioritisation of various elements of a project according to those responsible for managing the evaluative process.

3.2. Power and Place: Evaluation as a Neoliberal Tool

The subjective nature of relative “success” and “sustainability” has been discussed in detail [19]. Given their subjective nature, success and sustainability can be considered social constructs and are therefore entirely influenced by power relations within a social system [20]. Traditional frameworks of evaluation are generally mandated by those in positions of authority [21]. Consequently, evaluation is a means of assessing the degree to which the subject of said evaluation aligns with the ideals of those mandating the evaluation [22]. Through this means, metrics risk being (constantly re-) aligned with the priorities of those in positions of power with the terms “constraint”, “control” and “compliance” frequently associated [23].

Alongside being a product of power relations, evaluation can also itself act as a means of perpetuating and/or reshaping power relations within UBCs [24]. The inert nature of metrics has also been questioned, with metrics themselves described as having their own agency through the influences they indirectly exert [21]. Another consequence of authoritative evaluation methods is a focus on short-term, measurable results. Neoliberal thinking relies on rational decision making whereby actors require impartial and comparable data, specifically for the purpose of constructing (quasi-)markets and spheres of competition as the supposedly “optimal” governance structures for the good or service in question [25]. Quantitative data has a tendency to be viewed as objective and relatively free from external influences. This, combined with the difficulty in accurately forecasting long-term sustainable effects of projects, makes them frequently preferred sources within modern evaluative frameworks [24,26]. Yet the very concentration on short-term metrics tends to entrench such short-term concerns as the primary drivers of decision-making.

3.3. Local vs. Global Context

Within UBC for sustainability, regional development is frequently a core parallel aim of a project [2]. Despite this, although the actions within a UBC programme are frequently reported with some degree of reference to local region, the impacts or outputs of these projects are usually reported on a universal level [27]. For example, Carragher et al. [28] report a study of an Irish community which participated in a co-creation activity incorporating place-focussed methods. Results were reported as reductions in ecological footprint, aligning with universal metrics, yet there is no measurement of local impact, despite anecdotal reports that a significant contributor to this reduction in ecological footprint was achieved on a local level through reductions in car usage and household waste. Given that the socio-economic effects of UBC are typically concentrated around their host localities, it can be expected that therefore at least some degree of environmental benefit would be expected to be felt locally [29,30].

This example is indicative of several studies, with the regional context being mentioned only in certain areas of the process (in a typically anecdotal manner) rather than followed through from start to finish [31]. Frequently, local environmental issues are seen as separate from global issues and consequently, local action (though recognised as important) is considered a separate battle from widespread issues such as climate change. In reality, the two are not mutually exclusive but complementary, with local agendas acting as a necessary step to achieving global aims [31]. Crucially, this reporting method has a tendency to undervalue the effects of UBC by undermining the importance of local sustainability.

Evidently, sustainable UBC projects have a higher purpose on the unforeseen journey to sustainability as hubs of innovation, research application and local action, yet traditional sustainability metrics fail to capture the significance of this [32,33,34]. Equally, university-industry research output metrics have been found to have prominent inconsistencies that do not accurately depict their value within the institutional landscape [35,36]. Without urgent reconsideration of metrics and evaluation format, sustainable UBC projects risk being significantly undervalued and reported unfavourably within the policy landscape.

4. Discussion

One potential consideration for fulfilling some of the shortfalls discussed is an evaluative method more grounded in place. This approach has been proposed on a small number of occasions in various case studies (see Bentley and Pugalis [37] Hart et al. [38] Tan et al. [39]). Although the mechanisms vary, these place-grounded approaches commonly follow a way of thinking which gives greater priority to local and regional actions on different scales and considers this ‘context’ of place as an active and dynamic variable within a UBC project as opposed to a passive site of activity. Here, a place is not simply considered a geographical location but instead a space entrenched in meaning, encompassing social relations, power balances and the people themselves who interact with and within this space. This approach offers a different perspective in the context of UBC evaluation for sustainability alongside various considerations, which will be discussed in due course.

4.1. Reframing Problems and Solutions

The rationale for exploring a place-centred approach more seriously arises directly from the characteristic nature of most university-business collaboration projects. UBC projects frequently target regional development, focusing efforts within the locality of the partnership in question [2]. These projects are often relatively small-scale initiatives working in partnership with small-to-medium enterprises (SMEs) [2]. It is unreasonable to expect that these regional, small-scale projects have a tangible global impact. Evidently, the greater value of UBC for sustainability lies elsewhere on a regional or local level [27].

A consideration of place lends itself to allowing outputs and processes to be investigated from a regional perspective [40]. This takes steps to avoid some of the aforementioned fixation on universal sustainability metrics. In addition, it is equally mistaken to consider that UBC projects for sustainability are not influenced by the place in which they are situated. Instead of following traditional evaluative approaches, whereby ‘context’ is a variable to be controlled to maximise comparability, consideration of place within evaluation shifts this perspective.

By considering the dynamic nature of a project’s local area, such as seasonal or economic fluctuations, the project can be better tailored for regional development, for example, rapid responses to the regionally-specific effects of the COVID-19 pandemic [41]. Boundary navigation, the interface whereby knowledge translates into action, must be considered a dynamic, continually changing process amidst the equally dynamic backdrop of place [42]. In addition, by avoiding the need for absolute comparability, which renders regional disparities a variable to be controlled, the regional elements of a project can be individually considered. Here, more realistic predictions can be made for transferring practices between regions, allowing for regional differences [43].

Taking this approach further, when considering the embeddedness of social relations within the place, it becomes impossible to think of a place without considering the power relations which shape it. Equally, the influence of place and the way in which its structures shape power relations become inescapable from consideration.

Although this is far from remedying the mechanisms by which evaluation is entrenched in power relations, an understanding of the power relations at play within a project makes some steps towards addressing any underlying power tensions. To make further progress, assumptions regarding the agents responsible for evaluation need to be challenged, potentially through the consideration of participatory evaluation methods.

4.2. Participatory Evaluation

As UBC projects aim for partnership and more equitable power distribution, evaluation thereof should not escape such scrutiny. Beyond reinforcing existing tropes and power relations, such top-down methods of evaluation pose an ever-greater threat to sustainable development. Top-down evaluative methodologies limit the potential for wider and more equitable knowledge sharing. Reed and Abernethy [44] identified the benefits of lateral knowledge sharing (within similar initiatives) alongside vertical knowledge exchange and its value within co-production efforts such as UBC. The co-production of metrics can result in not only a more balanced representation of a project but the opportunity for stakeholder perspectives to shed light on previously unseen elements of a project which can yield valuable learning opportunities for future projects [23]. There are some concerns that top-down evaluation also results in under-reporting of project failings, limiting future learning opportunities [27].

However, the dangers of shallow democratisation must be recognised. Although a popular topic within evaluation studies, some argue that participatory evaluation as a concept is overly ambiguous and insufficiently theorised [45]. The language surrounding participation, democratisation and fairness can also be deployed as a superficial buffer to avoid scrutiny whilst keeping neoliberal, top-down approaches firmly in place. For example, Turreira-García, et al. [46] reviewed the so-called participatory evaluation of environmental projects found that the majority of projects only involved local communities in data collection whilst the creation and design of projects continued to take place in an authoritative manner; this is an effective compromise for authorities as it reduces the costs involved in data collection without sacrificing any material power within the project. Voorberg, et al. [47] demonstrate that participation of citizens in the co-creation processes is often viewed purely as an objective to be met with relatively little thought given to the beneficial contributions that citizens can add to the process itself. Co-creation alone is not sufficient and must be incorporated alongside co-governance to ensure a true democratisation [48]. Participatory evaluation measurements can prove to be useful in ensuring that democratisation goes beyond well-worded intentions [49], but even within these frameworks the danger of superficiality remains and can in practice act as little more than a tick-box exercise for institutions [50].

Greater consideration of the local lends itself to participatory evaluation through the greater weighting of the importance of local and lived knowledge [8]. Involvement of local communities allows greater tailoring of projects to the individual needs of a locality, increasing value gain both for regional economic and sustainable development [1,51,52]. In addition, local knowledge and stakeholder input can aid boundary navigation and greater diffusion of knowledge between universities and local businesses, acting as a valuable intermediary with shared perspectives [53]. Co-production of indicators and metrics incorporating local social and spatial contexts can also act as a valuable learning tool for developing shared understandings and building collaborative strategy [23,54].

More generalised sustainable development strategies have been found to be ineffective and impractical when faced with a particularity of individual places; local actors have been shown to possess the necessary skill set to map local decision processes and identify otherwise unforeseen regional characters [39]. Place-minded evaluation not only prioritises local knowledge and stakeholder perspectives but gives a greater voice to place itself through valuing local input.

4.3. Navigating Boundaries

Boundary navigation is frequently cited when discussing UBC, particularly regarding the translation of knowledge into a workable action [42,55,56]. In addition, sustainability is noted as an interdisciplinary and indeed transdisciplinary issue [57,58,59]. The navigation of disciplinary and sectoral boundaries, alongside the facilitation of mutual knowledge and resource exchange between social worlds, is one of the cornerstone challenges we must confront to make significant strides towards a sustainable future.

Place offers a unique opportunity here to transcend disciplinary boundaries and the knowledge-action barriers. Hart et al. [38] found that working at a scale of a single region and local place provides more frequent learning opportunities; Chammas et al. [55] identified common interest between researchers and the community as a core value in successful co-creation, citing participatory mapping and place-based study as a key means to achieve this. Shrivastava and Kennelly [29] have emphasised the utility of regional approaches to evaluation in navigating boundaries, maintaining that place-based enterprises are more likely to engage in UBC to pursue environmental, social and economic outcomes when compared with conventional enterprises. Place-based opportunities have also been highlighted through the ability to combine researchers’ explicit knowledge with the tacit knowledge of local stakeholders [39,51,60]. It has been reported that many key performance indicators are not relevant to all stakeholders within UBCs [33]. Local indicators for project success could provide an opportunity for a common goal which has higher relevance for all parties involved [61].

To break down the disciplinary and sectoral boundaries which have been constructed over decades (if not centuries), transdisciplinarity needs to be introduced within traditional education methods. At present, the relative isolation of disciplines can make it difficult for students to gain an interdisciplinary skillset [18] with sustainability students reporting feelings of disconnect from the climate story with a lack of ownership [62]. Place-based teaching and learnings foster unique opportunities to diversify learning experiences across a range of disciplines with the place and its social relations shown to have inherently pedagogical qualities [63,64]. By combining industry and academia in the crafting of curricula, both tacit and codified knowledge can be utilised to maximise learning potential [65,66]. Living labs, whereby students work directly across academia and industry to understand and tackle situational problems across campuses and local regions, have shown to be highly effective, particularly in the context of sustainability education [67,68]. Living labs have additionally been shown to be effective in the evaluative process, allowing the wider involvement of stakeholders in both the proactive and reflective stages of evaluation through clear relation to local and regional contexts [69].

5. Conclusions

From this study it can be concluded that, considering the significance of UBC for sustainability, significantly greater attention should be given to critically re-evaluating current evaluation methods. Without considerable effort, UBC projects for sustainability risk being undervalued (through lack of consideration of local and regional impact), exploited for individual gain (through the top-down, power-entrenched nature of evaluative mechanisms) and over-generalised (through the extensive use of reductionist approaches to evaluation). This study finds one potential ameliorator: greater consideration of local and regional contexts across all elements of UBC projects for sustainability through a place-centred approach.

This paper is by no means a guide for evaluating UBC sustainability projects, nor does it pretend to have all the answers for crafting a place-focussed evaluative framework. Rather, it merely aims to emphasise the importance of the evaluation of UBCs within the wider sustainability debate and identify potential areas for improvement. Although focussing on place fosters opportunities for improved evaluative practice, this perspective should certainly not be viewed as a silver bullet; indeed, embracing the complexities and individualities of place can illustrate the infeasibility of such silver bullet thinking. Practical considerations of scalability, comparability, longevity, and accessibility must be confronted with regional considerations likely used as a complementary element to a framework rather than an all-encompassing ethos.

There is clearly great scope for future research within this area. First, consideration of how regional and local focus could manifest in a practical evaluation methodology. Secondly, there is a demand for consideration of how to navigate the trade-off between specificity, scalability and comparability within the evaluation. Thirdly, mechanisms are required to ensure true participatory evaluation within UBCs and avoid superficial democratisation of evaluative methods. Other key areas include the use of evaluation as a neoliberal tool, the role of UBCs as boundary facilitators within the sustainability debate, potential remedies to the ambiguity of sustainability across disciplines and sectors and the greatest barriers to place-centred evaluative practices. Significant attention is required in this field to improve the potential for UBCs to deliver equitable and sustainable outcomes.