From Data to Impact: Assessing the Value of Cultural Heritage in the Digital Age

Abstract

This paper explores the complex landscape associated with the evaluation of the impact of digital cultural heritage initiatives within the European Union. While the present body of research has so far addressed various facets of digital culture and heritage, including digital humanities, a comprehensive understanding of the impact of digital heritage (i.e., LAM) projects on broader cultural, social, and economic contexts remains a critical gap. This is particularly important given the increasing emphasis on demonstrating value of and securing support for these initiatives. The EU recognizes this strategic importance, promoting digital transformation within the cultural heritage sector and setting ambitious digitization goals. However, the shift from digitization to digital transformation, alongside the more traditional concerns of access and preservation, requires a focus on sustainability, encompassing social and environmental impact, long-term preservation, and economic viability. By employing critical desk research, this paper examines the EU policies concerning digital cultural heritage and the challenges of measuring impact, discussing key concepts like sustainability and digital maturity. It provides an overview of prominent impact assessment frameworks, analysing their strengths and limitations and considering their appropriateness for today policy context. We conclude by arguing the importance of developing and applying holistic IA frameworks that consider the diverse values and long-term sustainability of digital cultural heritage initiatives, facilitating a shift from simply collecting data to demonstrating meaningful change.

1. Introduction

Cultural heritage is a complex notion that integrates the significant experiences across various dimensions of human existence. This article provides an overview of the current situation regarding the digital transformation of the heritage sector, with a focus placed on LAM institutions—libraries, archives and museums—that in their safekeeping have diverse forms of (digital) heritage, and examines the challenges and opportunities associated with assessing the impact of digital cultural heritage initiatives in Europe. When taking stock of academic research concerning digital culture and heritage, it becomes clear that this is a complex concept that does not conform to a singular definition that would encompass all of its facets and provide comprehensive reasons why it is important. Recent academic research has delved into various aspects of digital cultural heritage, including challenges of digital transformation in LAM institutions [1], digital humanities and digital heritage in AI environments [2], linked data and cultural heritage [3], cultural heritage data from the perspective of digital humanities [4], crowdsourcing and heritage [5], cultural heritage on social media [6], etc. Lian and Xie point out that the main themes currently concerning digital-heritage-related research cover issues of relevant technological innovation and its application, information management and technical support, and digitization and preservation of cultural heritage [7]. While such studies highlight the use and potential of digital tools in cultural heritage, the specific impact of digital cultural heritage projects on broader cultural, social or economic contexts remains a relatively unexplored area [8].

However, the evolving role of cultural heritage institutions nowadays is closely tied to their ability to create a meaningful impact on society. Clearly communicating the value of digital collections and demonstrating their impact to decision-makers, funders and users is becoming essential to justify current activities and secure continued support for their development and upkeep [9]. Equally important is that performing internal self-evaluations should allow heritage institutions to check if they are on the right track. Yet, the question remains: How can the success of digital cultural heritage resources and projects be measured, and against which underpinning values should they be evaluated?

In approaching this subject, we consider that culture is “collective memory, dependent on communication for its creation, extension, evolution and preservation” [10] (p. 19). This implies that whenever we discuss culture, communication is always implicit, as our collective knowledge has always been communicated and preserved through the existing cultural communication structures, with technologies playing a crucial role in enabling and facilitating those processes [11]. In other words, the ability to acquire, share, and innovate knowledge is essential for the preservation and development of any culture [12]. The digital context has paved the way for numerous collaborative and creative processes, with data now serving as a valuable resource for education, arts, etc., while enabling us to share and preserve our cultural memory. Vast amounts of data about human society and culture, both present and past, form a new frontier for digital initiatives in institutions worldwide. This is the reason that we consider digital heritage to be truly important and where we find its value—in considering it as a knowledge resource that should be used in the development of creativity and in the creation of an enabling environment for empowering citizens.

In recent decades, digital culture and digital cultural heritage have gained a more prominent place in the political agenda of the European Union (EU). The EU considers cultural heritage to be a strategic resource for sustainable development, something which also encompasses digital heritage. Ambitiously, the European Commission [13] (p. 8) set the goals for the digital future, as follows: “by 2030, Member States should digitise in 3D all monuments and sites that are considered to be cultural heritage at risk, and 50% of the most physically visited cultural and heritage monuments, buildings and sites”. The official EU discourse has shifted from focusing on “preservation” and “access” to emphasizing “digital transformation”. The digitization of cultural heritage has been seen as central to its transformation into a new knowledge resource, unlocking opportunities for the development of new services and content [14]. Moreover, the potential of digital cultural heritage to drive job creation across various economic sectors has become the focus of policymakers, and expectations that providing access to and enabling the reuse of digital content can generate additional revenue streams for cultural heritage institutions have been raised [15].

However, ensuring that digitized content reaches its intended audience is a challenging endeavour. In practice, heritage institutions strive to balance their missions of preserving and providing access to our shared heritage with the opportunities and challenges brought by the digital age, emphasizing the significance of their collections by digitizing them for both preservation and accessibility [12]. Cultural heritage institutions are increasing their digital offers with an aim to enhance user experience and attract new audiences. It is evident that participatory practices have become increasingly valuable in the digital heritage sector, transforming public engagement with cultural heritage institutions [16].

2. Materials and Methods

This paper explores the complex landscape of impact assessment (IA) for digital cultural heritage initiatives within the European Union. Based on a desk research methodology, the research provides a systematic review encompassing (1) relevant EU-funded projects focused on (digital) heritage impact assessment, with particular attention paid to their outputs such as reports, guidelines, frameworks, and toolkits; (2) relevant academic literature on the subject of impact assessment for digital cultural heritage, identified through a multi-faceted search approach using traditional academic databases such as Google Scholar, Zenodo, OpenAIRE, and the AI-powered research platform scite.ai; and (3) the current EU policy framework related to digital cultural politics, and the digital transformation of cultural heritage institutions, including key initiatives like Europeana and the Data Space for Cultural Heritage. This three-pronged approach aimed to ensure a comprehensive and contextualized understanding of the challenges and opportunities associated with measuring the impact of digital cultural heritage within the evolving European policy landscape.

A Google Scholar database search conducted between November and December 2024 using specific keywords “impact of digital cultural heritage,” “evaluation of digital cultural heritage” and “impact assessment framework for cultural heritage” yielded 66 articles. Nevertheless, less than 10 of these were pertinent to the scope of our research, which encompasses LAM sector initiatives. The search of OpenAIRE provided a total of 133 results based on the three used keywords, while the Zenodo search provided 36 references, with 36 overlaps across the two databases. However, the majority of references found were not relevant for our research as they were focused on topics unrelated to LAMs, such as archaeological data, architecture, indigenous digital cultural heritage, tourism, games, the engineering and technology fields, etc. or they were not written in the English language. Google Scholar was the most valuable resource for this research, as it provided a greater number of academic and project results focused on cultural heritage initiatives related to LAM institutions.

In addition, the AI-powered platform scite.ai was utilized to broaden the search, with a central inquiry aimed at summarizing existing research on the evaluation and impact assessment of digital culture projects. This search identified eight references, of which two further relevant articles have not been identified in the Google Scholar search results and have been used in our analysis. The results of databases research are presented in

Table 1. Results of keyword searches across databases.

DATABASE	Impact of Digital Cultural Heritage	Evaluation of Digital Cultural Heritage	IA Framework for Cultural Heritage
Total vs. relevant results according to the used keywords
Google Scholar	40/7	24/1	2/0
Zenodo	20/1	16/0	0
OpenAIRE	73/1	44/0	16/0
Platform scite.ai	8/2

.

Recognizing the limitations of the initial database searches, an additional effort has been made using snowball sampling methodology to identify additional relevant literature by carefully examining the references provided, allowing us to identify some additional literature and spot connected topics. This method proved useful in further expanding the literature. It yielded three more articles related to the topic of impact assessment frameworks from two identified resources found by scite.ai. In addition, the relevant literature identified in our earlier research have also been used.

Furthermore, during the research on impact assessment frameworks, several other crucial and interconnected concepts emerged, including digital maturity, accountability, resilience, and sustainability. These concepts were considered essential to the broader understanding of the relevance of impact assessment and, consequently, were further explored through a combination of EU policy documents and academic literature to establish their interrelationships within the context of digital cultural heritage. Database searches on the topics of accountability, resilience, and sustainability in the cultural heritage field yielded numerous results. The three results from Google Search were included based on their inclusion of digitality in the management of cultural heritage.

Given the relative novelty of impact assessment research in the LAM sector, the literature on the topic in our focus is quite limited. However, in recent times several EU projects have placed this topic under focus and have proposed various evaluation frameworks, which has provided us with relevant information for our analysis.

Therefore, a critical component of the research involves a comparative analysis of six impact assessment frameworks, including the balanced value impact (BVI) model, the Europeana Impact Playbook, the change impact assessment framework, Impactomatrix, the MOI Framework, and the SoPHIA model. These frameworks are evaluated based on their proposed methodologies, specific thematic focus, and the inclusion (or lack thereof) of proposed indicators.

A critical lens was applied throughout the analysis, allowing for a nuanced understanding of the interplay between policy, practice, and conceptual IA frameworks. The analysis focuses on the strengths and limitations of each framework and assesses their suitability for the current capacities and needs of cultural heritage institutions. The insights gained through the reviewed literature, project outputs, and the analysed IA frameworks enabled us to identify key challenges and propose directions for developing more standardized and holistic approaches to impact assessment for digital cultural heritage, emphasizing the need to move beyond data collection to demonstrating meaningful change and long-term sustainability.

3. Results

3.1. Digital Transformation and Data Policies

When the discourse shifted from digitization and digitalization to digital transformation, the issue of sustainability became more prominently the focus of both policymakers and heritage practitioners. However, this is not a straightforward issue. Digital transformation goes beyond the operational aspects of cultural heritage institutions; it reshapes their way of thinking. It is not only about technology and resources but also about the people and skills involved [17]. It is about turning digital cultural assets into products and services that make a meaningful impact on society while ensuring the long-term sustainability of digital resources.

Digitalization implies the preservation of cultural heritage for future generations (cultural memory objective), as well as reducing costs and energy consumption associated with physical visits to cultural heritage sites (environmental sustainability objective), and this is clearly recognized in the recommendation of 10 November 2021 on a common European data space for cultural heritage [13]. Nevertheless, the heritage sector needs to be mindful of the fact that digital infrastructure has a significant environmental impact.

Digitization is inextricably linked to augmenting the volume of digital data. LAM institutions are involved in processes of digitization and building digital archives, and these seemingly technical acts in fact involve the political acts of prioritization on different levels, from decisions concerning building collections, to providing access, search functions, metadata, etc. [18]. In this process “each type of content is treated as data… that can now be tinkered with, circulated, and traced” and thus they are contributing to the process of datafication [18] (p. 120). As the volume of data exponentially increases, it raises energy consumption due to the needed infrastructure, which has significant ramifications for environmental impact. The recently introduced concept of digital sobriety seeks to advocate for reducing the environmental impact of digital projects. This, among other things, entails minimizing energy usage throughout the digitization process, implementing efficient storage solutions, establishing sustainable practices for digital archive management, etc. [16]. To strike a necessary balance, developing metrics to evaluate the cultural sector’s performance, including its contribution to ecological sustainability is becoming a necessity.

Policies play a crucial role in shaping how the cultural sector adapts its communication methods [12,19] and data issues are increasingly becoming the subject of European digitization policies, influencing the creation, distribution and consumption of digital content. The numerous EU strategies around data, such as the open data strategy [20], open data directive [21] or the copyright directive [22]—which aim to regulate data issues (or content), etc.—are at the very core of cultural sector policies. The convergence of digital and sustainability issues is at the core of the European Strategy for Data “… making more data available and improving the way in which data is used is essential for tackling societal, climate and environment-related challenges, contributing to healthier, more prosperous and more sustainable societies” [23].

The recommendation of 10 November 2021 on a common European data space for cultural heritage [13] highlights the importance of data in cultural heritage and points towards the EU’s future direction. The openness of mediatized memory has been described as offering an alternative memory boom, characterized by an unfinished past and a revitalized future (Lunenfeld 2011, as cited in [24]). Linking policy with practice, platforms like Europeana and DARIAH EU drive digital transformation of the sector, facilitating access, reuse, and sharing of digitized heritage. Europeana, a central access point to European online heritage, is seen as the cornerstone for creating the “common data space” for the cultural heritage sector and serves as a platform for reusing and sharing digitized heritage materials. In addition to competence pooling from the heritage sector, transforming cultural content to social and economic assets, and informing EU digital cultural policy, Europeana plays a role in fostering European citizenship by promoting European identity through its rich repository of cultural content [25]. In 2021, Europeana provided access to 52 million cultural heritage assets, 45% of which could have been reused in various sectors [13]. Evidently, the value of data is in its openness for use and reuse. To support the use and reuse of digital resources, DARIAH EU serves as a platform to support digitally enabled research in the arts and humanities, facilitating the exchange of knowledge in digital humanities regarding content, methods, tools, and technologies. It assists researchers in utilizing these resources for building, analysing, and interpreting digital materials while ensuring adherence to best practices, as well as to methodological and technical standards [4].

It has been suggested that the success of (digital) technology should be measured by its openness to unanticipated uses—that is, its ability to enable change (Lunenfeld 2011 as cited in [24]). The data value chain proposed by the Open Data Watch (Figure 1) points towards a crucial step between data use and reuse—the “change” step. This step emphasizes the importance of tracking tangible behavioural changes resulting from data utilization. This framework provides a comprehensive view of the data lifecycle, from initial collection to the ultimate impact of data reuse. How to translate this into an evaluation grid for heritage institutions remains an issue to be resolved.

3.2. Managing Digital Change Responsibly: Key Concepts

In the context of the social crisis we are currently facing, it is essential to highlight concepts that make up the framework for managing cultural heritage institutions today—namely, that accounting, resilience, and sustainability are closely interrelated concepts relevant to cultural heritage management [26], particularly in today’s digitally infused reality in which we have still not fully reached digital maturity.

The recommendation of 10 November 2021 on a common European data space for cultural heritage [13], established to maximize the opportunities created by the digital transformation and encourage member states to help cultural heritage organizations to become more accountable, resilient and sustainable in the future. It underlines that the member states and cultural heritage institutions should take a “holistic approach” when planning digitization. This involves considering “the purpose of the digitisation, the target user groups, the highest quality affordable, the digital preservation of the digitised cultural heritage assets, including aspects such as formats, storage, future migrations, continuing maintenance and the necessary long-term financial and staffing resources” [13].

According to Thomas & Lamm, accountability is “an important component of ensuring pragmatic legitimacy for cultural enterprises that create value, respect the principles of sustainability (moral legitimacy), follow their mission, and deliberate and implement strategies (cognitive legitimacy)” [27]. Especially in the digital environment, accountability is an important aspect for public institutions. In other words, data are “the raw material for accountability” [28].

Resilience and sustainability are related but distinct concepts: resilience emphasizes short-term adaptability, while sustainability ensures long-term viability. Cultural resilience represents the ability of a cultural system to withstand adversity, adapt, and evolve [29]. The UN Brundtland Commission has defined sustainability as “meeting the needs of the present without compromising the ability of future generations to meet their own needs” [30]. This broad definition points to the heritage sector’s responsibility to ensure that heritage resources which is our inheritance from past generations remains a legacy we pass on to the future [31]. By making efficient management choices and assuming responsibility for their outcomes (being accountable), it is expected that organizations can foster both their resilience and sustainability [26].

Translating these concepts to the digital context points to the importance of the term digital maturity: “An individual’s or an organisation’s ability to use, manage, create and understand digital, in a way that is contextual (fit for their unique setting and needs), holistic (involving vision, leadership, process, culture and people) and purposeful (always aligned to the institution’s social mission)” [32]. Digital transformation is evidently a challenging process. Digitizing cultural heritage brings profound changes for heritage institutions, requiring them not only to adopt digital technologies but also to adapt to the digital landscape [33]. This involves reconsidering the representation, meaning, and, most importantly, the value of digital heritage and its impact on the community.

However, in practice it does not seem that we have reached the maturity point yet. Tanner points out that the assumptions continue to pervade that all digital things are innovative, that agile development can substitute planning, and that if competitors (or Google) are doing something, it is imperative to do it [34]. On the other hand, the case study by Marsh and others suggests that the impact concerning digitising collections can be defined by the question “What is meaningful and to whom?” [35]. Thus, cultural heritage institutions need to consider their digital activities in the context of their social missions and potential value for their users’ communities and try to make sense of how digital context supports or hinders this.

3.3. From Digitalization to Impact: Approaches and Challenges

Impact is usually described as something that brings change [35,36]. According to Tanner and Deegan [36], impact can be defined as “the measurable outcomes arising from the existence of a digital resource that demonstrate a change in the life or life opportunities of the community for which the resource is intended”. The concept of impact is rather complex. It can be tangible or intangible, positive or negative. It happens at multiple levels and magnitudes/scales. Moreover, the impact happens over time (e.g., short-term impact or long-term impact) [35]. Impact can range from individuals to communities or general society and can have various aspects, from social to economic [37].

By its very nature, the impact is difficult to assess. Most cultural heritage institutions can obtain figures on how much they have spent on their web services, how many visitors visit their site, or how the visitors navigate through their site. However, they struggle with how to relate, for example, high engagement rates, to an exact measure of impact. The impact of digital cultural heritage is assessed by evaluating the value of the changes it generates. For this, it is necessary to have assessment tools or frameworks that would enable meaningful evaluations.

Various evaluation methods are used today, including impact assessment. In general, evaluation processes refer to broader approaches that assess a project’s overall performance, effectiveness, and efficiency during or after implementation. In contrast, impact assessment focuses on the specific, long-term effects and outcomes of a project.

Impact assessment is defined in the literature as “research that requires setting questions and choosing methods to answer them” (ISO 2014 as cited in [38]); a “process of identifying the future consequences of a current or proposed action” [39]; “a tool to foster understanding of how strategic decisions about digital resources may be fostering change within our communities” [34]; or simply “thinking before acting” (Morrison-Saunders 2018 as cited in [40]). These definitions emphasize the complex, layered process involved in impact assessment, which demands careful planning and execution. From the definitions above, we can conclude that impact involves an intervention that creates change where the effects of the intervention/project/resource are assessed in relation to its intended purpose and the potential needs of its stakeholders [41]. It reflects the difference between what would have occurred naturally and what resulted from a specific action or project.

Impact assessment was first introduced in the heritage sector in the 1980s focusing on UNESCO’s World Heritage sites [40], but has gained more prominence in recent years. Heritage IA is closely linked to the concept of cultural capital, which includes the economic value of heritage assets [42], as well as the connection to the history of landscapes and communities, meaning the cultural values derived from heritage [43]. Additionally, cultural and social values are created through community involvement and everyday participation. Nevertheless, it is difficult to assign a tangible sense of value to a digital resource or project.

Measuring impact supports an evidence-based approach to managing heritage. Evidence of (digital) impact can be inferred from various sources, including output data, user satisfaction measures, or performance indicators [8]. Thus, impact assessment involves both quantitative and qualitative methods and indicators. In this regard, Shaw [9] argues that a comprehensive assessment of digital collections requires a multi-faceted approach, combining statistics, surveys, user studies, usability testing, and web analytics. However, Tanner [34] remarks that, so far, primary measures of digital heritage have predominantly focused on web statistics, anecdotal information, or evaluations of outputs, such as the quantity of digitised materials, rather than assessing the value and change derived from these efforts.

The basic elements in measuring impact are indicators or data that we interpret. Tanner proposes that indicators are the most critical part of the impact assessment because they measure progress toward set goals [34] and should reveal the change triggered by the project/action that is being evaluated. They are usually specifically developed depending on the proposed action. For example, Fukuyama and Tanner proposed a set of 13 potential indicators for the UK Web Archive (UKWA), but which can also be used by other web archiving organisations [38]. To be usable, they must be specific, measurable, achievable, realistic, and timely (SMART) and must meet quality criteria. This means that the functionality of each indicator needs to be evaluated against established benchmarks for effective and ineffective indicators [38]. Finally, for indicators to be useful, they must be set as opposed to some baseline values that have been registered before the project has begun, and against which change can be measured. Tanner underscores that the digital domain is a challenging environment for identifying suitable indicators due to the limited availability of historical data, i.e., the absence of effective baselines, which can hinder meaningful analysis [34].

To successfully perform an impact assessment, the initial step is deciding what will be monitored and how: what data sources are relevant and available to serve as indicators, which are the relevant questions for qualitative analysis, which methods will be used [41]. All of the above indicates that this is not a simple task and requires some underpinning models that would provide the framework for the IA analysis.

3.4. Exploring EU Initiatives for Impact Assessment and the Challenges of Measurement

As argued above, the adaptability of content to different types of audiences is important for the heritage sector, as well as to the increase of its engagement with digitised cultural heritage. To achieve this, we concur with Shaw’s claim that the development and preservation of high-quality digital collections that respect community standards and follow best practices is a complex and resource-intensive endeavour and that engaging in their meaningful assessment further compounds this complexity [9].

The heritage sector is increasingly aware of the need to evaluate the success of its projects and the impact they have achieved on the community they serve. However, they lack the knowledge and skills to fully embrace this practice. To support the heritage sector in reaching digital maturity, Culture24’s flagship collaborative action research program Let’s Get Real has, since 2010, provided capacity building that enables better understanding of what impact means in the context of digital heritage. The first edition addressed the question “How to Evaluate Online Success” by exploring what success looks like for different organizations and the tools available to measure it [44]. Subsequent editions focused on various aspects of digital activities, such as aligning digital practice with social purpose, assessing whether institutions’ content is “fit for purpose”, fostering deeper human connections through digital channels, and understanding and measuring digital engagement. Such in-depth exploration helped participating organizations better understand what impact means for them, by enabling them to measure their online performance more accurately and meaningfully and thus reaching better informed decisions regarding their online activities [44].

To assess the present state of development of tools for the impact assessment of (digital) heritage projects we have conducted a desk review of recent project reports that have aimed at proposing new frameworks or methods of evaluating the impact in the heritage domain. We analysed six impact assessment frameworks: the balanced value impact (BVI) model, the Europeana Impact Playbook, the change impact assessment framework, Impactomatrix, the museums of impact (MOI) framework, and the social platform for holistic heritage impact (SoPHIA) model. We looked into their methodologies, thematic focus, and the presence or absence of proposed indicators and tried to detect challenges of measurement. We shall present our findings in the

Table 2. Overview of the IA frameworks. Source: data assembled by the authors.

OVERVIEW OF THE FRAMEWORKS FOR (DIGITAL) HERITAGE IMPACT ASSESSMENTS
BVI—BALANCED VALUE IMPACT MODEL
DESCRIPTION One of the first models that challenges cultural heritage organisations to be more “evidence based” and to measure the impact of their digital resources is the balanced value impact (BVI) model developed by Simon Tanner [34,38,45,46,47]. The BVI model is focused on identifying the change in a community that arose from the existence of digital resources that are proven to be of value to the community [34]. Specially designed for cultural heritage institutions and their digital resources, it provides a conceptual framework that comprises a five-stage process guiding the IA. The BVI model distinguishes itself through its five value lenses. These lenses are specifically designed to capture the diverse types of value commonly associated with digital cultural heritage experiences. The five value lenses are the utility lens, the existence lens, the legacy lens, the learning lens, and the community.
SPECIFIC THEMATIC FOCUS NO
PROPOSED STEP-BY-STEP METHODOLOGY YES—a five-stage process guiding the IA. 1. Set the context; 2. Design the framework; 3. Implement the framework; 4. Narrate the outcomes and results; 5. Review and respond.
PROPOSED INDICATORS NO
EUROPEANA IMPACT PLAYBOOK
DESCRIPTION The BVI model has been further promoted, adapted and applied by the Europeana community. Based on BVI, the Europeana Impact Playbook (2017–2022) aims to help heritage organisations in their own impact planning and assessment by providing a step-by-step method to assess the impact of their digital resources consisting of four phases: 1. Designing the impact (figuring out which information is valuable for the organisation); 2. gathering data; 3. narrating and sharing the story; and 4. evaluating [48,49,50,51]. To encourage the use of the playbook, additional training and resources have been provided to Europeana users. Europeana highlights that the “Europeana Impact Community” has been active in creating a platform for learning and discussion around the impact issues and its community of professionals interested in the impact of cultural heritage has significantly increased in recent years. Sources: https://europeana.atlassian.net/wiki/spaces/CB/overview?homepageId=2256699653 (accessed on 11 February 2025) https://pro.europeana.eu/page/webinars#impact (accessed on 10 December 2024)
SPECIFIC THEMATIC FOCUS NO
PROPOSED STEP-BY-STEP METHODOLOGY YES—four-step method, each described in the related toolkit. 1. Designing the impact; 2. Gathering data; 3. Narrating and sharing the story; 4. Evaluating to increase the impact and develop new ideas for improvement
PROPOSED INDICATORS NO
CHANGE IMPACT ASSESSMENT FRAMEWORK
DESCRIPTION The change impact assessment framework has been created within inDICEs, a Horizon 2020 project that aimed to help cultural heritage professionals, practitioners, and policy-makers understand the social and economic impact of digitisation [52]. Based on the Culture 3.0 theory [53] and backed by ample research, its conceptual map “the 8 Impact Areas of active digital cultural participation” is assisting cultural heritage institutions to understand the potential impact of active digital cultural participation across eight areas of impact. This framework does not include the methodology for cultural heritage organisations to use when assessing the impact of their digital resources or projects, but rather addresses the areas in which digital culture has an impact. However, it includes a set of exemplary indicators that help the measurement of impact in specific areas. The framework can offer a new perspective on designing digital cultural activities that benefit participants’ mental health, environment, and creativity.
SPECIFIC THEMATIC FOCUS Eight impact areas: 1. Innovation and knowledge; 2. Welfare and well-being; 3. Sustainability and environment; 4. Social cohesion; 5. New forms of entrepreneurship; 6. Learning society; 7. Collective identity; 8. Soft power.
PROPOSED STEP-BY-STEP METHODOLOGY NO—It provides references to other existing methodologies that could be applied.
PROPOSED INDICATORS YES—a set of exemplary indicators
IMPACTOMATRIX
DESCRIPTION To assess how digital tools and infrastructure in the digital humanities influence research practices across the humanities and other disciplines, the Impactomatrix identifies key impact factors and success criteria for evaluating projects in the arts and humanities. It explores the value these tools bring to the scientific community and how to maximize the efficiency of funding allocation. By analysing these impacts, the digital humanities community is expected to be able to enhance visibility and transparency, effectively communicate their benefits to researchers and funding bodies, and strengthen the role of digital research in the humanities. Through its interactive website, Impactomatrix provides a methodological framework for evaluating developments in the digital humanities, incorporating both quantitative and qualitative criteria in the assessment process. Source: https://dariah-de.github.io/Impactomatrix/ (accessed on 16 February 2025)
SPECIFIC THEMATIC FOCUS A selection of 21 impact areas is provided, as follows: External impact; education; data security/safety; dissemination; effectivity; efficiency; funding perspective; innovation; integration; coherence; collaboration; communication; transfer of expertise; sustainability; usage; publications; relevance; reputation; transparency; competitiveness; transfer of knowledge. Each impact area is provided with a list of corresponding factors that influence that specific area.
PROPOSED STEP-BY-STEP METHODOLOGY NO
PROPOSED INDICATORS YES. A list of ‘criteria’ is proposed to help measure changes within the chosen impact area.
MOI FRAMEWORK
DESCRIPTION The Creative Europe project, through the impact MOI! Museums of Impact (2019–2022), has developed the MOI framework—especially designed for museums in order “to help museums discuss, evaluate, and choose development goals to increase their impact in society” [54,55]. It is focused on the societal impact of museums, which also includes the digital component (digital engagement) as an important part of the whole framework. The MOI Framework consists of eight modules, which contain 151 impact statements that the framework asks the participants to evaluate. The modules are divided between enabler and impact modules. Source: https://www.ne-mo.org/resources/moi-self-evaluation-tool (accessed on 2 February 2025)
SPECIFIC THEMATIC FOCUS Enabler Modules: 1. What we do—impact goals and strategy; 2. How we work—organisational culture and competences; 3. How our organisation functions—resources and service development; 4. How we embed digital into services and processes—digital engagement. Impact Modules: 1. Communities and shared heritage; 2. Relevant and reliable knowledge; 3. Societal relevance; 4. Sustainable organisations and societies.
PROPOSED STEP-BY-STEP METHODOLOGY YES—it provides self-evaluation workbooks for each module.
PROPOSED INDICATORS NO
SoPHIA MODEL
DESCRIPTION The H2020 project, SoPHIA—the social platform for holistic heritage impact assessment —developed a comprehensive model for evaluating the impact of heritage interventions on the development of communities. This model responds to the need for a comprehensive, multidimensional approach in heritage project assessments, considering social, cultural, economic, and environmental factors. The project had aligned with the EU’s strategic goals of promoting sustainable and inclusive growth, recognizing cultural heritage as a key resource for resilience and innovation [31,56]. The model is structured along three axes: (1) time—assessing impacts of heritage interventions before, during, and after interventions; (2) stakeholders—ensuring inclusive participation; (3) domains—integrating multidisciplinary perspectives for evaluating the social, cultural, economic, and environmental impacts of heritage projects. The model is divided into six main areas of impact, i.e., themes of assessment that need to be considered when assessing cultural heritage interventions. Each theme is further divided into several subthemes accompanied by a proposed list of possible indicators that support the IA analysis and a list of guiding questions for qualitative analysis and stakeholder input. By including both qualitative and quantitative indicators, the model provides a framework for measuring the effectiveness of heritage projects in contributing to social cohesion, cultural diversity, economic growth, and environmental sustainability. Source: https://shorturl.at/EYZB3 (accessed on 3 February 2025)
SPECIFIC THEMATIC FOCUS Six assessment themes/twenty-eight subthemes (1) Social capital and governance, (2) Identity of place, (3) Quality of life, (4) Education, creativity, and innovation, (5) Work and prosperity, (6) Protection. Each theme is further divided into subthemes (28 in total)
PROPOSED STEP-BY-STEP METHODOLOGY YES—a Toolkit which explains the purpose, logic, and conceptual framework of the SoPHIA model and describes its implementation phases.
PROPOSED INDICATORS YES—a proposed list of possible indicators and a list of guiding questions for qualitative analysis and stakeholders’ inputs.

. below.

4. Discussion

Starting from the different premises, the six examples described above provide various frameworks that enable heritage institutions to rethink what impact means for them in the context of their missions by providing them with tools that helps them asking relevant questions and choosing methods to answer them. While the first four examples focus exclusively on heritage in a digital context, the last two examples address the heritage sector in general but are still relevant for the digital heritage projects and resources.

IA frameworks, such as the SoPHIA model, MOI framework, and change impact assessment framework, primarily emphasize tangible outcomes like knowledge creation, creativity, innovation, sustainability, etc., directly addressing the scope of impact through their thematic focus. In contrast, the BVI model and the Europeana Impact Playbook, which is based on it, focus on the fundamental values that drive these outcomes. For instance, the BVI model utilizes five value lenses to guide impact assessment, ensuring a comprehensive evaluation of the core values linked to digital cultural heritage experiences. Meanwhile, Impactomatrix appears to integrate elements of both approaches.

While those IA frameworks provide different lenses on a variety of impact areas and provide general guidelines for potential applications of the framework, only some of them have developed a concrete methodology guiding the assessment plan that facilitates implementing the specific approach that depends on the needs of the specific heritage institution. This highlights a key challenge: effectively translating the framework into concrete evaluation action within the unique context of each institution by deciding on evaluation goals and timeframe, research methods to be used, adequate indicators that need to be collected and interpreted, etc. This leads us to the conclusion that standards for IA concerning digital heritage are not yet agreed upon and that there are numerous avenues of research that are yet to be performed to bring forward consolidated frameworks that would guide heritage sector evaluations of their impact on community they are serving.

We can conclude that, despite decades of continuous investment in the development of digital cultural resources within an ever-evolving digital landscape, there is still no clear consensus on how to assess the impact of digital heritage resources and projects. The European Commission [13] has highlighted the need for a “holistic approach” for member states and cultural heritage institutions when planning digitization initiatives. This is reflected in the IA frameworks described, which take a broader view of the wider implications of such efforts. However, progress is hindered by a combination of factors: financial constraints often limit the resources available for IA, a lack of skilled personnel and institutional understanding poses a significant barrier, and the complexity of digital heritage projects makes effective assessment a challenging process.

This indicates that understanding the impact of digital resources and demonstrating the change they have produced remains a challenge. As the data value chain (Figure 1) illustrates, the ultimate goal of data use is to achieve change (through reuse), which aligns with the fundamental understanding that impact is realized through meaningful change. This principle is well reflected in the IA models presented in Table 2. The point is that, by identifying the value of that change or by describing it as a concrete outcome, cultural heritage institutions aim to demonstrate the true significance and effectiveness of their digital efforts.

The described examples of IA frameworks developed recently point to the fact that the heritage sector increasingly understands the importance of demonstrating the impact of their activities and resources on the communities they are serving. However, most cultural institutions have not yet mastered the IA tools and methods to appropriate them in their work, as even with several existing IA frameworks in place, current assessments of digital heritage in the majority of heritage institutions predominantly rely on metrics like web statistics and the quantity of digitized materials, overlooking the crucial need to evaluate the true value and impact of these efforts. Ultimately, at this point the key benefit of utilizing IA frameworks is the valuable learning process inherent in the evaluation itself.

To translate proposed frameworks into practical applications and strengthen the sector, achieving digital maturity and enhancing digital skills in the heritage field is essential. This requires a strategic approach that extends beyond merely sharing information on the topic. Navigating this complex landscape demands digitally literate leadership to ensure that heritage professionals acquire the necessary digital competencies and can effectively adapt to ongoing changes. Cultivating these skills across the entire team requires dedicated time and resources. In essence, cultivating a digitally literate workforce, investing in resources, and conducting evaluations thorough impact assessment frameworks are all crucial elements for building resilience in cultural heritage institutions and ensuring their sustainability.

Finally, as sustainability is becoming an integral part of EU politics, (digital) cultural heritage impact assessment frameworks should enable lenses through which digital cultural heritage initiatives can be evaluated, not only to achieve their short-term objectives but also to foster long-term sustainability and thus contribute to the preservation and vitality of cultural heritage in the digital age. It is clear that the impact assessment of digital cultural resources is relevant for decision-making processes in culture and are equally of interest for the policy makers, as such assessments provide relevant data for the evidence-based policy making that we are striving for.

When conducting this research, it became clear that comprehensive literature specifically addressing digital cultural heritage IA is not abundantly present. While IA frameworks are emerging in recent years, they remain at an initial stage. Although they provide a good starting point, substantial research is still needed. Further research is needed to fully evaluate the effectiveness and applicability of these IA frameworks and should prioritize critical case studies that incorporate qualitative and quantitative data, comparative studies across different types of LAM institutions and longitudinal studies to identify trends. Such studies should provide the information needed for a comprehensive understanding of digital initiatives that requires exploring not only measurable outcomes but also the less tangible, value-driven impacts and their long-term sustainability implications.