Artificial Intelligence for Digital Heritage Innovation: Setting up a R&D Agenda for Europe

Abstract

Artificial intelligence (AI) is a game changer in many fields, including cultural heritage. It supports the planning and preservation of heritage sites and cities, enables the creation of virtual experiences to enrich cultural tourism and engagement, supports research, and increases access and understanding of heritage objects. Despite some impressive examples, the full potential of AI for economic, social, and cultural change is not yet fully visible. Against this background, this article aims to (a) highlight the scope of AI in the field of cultural heritage and innovation, (b) highlight the state of the art of AI technologies for cultural heritage, (c) highlight challenges and opportunities, and (d) outline an agenda for AI, cultural heritage, and innovation.

1. Introduction

Digitization is key for protecting, preserving, documenting and opening up European and global cultural heritage (CH) to meet pressing sustainability threats, including environmental ones and increasing social inclusivity. Within the CH sector, economic activities related to digital collections in cultural institutions are a market worth ten bn EUR in 2015 [1]. These developments have been accelerated by the COVID-19 pandemic [2]. Digital technologies can transform the entire value chain model in CH institutions—from capturing and digitizing tangible and intangible heritage and long-term preservation over innovative digital research methods to digital channels allowing people across the globe to interact with digital objects. These channels enable connections to other collections published on the web and accelerate the creation of new artistic works, unearthing new narratives in collections. While all these areas of work could be improved by applying the latest digital technologies, a significant increase is expected during the next few years.

The Strategic Topic Group (STG) Cultural Heritage in Green and Digital Transitions for Inclusive Societies was formed in 2022 within the European Institute of Innovation and Technology’s (EIT) Knowledge and Innovation Community for Culture & Creativity and seeks to unlock the potential of CH for the green and digital transitioning of Europe encompassing societal challenges on this key policy topic. The group includes 32 partner organizations in mid-2023 and focuses on four closely connected areas, including (i) upskilling and capacity building; (ii) environmental impact of operations of CH institutions; (iii) increasing outreach and community engagement; and (iv) creation of new business models. This article investigates the state of the art and proposes future steps to leverage artificial intelligence (AI), particularly machine learning (ML), for CH innovation. The purposes of this article are:

• To highlight the scope of ML in CH and innovation;
• To present the state of the art of ML technologies for CH;
• To identify challenges, risks, and opportunities;
• To draft a mitigation strategy and agenda for AI in CH and innovation.

This article summarizes and reviews published research papers and expert opinions. It addresses the stakeholders such as authorities, innovators, and researchers dealing with cultural heritage and AI.

1.1. Methodology

Input for this article came from (a) a desk review of recent project reports, research articles and agendas at the EU level, (b) domain experts to provide community and media type-specific input, and (c) an online workshop to collect feedback, in particular, regarding the proposed roadmap.

The desk research included the review of deliverables from recent EU projects on digital heritage (see, e.g., [3]); document collections and working papers compiled by, e.g., the Council of Europe [4], the JPI [5]; and the review of AI-related roadmap documents from the EC and AI-related associations. The review was conducted as a thematic inquiry (approach [6,7] and application [8]) as a topic-oriented qualitative research paradigm. The 8 domain experts involved as authors compiled media type-specific overview sections and reviewed the overall sections based on desk research. The online workshop took place in December 2023 and involved 26 individuals from member associations of the STG. Feedback on the draft roadmap was provided through verbal comments, Zoom chat contributions and an online survey conducted via the EC Survey tool. This feedback was reviewed and incorporated into the revision of the roadmap.

1.2. Definitions

Cultural heritage can be understood as traces and expressions from the past that attribute values and are used in contemporary society, cf. [9]. CH has traditionally focused on tangible objects, though its complete understanding also implies the inclusion of intangible, natural, and—most recently—born-digital heritage, such as computer games and websites. [10].

While CH traditionally focuses on tangible objects, a broader understanding adds intangible heritage (e.g., dances, customs, and crafts) and natural heritage (Figure 1). Another important concept is digital (cultural) heritage. It comprises technologies to preserve, research, and communicate CH [11], which includes materials like texts and images created digitally or digitized, as well as digital resources of human knowledge or expression (e.g., cultural, educational, or scientific) [10]. This latter facet also comprises various digital technologies to study CH [12].

Innovation: In CH, digital innovation plays a key role in adding economic value [13,14]. Innovation is the “multi-stage process whereby organizations transform ideas into new/improved products, services or processes, to advance, compete and differentiate themselves successfully in their marketplace” ([15] p. 95). This complex endeavor occurs in multi-stakeholder environments defined as innovation ecosystems [16]. Policies play a key role in steering innovation (e.g., for the EU level [17,18]) and vice versa, being influenced by innovations (e.g., research results and new products) as well as global trends (climate change, health, digitization, diversity, and intangible heritage).

ML and AI: ML, AI, and big data are interconnected fields that have gained significant attention in recent years [19]. In today’s era of rapid technological advancement and exponential increases in extremely large datasets (“big data”), AI has transitioned to tangible applications on a large scale [20].

Figure 1. Types of cultural heritage [20] (Images: Münster except for the right-hand image: https://www.europeana.eu/de/item/916118/S_TEK_object_TEKS0057154, accessed on 1 February 2023).

Figure 1. Types of cultural heritage [20] (Images: Münster except for the right-hand image: https://www.europeana.eu/de/item/916118/S_TEK_object_TEKS0057154, accessed on 1 February 2023).

• Artificial intelligence (AI) refers to the development of computer systems capable of performing tasks that would typically require human intelligence, such as pattern and speech recognition, game playing and decision-making, problem-solving, and learning from data (cf. [21,22]). AI encompasses subfields, including ML, natural language processing, computer vision, and robotics. AI is now being used across all disciplines, including information science, mathematics, medical science, geoscience, physics, and chemistry [23].
• Machine learning (ML) is a subset of AI that focuses on developing algorithms and models that enable computers to learn from experience without being explicitly programmed. ML algorithms learn patterns and relationships from large datasets and use this knowledge to make predictions, classify data, or make decisions (cf. [24]). ML is traditionally divided into three categories: supervised, unsupervised, and reinforcement learning [25]. An algorithm learns from labeled training data to make predictions or decisions in supervised learning. The goal is to learn a mapping function to accurately predict the correct output label for new, unseen input data. Unsupervised learning aims to find structure and regularity in an unlabeled dataset. In reinforcement learning, the algorithm learns a policy for maximizing rewards given as feedback within a dynamic environment [26,27]. While originally algorithmic approaches were used for solving ML problems, the advent of deep learning and neural networks almost completely replaced these traditional methods [28].
• Big data refers to large and complex datasets that cannot be effectively processed or analysed using traditional data processing techniques (cf. [29,30]). In contrast to other approaches, big data processes full-scale data instead of samples to uncover patterns, trends, and insights. Big data often involves using advanced technologies and techniques, such as distributed computing and data mining.

2. Application Fields of AI in CH

In CH, AI is being used in a variety of research areas. These include:

• Image analysis and restoration: AI algorithms can analyze and restore old, damaged, or degraded (moving) images, sounds, paintings, and photographs. These algorithms can enhance image quality, remove noise, and even reconstruct missing parts of the artwork, aiding in preserving and restoring cultural artifacts. Examples listed in [27] are the prediction of the painting’s style, genre, and artist, the detection of fake artworks by stroke analysis, and the artistic style transfer using adversarial networks to regularize the generation of stylized images.” Further research deals with the automatic colorization of images [31] and the restoration of ancient mosaics [32].
• Object recognition and classification: AI-powered computer vision techniques enable automatic recognition and classification of cultural objects. By analyzing visual features and patterns, AI algorithms can identify and categorize artifacts, sculptures, and architectural elements [33], facilitating the organization and cataloging of museum collections. Examples are the prediction of color metadata, e.g., for textile objects [34], of technique, timespan, material, and place metadata for European silk fabrics [35], and the recognition and classification of symbols in ancient papyri [36].
• Translation and transcription: AI language models are capable of translating. e.g., ancient texts, inscriptions, and manuscripts into modern languages. They can also be used for modern languages by translating metadata or full-text content of heritage objects and related information, making sharing cultural heritage across languages easier. Other models can transcribe handwritten texts, allowing researchers and historians to access and understand historical documents and perform automated analysis (e.g., [37]).
• Automatic text analysis: This comprises various approaches [38]. An example is the automatic semantic indexing of pre-structured historical texts, which enables historians to mine large amounts of text and data to gain a deeper understanding of the sources (e.g., [39]); for example, tax lists or registers of letters sent to a historical entity [40].
• Virtual Reality (VR) and Augmented Reality (AR): AI technology supports the creation of immersive VR and AR experiences for CH sites and museums. Visitors can virtually explore ancient ruins, historical sites, or museum exhibitions, interacting with AI-generated virtual characters or objects to enhance their understanding and engagement with the cultural context [41,42].
• Recommender systems for personalized experiences: AI algorithms can analyze user preferences, historical data, and contextual information to provide personalized recommendations for CH experiences. Despite the risks of information filtering (e.g., [43]), use is to suggest relevant exhibits, customized tours, or tailored content, AI-powered recommender systems enhance visitor engagement and satisfaction, or—triggered by the advent of large language models (LLMs) such as GPT—dialogue and chatbot systems. Examples are the use of chatbots in museums [44,45] or recommender systems for CH collections (e.g., [46,47]).
• Cultural content analysis and interpretation: AI techniques, such as natural language processing (NLP), are used to analyze large volumes of cultural content, including literature, music, and artwork. This analysis can reveal patterns, themes, and cultural influences, providing valuable insights into historical contexts and artistic movements. Examples are metadata enrichment (e.g., [48,49,50]) and linking to open data sources (e.g., [33]).
• Heritage digitization and preservation: AI can be crucial in digitizing cultural artifacts and archives. By automating digitization processes and extracting knowledge, AI speeds up the preservation of CH, allowing researchers and the public to explore and study rare artifacts remotely. Several articles provide an overview of particular technologies, e.g., for 3D acquisition, such as laser scanning [51] or photogrammetry [52], and quantify their use [53]. AI-powered systems can monitor and analyze CH site environmental conditions, helping with early detection of potential threats such as humidity, temperature fluctuations, and structural damage. This real-time monitoring aids in the proactive conservation and protection of cultural landmarks (e.g., [54,55]).
• Multimodal analysis: AI is capable of bringing together different sources and types of data. Approaches include text, images [35], 3D models [56], audio [57], and video [58].
• AI supports or creates artistic expressions: Applying algorithms that analyze heritage objects (or entire collections) and extract information that either artists and other creators can use to create new works [59] or AI creating “artistic” expressions (review article: [60]; empirical study: [61]).

3. Project Examples

To date, there are some impressive examples of the utilization of AI technologies in the field of CH (

Table 1. Project examples of AI application in CH (all links accessed on 1 December 2023).

	Art Transfer by Google Arts & Culture Using AI algorithms, Art Transfer allows users to transform their photos into the style of famous artists such as Van Gogh or Picasso. Link: https://artsandculture.google.com/camera/art-transfer
	MicroPasts by the British Museum MicroPasts is a project that combines crowd-sourced data with AI technology. Volunteers contribute by digitizing and tagging images while AI algorithms analyze the data. Link: https://micropasts.org/
	4Dcity by the University of Jena This application uses AI to automatically 4D reconstruct past cityscapes from historical cadastre plans and photographs. This 4D model is world-scale and enriched by links to texts and information, e.g., from Wikipedia, and accessible as mobile 4D websites [62]. Link: https://4dcity.org/
	SCAN4RECO This EU-funded project combines 3D scanning, robotics, and AI to create digital reconstructions of damaged or destroyed CH objects. Link: https://scan4reco.iti.gr/
	AI-DA by Aidan Meller Gallery AI-DA is an AI-powered robot artist developed by Aidan Meller Gallery in the United Kingdom. The robot uses AI algorithms to analyze and interpret human facial expressions, creating drawings and paintings inspired by the emotions it perceives. AI-DA’s artworks have been exhibited in galleries across Europe. Link: https://www.ai-darobot.com/
	Transkribus by Read Coop SCE Transkribus is a comprehensive solution for digitization, AI-powered text recognition, transcription, and searching historical documents. A specific emphasis is on handwritten text recognition. https://readcoop.eu/transkribus/
	Transcribathon The Transcribathon platform is an online crowd-sourcing platform for enriching digitized material from Europeana. It applies the Transkribus handwriting recognition technology to input documents, performs some automatic enrichments (including translation) on the obtained text and metadata, and lets volunteers validate the results. https://transcribathon.eu/
	The Next Rembrandt by ING Bank and Microsoft This project employed AI algorithms to analyze Rembrandt’s works and create a new painting in his style. https://www.nextrembrandt.com/
	Rekrei (formerly Project Mosul) Rekrei is a crowd-sourcing and AI project aimed at reconstructing CH sites that have been destroyed or damaged. Users can contribute photographs and other data, and AI algorithms help in reconstructing the lost heritage digitally. https://rekrei.org/
	Notre Dame reconstruction After a fire destroyed parts of the Notre Dame Cathedral in Paris in 2019, a digital twin model was created to experiment—physical anastylosis, reverse engineering, spatiotemporal tracking assets, and operational research—and create a reconstruction hypothesis. The results demonstrate that the proposed modeling method facilitates the formalization and validation of the reconstruction problem and increases solution performance [63]. https://news.cnrs.fr/articles/a-digital-twin-for-notre-dame
	Finto AI by the National Library of Finland Finto AI is a service for automated subject indexing. It can be used to suggest subjects for text in Finnish, Swedish, and English. It currently gives suggestions based on concepts of the General Finnish Ontology, YSO. Link: https://ai.finto.fi
	Europeana Translate This project has trained translation engines on metadata from the common European data space on cultural heritage in order to obtain a service that can translate CH metadata from 22 official EU languages to English, improving the multilingual experience provided to its users. It has been applied to 29 million metadata records so far. Link: https://pro.europeana.eu/post/europeana-translate-project-brings-together-multilingualism-and-cultural-heritage
	MuseNet by OpenAI MuseNet composes original music in a wide range of styles and genres. It can create music inspired by different cultural traditions and historical periods, demonstrating the potential of AI in generating new compositions that reflect CH. Link: https://openai.com/research/musenet
	The Hidden Florence by the University of Exeter The Hidden Florence is an AI-enhanced mobile app that guides visitors through the streets of Florence, Italy, offering insights into the city’s rich CH in an engaging way. The app utilizes AI algorithms to provide location-based narratives, AR experiences, and interactive storytelling. Link: https://hiddenflorence.org/
	Smartify App by Smartify Smartify utilizes AI to provide interactive experiences with artworks in museums and galleries. The mobile app uses image recognition to identify artworks, delivering detailed information, audio guides, and curated tours. It is compatible with numerous cultural institutions across Europe and beyond. Link: https://smartify.org/
	Second Canvas App by Madpixel and the Prado Museum The app uses AI technology to enhance the visitor experience. It provides high-resolution images of artworks, along with interactive features that allow users to explore the details and stories behind the paintings. Link: https://www.secondcanvas.net/
	WAIVE WAIVE is a smart DJ system utilizing AI to create unique music samples, beats, and loops from the digitized audio archives of the Netherlands Institute for Sound & Vision. Link: https://www.thunderboomrecords.com/waive

).

4. AI Technologies for CH State of the Art

The state of the art of AI for CH has been analyzed in various publications.

• Fiorucci et al. analyzed the current situation on AI for CH in 2020 with regard to both ML approaches and application examples [27].
• A high-level view on overall challenges and examples for AI for CH and museums was compiled by the European Commission in 2022 ([64], pp. 143), similarly about challenges and institutional positions as a briefing for the European Parliament in 2023 [65].
• The EuropeanaTech AI task force conducted a survey amongst professionals to examine the usage and prospects of AI in that field [66].
• A curated list of policy documents—with only a few links to CH currently–is maintained by the Council of Europe [4].
• Gasparini and Kautonen examined the state of AI for libraries [67], and Mishra for building heritage monitoring [54]. The AI4LAM maintains a list of resources and projects, particularly on AI for CH [3].

The following paragraphs describe the state of the art of AI in several fields of CH with regard to the type of material.

4.1. AI and Images

Historical images hold immense value in documenting our collective heritage. However, analyzing and extracting information from these images manually can be limited, e.g., due to the required effort. Current evolvements in computer visualization are closely coupled to the massive renaissance in ML [68] with the use of convolutional neural networks (CNNs, cf. [69]). There is a large number of computer vision techniques employed in historical image analysis [70,71], including:

• Content-based image retrieval: Efficient retrieval and exploration of historical images based on visual similarity and content-based features. However, traditional ML technologies currently require large-scale training data [27,72,73,74], which are only capable of recognizing well-documented and visually distinctive landmark buildings [62] but fail to deal with less distinctive architecture, such as houses of similar style. Even using more advanced ML approaches or combining different algorithms [75] only allows the realization of prototypic scenarios [76,77].
• Image-based localization: Connecting images with the 3D world relevant for AR/VR applications requires estimating the original six-degree-of-freedom (6DOF) camera pose. While several methods exist for homogeneous image blocks [78,79], the problem becomes increasingly complex for varying radiometric and geometric conditions, especially relevant for historical photographs [80].
• Image recognition and classification: Identifying objects, scenes, or people depicted in historical images using deep learning models, such as CNNs. This field ranges from the detection of WW2 bomb craters in historical aerial images [81], via historical photo content analysis [82] to historical map segmentation [83,84,85].
• Semantic segmentation and object detection: Locating and recognizing specific objects or regions of interest within historical images using techniques like Faster R-CNN and YOLO. In semantic segmentation, to classify parts of images [74,86,87].
• Image restoration and enhancement: Repairing and enhancing degraded or damaged historical images through techniques like denoising, inpainting, and super-resolution [88,89].

4.2. AI and Text

Historical texts provide a rich source of information for understanding the past. However, the sheer volume and complexity of historical archives make manual analysis laborious and time-consuming [90]. ML algorithms supported these processes in various ways—from optical character recognition (OCR) to automating the extraction of knowledge and patterns from historical texts [90,91,92]. Approaches include these ML approaches commonly used in historical text analysis:

• NLP techniques: Named entity recognition, part-of-speech tagging, sentiment analysis, and topic modeling. The most recent applications of CNNs and Transformer [93] are consistently successful in accurately extracting and reducing the number of errors even with unsupervised pre-training.
• Text classification algorithms: Naive Bayes, Support Vector Machines, and Random Forests.
• Sequence models: Hidden Markov models, conditional random fields, and recurrent neural networks.

In addition, various preprocessing techniques are used for historical texts to enable their digital processing and respond to challenges such as linguistic variations, archaic vocabulary, and textual degradation:

• Preprocessing: Includes character recognition (e.g., OCR), unification, processing of spelling variations and alignment to controlled vocabularies (e.g., [94]).
• Postprocessing: Used to check and correct any OCR reading errors via neural network approaches [95].

4.3. AI and Virtual 3D Objects

The application of AI in 3D for CH has gained significant attention in the research community to enhance the analysis, interpretation, and preservation of CH in 3D environments. Here are some key areas of scientific analysis:

• Object recognition and classification and semantic segmentation: In 3D/4D reconstruction of CH, ML-based technologies are currently used primarily for specific tasks. This involves AI models to identify specific architectural elements, artifacts, or decorative motifs, to recognize specific objects [72,73,74,96], and to preselect imagery [97,98]. Other tasks include AI-based semantic segmentation techniques to partition 3D models into meaningful regions or components [99].
• 3D model creation: Research has focused on developing AI-based algorithms for efficient and accurate 3D reconstruction of CH objects, buildings, and sites. Traditional algebraic approaches, as in photogrammetry, employ algorithms within equations, e.g., to detect, describe, and match geometric features in images [100] and to create 3D models. ML approaches are currently heavily researched and used for image and 3D point cloud analytics in CH (recent overview: [27]), but increasingly for 3D modeling tasks. Generative adversarial networks (GAN), a combination of the proposal and assessment components of ML, are frequently employed as approximative techniques in 3D modeling, e.g., for single photo digitization [101], completion of incomplete 3D digitized models [102,103] or photo-based reconstructions [104]. Recent approaches include neural radiance fields (NeRF) [105,106,107,108], which have shown strength in creating 3D geometries from sparse and heterogeneous imagery and short processing time [109,110].
• Image to visualization approaches: Approaches bypass the modeling stage to generate visualizations directly from imagery [72,111,112], e.g., by transforming or assembling image content (recent image generators like DALL-E [113], Stable Diffusion or Midjourney). Other approaches based on NeRF to predict shifting spatial perspectives even from single images [114] can predict 3D geometries.
• Use of ML algorithms to detect patterns, anomalies, or changes over time within 3D models (e.g., [54]). The analysis involves assessing the effectiveness of AI in extracting meaningful information from large-scale 3D datasets, supporting archaeological research, conservation efforts, or architectural analysis.

4.4. AI and Maps

The application of AI to cartographic corpora is relatively new and for now primarily addresses the need to segment historical cartography to extract graphs and assign semantic classes to them. To date, these approaches are still entirely manual in many cultural institutions, making it possible to extract useful information on the stylistic-graphic evolution of cartography or graphical elements of the past, such as the road network [115] or the footprints of buildings on a large scale. Recently, the CNN approach has inaugurated some promising lines of study on segmentation [116,117,118]. Historical cadastres provide a stable geometric medium to infer procedural 3D reconstructions [119]. Because of their visual homogeneity, they can be segmented and annotated using CNN and Transformer approaches [120,121].

Another approach to automatically generating 3D/4D models comprises building footprint recognition and parametric modeling. Footprint recognition via semantic segmentation for aerial/satellite imagery [122,123,124,125] or from current cadastral data [126] and for contemporary photography [127] has been frequently researched. One issue in boundary detection workflows is overlapping building boundaries and texts. Consequently, many approaches combine text recognition and boundary delineation [128,129,130,131] to trace building footprints.

4.5. AI and Music

The International Society for Music Information Retrieval defines Music Information Retrieval (MIR) as “a field that aims at developing computational tools for processing, searching, organizing, and accessing music-related data” [132]. MIR utilizes various computational methods such as signal processing, ML, and data mining (i.e., [133]). MIR may use various forms of music data such as audio recordings, sheet music, lyrics, and metadata. Supervised ML relies on the accessibility of large datasets of annotated data. However, the dataset size can be increased by data augmentation. For sound, two data augmentation methods may be used: transformation and segmentation. Sound transformation transforms a music track into a set of new music tracks by applying pitch-shifting, time-stretching, or filtering. For sound segmentation, one splits a long sound signal into a set of shorter time segments [134].

In terms of digital CH and its research, the following areas of MIR are relevant:

• Automated music classification utilizes computer algorithms and ML techniques to automatically categorize music into classes or genres based on features extracted from the music data. Automated music classification has various applications, such as organizing music libraries and archives, and assisting in music research. Music-related classification tasks include mood classification, artist identification, instrument recognition, music annotation, and genre classification. For instance, one study investigates automatic music genre classification model creation using ML [135].
• Optical Music Recognition (OMR) research investigates how to computationally read music notation in documents [136]. OMR is a challenging process that differs in difficulty from OCR and handwritten text recognition because of the properties of music notation as a contextual writing system. First, the visual expression of music is very diverse. For instance, the Standard Music Font Layout [137] lists over 2440 recommended characters and several hundred optional glyphs. Second, it is only their configuration—how they are placed and arranged on the staves and with respect to each other—that specifies what notes should be played. The two main goals of OMR are:

  1.

  Recovering music notation and information from the engraving process, i.e., what elements were selected to express the given piece of music and how they were laid out. The output format must be capable of storing music notation, e.g., MusicXML [138] or MEI [139].

  2.

  Recovering musical semantics (i.e., the notes, represented by their pitches, velocities, onsets, and durations). MIDI [140] would be an appropriate output representation for this goal.

• Automatic Music Transcription (AMT) is the process of automatically converting audio recordings of music into symbolic representations, such as sheet music (e.g., MusicXML or MEI) or MIDI files. AMT is a very useful tool for music analysis. AMT comprises several subtasks: (multi-)pitch estimation, onset and offset detection, instrument recognition, beat and rhythm tracking, interpretation of expressive timing and dynamics, and score typesetting. Due to the very nature of music signals, which often contain several sound sources that produce one or more concurrent sound events that are meant to be highly correlated over both time and frequency, AMT is still considered a challenging and open problem [141].

4.6. AI and Audiovisual Material

Audiovisual heritage includes various materials such as films, videos, and multimedia content. AI for audiovisual heritage supports various aspects of preserving, analyzing, enhancing, and making accessible audiovisual content of historical and cultural significance. Key areas of application for AI in audiovisual heritage include:

• Digitization and restoration: AI assists in digitizing and restoring deteriorating audiovisual materials, improving their quality and preserving their historical significance.
• Video summaries: Can speed up the process of finding content in audiovisual archives [142].
• Content analysis and knowledge extraction: AI algorithms analyze audio and visual elements within content to identify patterns, objects, scenes, speakers, and other relevant information. It can also help to spot biases and contentious terms and track semantic drift in metadata, supporting curators, cataloguers, and others in deciding on potentially updating catalog records [143].
• Metadata enhancement: AI enriches metadata for better content organization, search, and context by extracting keywords or using LLMs to organize and enrich metadata records at scale.
• Transcription and translation: AI-powered speech-to-text transcription and translation services make audiovisual content more accessible and understandable to a wider audience [144].
• Partial audio matching: Supports framing analysis in identifying segments in one source audio file that are identical to segments in another target audio file. Framing analysis can reveal patterns and biases in the way content is being recontextualized in the media to shape public discourse [145].
• Cross-modal analysis: AI techniques analyze both audio and visual components of content, facilitating holistic interpretation and understanding.
• Interactive storytelling and content-generation interfaces: AI-powered interactive narratives and documentaries engage users with historical events and cultural context. AI can further enhance access by using fine-grained and time-based data extracted by AI systems as a basis for creating “generous interfaces” that allow for the rich exploration of CH collections [146,147] and using conversational speech to provide new ways of interacting with audiovisual collections [148].

5. Challenges and Opportunities for AI and CH

5.1. Quality

The analysis of historical images presents unique challenges. These include (a) source degradation and preservation issues related to fading, noise, scratches, and other types of damage present in historical sources; (b) handling diverse formats, resolutions, and color spaces of historical images captured using different cameras and techniques over time; (c) dealing with the scarcity of historical sources poses challenges, e.g., for training ML models.

5.2. Quantity and Historical Singularity

High-quality and diverse datasets are fundamental for training AI models. Particularly in CH, datasets often suffer from limited availability, data gaps, and challenges related to data annotation and standardization [149,150,151]. Unlike in, for instance, medical AI [152], an optimized heuristic interpretation is not sufficient for historical sources and their singularity [27]. Current approaches to employing AI in heritage validate their results only for some examples [153]. Considering the singularity of history, there is a need to establish full-scale cross-validation of AI-based predictions of historical situations. Examples are cross-validating mixed-methods [154] or human-in-the-loop approaches.

5.3. Time and Temporal Transition

Time and non-linear temporal change are the main elements of history and heritage. Current approaches focus mostly on specific timestamps. This is challenging since it requires multiple sources for these reconstructions, often taken at very different times and with each source being a singular document of the represented state. In addition to the issue of inter- or extrapolating sources of different times to gain a coherent historical view, the dating of sources is challenging. Historical imagery is still primarily classified by time via metadata captured at recording or amended at later points. Where metadata are unavailable or uncertain, change detection can be applied to image series, e.g., to assess if undated images show corresponding states of construction to the dated ones. Time and non-linear temporal change are the main elements of history and heritage. Current algorithmic change detection focuses on homogenous quality images, such as time series of satellite images [155,156,157] or aerial photos [158,159]. Approaches for heterogeneous photographs can deal with large-scale changes but are limited to subtle changes (overviews: [160,161,162]). Other change detection approaches work with 3D geometries (overview [163]) or segmentation and feature-based comparison between different images to identify changes in architectural features [80].

5.4. Transparency and Explainable Artificial Intelligence for History and Heritage

As AI models become more complex, explainability and interpretability become crucial in the CH domain. Since algebraic approaches are reproducible, ML approaches are still primarily applied within black box settings with non-transparent decision-making [27,164]. Consequently, a key research focus is explainable AI [165]; understanding the decision-making process of AI systems is essential for building trust and for enabling human experts to verify, validate, and interpret the outcomes generated by AI algorithms.

5.5. Ethical Considerations and Bias

Numerous policy documents target ethics for AI [166,167,168,169]. As a consensus, AI applications in CH should address ethical considerations such as privacy, data security, and cultural sensitivity [170]. AI algorithms should be designed and evaluated to mitigate biases and ensure fairness and inclusivity in CH representation and interpretation.

5.6. Data Availability, Accessibility and Quality

Accessibility and availability of data are big challenges of digital humanities and heritage [171,172], including when data access is limited by legal barriers or company ownership. Privately owned data is potentially at risk of being locked away and inaccessible [173]. In addition, much data is currently not properly accessible due to insufficient tagging, indexing or linking [172]. Despite many attempts to increase the amount of high-quality online data, e.g., through massive digitization campaigns, art historians still have limited access to digital resources containing primary material and good-quality open access visual information, which is digitized and presented according to their preferences and needs. Areas of art history subject to little research, such as digital art history and non-Western art, face greater difficulties with availability. Developers need to understand these scholars’ needs to build appropriate digital resources. In addition, social media companies determine who can access their vast datasets necessary for model training. In the next ten years, we hope to see heritage organizations emerging as strong competitors in this domain, offering access to high-quality, culturally aware, and contextualized datasets. To get there, we need to see concerted advocacy efforts from the European media industry and the research community to radically increase open access to media collections, ensuring that scholars and ML engineers have the right resources and skills to develop AI tools [143].

5.7. Interdisciplinary Collaboration

Promoting collaboration between AI researchers, CH experts, archaeologists, historians and other relevant disciplines is crucial. Bridging the gap between technology and domain expertise can foster innovation and ensure that AI solutions are tailored to CH’s specific needs and contexts. Insights could be retrieved via different methods such as generating, quantifying, and explaining phenomena (e.g., [174]). As concerns grow about biases and social injustices replicated and amplified by commercial AI systems, the interaction of AI experts with social sciences and humanities scholars becomes more significant. The goal is to question current practices and collaboratively develop more equitable solutions. The critical analytical approach that scholars apply when working with AI tools would result in better-tailored research tools and better AI models and practices that could be transferred to wider societal contexts.

5.8. Education

Educational programs on digital heritage are driven by traditional fields such as digital archaeology, digital curation, or digital conservation, as well as related areas, including digital humanities. In addition to higher education, there is a wide spectrum of courses in vocational education (e.g., the EU Codeweek program, DARIAH Teach, PARTHENOS, DHSI, etc.) and frameworks for training and qualification activities (within ERASMUS+, COST etc.) [175]. Due to the rapid technological development in AI and the multitude of tasks for heritage professionals, there is a high demand for multidisciplinary skills and continuing professional development.

5.9. Customization

Users (such as humanities scholars) should be able to tailor and experiment with the parameters of tools, allowing them to refine existing models by incorporating custom concepts relevant to their research. For example, they should be able to fine-tune models through methods like few-shot learning. Additionally, these users should be able to create collaborative experimentation environments, facilitating comparative analyses. This approach would empower researchers to attain more gratifying and meaningful results and enhance their overall confidence in AI techniques, enabling them to engage with them critically [143].

5.10. AI for CH as a Business Sector

Digital heritage is an important business sector, and the provision of digital tools and applications for CH institutions has contributed to the development of many SMEs. The market structure in digital heritage is different from other sectors due to the intangible nature of assets, the niche nature of some markets, and the importance of public funding [176]. Very few medium-sized enterprises are extant, which contrasts with many micro and small enterprises. Consequently, the CH sector needs tailored support instruments, e.g., funding or training, to address its specific AI needs.

6. Strategy and Agenda for Digital Heritage Innovation

We propose a strategy and agenda for AI for heritage and innovation to meet the mentioned challenges. Forecasting actions on AI include:

• The FUTURES4EUROPE, conducted on behalf of the European Commission DG RTD, was a Delfi-like expert review to identify and scope future AI directions [165]
• The Millennium Project developed ideas, strategies, and global governance models for Artificial General Intelligence (AGI) [177].
• AI for archives [178] provide views and demands of this particular subdomain of the heritage sector.
• The Time Machine FET-Flagship CSA conducted various workshops, surveys and scoping activities in 2019 and 2020 to develop a roadmap for large-scale research initiatives [179].
• The ARCHE project reviewed future-oriented literature spanning the environment, economics, health, education, arts and culture, and heritage to identify megatrends, cross-cutting themes and possible opportunities for action for the heritage sector [180]
• ELISE’s 2021 Strategic Research Agenda set out the research challenges that needed to be addressed to strengthen the technical capabilities of AI, improve its performance in deployment, and align AI development with societal interests [181]

Based on these strategies, topics, and challenges highlighted in the previous sections, we would like to propose an AI agenda for CH (cf.

Table 2. R&D Agenda for AI for Cultural Heritage.

R&D AGENDA FOR AI FOR CH Understanding the challenges and opportunities of AI and CH Despite much research, a full understanding of how AI and CH could contribute to each other is still limited. The challenge is to understand the specific challenges and opportunities within the field and identify key research questions and problems that AI can address, such as artifact analysis, preservation, restoration, historical context understanding, and public engagement. Vice versa, CH could contribute to the development of AI regarding specific data and problems, problem authoring, and results interpretation.

Data collection and curation Since data collection and suitable training data is an all-time challenge of AI, CH applications increase the complexity of gathering, annotating, and curating the data to create training sets for AI models taking into account specific CH aspects, e.g., time variance, digitized analog material, or heterogeneous media sets.

Domain-specific AI challenges CH poses some unique challenges to the development of AI applications: Non-linearity of history: Specifically, time variance and singularity of historical sources. Current AI approaches heavily rely on approximation. Heterogeneity of heritage objects: Sparse, incomplete, and heterogeneous data, metadata, and paradata. Complex nature of CH objects: Multiple and often conflicting meanings arise from the historical sources and data.

Domain-specific AI applications Develop and fine-tune AI models tailored to CH tasks such as: Image recognition models for identifying artifacts, styles, and artistic techniques. NLP models for analyzing historical texts and documents. 3D modeling and computer vision for virtual reconstructions. AI tools that assist in analyzing artifacts, identifying patterns, and extracting insights. Algorithms that can determine provenance, age, authenticity, and stylistic influences. Imaging techniques to identify deterioration and suggest restoration approaches. Frame analysis to support media studies research.

Cross-domain opportunities CH comprises a wide variety of AI usage scenarios— from tourism to research and education. A cross-cutting demand and prerequisite for employing AI is to make data connectible and, therefore, employ metadata schemes and vocabularies capable of dealing with different data types and domains.

Context understanding and information enrichment There is an increasing move towards multimodality to include images, texts, and audio into a joint frame of reference, mixed methods combining AI with algebraic approaches, and information enrichment using domain and object-specific understanding to enhance the quality of information (e.g., [182]). Together, these can be used to build AI systems that can contextualize historical artifacts.

Ethical considerations and transparency Biased collections and dominating cultural narratives have been flagged as a major challenge of CH [183]. AI intensifies this challenge by the tendency to replicate dominant features and create limited explainable results [184]. A resultant challenge is to ensure that AI systems respect cultural sensitivities and do not perpetuate biases [167].

Interdisciplinary collaboration CH as a field is marked by high complexity and “fuzzy” problems, which are challenging to transpose into computable approaches [185]. A resultant challenge is to foster collaboration between AI researchers, CH experts, computer scientists, and ethicists to ensure appropriate, high-quality, and meaningful results.

Human in the loop Dealing with CH is still highly influenced by personal expertise and tacit knowledge [173]. It is therefore important to rigorously evaluate AI models’ performance against established benchmarks and human expertise and continuously improve models based on feedback from domain experts.

Long-term sustainability Currently, most heritage data, AI models, and resources are held by companies outside Europe [50]. It is a major challenge to ensure the long-term maintenance, availability, and sustainability of AI tools, data, and platforms and foster open-source and open-data initiatives to not lose control and access to heritage and culture.

Legal and intellectual property considerations CH in Europe is faced with a currently heterogeneous and highly complex legal situation (recently: [185,186]); thus, it is also challenging for AI technologies [4]. A resultant demand is to create and maintain an appropriate legal framework when working with AI for CH.

AI for heritage education Adequate skills have been named as the most important challenge for heritage institutions in the digital realm [187]. Currently, qualifications and skills are mainly taught within academic programs [175]. Against the background of rapid technological developments CH stakeholders need continuous professional development and lifelong learning to be skilled to assess, apply, and reflect on AI.

Heritage innovation support Due to the specifics of the heritage sector, most extant programs to support AI implementation in the European innovation landscape are limited and only applicable to this domain [176]. Intermediaries and tailoring of support offers are needed to successfully connect AI infrastructures, technology providers, financers and the CH sector.

).

7. Summary

Although AI technologies have already been adopted in the CH sector with a multitude of applications all over Europe, the full potential of AI for economic, social, and cultural change is not yet fully visible. This article provides an overview of current AI technologies and applications in the cultural sector and their challenges. Since CH is currently primarily another field of application for AI technologies, it poses several unique challenges for AI research and development and application in innovation contexts. The authors sketch a R&D agenda to guide the next steps of the EIT Climate & Culture STG towards a European AI for CH.

7.1. Discussion

Current trends in AI development such as AGI [177], Explainable AI [188], Human-in-the-Loop [189,190], or recent developments regarding LLMs or computer vision are also of high relevance and applicability to the heritage sector. In addition, several challenges already identified by generic AI roadmapping initiatives are important for the CH sector, too, for example, the need for qualification, evaluation and benchmarking, e.g., of expertise, or the establishment of sufficient legal and ethical frameworks. In addition, there are several unique challenges and opportunities in this area. Specific challenges arise from the diverse, complex and fuzzy nature of heritage and humanities paradigms [185], the incomplete, heterogeneous and sparse information available and the diversity of applications. Vice versa, there may be a unique contribution to the paradigm of humanities and cultural heritage can make to the interpretation and understanding of causalities and singularities, which is still a challenge for AI today (see, e.g., [174]).

7.2. Limitations and Implications

The main scope of this article is to provide an overview of the current state of play and to highlight the specific challenges and opportunities of intertwining cultural heritage and AI. Although this article is based on various mapping activities and research studies, it is not a meta-analysis [191] but designated to provide stakeholders in the field of cultural heritage and AI and authorities and innovators a current and comprehensive overview. As another limitation, the article focuses mainly on a European community, although the challenges and needs highlighted in the article are globally relevant.