ChatGPT and the Generation of Digitally Born “Knowledge”: How Does a Generative AI Language Model Interpret Cultural Heritage Values?

Abstract

The public release of ChatGPT, a generative artificial intelligence language model, caused wide-spread public interest in its abilities but also concern about the implications of the application on academia, depending on whether it was deemed benevolent (e.g., supporting analysis and simplification of tasks) or malevolent (e.g., assignment writing and academic misconduct). While ChatGPT has been shown to provide answers of sufficient quality to pass some university exams, its capacity to write essays that require an exploration of value concepts is unknown. This paper presents the results of a study where ChatGPT-4 (released May 2023) was tasked with writing a 1500-word essay to discuss the nature of values used in the assessment of cultural heritage significance. Based on an analysis of 36 iterations, ChatGPT wrote essays of limited length with about 50% of the stipulated word count being primarily descriptive and without any depth or complexity. The concepts, which are often flawed and suffer from inverted logic, are presented in an arbitrary sequence with limited coherence and without any defined line of argument. Given that it is a generative language model, ChatGPT often splits concepts and uses one or more words to develop tangential arguments. While ChatGPT provides references as tasked, many are fictitious, albeit with plausible authors and titles. At present, ChatGPT has the ability to critique its own work but seems unable to incorporate that critique in a meaningful way to improve a previous draft. Setting aside conceptual flaws such as inverted logic, several of the essays could possibly pass as a junior high school assignment but fall short of what would be expected in senior school, let alone at a college or university level.

1. Introduction

At the time of writing, artificial intelligence (AI) has reached public consciousness, with a wide-ranging debate on its present and potential future abilities, its dangers and the ethics of its usage. All of this was brought about by the public release of DALL-E (an image generator) and ChatGPT in early 2022. Chat Generative Pre-trained Transformer (ChatGPT), a generative AI language model developed by OpenAI, is a type of deep learning model that uses transformer architecture to generate coherent and contextually relevant, human-like responses based on the input it receives [1].

From its formal release in 2018, ChatGPT has undergone several iterations and improvements. In November 2022, ChatGPT 3.5 was released to the general public as part of a free research preview to encourage experimentation. The current version, GPT-4, released in March 2023, was trained (by human trainers) on a dataset of 175 billion parameters and reputedly exhibits greater factual accuracy, reduced probability of generating offensive or dangerous output and greater responsiveness to user intentions as expressed in the questions/query tasks [2]. An analysis has shown that the ChatGPT language model memorized a wide collection of books (fiction, non-fiction and scientific), with the degree of memorization correlated with the frequency with which passages of those texts appear on the web [3,4]. The temporal cut-off for the addition of training data was September 2021, which implies that ChatGPT cannot integrate or comment on events, discoveries and viewpoints that are later than that date. It is asserted in public media, however, that GPT-4 has the ability to search the Internet in real time.

ChatGPT has been shown to be capable of writing lines of code [5], producing poetry [6], short stories and plays [7,8,9] and writing English essays [10], as well as producing simulated scientific content (see below)

1.1. The Use of ChatGPT in Academic Disciplinary Research

There is a growing body of literature that examines the level of knowledge of ChatGPT as reflected in its responses to several fields of research, such as chemistry [11], the use of remote sensing in archaeology [12], architecture [13], diabetes education [14], medicine [15,16,17,18,19], nursing education [20], agriculture [21] and computer programming [22]. Some papers looked at the use of ChatGPT in writing literature reviews [23,24,25].

In addition, some work looked at the “perceptions” of ChatGPT about its future role in some disciplines and professions, such as accounting [26], libraries [27], academia in general [27], medicine [15], medical research [28], health care [29] digital leadership and technology integration [30] or textile manufacturing [31]. Several of these papers make use of the interactive nature of ChatGPT and frame their research as a conversation about the topic [11,12,15,32]. Other research has examined the role and usefulness of ChatGPT in advising or guiding professionals, for example in fields of medicine such as arthroplasty [33,34], nursing [6], dentistry [35], orthopedic [36] and pediatric research [28].

A growing body of research has been examining the effects of ChatGPT on education and academia in general. At the time of writing, there are two discrete strands of thinking: one that considers ChatGPT as a potential device to enhance learning [35,37,38,39,40,41,42,43,44] and one that considers its effect on assignment writing and associated student misconduct [32,40,41,45,46,47,48,49], as well as the integrity of academic writing and publishing in general [50,51,52,53,54,55,56,57,58,59].

As several authors noted, ChatGPT is the typical double-edged sword presented by many new technologies: both useful and detrimental [60]. In response to the threat of AI-generated text to the integrity of assignments and other text, tools have been developed (and are being refined) to assess a block of text for it human vs. AI authorship [61,62]. On the other side, techniques to evade detection are also being examined [63,64].

1.2. ChatGPT in Cultural Heritage Research

Compared to disciplines such as medicine, there has been little formal exploration of the use of ChatGPT in cultural heritage research and management. Thus far, it has been explored in terms of its knowledge of the potential for remote sensing in archaeology [12] and creating plain-language summaries of archeological research reports [65]. Other work looked at the ethical implications of generating “new” content in textual and pictorial form and the limitations ChatGPT (and its image-generation cousin DALL-E) possesses in the cultural sphere [66,67].

The majority of the current discussion on the use of ChatGPT in the cultural heritage contexts occurs in the unrefereed blog sphere, with most of the activity in the museums and collections sphere.

The potential of ChatGPT as a tool for message generation, content marketing and audience interaction can be readily ported to a museum context [68]. A single paper examined the use of ChatGPT in the conceptualization of an entire exhibition [69], highlighting its usefulness while at the same time showing that curator expertise remains required.

It has been posited that that ChatGPT (or future iterations) can be readily used in generic document and content analysis. There is an emerging potential to use sentiment analysis of general visitor enquiries, specific queries and overall feedback to provide an integrated understanding of visitor interest and reactions to specific exhibitions or the museum overall and to track visitor satisfaction [68,70]. In addition, ChatGPT appears well suited to extract pertinent data from longer documents and to provide a succinct summary [71], which could be employed in the creation of exhibition texts [72,73] and scripts for audio guides [73], while data extraction from object inventory data can result in the creation of exhibit labels and catalogue information [73] and museum guides [74]. This can be extended in the form of customized tours of exhibits and museum holdings that reflect a visitor’s personal interests or information needs [68,70]. This is readily implementable given the increased digitization of museum holdings and their presentation to the public in the guise of digital exhibits, which was supercharged by the digital pivot required in response to the shuttering of museums during the COVID-19 pandemic [75,76,77,78,79]. Not surprisingly, some concerns have been raised as to how ChatGPT could affect museum studies projects in school curricula by encouraging plagiarism and reducing active enquiry [80].

While ChatGPT appears to have considerable success in responding to fact-based tasks [37,47], the question arises whether it can provide similarly valid responses when tasked with explaining theoretical concepts.

While some of the limitations of large language models such as ChatGPT are known, with such models occasionally labeled “stochastic parrots” [81,82] (but see [83]), and while it has been argued that ChatGPT cannot reason [84,85,86], there is, at least at the time of writing, no published or preprint information on ChatGPT′s capability of synthesizing and presenting complex concepts beyond its well-documented capacity to extract and summarize factual information. Yet it is this kind of capability that will be very useful, if not essential, to a user who is generally “naïve” about a topic and wants to engage in a conversation-like question-and-answer fashion rather than reading a summary page provided on a general user platform such as Wikipedia.

This paper is informed by the researcher’s longstanding interest in cultural heritage theory and management, and his role as a teaching academic in that discipline (with multi-decade experience) being concerned by the possible (ab-)use of ChatGPT as a tool to generate undergraduate assignments.

1.3. Background: The Nature and Assessment of Values Attributed to Cultural Heritage Assets

In broad terms, cultural heritage is the result of peoples’ interactions, both with each other and with the environment in which they live. The outcome of these interactions expresses itself in a number of forms. Peoples’ interactions with each other result in intangible manifestations which find their expression, inter alia, in language, folklore, sounds, skills, cultural knowledge, performing arts and customs [87,88,89,90], whereas people’s interaction with the environment manifests itself in a tangible form, such as the built and constructed environment, cultural landscapes, resource extraction and refuse sites, as well as in moveable artefacts and objects [88]. Both spheres can overlap where multisensory experiences occur in tangible spaces [91]. Whether these manifestations are deemed cultural heritage depends on the level of importance ascribed to them and the extent to which they are deemed to be important enough to be transmitted on an intergenerational scale by a community or section thereof.

Management processes aim at fostering the conservation and preservation of tangible heritage assets through the identification and intervention of decay processes [92,93,94], as well as the identification of adaptive reuse options for places that have lost their original function [95,96,97] and the maintenance of intangible heritage assets through documentation and in particular through ongoing practice. Despite public rhetoric that espouses the notion that heritage is being preserved for the future [98,99], this preservation of cultural heritage assets occurs to service present-day desires and aspirations. As such, management cannot presuppose future perceptions of heritage significance [100]. Any preservation of assets, however, affords the next generation the ability to make decisions [101].

Whether heritage assets are “worthy” of preservation and conservation is underpinned by how and to what degree individuals, groups of individuals and entire communities value these assets and what level of importance they ascribe to their preservation. Whether cultural practices and tangible assets inherited from past generations are valued by a present-day community is dependent on the degree to which these define or circumscribe the cultural identity of the group or community and to what extent the current cultural identity is anchored in and dependent on these experiences and traditions.

All attributions of value are anthropogenic, and often also anthropocentric constructs that are projected on an inanimate (e.g., tangible heritage sites and objects) or animate human (e.g., intangible cultural heritage) and non-human world (e.g., the natural environment). Consequently, nothing possesses “intrinsic value” (i.e., valuable in and of itself) unless human individuals, singly or as a group, attribute such value onto a given entity. While the basic needs to be met on Maslow’s pyramid are often attributed intrinsic value (such as food, water and shelter), the history of the industrial pollution of air and water demonstrates that the intrinsic values are not universally held. The same applies to the natural environment, which, although ascribed intrinsic value by ecologists and ethicists [102], has long been regarded as exploitable in a knowing fashion, even to the extent of the extinction of species or the exhaustion of natural “resources”. In view of their anthropogenic origin, both the nature and the perceived importance of values will differ between individuals and groups of people, wherein cultural traditions and group identity shape, define and consolidate these values. Humans as social creatures are embedded in a complex web of past and present interpersonal, intra- and inter-group and community relationships. These relationships result in—and shape—a person-centered equally complex web of values that we project on the various expressions of heritage and that are contingent in strength on the valuer’s position in their social space at that point in time. Differences in the perception of values, which can be perceived as positive, neutral or negative, may result in value trade-off or, where non-negotiable, in value conflicts [103,104,105].

The authorized heritage discourse (sensu Smith) is founded on four axiomata, namely that tangible cultural heritage assets are scarce; finite; non-renewable and valuable. It posits that heritage assets are inherently valuable, with some authors even attributing intrinsic value [106]. The authorized heritage discourse commonly relies on the identification and assessment of four sets of heritage values, aesthetic, historic, scientific and social value, which are enshrined in many formal charters, standards, regulations and guidelines [107,108,109,110,111,112]. The assessment of these values leads to a determination of heritage significance, from which all management and conservation actions flow (or the abstention therefrom). The assessment of these values, however, is prone to be influenced by the dominant culture and by professional practitioners [88,113,114,115,116]. While some of this can be overcome with broad-based consultation or even community-based assessment and co-creation [109], intra-group dynamics will influence the process [117].

Heritage values and the significance derived therefrom are neither universal nor static. Rather, as expression of a culture, they are culturally relative entities [118,119] and subject to an individual’s cultural positioning [120] and also mutable entities due to changing professional [121,122] and intergenerational perceptions and perspectives [123]. Indeed, the past two decades have seen a widened understanding of the interpretation and application of values, in particular in terms of gender [124,125] and Eurocentrism [126,127,128], with studies looking at the epistemological basis of the various conceptualizations and descriptors of values attributed to heritage (instrumental, authenticity, etc.) [129].

While most scholars see the instrumental value of heritage in terms of identity [130,131,132,133], the tourism literature in particular examined the instrumental value of heritage, both in its tangible and intangible manifestations to the tourism product and the economics derived therefrom [134,135,136].

Any discussion of the specifics of heritage values projected on tangible and intangible manifestations will depend on the cultural positioning and standpoint of the valuer. Although numerous approaches are possible, all discussions should cover aspects of the authorized heritage discourse, cultural relativity, the subjectivity of assessment and the mutability of values.

While people can balance these aspects and can provide an essay with a nuanced exposition, it is unclear to what extent AI can do the same. This paper will examine the ability of ChatGPT to write a comprehensive essay which discusses the nature of values used in the assessment of cultural heritage significance and to assess the extent of ChatGPT’s understanding of the topic and its ability to provide a nuanced discussion.

The approach taken was to emulate a generic query akin to that posed by a generally “naïve” user, such as a member of the general public or an uncritical undergraduate student pressed for time, requesting a longer (1500 word) exposition on the topic, rather than to generate a brain-storming response for an already informed and knowledgeable user who would be able to adjust the response output through informed, iterative prompting.

2. Methodology

2.1. Data Generation

2.1.1. Essay Task

The study used OpenAI’s ChatGPT 4.0, May 24 version (https://chat.openai.com accessed 16 June 2023), to generate an essay with the following set task: “Write a 1500 word essay that discusses the nature of values used in the assessment of cultural heritage significance. Provide references”.

Run Set A: Twelve iterations of the task were run on 16 June 2023 between 12:45 and 13:15 AEST (2:45–3:15 GMT). The system was prompted to “continue generating” until the essay was completed. Once completed, the text was copied and the system was requested to regenerate the response using the provided button. No feedback was provided to the system as to whether the completed task was deemed to be adequate or not. During the run set the “chat” was left open, whereby ChatGPT added the new versions and retained the previous ones, allowing the user to backtrack if needed. At the end of the run set the “chat” was deleted.

Run Set B: This run set was a replicate of Run Set A, with twelve iterations of the task run on 16 June 2023 between 14:00 and 14:15 AEST (4:00–4:13 GMT).

Run Set C: Twelve iterations of the task were run on 17 June 2023 between 12:45 and 13:00 AEST (2:45–3:00 GMT). As before, the system was prompted to “continue generating” until the essay was completed. Once completed, the text was copied. In this set, however, the “chat” was deleted after each task completion and each new “chat” was initiated.

2.1.2. Reference Query

On 17 June 2023 at 14:35 AEST (4:35 GMT) ChatGPT was tasked with the following request: “Cite 20 references on cultural values in cultural heritage management”. Once these were delivered (including being prompted to “continue generating”), ChatGPT was tasked with the follow up request “can you cite 20 more?”, following which 20 more references were delivered (including being prompted to “continue generating”). Following the delivery, ChatGPT was asked for detail on the origin of the references (“Where did you get these references from?” and “Did you source some of these references from Wikipedia?”).

After the previous chat was deleted, a new chat was initiated at 14:43 AEST (4:43 GMT) with the query “Cite 50 references on cultural values in cultural heritage management”. ChatGPT baulked at the request due to server-demand issues (“providing 50 references in a single response would be quite overwhelming and space-consuming”) and offered 10 references. It then provided additional references in response to the follow-up request of “can you provide me with 30 more?”.

The veracity of all references was ascertained through title searches in GoogleScholar.

2.2. Data Analysis

The technical aspect manuscript files were analyzed using the editorial reporting functions in Word for Microsoft 365 that provide descriptive data on the text, such as the number of words, paragraphs and sentences, as well as reading levels.

The content of the essay manuscripts was analyzed in terms of the coverage of topics and the structure, coherence and complexity of the argument.

2.3. Data Documentation

All conversations with ChatGPT used in this paper have been documented according to a protocol [137] and have been archived as a supplementary data file at XYZ (to be inserted upon publication).

3. Results

The results will be presented in the following order. First, we will address the technical aspects of essays generated by ChatGPT in terms of the reading age and word count, as well as the references that were cited in these essays. This is followed by a discussion of the results of the reference queries. The section concludes with an examination of the essay response with specific focus on the coverage of topics and the structure, coherence and complexity of the argument.

3.1. Technical Aspects of the Essay Task

3.1.1. Reading Age and Wordcount

Although ChatGPT had been tasked with writing a 1500-word essay, three quarters of the generated essays were of less than half the required length (Figure 1), with an overall average of 710.9 ± 60.0 words (n = 36). The longest essay comprised 933 words (

Table 1. General statistics of the documents analyzed in this paper.

	Word Count of Text		Flesch–Kincaid Grade Level
Run Set	Avg ± StdDev	Min–Max	Avg ± StdDev	Min–Max
A	723.17 ± 62.92	646–885	17.94 ± 0.97	16.50–19.50
B	716.75 ± 94.81	601–933	17.33 ± 0.76	16.20–18.90
C	692.67 ± 40.56	627–747	18.03 ± 0.96	16.00–19.30

) (for general descriptor of each essay, see Appendix A,

Table A1. General descriptors of the documents analyzed in this paper.

Iteration	Word Count	References	MS Word Editor Score %	Paragraphs	Sentences	Sentences/ Paragraph	Words/ Sentence	Flesch Reading Ease Level	Flesch– Kincaid Grade Level
A01	695	5	97	10	38	4.2	18.2	5.6	17.0
A02	885	5	100	18	52	3.0	16.5	0.0	17.5
A03	675	4	97	13	44	3.6	15.2	0.0	18.2
A04	752	4	98	9	39	4.8	19.2	0.0	19.1
A05	705	6	96	9	32	4.0	22.0	0.0	19.5
A06	726	5	95	11	37	3.7	19.5	2.9	17.7
A07	690	5	100	11	39	3.9	17.6	0.9	17.5
A08	766	4	98	13	43	3.5	17.7	5.2	16.9
A09	698	4	97	15	36	2.5	18.5	1.0	17.7
A10	762	6	94	12	44	4.0	17.2	7.4	16.5
A11	678	4	100	9	39	4.8	17.8	0.0	18.5
A12	646	5	99	9	30	3.7	21.3	0.0	19.2
B01	667	5	100	26	44	2.2	14.0	2.0	16.4
B02	766	5	100	29	39	2.6	18.1	0.0	17.9
B03	750	5	95	21	38	3.4	18.2	4.1	17.2
B04	677	5	100	23	38	3.4	16.6	8.2	16.2
B05	764	5	100	28	46	2.0	15.7	3.5	16.6
B06	933	2	97	26	44	2.3	19.4	4.4	17.4
B07	615	1	100	17	34	4.2	17.3	5.0	16.7
B08	723	4	96	12	37	3.3	19.2	0.0	18.9
B09	637	5	95	8	31	4.4	20.5	4.1	17.6
B10	658	5	97	19	33	2.5	19.1	0.0	17.9
B11	810	6	96	13	42	3.5	19.2	3.5	17.5
B12	601	5	97	8	34	4.8	17.6	0.2	17.6
C01	747	5	93	22	37	3.0	19.5	14.9	16.0
C02	713	5	100	10	34	3.7	20.9	0.0	19.1
C03	729	5	100	17	38	3.4	18.4	1.2	17.6
C04	627	5	96	7	29	4.8	21.5	0.0	18.9
C05	664	4	100	11	35	3.5	18.9	9.0	17.7
C06	641	5	97	13	33	2.7	19.0	6.8	16.8
C07	658	4	99	12	35	3.1	18.7	0.6	17.6
C08	735	5	95	10	36	4.0	20.3	0.5	18.2
C09	661	3	99	19	32	1.7	20.1	0.0	19.3
C10	727	8	92	10	38	4.2	19.1	0.0	18.5
C11	697	2	97	14	38	1.9	18.3	0.0	18.6
C12	713	4	96	8	37	5.2	19.2	0.0	18.1

).

The complexity of a given text and the associated reading age or grade level can be assessed with numerous metrics. The most common of these is the Flesch–Kincaid readability test, which takes into account the number of sentences, words and syllables in the words [138] and which expresses the complexity in terms of (US) grade levels. This ranges from 4.8 to 6.5 for recreation books, 13 to 15 for textbooks used in tertiary education and in excess of 17 for scientific writing [139,140,141]. Among the latter category, a trend of increasing complexity has been observed [141]. The Flesch–Kincaid score for the essays written by ChatGPT ranged between 16 and 19.5, with an overall average of 17.8 ± 0.9 (n = 36; Table 1).

3.1.2. References Cited in the Essay Task

As part of the essay task, ChatGPT was asked to provide references. The majority of the iterations generated by ChatGPT cited between four and five references per iteration (Figure 2), which, with few exceptions, were presented in alphabetical sequence. Where not in alphabetical order, the sequence of genuine references seems to reflect the order in which sources were drawn upon.

With two exceptions, none of the statements in the essay texts were referenced within the paragraphs, with references given in the form of a bibliography at the end. In both instances, where essays referenced assertions made in the text within the paragraph (essays A4 and B12), each paragraph of the body of the essay (excluding the introduction and the conclusion) carries one reference. The references are broadly contextual, but the relevance of each citation was drawn from a key term in its title (with one additional fictitious reference in essay B12).

ChatGPT explicitly commented in one essay (A1): “It will draw upon scholarly references and case studies to support the discussion”. The reader has to assume that the references are genuine and generally there is no indication by ChatGPT that this may not be the case. Only one iteration (B9), which cited a single, fictitious reference, added the following caveat at the end the manuscript text: “Note: This essay is a product of an AI language model and the references provided are fictional. Please consult academic sources for authentic references on the topic”.

To a casual user who is not firmly familiar with the literature, however, all references appear valid because (i) the titles appear plausible; (ii) journal titles are those of genuine publications and (iii) and the vast majority of authors’ names were those of academics publishing in the fields of cultural heritage or archaeology or were from cultural heritage organizations such as UNESCO or ICOMOS. Only 6.0% of the authors “cited” did not publish in the discipline, with another 1.8% being dual non-existent author combinations where one author did publish in the relevant fields.

The veracity of all references was verified through title searches in GoogleScholar. Depending on the run set, between 40% and 56.4% of the references existed, while between 23.6% and 40% of the references were entirely fictitious (

Table 2. Nature of the references cited in the documents analyzed in this study.

	Run Set
Reference	A	B	C
exists	40.0	52.8	56.4
exists, but wrong year	13.3	20.8	12.7
exists, but wrong URL	3.3	—	3.6
exists, but wrong year and URL	3.3	—	3.6
fictitious (constructed)	40.0	26.4	23.6
n	60	53	55

).

Seven references occurred more than twice, all of which are genuine:

• Smith, Laurajane. Uses of heritage. Routledge, 2006 (25 “citations”).
• ICOMOS Australia. (2013). The Burra Charter: The Australia ICOMOS charter for places of cultural significance 2013. Burwood, Vic: Australia ICOMOS Inc. International Council of Monuments and Sites (15 “citations”).
• United Nations Educational, Scientific and Cultural Organization (UNESCO) (1972). Convention Concerning the Protection of the World Cultural and Natural Heritage. version: Paris (1972) (eight “citations”).
• United Nations Educational, Scientific and Cultural Organization (UNESCO). “Convention for the safeguarding of the intangible cultural heritage”. Paris (2003) (seven “citations”).
• Waterton, Emma, and Laurajane Smith. “The recognition and misrecognition of community heritage”. International journal of heritage studies 16.1-2 (2010): 4-1 (seven “citations”).
• Waterton, Emma, and Steve Watson, eds. Heritage and community engagement: Collaboration or contestation? Routledge, 2013 (five “citations”)
• Bandarin, Francesco, and Ron Van Oers. The historic urban landscape: managing heritage in an urban century. John Wiley & Sons, 2012 (four “citations”).

An examination of the non-existent references shows that these were generated using the names of real authors working in the field (in the main) with fragments of real article titles or journal or publisher names and DOIs to construct false but realistic looking references. An example is the following reference (iteration A5, reference 4):

Matero, F. (2010). Cultural Heritage Conservation and Environmental Impact Assessment by Nancy Odegaard, Scott Carroll, Werner Zimmt, with Katherine Rankin. Journal of the American Institute for Conservation, 49(1), 65–66. doi:10.1179/019713610803315317

This reference can be deconstructed as follows:

Matero, F.	Frank Matero (genuine author)
(2010).	Plausible year
Cultural Heritage Conservation and Environmental Impact Assessment by	Fragment taken from Van Grieken, R., & Janssens, K. (Eds.). (2004). Cultural heritage conservation and environmental impact assessment by non-destructive testing and micro-analysis. CRC Press.
Nancy Odegaard, Scott Carroll, Werner Zimmt.	Fragment taken from Nancy Odegaard, Scott Carroll, and Werner Zimmt. Material characterization tests for objects of art and archaeology. 2000
with Katherine Rankin.	various sources possible
Journal of the American Institute for Conservation	genuine journal title
49	correct journal volume number for the year 2010
(1)	issue 1 exists
65–66.	formal issue ends with page 64, end matter on pp. 65–66
doi:10.1179/019713610803315317	non-existent DOI

3.2. Results of the Reference Queries

3.2.1. Query 1: 20 plus 20 References

Upon request, ChatGPT provided a set of twenty references and, when prompted, another set of twenty. The veracity of these references was again ascertained through title searches in GoogleScholar. The accuracy references were classified according to four categories: title, author, year and publisher (or journal).

Of the forty references, twelve publications did not exist at all, but their fictitious titles had been constructed (set 1: four; set 2: eight). As before, all these fictitious titles sounded quite plausible. Nine of the references provided an existing title but incorrect authors (set 1: five; set 2: four), while another ten provided an existing title with the correct authors (and commonly the correct publisher) but offered the wrong publication year (set 1: five; set 2: five). Only nine references had correct publication details (set 1: six; set 2: three). The references included the names of 50 authors, 4 (8%) of which have not published in the fields of cultural heritage or archaeology.

ChatGPT provided the following caveat at the end of the first set of 20 references: “Please note that while I have provided the references, it is always a good practice to review and evaluate the sources for their relevance and credibility before using them in academic or professional work” and the following caveat after the second twenty references “Remember to evaluate the sources for their relevance and credibility before using them in academic or professional work.” At no point did ChatGPT offer any indication that over half of the references were fictitious or seriously flawed (incorrect authors).

3.2.2. Query 2: 50 References

When ChatGPT was tasked to generate “50 references on cultural values in cultural heritage management”, the system baulked and offered ten references (set 1) but provided an additional thirty (set 2) when prompted. Of the forty references thus generated, six publications did not exist at all but had constructed fictitious titles (set 1: four, set 2: two). As before, all of these fictitious titles sounded quite plausible. An additional two titles were mis-constructed by merging the titles of the article and the book into one (both set 2). Four of the references provided an existing title but incorrect authors (set 1: one; set 2: three). Another four provided an existing title with the correct authors (and commonly the correct publisher) but offered the wrong publication year (all set 2). Fourteen of the references had had correct publication details (set 1: five; set 2: nine), with two publications doubled up (both set 2)

The references included the names of 40 authors, 8 (20%) of which have not published in the fields of cultural heritage or archaeology.

3.3. Nature of the Essay Response

3.3.1. Coverage of Topics

The topics covered in the exposition section of each essay were extracted and classified as value descriptors and as assessment concepts using the primary terms provided by ChatGPT. No attempt was made to integrate or assign them to classes (for individual data see Appendix A,

Table A2. Coverage of value topics in the various iterations: ◊—mentioned as subset of another value term; •—prominently and explicitly mentioned.

Iteration	Historic Value	Scientific Value	Social Value	Aesthetic Value	Spiritual/Religious Value	Artistic Value	Educational Value	Economic Value	Tourism Value	Minority/Multicultural Values	Environmental Values	Indigenous Values	Individual/Personal Values	Community/Collective Values	Identify Value	Nostalgia/Emotional Value	Tangible vs. Intangible Values	Cultural Value	Heritage Value	Instrumental Value	Utilitarian Value/Use Value	Intrinsic Value	Extrinsic Values	Associative Value	Contextual/Relational Value	Universal Values
A1	•		•			•
A2	•	•	•	•		•		•	•	•				•
A3	•		•								•											•
A4	◊	◊	◊		•		◊	◊		•									•	•		•
A5												•					•
A6	•		•	•				•		•								•
A7	•	•	•	•	•					•	•											•
A8	•	•	•	•
A9			•																		•	•
A10	◊	◊		◊										•			•
A11	◊	◊	◊	◊	◊		◊	◊						◊				◊			•	•		•	•
A12	•	•	•	•
B1	•		•	•				•										•
B2	◊	◊	◊	◊	◊	◊		◊	◊											•	◊	•			•
B3	◊	◊				◊							•			•	•
B4	◊	◊	◊	◊	◊	◊																•		•	•
B5	•	•	•	•	•
B6	•	•	•	•									•
B7	◊	◊	•	◊	•			•					◊				•					•		•	•
B8										•										•		•
B9	◊	◊	◊	◊				◊		◊
B10	•	•	•	•				•																•
B11	◊		◊	◊	◊		◊	◊	◊			•								•		•			•
B12	•	•	•	•				•
C1	•	•	•	•																				•
C2	•		•			•	•	•	•								•	•
C3	◊		◊	◊	◊
C4								◊	◊								•
C5	◊	◊	◊		◊					•			◊	•			•									•
C6	•		•	•														•
C7	◊	◊	◊	◊	◊																				•	•
C8	•		•				•	•	•		•				•		•	•
C9	•		•	•				•			•
C10												◊					•					•	•		•
C11														•						•		•		•
C12

).

When considering the value descriptors, those four values which are associated with the authorized heritage discourse (aesthetic, historic, scientific and social value) dominate, with all four being reflected in over half of all essays (

Table 3. Coverage of value descriptors in the various iterations.

Concept	A	B	C	Overall
Historic value	83.3	91.7	66.7	80.6
Social value	83.3	83.3	66.7	77.8
Aesthetic value	58.3	83.3	41.7	61.1
Scientific value	58.3	75.0	25.0	52.8
Economic value	33.3	58.3	33.3	41.7
Intrinsic value	41.7	41.7	16.7	33.3
Spiritual/religious value	25.0	41.7	25.0	30.6
Tangible vs. intangible values	16.7	16.7	41.7	25.0
Minority/multicultural values	33.3	16.7	8.3	19.4
Contextual/relational value	8.3	33.3	16.7	19.4
Value hierarchies/conflicts	27.8	41.7	19.4	19.4
Artistic value	16.7	25.0	8.3	16.7
Tourism value	8.3	16.7	25.0	16.7
Cultural value	16.7	8.3	25.0	16.7
Associative value	8.3	25.0	16.7	16.7
Educational value	16.7	8.3	16.7	13.9
Community/collective values	25.0	—	16.7	13.9
Instrumental value	8.3	25.0	8.3	13.9
Environmental values	16.7	—	16.7	11.1
Individual/personal values	—	25.0	8.3	11.1
Indigenous values	8.3	8.3	8.3	8.3
Utilitarian value/use value	16.7	8.3	—	8.3
Universal values	—	—	16.7	5.6
Identify value	—	—	8.3	2.8
Nostalgia/emotional value	—	8.3	—	2.8
Heritage value	8.3	—	—	2.8
Extrinsic values	—	—	8.3	2.8
n	12	12	12	36

). Among these, historic value dominates, followed by social value. The only other value concept that has more recently been recognized in the heritage assessment literature, that of spiritual and religious value, is mentioned in just under a third of the essays. All other value descriptors are either uncommon constructs (e.g., “identity value”, “tourism value” and “tangible vs. intangible values”) created by ChatGPT, are concepts that are discussed but not generally recognized in the heritage assessment literature (e.g., “intrinsic value” and “economic value”) or are value descriptors that are commonly regarded as subsets or exemplifications of overarching concepts (e.g., “educational value” and “artistic value”).

When considering concepts for assessing values and their limitations, three concepts recur in more than half the essays: the subjectivity of values, the concept of the evolving societal perspectives and stakeholder engagement (

Table 4. Coverage of value assessment concepts in the various iterations.

Concept	A	B	C	Overall
Subjectivity of values	83.3	50.0	66.7	66.7
Evolving societal perspectives	50.0	58.3	50.0	52.8
Stakeholder engagement	50.0	50.0	50.0	50.0
Cultural relativism	75.0	33.3	33.3	47.2
Value hierarchies/conflicts	58.3	33.3	25.0	38.9
Authenticity	41.7	16.7	33.3	30.6
Heritage significance	25.0	33.3	25.0	27.8
Ethical considerations	25.0	25.0	25.0	25.0
Assessment frameworks	16.7	16.7	41.7	25.0
ICOMOS	16.7	8.3	41.7	22.2
Community perspectives	—	33.3	25.0	19.4
Intergenerational mutability	16.7	33.3	—	16.7
Multidimensionality of values	8.3	25.0	16.7	16.7
Community identity	25.0	16.7	8.3	16.7
Ownership and repatriation	8.3	16.7	16.7	13.9
Plurality of values	8.3	8.3	16.7	11.1
Global vs. local perspectives	8.3	16.7	8.3	11.1
Professional perspectives	—	16.7	16.7	11.1
Integrity	8.3	8.3	16.7	11.1
Contemporary significance	—	16.7	8.3	8.3
Sustainable development	—	16.7	8.3	8.3
Eurocentrism	—	8.3	—	2.8
Objective criteria	—	8.3	—	2.8
n	12	12	12	36

). A fourth, cultural relativism, was mentioned in just under half of the essays (47.2%), followed by value hierarchies and value conflicts (38.9%).

A correlation of the inclusion of the values of the authorized heritage discourse against the five most frequent concepts for assessing values follows the pattern of overall representation, and no deviations from the expected pattern emerge (

Table 5. Correlation of the inclusion of values of the authorized heritage discourse against concepts for assessing values (n = 36).

	Value
	Historic	Social	Aesthetic	Scientific
Subjectivity of values	52.8	52.8	41.7	33.3
Evolving societal perspectives	38.9	38.9	30.6	30.6
Stakeholder engagement	36.1	36.1	30.6	22.2
Cultural relativism	36.1	36.1	33.3	30.6

).

3.3.2. Structure of the Essay Argument

Given the concept of ChatGPT as a generative AI language model, it is not surprising that each of the 36 essays are different. All essays written by ChatGPT follow the generally accepted practice of constructing an essay (introduction, exposition and conclusions), with the exposition section commonly structured as a description and definition of a select set of values to be followed by broader concepts and limitations.

In all three iterations, ChatGPT starts the essay with a paraphrased definition of cultural heritage, which is expressed in terms of “Cultural heritage encompasses…” (12 iterations), “Cultural heritage is a collective expression of …” “human creativity and history”, “reflection of a society’s history, values, and identity”, “a rich and diverse tapestry of human achievements”, ”a vital aspect of human civilization”, “Cultural heritage plays a crucial role in defining/preserving/shaping a society’s identity” and “Cultural heritage holds immense value…”. This is then followed by a sentence that refers to cultural heritage significance and is relevance. The final sentence of the introduction routinely commences with “[t]his essay aims to explore/examines/delves into the nature of values used in the assessment of cultural heritage significance”, heavily drawing on the set task (“Write a 1500 word essay that discusses the nature of values used in the assessment of cultural heritage significance”).

The expositions of essays fall into four groups: one group focusing on the values espoused by the authorized heritage discourse (sensu Smith); one group that framed their exposition in overarching, more fundamental concepts of values (intrinsic vs. instrumental) values; one group that framed their exposition in terms of heritage significance; and one group that started their exposition with commentary on the subjectivity of values.

Slightly more than one third of the essays (14) focused on an exposition of various discrete values as utilized in the authorized heritage discourse. Half of these prefaced the exposition with a framing that defined cultural heritage values and their multi-faceted nature (

Table A4. Essay structures Group 1, Set A.

Essay C1	Essay C6	Essay C9	Essay A2	Essay B6
Defining Values	Defining Values	Defining Values	Multifaceted Nature of Values	Multifaceted Nature of Values
Associative Aesthetic Historic Scientific Social	Historical Aesthetic Social and cultural	Historic Aesthetic Social Economic Environmental	Historic and historical Social and community Scientific and research Economic and tourism	Historic Artistic Social Scientific
Relativity of values Balancing conflicting values	Multiple perspectives and stakeholders Cultural relativism and contextual understanding Evolving values and reassessment	Subjectivity and multiple perspectives Indigenous and local perspectives Global perspectives Ethical considerations Ownership and repatriation	Subjective nature Shifting societal values Stakeholder engagement Ethical considerations Challenges in assessing Value hierarchies and trade-offs Lack of data and knowledge gaps Global vs. local perspectives	Subjectivity Temporal perspectives Individual interpretations Challenges in assessing Conflicting values Power imbalances Global vs. local dynamics Incorporating values in heritage assessment Participatory approaches

). Every essay either commented on subjectivity or cultural relativism. Of the others, seven framed the exposition in terms of cultural heritage significance, following the standard heritage discourse of assessment and role played by the various values (

Table A5. Essay structures Group 1, Set B.

Essay A8	Essay B5	Essay A1	Essay B1	Essay A7	Essay C2	Essay B10
						Subjectivity of heritage assessment
Aesthetic Historic Scientific Social	Historic Aesthetic Social Spiritual and religious Scientific	Historic Artistic Social	Historical Cultural Aesthetic Social	Intrinsic Historical Aesthetic Social Spiritual and religious Scientific	Cultural, historical and aesthetic v. Authenticity and integrity Social and community v. Educational and interpretive v. Economic and touristic v. Ethical and intangible v.	Historical and associative Aesthetic Social and community Scientific and research Economic
Subjectivity Cultural and contextual relativity Shifting and evolving values	Subjectivity Cultural relativism Challenges in assessing values Ethical considerations Evolving values Conflicting values	Subjectivity and cultural relativism Changing values Contemporary significance Challenges in assessing values	Economic decision-making processes Stakeholder engagement Sustainable management	Cultural Diversity Multicultural values		Challenges in assessing values Inclusive and intergenerational perspectives

). Three commented on subjectivity or cultural relativism, while the others diverged considerably. The remaining essays simply described the heritage values without specific framing, one of which did so without any contextualization or qualification (

Table A6. Essay structures Group 1, set C.

Essay A3	Essay A6	Essay B12
Intrinsic Historical Social Environmental	Cultural Historical Social Economic Aesthetic	Historic Artistic Social Scientific Economic
Subjectivity and context Cultural relativism Community engagement Ethical consideration Ownership and repatriation Sustainability and conservation	Challenges and implications

).

A second group of seven essays framed their exposition in overarching, more fundamental concepts of values. These essays lead off with a description of intrinsic values followed by a section on instrumental values (five essays). Thereafter, the expositions diverge considerably without any discernable clear pattern, generating individual essays that all cover different aspects with a different emphasis (

Table A7. Essay structures Group 2.

A4	B2	B8	B11	C11	B4	A11
Intrinsic	Intrinsic	Intrinsic	Intrinsic	Intrinsic	Intrinsic	Intrinsic
Instrumental	Instrumental	Instrumental	Instrumental	Instrumental	Contextual	Contextual
Heritage	Relational	Process	Contextual	Associative	Associative	Instrumental Associative
Cultural relativism and values Challenges and controversies	Sources of value Cultural and community perspectives Expert and institutional perspectives Contemporary contexts and public opinion Challenges in assessing cultural bias and Eurocentrism Balancing universal and local values Ethical considerations	Implications Balancing priorities Cultural diversity and inclusivity Sustainable development Ethical considerations	Subjectivity Stakeholder perspectives Local communities Government and regulatory bodies Indigenous groups	Multiple perspectives Experts and professionals Government and policy makers Assessing values Burra Charter Interpretation Charter *)	Cultural perspectives Stakeholder engagement Evolving nature of values Role of legislation and international charters	Cultural relativity and subjectivity Inclusive and holistic approaches

* ICOMOS, International Cultural Heritage Charter for the Interpretation and Presentation of Cultural Heritage Sites.

).

A third group of five essays framed their exposition in terms of heritage significance (Table A7), but then widely diverged in their line of argument (A10, B3, B7, C5 and C10). Three additional essays framed their exposition as “understanding cultural heritage values” (A5, A9 and C7). Common to all is that the level of exposition also involves overarching, more fundamental concepts of values, with common aspects of subjectivity and cultural relativism. Again, ChatGPT generated individual essays without any discernable clear pattern that all cover different aspects with a different emphasis.

The fourth group started their exposition with commentary on the subjectivity of values, with two essays progressing to comment on the diversity of values and their evolving nature (Appendix B,

Table A9. Essay structures Group 4.

Essay B9	Essay C4	Essay C8	Essay C12
Subjectivity of v.	Subjectivity of v.	Subjectivity of v.	Subjectivity of v.
Diversity of v. Evolving nature of v.	Diversity of v.	Tangible and intangible v. Cultural, social and identity v. Historical and educational v. Economic and touristic v. Environmental and natural v. Evolving and dynamic nature of v.	Contextuality of v. Evolving nature of v. Multiple dimensions of v. References in assessing v.
Cultural heritage Legislation and policies Case study	Role of cultural heritage v. in preservation Challenges in assessing

). Again, all essays are individual in their approaches and emphasis. One of these included a case study (Sydney Opera House), an approach that had not been taken by any of the other iterations (essay B9).

The remaining two iterations (essays A12 and C3) commenced with a discussion of the multidimensional nature and the subjectivity cultural heritage values and then also diverged, again discussing overarching, more fundamental concepts of values.

3.3.3. Coherence and Complexity of the Argument

When considering the essays, the two framing sentences in the introduction set out the general focus of the paper, while the following sentence merely repeats the essay task. While the essays broadly follow what the first two sentences of the introduction signal is the focus of the essay, the exposition itself often does not follow this. While the exposition section is generally structured as a description and definition of a select set of values (or criteria) followed by broader concepts and limitations, there is no coherent line of argument. Instead, the exposition is a sequence of discussion points that are commonly presented in an unconnected form. The discussion section seemingly provides a generic summary, but when examined more closely, reflects only part of the exposition and does not tie back to the aims as stated in the introduction. The following example (essay A7) is one such example.

“Introduction: Cultural heritage plays a vital role in shaping our identities, understanding our history, and fostering a sense of community. As societies evolve, it becomes crucial to assess and evaluate the significance of cultural heritage sites and objects. However, the assessment of cultural heritage significance is a complex process that involves a range of values. This essay explores the nature of values used in the assessment of cultural heritage significance, highlighting their diversity and the influence they exert on the preservation and interpretation of our shared heritage”.

The introduction is straightforward and sets up expectations regarding a discussion of the nature and diversity of values, but also their role in heritage preservation and interpretation. The following exposition presents eight topics as numbered dot points in the following order:

• Cultural Heritage Significance.
• Intrinsic Values.
• Historical Values.
• Aesthetic Values.
• Social Values.
• Spiritual and Religious Values.
• Scientific Values.
• Cultural Diversity and Multicultural Values.

The lead paragraph, which clearly follows the authorized heritage discourse, notes that “Cultural Heritage Significance: Cultural heritage significance refers to the value and importance ascribed to a particular heritage site or object. Significance assessment enables us to understand and prioritize cultural heritage resources for conservation, management, and interpretation. It involves examining both the tangible and intangible aspects of cultural heritage, including historical, aesthetic, social, spiritual, and scientific values”.

Instead of following its outline sequence, ChatGPT first proceeds to discuss “intrinsic values”, which it constructs as follows: “Intrinsic values refer to the inherent qualities and characteristics of cultural heritage that make it valuable in and of itself. These values are independent of human perceptions and interactions. For instance, the age, rarity, and authenticity of an artifact contribute to its intrinsic value. Intrinsic values are essential in determining the uniqueness and irreplaceability of cultural heritage, guiding decisions on preservation and conservation.”

Setting aside the problematic definition of “intrinsic value” (see Discussion), there has been no lead-in that considers the anthropogenic, and often anthropocentric, nature of values, within which the concept of intrinsic value needs to be situated. Rather than building on this by considering and outlining instrumental values, ChatGPT picks up its outline and presents the “standard” values as prescribed by the authorized heritage discourse: historical, aesthetic, social, spiritual and religious and scientific values. Each of these sections is written as a standalone paragraph without any connection to the previous or the following section. There is also no attempt made to arrange the sequence in in a logical structure, such as presenting the values with increasing or decreasing levels of subjectivity.

The final numbered section, which covers “cultural diversity and multicultural values”, highlights that cultural heritage encompasses a “diversity of human experiences and perspectives” and the “importance of different cultures, traditions, and languages”. This section does not follow from the previous section, nor is it signposted in the beginning. It is also purely descriptive and fails to comment on cultural relativism.

The conclusion section briefly summarizes the exposition, but does not draw out any common threads, nor does it highlight the importance of the nature of values or any commonalities they may have: “The assessment of cultural heritage significance is a multifacet [sic] process that requires the consideration of various values. Intrinsic, historical, aesthetic, social, spiritual, scientific, and cultural diversity values collectively contribute to our understanding of cultural heritage and inform decision-making processes. Recognizing and valuing this diversity of values is essential to ensure the preservation, interpretation, and enjoyment of our shared heritage for present and future generations”.

Overall, there is little coherence to the argument within a ChatGPT-written essay, as the sequence is arbitrary, without any defined thread, and the text is basic, without any depth or complexity. In terms of Bloom’s revised taxonomy, the essays produced by ChatGPT, which are essentially a reconfiguration of data extracted from various sources, can be situated at the “understand” level [142] as ChatGPT has shown to be unable to analyze and evaluate concepts related to heritage values. Setting aside conceptual flaws such as inverted logic in some of the sections, the essay could possibly pass as a junior high school assignment where the collation and selective presentation of data on a topic is an acceptable standard, but falls short of what would be expected in senior school, which requires the ability to analyze, let alone at a university level, which requires essays that are based on the perusal of diverse literature and that demonstrate reasoning and an ability to evaluate, to critique and to advance a balanced discussion [143].

3.3.4. Iterations vs. Fresh Starts

After the provision of the first response to a prompt, the ChatGPT interface gives the user the opportunity to “regenerate” that response. Any request to regenerate carries with it an implied level of dissatisfaction on behalf of the user with content or detail provided by ChatGPT in its first response. Potentially, therefore, ChatGPT can learn from any request for regeneration of a response. After the first regeneration, the user is provided with a prompt (“Was this response better or worse?” with options: better, worse and same) to judge the quality of that new response and thereby “tweak” and influence the nature of the next query or iteration.

The sample essays were generated in two different ways. In Sets A and B, the chat was left open while ChatGPT was asked multiple times to regenerate the essay text, but without using the judgement option. In Set C, each chat was deleted after the essay was generated, thus creating 12 essays where ChatGPT had no reference to its previous work.

Each of the three sequences generated essays that differed from the preceding option both in structure, the nature of the values covered and the level of detail. There was no discernible pattern of change within the “open-chat” generated iterations of Sets A and B and there was no discernible substantive difference between any of the three sets. This suggests that regeneration of an essay while the chat was left open without quality prompting has no significant impact on the nature of the regenerated essay.

4. Discussion

As noted, ChatGPT returned essays with an average word count of 710 ± 69 words, although it had been tasked with writing a 1500-word essay. None of the essays were longer than 62% of the prescribed length. The limitation of ChatGPT to provide answers well below the stipulated word range has been observed by other authors [45,47,49,144]. It can be speculated that this is due to instructions delivered during the training phase to generate comprehensive and succinct responses in favor of more detailed and nuanced discussions, even though the word count would permit this.

4.1. References

The small number of genuine references, which are “cited” more than twice, suggests that these were part of the input that may have been used to train the model. It was anticipated that ChatGPT would generate its references using keywords or word combinations derived from the query task and connecting these to select sources that were fed into the model during its training. Yet, as the results show, a large number of references are entirely fictional or flawed in terms of authorship or year of publication. It is of interest to note that in only 1 of the 36 iterations of the essay task the reader was advised that “the references provided are fictional”. In the first query task that required ChatGPT to provide 20 +20 references, the reader was exhorted to “evaluate the sources for their relevance and credibility before using them in academic or professional work.” That would, of course, expose the false references. In a plagiarism and academic misconduct setting, however, it can be posited that such references would have been inserted without verification.

In the second query run, ChatGPT baulked when it was required to generate “50 references on cultural values in cultural heritage management”. As noted, it offered only 10 references, apologetically commenting that “providing 50 references in a single response would be quite overwhelming and space-consuming”. Significantly, it prefaced the supply of these 10 references with the comment “However, I can certainly provide you with a list of 10 reputable references on cultural values in cultural heritage management”. Despite the claim of having provided “reputable references”, four of the ten did not exist at all and represented publications constructed of fictitious titles, with an additional reference that cited an existing title but listed incorrect authors.

Again, this poses serious problems, as the casual, non-specialist reader is being misled. The use of false references in ChatGPT written essays has been observed in other discipline areas [40,145,146,147,148,149].

4.2. Constructs

Three essays claim a level of authoritativeness when they asserted, in their introduction, that they were “[d] rawing on scholarly literature/sources” (A8, B11 and C1) and “expert opinions” (B11) or “examples from cultural heritage practices worldwide” (C1), yet two of the three essays each contain a fictitious reference.

At a superficial glance, the types of values that are included in the essay responses make sense, but the responses are patchy, and a more in-depth examination exposes problems that seem to be due to the generative nature of ChatGPT and which highlight its limitations. In the following we look at a select number of examples, which are representative but by no means exhaustive.

Consider the following, which at first sight appears reasonable: “Historic values are concerned with the historical context, narratives, and events associated with cultural heritage. These values emphasize the significance of heritage in conveying historical knowledge and understanding. Historical values often emerge from the connection between heritage and significant events, figures, or periods in history. For example, a site where a historic event occurred may be considered culturally significant due to its historical value” (essay A8). Upon closer examination, the second sentence is fundamentally flawed, as it inverts the conceptual sequence. It constructs historic value as “emphasiz[ing] the significance of heritage in conveying historical knowledge”. In common usage by the heritage profession, heritage assets may indeed be attributed historic value if they contribute significantly to our understanding of the course or pattern of an area’s history [109]. The overall heritage significance, however, is derived from the nature and strength of that contribution and not the other way round.

In another example, one essay presents the “category” of “Historical and Associative Values” and expounds as follows: “Historical values pertain to the significance of a heritage element in relation to past events, periods, or people. Associative values refer to the connections between a heritage element and individuals or groups who have interacted with it, such as cultural, religious, or political associations. These values highlight the importance of heritage as a record of collective memory and identity” (essay B10). This highlights two fundamental problems. The first sentence again inverts the conceptual sequence. It again constructs value as to whether a “heritage element” has significance “in relation to past events, periods, or people”. The second sentence is a mélange derived from British approaches to heritage assessment [111,150]. While these sources use historic associative value, this does not relate to “cultural, religious, or political associations” as constructed by ChatGPT, but relates to an asset’s “association with a notable family, person, event, or movement” [111]. The ChatGPT AI language model appears to have split “historic associative value” into historic and associative value and then proceeded to split “people” into “individuals or groups”, while also assuming an equivalence of “associative” with “associations”.

A third example is the treatment of “social value” in a different essay (A7). ChatGPT states that “[s]ocial values highlight the role of cultural heritage in society and its impact on communities. They encompass the sense of identity, pride, and belonging that individuals and communities derive from their heritage. Social values also include the educational, inspirational, and recreational opportunities provided by cultural heritage”. Once again, this is an inverted logic. Social value does not “highlight the role of cultural heritage in society” but is derived from the importance that communities (or sections thereof) attribute to heritage assets or practices and that are instrumental to the social or cultural wellbeing of that community. While cultural heritage undoubtedly provides educational, inspirational and recreational opportunities, these are not social values as used in heritage assessment.

The same essay notes that “[a]esthetic values pertain to the artistic and visual qualities of cultural heritage. They encompass the beauty, craftsmanship, and creativity of artifacts, architecture, and landscapes. Aesthetic values evoke emotional responses and are instrumental in creating a sense of awe and appreciation”. Once again, aesthetic values do not evoke “emotional responses” and a “sense of awe and appreciation” to cultural heritage assets, but the perception of such responses may contribute to a notion that aesthetic value can be attributed to a cultural heritage asset. Yet, when examining other values discusses in the same essay, such as scientific value or spiritual values, the logic is not inverted, and the explanations are quite reasonable.

While many essays make reference to the subjectivity in evaluation (66.7%), cultural relativism (47.2%) and evolving societal perspectives (52.8%) (Table 3), none of them frame values as anthropogenic, and often anthropocentric constructs are projected on inanimate objects, structures and landscapes or on people’s cultural activities and practices. This lack of fundamental understanding results in ChatGPT presenting the concept of “intrinsic value” in a third of the essays (Table 3), with formulations such as “[i]ntrinsic values pertain to the inherent qualities and attributes of cultural heritage, such as its aesthetic, historic, scientific, or spiritual significance” (essay C11). In almost all instances, the aesthetic/artistic, scientific, spiritual, or historical qualities of heritage items are drawn upon to exemplify “intrinsic value” (essays A4, A11, B2, B4, B8, B11 and C11). The definition of intrinsic value as inherent worth is derived from finance theory [151], with wide application among environmental ethicists [152,153]. Except possibly for a small segment of fundamentalist heritage professionals, aesthetic, historic, scientific or spiritual values would not be construed as intrinsic, but as instrumental and part of community wellbeing [131].

A more elevated comment is made in essay A7, where ChatGPT expounds that “[i]ntrinsic values refer to the inherent qualities and characteristics of cultural heritage that make it valuable in and of itself. These values are independent of human perceptions and interactions. For instance, the age, rarity, and authenticity of an artifact contribute to its intrinsic value”. Contrary to its assertions that the cite value examples “are independent of human perceptions”, both “rarity” and “authenticity” are subjective and mutable constructs, while “age” can also subject to evidentiary biases—all of which are “independent of human perceptions and interactions”. It appears that ChatGPT conflates the ontological components of artefacts with their value.

ChatGPT juxtaposes these intrinsic values with instrumental values, which refer “to the usefulness or benefits that cultural heritage provides to society beyond its inherent worth. It includes economic, social, educational, and environmental dimensions” (essay A4). Again, in all instances where instrumental values were referred to (essays A4, B2, B8, B11 and C11), the exemplification involved economic (incl. tourism), educational and social benefits, including “community cohesion” and “sustainable livelihoods” (the latter essay B2).

A further example of confused logic is a section in an essay, which notes that “[e]thical and spiritual values are integral to the assessment of cultural heritage significance, particularly when considering indigenous or sacred sites. These values encompass the rights, beliefs, and practices of communities that have a deep spiritual or ancestral connection to the heritage element. Respecting these values is crucial for ethical and culturally sensitive heritage management” (essay B7). While it is appropriate to respect spiritual values held by First Nations communities when engaging in ethical and culturally sensitive heritage management, these are not ethical values associated with heritage assets, but professional ethics. That this confused logic is not aberration is underlined by another essay which noted that “[e]thical considerations and intangible values are integral to the assessment of cultural heritage significance. Ethical values address issues such as ownership, repatriation, cultural appropriation, and the impact of heritage management practices on local communities” (essay C2).

The concern with these, and a plethora of other examples, is that such formulations can pass careless or cursory reading. Moreover, members of the general, non-specialist public are less likely to identify such shortcomings.

One of the limitations of ChatGPT seems to be that while the resulting product is plausible in language and expression, it suffers from inverted logic, as shown above. The following observation suggests that this is, at least in part, caused by initial word substitution, which then sends ChatGPT off track.

As noted in the introduction, cultural heritage manifests itself in tangible and intangible forms. All of these relate to the outcomes of peoples’ interactions, both with each other and with the environment in which they live. In several essays discussing cultural heritage values, ChatGPT commingled the concept of tangible and intangible “heritage” with tangible and intangible “values” (essays A5, B7, C2, C5, C8 and C10). Clearly, as all values are intangible concepts that are projected on tangible (and intangible) products of a community, the concept of “tangible value” is an oxymoron. In most instances, ChatGPT drew on definitions of tangible and intangible heritage and substituted “value” for “heritage”.

One essay includes a section on environmental and natural values, where ChatGPT expounds that “[c]ultural heritage is not limited to man-made structures and traditions but also includes natural landscapes, ecosystems, and biodiversity … Evaluating the environmental values of cultural heritage ensures the integration of conservation efforts with cultural preservation” (essay C8). Several papers comment on the fact that there is a continuum between natural and cultural heritage [154,155], even though value conflicts are not uncommon [105]. In this example, ChatGPT seems to have used a snippet of information related to this nexus, and, possibly triggered by the mention of cultural landscapes, focused on the noun and substantive term of “landscapes”. From this it then constructed the notion that natural landscapes, ecosystems, and biodiversity are manifestations of cultural heritage.

Another example is the assertion by ChatGPT that heritage assessments can be based on objective criteria, but then proceeds to exemplify this with “historical importance, architectural or artistic merit … and scientific value”, all of which are subjective criteria held by sections of society. In the interpretation of the ChatGPT model, these are somehow different from the subjective values which the essay addresses in the subsequent paragraph, and which are noted as being “deeply rooted in cultural, social, and personal perspectives” (essay B3).

As noted in the discussion of the example presented in the results section, and which applies to all essays assessed in this paper, there is, overall, little coherence to the argument made in each essay. The sequence of the argument in each exposition seems arbitrary, without any defined thread, and appears to be triggered by the sequence of terms and concepts enumerated in the introduction. In essays written by humans, the argument is structured, and commonly completed, before the introduction and conclusions are written, which can then tightly frame and summarize the arguments made. At this point in its development, ChatGPT does not seem to be able to reflect on the coherence of its argument and improve the text before it is presented. The complexity of the argument is also left wanting.

One of the potential limitations of the approach taken in this paper is that ChatGPT was given a set task, and that multiple iterations of the same task were generated rather than systematically tweaking and adjusting the prompt to engineer “better” outcomes. This approach was deliberate, however, as the aim of the paper was to emulate a generic query by a “naïve” user, rather than to generate a response for an already informed and knowledgeable user.

4.3. Can ChatGPT Reflect on and Improve Its Own Work?

While writing this discussion section, it seemed apposite to assess whether ChatGPT can provide some reflection and then integrate that reflection into a better version. A separate conversation was carried out (for full text see Appendix C). As before, ChatGPT was tasked to “Write a 1500 word essay that discusses the nature of values used in the assessment of cultural heritage significance” and to provide references. In response, it generated a 673-word essay (incl. six references) that framed the exposition by stating that “determining the value of cultural heritage involves a complex interplay of subjective and objective factor” and that the essay would “explore the nature of values … highlighting the multidimensional and dynamic nature of these values”.

The exposition then presented the reader with the following seven groups of values: “tangible and intangible values”; “historical and social values”; “aesthetic and artistic values”; “symbolic and spiritual values”; “environmental and sustainable values”; “economic and tourism values”; and “community and identity values”. Each of these comprised a descriptive sentence or two followed by a sentence that outlined what would be required to assess the respective value (e.g., “Evaluating these values requires an understanding of the historical context and societal dynamics that shape the heritage’s significance”).

In common with all previous essays presented by ChatGPT, the essay lacked coherence and linkages, and also lacked overall complexity. In the same session, ChatGPT was then provided with the following prompt: “What are the shortcomings in the essay that you just wrote?”

To this, ChatGPT responded thus: “To enhance the essay, it is recommended to revise the content by addressing these shortcomings, incorporating in-text citations, providing more diverse perspectives, including contemporary examples, discussing the challenges of balancing values, and exploring the dynamic nature of cultural heritage values”. It then proceeded to provide explanatory two-sentence paragraphs to each of these points. Two of these relate to the mechanics of an essay (“lack of in-text citations” and “absence of contemporary examples”), while the remaining three are apposite critiques, such as “limited coverage of perspectives”, “insufficient discussion on balancing conflicting values” and “limited discussion on the dynamic nature of values”. The explanations of these three points, as provided by ChatGPT, provide reasonable pointers to revise the essay. There are obviously numerous other concepts that have been considered that ChatGPT used in other iterations of the essay, such as cultural relativism, the subjectivity of values or the concept of evolving societal perspectives.

ChatGPT was then provided with the following prompt: “Can you rewrite and improve your initial essay by incorporating the critique that you just wrote? Keep in mind that the word limit is 1500 words, so have more room to expand your discussion”.

Based on this instruction, ChatGPT returned an 844-word essay (incl. ten references). The new essay followed the first attempt in structure and content. A side-by-side comparison revealed that while the introduction had changed, the bulk of the text of the exposition remained the same. For each of the points made, ChatGPT rewrote the final sentence that outlined what would be required to assess the respective value. This reformulation, which rarely goes beyond paraphrasing, does not introduce any new dimensions of the values discussed. In the case of aesthetic and artistic values, it does add the comment that “[i]ncorporating contemporary examples, such as modern art installations, helps to highlight the dynamic nature of aesthetic values”. Throughout the revised text it fails to incorporate in-text citations, fails to provide more diverse perspectives, fails to discuss the challenges of balancing values and fails to explore the dynamic nature of cultural heritage values.

Based on the example provided in this paper, as well as repeat examples, ChatGPT seems to be able to highlight shortcomings in its own work, but then fails to integrate and incorporate these into a new essay that is substantively different from the initial attempt.

5. Conclusions

As noted in the introduction, ChatGPT has the capacity to extract and summarize factual information. The generation of multiple essays with the same topic has shown that its ability to synthesize concepts and to present them in the form of an essay, however, is limited. The concepts presented, which are often flawed and suffer from inverted logic, are presented in an arbitrary sequence with limited coherence and without any defined line of argument. Given that it is a generative language model, ChatGPT often splits concepts and uses one or more words to develop tangential arguments. At present, ChatGPT has shown to be able to provide some critique of its own work but seems to be unable to incorporate that critique in a meaningful way to improve a previous draft.

This then has implications on how a generative AI language model interprets cultural heritage values. An understanding of how values work in human society is fundamental to an interpretation of values, i.e., that values are broadly conditioned through enculturation and education but personalized through upbringing and life experience; that the strength attributed to a value is conditional on personal circumstances at the point of evaluation; and that, therefore, values are but tradeable and diachronically mutable entities. Any interpretation of values in general requires both an understanding of the fundamental nature of values in general as well as the specific types of values and their definitions as agreed upon by the respective discipline.

As members of society, humans are enculturated into value concepts from an early age and have an intrinsic understanding of value concepts and experience with application in their daily and professional lives. This understanding of and experience with values contextualizes any research into and discussion of cultural heritage values. As a large language model, generative AI, however, lacks the combined effects of enculturation and lived experience and thus cannot relate to and contextualize the factual information it collates. As a consequence, any interpretation is based on the process of collation, extraction, and summation of data gleaned from its training sources that are presented in an arbitrary sequence without a sense of hierarchy or order of significance.

It must be stressed that the observations made in the foregoing pages apply to ChatGPT and like software at this point of the development cycle. The critical inhibitor seems to be that the model needs to be able to integrate materials that are being fed into it, and that it needs to be trained in its responses. At present, this entails both a limited set of materials that were made accessible to the system and a set cut-off date (September 2021 for ChatGPT). The nature and quantity of the source material fed into it will determine the “knowledge base” and thus the responses provided. As noted, criticism has been leveled at the apparent biases that seem to be reflective of the choice of materials made available during the training phase.

When users prompt ChatGPT to regenerate its response to a task, they are prompted to judge whether the response is better than the previous one. While this allows ChatGPT to learn the user-perceived quality of its responses, it has the potential to introduce user-specific biases into the system that can be exploited by malevolent actors.

It can be posited that a future iteration of ChatGPT will be able to progressively and iteratively access newly published information, for example, new issues of academic journals that are being published via OpenAccess, and integrate these new data into its responses.

Also, and again at this point of the development cycle, ChatGPT has the fundamental limitation of being limited to integrating and summarizing information. While it possesses the capability to critique it is own work, it does not possess the ability to integrate this in a meaningful manner into a revision, even if asked.