When Augmented Reality Met Art: Lessons Learned from Researcher–Artist Interdisciplinary Work

Abstract

Over the last few years, Augmented Reality (AR) technology has quickly evolved and today there is commercial software and hardware that allow the creation of AR experiences. However, it might still be cumbersome to create rich AR experiences without a deep knowledge of the technology. Artists have collaborated with IT experts during the last few years in order to enhance artistic pieces embedding AR technology, leading to the emergence of interesting collaborations between artists and engineers, computer scientists, architects, etc. The resulting works can be referred as AR Art. However, the interdisciplinary work behind these collaborations is usually not addressed. The aim of this paper is to review some of the reported AR Art work since the AR term was first coined in 1990, focusing on collaborations between different disciplines, especially during the early years, where art and technology became one. Additionally, the author reports her own experience in contributing to AR Art from an interdisciplinary perspective.

1. Introduction

The relationship between science and art has fluctuated over the course of the history, with moments of proximity and moments of distance. Although they are currently usually understood as two different cultures, interdisciplinary situations of collaboration and interchange occur, often via a nexus of digital culture and the use of computers. According to [1], since the appearance of computers, scientists and artists have found a common space of work and understanding. Therefore, within the use of new technologies, the distance that separates both disciplines is becoming shorter. Today, multiple examples can be found where artists make use of new technologies in their artistic pieces (e.g., interactive videos, virtual worlds, and robots), and the Augmented Reality (AR) technology is no exception.

The term “Augmented Reality” was first coined by Caudell and Mizell in 1990 in order to describe a special kind of display used by electricians for aircraft repair and maintenance that was able to show virtual information on top of real images [2]. While early definitions of AR restricted the technology to systems that used Head Mounted Displays (HMDs), aiming at an exclusively visual space, later definitions extended the concept of AR to other devices and multimedia contents not necessarily visual. Today the definition given by Azuma [3] is widely accepted, who defines augmented reality as those systems that simultaneously contain the three characteristics: mixing of real and virtual, interactivity in real-time, and three-dimensional registration.

The idea of showing synthetic data related to a real environment is not new, nor did it start with AR technology. For example, there are books in which images of the real world are mixed with drawings. This effect was also used in some films prior to the 1990s, where real characters and cartoons are mixed, such as in Mary Poppins (1964), Bedknobs & Broomsticks (1971), and Who Framed Roger Rabbit? (1988); In the exhibition A Reflected World (1966) by M. Pistoletto [4], the artist mixed photographs of people with real people by using mirrors. In this sense, it can be said that augmented reality is a discipline that, although technologically novel, conceptually it is not. However, AR requires interaction in real-time, a characteristic that is missing from the previous examples.

Since the technology was invented, at the research level a lot of collaborations between artists and other experts have arisen. Thus, AR has been used as a means to create artistic pieces, which is referred as AR Art. Today, there are a variety of APIs at the consumer level that makes it possible to create AR experiences, most of them making use of mobile technologies such as smartphones or tablets. For instance, recently Apple has launched ARKit [5] that allows building AR environments with very little knowledge about the technology behind it. Another good example is the Vuforia developer library for Unity [6], which provides users with very good examples to build their own AR experiences. However, the poor maturity of this technology during the first few years since it was born made it inaccessible for the majority of prospective end-users, including artists, which looked for collaborations with engineers and computer scientists in order to materialize their artistic pieces. To date, there are many reported works focusing on research aspects, but little attention has been paid to the added value emerging from the interdisciplinary aspect behind these works.

It is the aim of this paper to offer a new perspective on AR applications related to art, by pointing out the importance of the interdisciplinary work behind them. To that end, a review is done on some artistic-related AR applications, paying special attention to works completed during the first few years of the AR technology development, where the poor maturity of the technology itself required computer scientists and other IT experts to work in collaboration with artists, while also influencing the final result of the artistic pieces. In these papers, however, the interdisciplinary nature of the work is not properly addressed, with very few exceptions. Therefore, the author reports her own experience in order to further delve into the added value emerging from the interdisciplinary work, essential to achieve results not possible with the involved disciplines working in an isolated fashion.

2. Related Work

It is said that until 1999 AR remained mainly in the hands of scientists and in their labs, as the technology needed to evolve before it could be ready to reach the consumer level. Still, for that period (1990–1999) some few works of AR Art can be found. This greatly changed in 2001, when the ARToolKit library by Hirokazu Kato was released to the open source community, making the technology available to wider audiences—though still in many cases at the research or experimental level (Figure 1). To get an idea, a search in Google Scholar of “augmented reality” & “artistic” between the years 1990–1999 produces 277 results. The same search for the period 2000–2009 produces 2440 results (an increase of almost 9-fold) and for the period 2010–2018 (today) produces 8990 results (an increase of more than 3 times that of the previous period). The important feature here is the rate of increase, as not all the found papers by Google Scholar actually refer to AR Art.

Early works of AR Art can be found in [7,8,9,10,11,12,13]. Augmented Groove (2000) is a musical AR-based interface that explores physical and tangible interaction for conducting musical performances [11]. It was carried out by a team of engineers and computer scientists in collaboration with artists Rodney Berry (music composer) and Jun Kurumisawa (3D modeling). Another early AR Art piece involving interdisciplinary collaborations is Can You See Me Now? (2001) [12,13], which arose from a collaboration of the group of artists Blast Theory and the Mixed Reality Laboratory of the University of Nottingham (UK). It consists of a game that takes place in the streets of a city in which different players are connected online and are pursued through a virtual model of the city by runners. Conceptually, the Blast Theory group sought to investigate the penetration of mobile technology, even in the poorest areas, rural environments, in adolescents and, more generally, in demographics that do not usually have access to new technologies.

A few years later, LifeClipper (2004) was developed, a mobile AR Art piece that consists of an audiovisual experience enriched with virtual elements [14]. It was developed by a group of researchers and artists from the University of Applied Sciences Northwestern Switzerland and coordinated by artist Jan Torpus. Technically it is based on a laptop that the user carries around a specific area of cultural interest. Through an HMD, users see and listen to a series of images and sounds superimposed on the real environment, feeling as if they are watching a movie in which they participate as active observers and in which the various virtual characters are directed to them. During the same year, the artistic piece Red Libre Red Visible (2004) was developed by artists Diego Díaz and Clara Boj [15,16] with the collaboration of computer scientists at the Mixed Reality Lab (Singapore). In this piece the AR environment was built in the urban space through the use of the MXRToolKit library [17], creating new landscapes in the public space through the visualization of the data flowing through digital networks. New virtual, personal and subjective configurations were sought, and access and exchange of information was encouraged through a free and open network.

Other AR Art works emerging from interdisciplinary collaborations reported in the following years can be found such as in [18,19,20,21,22]. For instance, in [19] the Debussy3.0 project is described. It is a ballet involving AR technology. The work was carried out by ESTIA research laboratory and the National Choreographic Centre Malandain Ballet Biarritz. In this case, the collaboration between artists and researchers is deeply described. Among others, the paper deals with the difficulty and the benefits of the collaboration, similarities between art and science, and the transfer from art to industry. In [18], a work is presented that allows people to digitally curate their own augmented reality art exhibitions in their own homes by augmenting the pictures they have on their walls with content from the Peter Scott Gallery in Lancaster. This research project involved a partnership between the gallery, design researchers, and a technology provider. In [20], industrial designers and artists collaborated in order to identify the methods and technologies for developing a prototype that determines the influence of spatial AR in the creativity of designers and artists.

To have a wider scope of other artistic pieces or projects involving AR Art, the reader is referred for instance to [23], where a survey is presented on the current trends in AR artistic interventions, or to [24], where there is a great collection of artistic works dealing with AR technology, many of them developed in the last 10 years as the maturity of the technology has made it available to wider audiences. The work presented in [25] is also interesting, in which there is a discussion of sound art projects involving AR and public spaces.

In the works described above, artistic and technical skills are intermingled, with interesting collaborations between artists and computer scientists, software designers, IT experts, engineers, multimedia scientists, etc. As can be seen, the relationship between AR and art arose very early. Here, I have focused my attention on a few early works—when the technology was still immature—reported by the research community involving collaborations between researchers and artists. I have also cited recent works have been cited, dealing with interesting collaborations and showing that, although the maturity of the technology makes it available to wider audiences, interesting collaborations are still sought.

However, the added value from the interdisciplinary work emerging from these collaborations is usually not described in detail, with very few exceptions such as in [19], a recent work tackling this issue. The testimonies behind interdisciplinary work are apparently given little importance. However, they are relevant, as interdisciplinary work conditions the artistic piece for many reasons. For instance, both the maturity of the technology and the skills in dealing with it have direct impact on the artistic pieces in a meaningful way. This fact is even more evident during the first years of the technology, during the transition from the lab to the consumer level. Other issues emerging from different point of views and way of working between different disciplines are also of relevance. To tackle these issues more deeply, in the next sections I report on my own experiences.

3. Reporting Own AR Art-Related Works

Most of the works here described were carried out during the period 2004–2008 as part of my PhD research at Laboratorio de Luz, Fine Arts School, Universitat Politècnica de València (UPV). To begin with, I should note that my background was in surveying and geo-information with specialization in photogrammetry, but I moved to Laboratorio de Luz to carry out my PhD as part of a project dealing with AR and art. In my doctoral thesis, I explored the capabilities of the AR technology for the artistic field, making use of different solutions for positioning in real-time, and the blending of real and virtual worlds, taking into account a variety of sensors, displays, types of data, interaction paradigms, and software solutions. The works are mainly experimental, although some have been exhibited.

Back to the 70’s (2005) is an AR application designed to (conceptually) depict the Vera University campus in Valencia city center as it was 30 years ago. Within this application, the user was immersed in a hypothetical environment where the campus buildings were seen together with some historical ones. I built the application in collaboration with artist Francisco Sanmartín using the Max/MSP software in combination with Jitter [26]. This software, although not intended to build AR applications, is widely used by artists as it allows to build highly interactive solutions with a simplified programming environment. I implemented the AR solution in collaboration with F. Sanmartín, who had a deep knowledge of the software. We tackled the occlusion problem with the use of 3D phantom models of the University’s buildings, which I acquired with a topographic total station with the aid of artist Francisco Giner, who I gave indications on how to use the device. Detection of user pose in real time was achieved via a combination of a visual and an inertial tracker, which I integrated into the software solution. The ancient buildings and phantom objects were modeled by F. Giner, who dealt with the final aesthetics of the AR environment. The results were experimental and were disseminated at the ACE conference (2005) [27].

Magic Tunnel (2005) and Un∞ (2006) are two artistic installations built following the concept of the Augmented User (2004) [28]. Each installation was composed of three AR mirrors, each one proposing different user augmentations in such a way that they acquired as many appearances or identities as the number of mirrors observing them. Users carried a set of markers (in their head and hands) that were replaced by virtual models in the mirror by means of the ARToolKit library [29]. In Magic Tunnel, we proposed three different versions to transform the face of users: famous actors (Charles Chaplin, Stan Laurel, and Oliver Hardy), polyhedral figures, and Señoritas de Avignon (inspired in the famous artwork of Pablo Picasso). For each version, only one user could interact with the mirrors and only the faces were augmented. Differently, in Un∞ we proposed three different transformations, one for each mirror, and two users could interact simultaneously. Additionally, the hands were augmented with text related to the theme of the mirrors. The mirrors’ themes were: emotional mirror, stereotype mirror and multicultural mirror. A posterior version of Un∞ was based on the Lewis Carrol’s famous novel Alice’s Adventures in Wonderland. The virtual models where designed by F. Giner, who also dealt with the aesthetics of the installation. We both worked on the concept behind the mirrors and the design and construction of the wearable markers. The installations were possible thanks to the work of the team Laboratorio de Luz, led by artist María José Martínez de Pisón; they also collaborated in testing the proposed experience. Additionally, the first version of the mirrors (the Augmented User) was tested at the Fine Arts school of the UPV [30], so we collected the feedback of tens of students and professors. Based on that, we worked on improving the contents, so the message sent by the mirrors was meaningful. Un∞ was exhibited at the Centro Párraga Murcia (2006) (with the initial version of the thematic mirrors) and at the festival Observatori (2009) [31] (with the version of Alice’s Adventures in Wonderland), among others. In Figure 2, an installation of the Magic Tunnel is shown, with a user augmented with virtual characters inspired in Las Señoritas de Avignon, from Pablo Picasso. Results were also disseminated in journals and conferences [32,33].

Other artistic pieces built following the concept of the Augmented User (but with some variations—a kind of AR mirror with a third user point of view) are Live LEGO House (2006) and AR-Jazz (2007). The aim of Live LEGO House was to explore coexistence and multicultural factors through gaming. The system consisted of a real house built with the LEGO blocks enriched with synthetic content (sounds, videos and 3D animations) by means of the MXR ToolKit library. I built this application during my research fellowship at the Mixed Reality Lab (Singapore), led by Prof. Adrian David Cheok. I had the opportunity there to meet the programmers of the MXR ToolKit library, mainly computer scientists. I also had the opportunity to exchange ideas and learn from artists Clara Boj, Diego Díaz and Rodney Berry. Their work in AR and personal experiences in Singapore were an inspiration to me to develop the concept of the Live LEGO House. Additionally, I recevied direct feedback from composer Carlos David Perales, who contributed to the sound and to define the role of dolls in the house. The results were disseminated in journals and conferences [34,35]. On the other hand, in AR-Jazz I worked with musicians to build an AR experience where sounds and movements were visualized in an AR mirror during a live jazz performance at the SedaJazz festival 2007. I built the AR experience with Max/MSP combined with Jitter. The sensors were a microphone (to capture live sound) and an Xsens inertial sensor (to capture movements of an interpreter). During the lab testing, I received advice from C. D. Perales. At the stage, during the rehearsals, I worked on adjusting some parameters (distortion, colors, etc.) of the shape representing sound, so to achieve appealing visual effects. The feedback of the musicians—some belonging to SedaJazz collective—was relevant for the final result that, in a certain way, conditioned part of the live performance (Figure 3). I was certainly satisfied with the great engagement of musicians with the AR piece; for instance, at the rehearsals they were interested in interacting individually with their instrument, playing with notes to see the effect produced in the virtual representation. The experience was documented and disseminated at a conference [36].

Other works related to AR Art are CatedrAR (2008) and AR-Immersive Cinema (2008), pieces merging different artistic styles or historical moments. In CatedrAR, art and technology were essential to build an AR scene where past (now partially or totally lost) and present elements are displayed together. It recreated two features, a Baroque vault and a Renaissance reredos, formerly present above the high altar of Valencia Cathedral in Spain. The Baroque vault—which was dismantled, as it hide a Gothic vault that provided the physical support for unique early Renaissance frescos of musical angels—was scanned with a 3D topographical scanner, a work done by topographers and architects. This shape was afterwards simplified for the visualization of the vault in the AR scene, which I built using the BazAR library [37,38]. Registration in BazAR is based on feature point detection and matching, where homography between a target image and the input image is estimated from correspondences by RANSAC. The experience was visualized with a HMD with an attached webcam, and users carried a laptop in a backpack. The results were disseminated in a scientific journal [39]. On the other hand, in AR-Immersive Cinema, augmented users are inserted into lineal video sketches, where important historical moments were presented in a novel way – mixing video sketches and AR – to engage users. The application was achieved using two technologies: the first was a cinema filming with authors that interpret a sketch; the second part concerned the AR, and was managed using the ARToolKit library. Users see themselves virtually transformed into one of the actors, thanks to the paradigm of the augmented user. To increase engagement, actors in the video sketches talk to and look at the user. Both technologies were visually integrated within a combination of spatially aligned projections. The installation is part of the permanent exhibition at the Aula Natura of the Gandia Marshlands, which was designed and realized by experts in tourism and construction. Experts in Valencian history designed the contents of the AR experience, while the narratives were built by a screenwriter; some actors represented the historical characters, and multimedia experts recorded and edited the sketches. The piece was also documented and disseminated through a scientific journal [40]. During my years at Laboratorio de Luz, I also dealt with another piece: Hot Images (2005) [41], mainly while I was completing a research fellowship at the Mixed Reality Laboratory of the University of Nottingham. However, that work was rather experimental and quite individual, so it has little interest for this paper.

4. Lessons Learned from Interdisciplinary Work and Discussion

The interdisciplinary aspects behind AR Art are usually not given enough prominence. This was also true in my case, as I didn’t put enough emphasis on my own experiences during the writing of my doctoral thesis. Looking back and seeing things with another perspective, I have noticed the impact that my years at Laboratoio de Luz have had on my way of approaching things, my way of thinking and thus my research—during and afterwards. Here I describe some of these issues, with the aim of highlighting the benefits that interdisciplinary work brings and the need to find a common space of understanding to achieve it.

I remember the first months at Laboratio de Luz, feeling as if artists and myself spoke different languages. For instance, talking about AR, I was very worried about the poor accuracies of the technology in the real-time positioning under certain setups. For a surveyor like myself, a misalignment error—between the real and the virtual worlds—greater that 1 cm certainly hurt, an aspect that that is really trivial for an artist. What was important for artists was that the system was stable, so it didn’t crash during live performances or exhibitions, an aspect not that relevant to me, as I was mainly interested with a laboratory validation of the developed applications. In the end, I had to look for technical solutions that fulfilled both needs, in many cases building AR environments from scratch (with the only exceptions being those built with vision-based libraries such as the ARToolKit or the MXRToolkit), where I explored the integration of different sensors, display devices, and interaction paradigms in software environments to be used in live performances (e.g., Max/MSP Jitter). As an example, I used the paradigm of augmented virtuality instead of AR to hide the poor accuracies of a GPS positioning in Hot Images, while in Back to the 70’s I made use of different tracking techniques.

Dissemination of the work of scientists and artists is usually done through different channels. While my interest was to present the results of our work through conferences and scientific journals, artists’ main aim was to show the work at museums or exhibitions. Therefore, while to me it was enough to validate the applications with non-sophisticated means, the aesthetics were quite important for my colleagues. Paradoxically, the relevance of end-user feedback was another issue of discordance. While I was interested in knowing if the technology was usable and if the message that we tried to transmit was well understood, less interest on these facts was shown by artists—to them the work finished with the realization and exhibition of the artistic piece. This puzzled me, as behind the artistic pieces there is a “message”, but they were not collecting feedback. In the end, I collaborated in the exhibitions, but additionally collected user feedback in some cases (e.g., in the Augmented User), which was relevant for building other versions of the artistic pieces. In parallel, the documentation of the pieces at the exhibitions was useful for papers, as I was able to collect nice pictures from outside the lab environment. On the other hand, some of my colleagues also participated in the papers I was interested in, and presented them together at conferences that, while focusing on computer science, had a panel related or close to the artistic field. Again, solutions were sought that fulfilled both needs.

During those years, I also experienced interdisciplinary work in lecturing. During the third year of my Ph.D. studies, my colleague F. Giner and I were lecturing a subject on AR for students in Fine Arts; while he was mainly working on the creation of 3D models and artistic concepts, I dealt with AR technology. I remember the puzzled faces of the students when I showed a slide with a rotation matrix. I explained this and other mathematical concepts in a very intuitive way, so they could really understand what is behind the AR technology. To show the students how to build AR pieces I made use of AMIRE [42,43] (no longer supported), a software that allowed the authoring of Mixed Reality applications without programming, using a visual creation tool. By the end, they were able to create an AR piece, some of them being exhibited that year in a gallery at the center of Valencia. One of the students that I had that year, Manuel Ferrer, realized an outstanding doctoral thesis on AR Art [44]. Additionally, he, together with artist Alena Mesárošová, have greatly contributed to AR Art with pieces such as Timetravellers, DanzAR and AR_VR_Putney [45].

Collaborative work involves a learning process, if all those involved are open to it. In this way, not only the work is benefited from different expertise, but the experts themselves learn from other disciplines, from theory to practice. It is said that engineers solve problems while artists make problems evident. It means we have different points of view in addressing the same issue, but all are needed to reach a wider scope, a holistic view. From my experience, the gridded mind of an engineer can be broken into pieces, while the sometimes chaotic exploratory work of artists (without meaning to generalize) can be ordered in a certain way. The resulting work is more than the sum of the individual parts; interdisciplinarity does not consist on doing isolated work and then putting some glue between the results, but really finds a way of working together, mixing methodologies, expectations, dissemination channels, etc. of different disciplines.

Since leaving Laboratorio de Luz, I have collaborated with experts from other areas, such as physicists, engineers in telecommunications, computer scientists, multimedia engineers, tourism planners, art historians, etc. Among others, I was part of the coordination team of FP7-SYDDARTA [46], a project related to the curation of cultural heritage, and with collaborations between social sciences and humanities (SSH), and ICT experts. Thanks to that experience and to my interdisciplinary background and interest, I have recently been able to lead the coordination of a proposal called SILKNOW [47], also involving SSH and ICT experts, which has been recently funded under H2020 in a highly competitive call. In [19] they point to a “proxy”, a person who has both artistic and technical abilities in order to overcome the difficulties of collaborating. This is a good definition of the role I’ve taken in these cases.

To me, the key issues to successfully collaborating with experts in other disciplines are fluent communication, a common interest to evolve, and an open-minded predisposition. A dialogue (bi-directional) between researcher and artist is essential, making efforts in trying to understand each other and forgetting egocentric views, in order to break self-established artificial barriers. Therefore, forget things such as “this has to be done in this way because this was what I was taught”, “that is not interesting to me as I will not be rewarded for it in my field”, “I do this, you do that”, etc. A strong commonality between scientists and artists is perhaps that both need to be highly creative in order to innovate in their work. Therefore, creativity comes from everywhere (researcher/artist), and when that happens the process involves co-creation, a truly interdisciplinary work.

I am a strong believer that interdisciplinary work is beneficial for research and for progress, generally speaking. Good examples of well-known interdisciplinary profiles come from Aristotle, Galileo Galilei, Leonardo da Vinci or Nikola Tesla. Today, it might be too cumbersome for a single person to embrace different disciplines with a deep understanding of all of them, but interdisciplinary work is still necessary, maybe led by the aforementioned “proxis”. A new figure like this is essential to build bridges between very specific experts, brining novel tools and methods to articulate the union between disciplines. However, this fact is yet not well understood and even rejected in many ways. In my case, the collaborations with other disciplines have opened my mind, given me a more holistic view, and brought great personal satisfaction. However, professionally speaking, I have found a lot of obstacles because of my interdisciplinary profile. At least in Spain, researchers are evaluated according to specific areas of knowledge, with profiles like mine strongly punished when applying for national grants, not mentioning lecturing. These facts hinder the conciliation between different areas, putting a barrier in front of a generation of new outcomes emerging from knowledge transfer. This is contrary to the logic that I have previously been exposed to; it leads to endogamic research, not aware and thus not taken benefit from other advancements. These and other aspects make me sometimes feel more a survivor than a surveyor in the current Spanish research realm. Even so, I will continue fostering interdisciplinary work as, in the end, research comes first for me.

5. Conclusions

AR is a transversal technology, meaning that it can be applied in many areas, such as training, maintenance, entertainment, mental health, engineering, science, and many others. Art is not an exception. Since very early on, AR has been used as a means for artists to create. Maybe because of the complexity of the technology during the first years, there have been interesting works of AR Art emerging from the collaboration of diverse disciplines, disciplines that traditionally have been considered distant. In my case, it meant collaboration between artists and an engineer in surveying and geo-information, something very rarely seen. From my own experience, the works here presented wouldn’t have be the same without a co-creation process, where a mutual understanding between the researcher and the artist was the key.

However, nowadays many artists are experts in multimedia technologies, including AR. Therefore, a question arises: are these interesting collaborations still needed? Today, it is the moment for artists and other audiences to really exploit AR, making profit of all the developments and advancements that the research community has made during the last almost 30 years. This does not mean that interdisciplinary work is not required any more, but it is expected that the final applications be not so much conditioned by the restrictions of the technology itself. In fact, due to the huge amount of (multimodal) sensors and displays, APIs, positioning strategies, realms (mixed reality), types of interaction (e.g., gestures) and many other factors, it is expected that experts in technology (e.g., multimedia engineers) will play an important role in developing applications involving AR, including for the artistic sector. Collaborations are not always easy, as different disciplines have different ways of working and different expectations, among other differences. In this sense, a new figure referred to as a “proxy” turns out to be relevant, a role needed to build bridges between disciplines in order to really bring added value from interdisciplinary work. Approaching the technology from a holistic view expands the possibilities in building richer AR experiences: a bunch of new worlds are possible with AR and interdisciplinary work.