How The Arts Can Help Tangible Interaction Design: A Critical Re-Orientation

Abstract

There is a long history of creative encounters between tangible interface design and the Arts. However, in comparison with media art, tangible interaction seems to be quite anchored into many of the traditional methodologies imported from human–computer interaction (HCI). How can the Arts help tangible interaction design? Building on Søren Pold’s Interface Aesthetics, a re-orientation of the role of the artist towards a critical examination of our research medium—tangible interaction—is proposed. In this essay, the benefits of incorporating artistic research and its methodologies into our field are described. With these methodologies it is possible to better assess experiential aspects of interaction—a relevant attribute which traditional HCI approaches cannot afford. In order to inform our community, three examples of critical artworks are comparatively studied and discussed.

1. Introduction: Digital Arts and Tangible Interaction Design

In 2016, the media art festival Ars Electronica [1] gave special relevance to the field of tangible interaction design, presenting the project Radical Atoms [2] together with other works by the MIT Tangible Media Group. A number of tangible interfaces were presented as “the artistic highlights of the year”. Indeed, the reacTable project [3] had won the “Golden Nica in Digital Musics” at the same festival in 2008. Certainly, there exists a history of intersections between the media arts and the field of tangible interaction design. In 2007, the first International Conference on Tangible, Embedded and Embodied Interactions (TEI) was stated “to be aimed at bringing together researchers, students, designers, practitioners, and artists” [4]. Just a few years later, this initial impulse for integrating artistic proposals in the conference resulted in a separate Arts Track for TEI. A much earlier example of this history of creative encounters is the development of the “Marble Answering Machine” by Durrell Bishop at the London Royal College of Art in 1992.

As Hornecker and Buur [5] have described, “tangible interaction tends to emphasize materiality, physical embodiment of data, bodily interaction and embeddedness in real spaces and contexts”. However, these are also concepts serving well to describe the intentions behind the work of many media artists [6,7]. Hence, novel developments in tangible and embodied interaction have gone hand in hand with new ground in the Arts and vice-versa. Interestingly, this crossroads of cultures was emphasized after the so-called practice turn in interaction design during the first decade of the 2000s [8]. This change of mentality motivated tangible interface designers to focus more on designing interactions than on the interface itself. Since then, explains Hornecker [9], human action, creativity, and social action are topics which have been increasingly adopted into our research agenda, thus unlocking definitely the door to artistic production within academia and interaction design labs.

As an active artist and researcher in the field of tangible interaction, the spark for writing this essay came from a process of inquiry about the actual role (we) artists have often assumed at this merge. If tangible interfaces—especially those created by engineers at academic labs—also seem to feature enough artistic contents to be presented at gallery spaces as aesthetic artifacts, why is the artistic collective not so much present at tangible interaction labs? Which role should artists assume at human–computer interaction (HCI) labs? Which is the real value that the Arts can now offer to tangible interaction design?

The contribution of this essay lies in suggesting a re-orientation of the role of the artist towards the creation of “critical interfaces”: interfaces which are critical with our own research medium (e.g., tangible interaction design). This re-orientation encourages the shift from a traditional vision of artists as mere producers of aesthetic content to a more updated and useful vision of the Arts fully engaged in research. This contribution is built on Søren Pold’s “Interface Aesthetics” [10] which recommends the incorporation of digital arts into HCI as a method to evaluate its long-assumed paradigms.

As will be explained further, “critical aesthetic artifacts” would be interfaces especially designed to criticize our own research medium and trigger reflection about assumed paradigms of our field. In fact, cultural interfaces have often been described as “critical” [11], as they can also convey a social or political role. However, the specific type of critical interface which is proposed here must be more, orienting its existing critical dimension towards our research paradigms and accepted assumptions about HCI and its applications. For instance, a critical interface of this type would be an interface designed to visualize and compare the degree of “invisibility” [12] which certain interface technologies promise to their users. (an issue studied by Donald Norman’s book “The Invisible Computer” where the author asserts that “technology should be invisible, hidden from sight”.).

To illustrate the benefits of this re-orientation of artists’ role in HCI, this essay first studies why interfaces specifically afford artistic exploration. Then, it presents Pold’s treatise on “Interface Aesthetics”. After that, the field of artistic research is introduced in the context of our topic, as well as some of its methodologies and methods. Finally, three examples of recent critical interfaces are analyzed, compared, and discussed in order to exhibit some of their design patterns.

2. Interface Aesthetics

2.1. Interfaces as Aesthetic Artifacts

The central point of this section is unfolding a vision of interfaces as non-neutral technologies for communication. As it will be shown, this non-neutral character is the key factor which makes tangible interfaces suitable candidates for becoming aesthetic artifacts.

The study of “interfaces” as an aesthetic category has occupied a large part of the Cultural Studies research agenda. During the 1990s, Don Norman answered the question of Why Interfaces don’t work?, Steve Johnson published his book Interface Culture, and Manovich released The Language of New Media and Interface as a New Aesthetic Category. Since then, a significant number of authors have continued studying the consequences of the massive use of interfaces from the perspective of Digital Humanities and Digital Arts.

Akrich and Latour [13] have informed on the ways interfaces are scripted with predefined roles of use and with specific politics. For Galloway [14], an interface is not a thing, it is mostly an effect: often we can only perceive the interface through its effects. For Don Ihde [15], interface technologies are never stable entities, as they are conformed from a multiplicity of black-boxed processes and assemblies we cannot fully achieve. Reinforcing this argument against neutrality, Manovich [16] asserts that “the computer interface acts as a code which carries cultural messages in a variety of media [...], and in cultural communication a code is rarely simply a neutral transport mechanism”.

Interestingly for us, this perception of interfaces as non-neutral technologies has definitely sparked an interest in artistic exploration. For Pold [16] and Galloway [14], interfaces can be considered “autonomous zones of aesthetic activity”. Certainly, if we are confronted with a medium of entangled non-neutral mediations between agents, this can surely be the perfect midst for enacting unexpected but meaningful relations. Through artistic exploration, critical aspects of this medium can easily provoke imagination. Thus, interfaces will not only adopt a particular functionality, but they can also extend themselves towards aesthetic experiences [10]. In short, interfaces extend themselves towards the Arts. Their permanent exhibition at media art festivals or their presence at art academies [17] illustrate that this phenomenon is already happening, while it still offers huge expectations for the future.

Especially for our community, the practical consequences of this extension are double:

• An extension of tangible interface design towards aesthetics shifts the perception of interfaces as strict functional objects to a vision of interfaces as artifacts with an expressive intention, as long as their components, representations, and interactions can afford aesthetic relationships. As a consequence, interfaces can appear highly disembodied of their original functionalities at the level of the computer system. For instance, a computer keyboard attached to a tennis racket which allows typing on facebook while its user is playing can probably afford aesthetic relationships (an idea inspired on the artwork Writing_racket by the artist César Escudero Andaluz). However, if some of its keys happen to get broken after the impact with a tennis ball, the aesthetic experience will not be affected decisively. The work is mostly about the act of random typing and some broken keys would not radically change its expressiveness.
  However, it is important to clarify that “the aesthetic” does not define itself as the opposite to “the functional”. Aesthetic extensions re-organize many elements of artifacts towards an expressive or artistic intention. This is a typical mechanism of the Arts. A classic example is Man Ray’s work Cadeau (Figure 1), where a flat iron of the sort that had to be heated on a stove is transformed into a dysfunctional, disturbing object by the addition of a single row of fourteen nails. The transformation of an item of ordinary domestic life into a strange, unnameable object with sadistic connotations exemplifies the power of the artistic methods.
  Indeed, the aesthetic dimension of an interface does not act against its functional part. In fact, aesthetics and functionalities always engage together at the game of expressiveness: they are both active agents at the dialectic nature of interfaces. That makes it possible, for instance, that a non-functional interface could be expressive too. In fact, a lack of function always alludes to the already known original functionality of the interface. Thus, the aesthetic reconfigures but never acts against functionality.
• If these aesthetic interfaces are also critical with their research medium, they can become a language and a valid medium for research. They can engage us in aesthetic experiences, but they can also serve us well to evaluate the validity of accepted assumptions regarding design and user experience in our research medium. In conclusion, they can be used as a methodology for research within HCI. This possibility is studied in the following section.

2.2. Critical Interfaces

In this section, the benefits of designing critical interfaces are explained. Therefore, this section defends the role of artists as suitable experts to create critical artifacts with our research medium.

The main objectives of creating critical interfaces are twofold:

• Evaluating our accepted assumptions about HCI design
• Proposing alternative models inside HCI which could better assess aspects of user experience and interaction

The theory about critical interfaces was elaborated by Søren Pold and presented in his articles The Critical Interface [18] and Interface Aesthetics [10]. For Pold, designing critical interfaces means unfolding the paradoxes of assumed interaction paradigms, their standardized limitations and preconceived ways to deal with tasks such as data manipulation or user experience. Hence, critical interface design would be suitable to evaluate the validity of our research methods and the experiential aspects of interaction.

A critical re-orientation of tangible interface design, or as Pold claims, an aesthetic approach to tangible interface design, allows us to explore tangible interfaces as a form, as a language, and as a medium for research. For Pold, the improvements of being critical to the assumptions regarding interfaces can be quite relevant. Through interface aesthetics, HCI becomes re-oriented as a critical field—a feature that traditional HCI methodologies do not tend to incorporate.

Critical interface design suggests the use of alternative methodologies open to the inclusion of cultural and idiosyncratic aspects of interaction, at least with the same weight that the search for efficiency at the computer level has traditionally sustained. As Søren Pold defends, art-based methodologies allow us to observe in a closer and more complete way the assessment of the various aspects and effects produced by interfaces. Not only their functional aspects but also their sociological, cultural, and economic features. In a world where interfaces are often becoming invisible, digital arts show us “how the interface changes what and how we see, how we experience and interact with reality and how this reality is reconfigured through the computer” [10].

From a vision where designing tangible interfaces would be aimed at increasing user-centered design and efficiency, art-based research proposes a perspective aimed at evaluating the validity of long time acquired assumptions and research methodologies within the field of study.

In conclusion, for Pold, digital arts are the adequate research companion for HCI [10]:

“Instead of focusing only on functionality and effects, digital art explores the current materiality and cultural results of the interface’s representational effects. What are the representational languages of the interface, how does it work as text, image, sound, space and so forth, and what are the cultural effects, for instance, of the way it reconfigures the visual, textual or auditory? How does the interface reconfigure aesthetics and what does it do to representation, communication and, in continuation of this, the social and the political?”

2.3. Traditional Visions about the Arts in HCI

A certain vision of artists at the end of the production research chain still exists at many HCI labs. For instance, after the first public exhibition of the tangible interaction project Transform [19] by the MIT Tangible Media Group, its director Hiroshi Ishii asserted to the press:

“This (Transform) is a white canvas, paintbrush and ink waiting for a Picasso” [20].

Contrasting it with Pold’s critical perspective, this vision diminishes the role of the Arts as a valid medium for research. As we will see in the following section, contrary to many methodologies inheriting from HCI, the artistic research object is not instrumentalized for the engineering apparatus. This means that art-based research does not function as a proof of the theories exposed, but towards creating unexpected relationships between the elements of the field of study. Thus, and following the concepts behind “critical interfaces”, the role of an artist doing research within tangible interaction would not be the direct acceptation of the artistic medium for the creation of aesthetic contents. This role is more related to the visualization of the paradoxes present in the actual design. For instance, applying this concept to Transform, a possibility would be questioning the design decision to focus all interaction only on the surface of the artifact.

Another traditional idea within HCI is formalizing the Arts as a source of inspiration. Certainly, artistic research can have an epistemic value, and artworks can inspire reflection. For Nam [11], art installations carry embodied knowledge; they can convey useful critical messages about their social and political situations and propose novel representational forms. However, for these authors the artistic artifact—the artwork—is always something external to HCI. It is an object of study, a case of analysis done by others. It is art, but it is not research, as the artwork does not follow any methodology towards improving some aspect within HCI. Therefore, the role of the artist is disconnected from the flow of HCI research.

2.4. Artistic Research Methodologies

Although there exists an evident mutual interest between art practitioners and the rest of our community, the application of artistic research methods in Human–Computer Interaction labs is still not generalized. For Edmonds [21] and Candy [22], while practice-based research in media arts is pushing the boundaries of our knowledge about experience design, arts-based research methods are little-known, if not unknown.

Artistic research has received academic interest for the last two decades [22,23,24], but interestingly Pold’s Interface Aesthetics do not mention its contribution.

An artistic practice is considered “research” if the aim of its production is creating some type of knowledge. For Julian Klein [23], the type of knowledge created by artistic research projects is sensual and physical; it is purely an embodied knowledge. This knowledge must be acquired through sensory and emotional perception—precisely, through artistic experience. Tere Vadén [24] explains the notion of artistic research as any artistic process which argues for a point of view —an act inside of the artistic practice carrying some contextual, interpretive, conceptual, or narrative work.

A decisive factor in artistic research is the acknowledge that it is not possible to define “the artistic method”. Instead, there exists a multiplicity of methods which can be applied and combined in artistic research projects. Interestingly, the foundations of science have been dominated by the idea that science is different because it follows the scientific method—a characteristic which separates “the scientific” from other fields of research and experience. In contrast, within artistic research it is assumed that a methodological abundance is the most productive approach.

As listing and fully describing relevant artistic methodologies is out of the scope of this essay, as it would certainly need an independent article, a long and well-documented list of research methodologies for the creative arts and humanities can be found at ECU’s on-line Research Methodologies for the Creative Arts and Humanities guide [25]. However, we can mention Action Research, Constructivism, Ground Theory, Critical Discourse Analysis, Ethnographic Research, Longitudinal Analysis, Positivism, Practice-Based Research, Qualitative Research, Social Constructivism, Survey Research, etc.

Interestingly for us, the concept of “experience” plays a central role in artistic research. Gary Knowles and Andrea Cole [24] have defined artistic research as the systematic use of the artistic process as a primary way of understanding and examining “experience”. This is coherent with Dewey’s notion of “art as experience” [26]. We can agree that the scientific methods implement “experience” in a way that can always be controlled, repeated, quantified, and manipulated. However, in cultural phenomena, “experience” is at odds with repeatability and control. Let us simply think of a live concert or a theater piece where non-systematic repeatability is awarded. For this reason, an “artistic experience”—or as it is also defined, an “aesthetic experience”—usually requires alternative means for being studied and evaluated. For this reason, artistic research is still a discipline slowly merging other fields of study which were established long time ago (e.g., scientific research).

Finally, aesthetic experiences deal with unreflective and non-conceptual contents which can be materially anchored to physical artifacts transcending their medium—for instance, when a research idea becomes embodied in an art installation. Thus, it is possible to say that aesthetic experiences can be good examples of embodied experiences. This fact is especially relevant for the field of tangible interfaces.

In summary, the goal of artistic research is to open up new visions and interpretations into some proposed questions and artistic experiences. In the special case of critical interfaces, as we will see a few sections below, the goal is the critical study of their own research medium.

3. Examples

In order to exemplify the different forms a critical interface design can adopt, three critical artworks by three different authors are presented in this section. For the selection of these examples, as the whole argument of this essay is built on Pold’s Interface Aesthetics, his three types of interface realism [10] (which will be explained next) were used to select the artworks. It is important to remark here that the data used to describe these examples was obtained through various academic papers published by the artists, two from TEI’s Arts Track conference (2014 and 2016), as well as from direct communication with the artists.

For Pold, critical interface artworks should go beyond the safe borders of the autonomous artwork. In other words, to serve as research, artworks should not only be made of autonomous aesthetic contents. In that case, he recognizes three types of works dealing with alternative realisms of the interface which can be combined in the same artwork:

• Artworks dealing with Functional realism: those visualizing the functional elements of the instrumental medium as components with aesthetic possibilities. Those artworks making use of operational elements of the interface which can be constituted of some aesthetic value. For example, artworks hacking hardware or software components or reconfiguring some functionality to create unexpected situations.
• Artworks dealing with Media realism: artworks going beyond the visual surface of the interface towards the imperceptible and unreadable code. Artworks of these type can, for instance, show the codes behind the screen and reveal the normally hidden flow of codes that the user interaction causes.
• Artworks dealing with Illusionistic realism: artworks beyond pure representation, interfaces maximizing reality towards immersive simulation. These artworks make the user forget about the interface and become immersed in the illusionistic world it presents.

Therefore, three artworks have been selected herein following Pold’s types of realism, and they will be presented in the next sections. A comparative analysis as examples of artistic research is described after its presentation.

3.1. Interfight: Functional and Media Realism

Interfight [27] by the artist César Escudero Andaluz is a set of data polluters living on top of devices incorporating touch screens (Figure 2). A number of small robots equipped with conductive plastic sheets attached to motors are designed to freely stroll on top of our tablets and smart-phones. These bug-shaped robots pollute our tangible interfaces by clicking, selecting, zooming, or scrolling indiscriminatingly. For instance, they can type random comments on your social networks or search non-existing concepts on Google.

Interfight was created with a critical intention with HCI in mind. It shows us the following critical characteristics present at our touch interfaces:

• Interface design homogenization: Currently, many commercial websites track human input activity with hidden services to automatically improve their GUI (graphical user interface) design using machine learning strategies. Interfight makes us aware of this issue and pollutes those tracking services with random data produced by physical bots. They are a mechanism against interface design homogenization.
• Against stable interfaces: These artificial bugs entangle and reconfigure the graphical interface in strange ways, re-arranging desktop appearances and graphical customizations. They work as a mechanism against the principle of “perceived stability” [28], which defend the idea that the elements in the computer interface should not be changed without the user’s involvement.
• Bodily interaction: Touch interfaces limit our bodily interaction to an extreme degree—we are radically reduced to the surface of our finger tips. That is what makes simulating physical human input so easy.Interfight questions why have we accepted that other parts of our body are not suitable for interaction. Or maybe, should we extend our fingers with other non-human artifacts?

To conclude, Interfight proposes a dysfunctional extension of a tangible interface for studying the actual interaction paradigms implemented in tablets and smartphones, and a critical re-orientation of the standard way we access information using touch-screens. Malfunctioning is here the key to re-orient our focus and to better address experiential aspects of interaction on these types of interfaces. Interfight deals with both functional and media realism. Functional realism because it rearranges the original elements of touchscreens into aesthetic components (e.g., desktop). The utilization of media realism is clear, as the artwork shows us the hidden mechanisms which commercial websites use for tracking our human activity.

3.2. Hatching Scarf: Illusionistic Realism

The Hatching Scarf [29] is a wearable by the artist Young Suk Lee (Figure 3). It is composed of a feathered scarf and pouch. The scarf is equipped with servomotors and sensors. It opens and closes itself with similar movements to those of baby birds requesting food. The artist has also included some pieces of jewelery in the form of larvae which vibrate and make sound when the nest moves (a video can be found at the artist’s website [30]. A flex sensor is used to detect when the user puts her arm close to her mouth as in the physical attitude to eat. The pouch contains chocolate.

Apart from its aesthetic dimension, Hatching Scarf holds a critical position with its research medium. For the artist:

"In HCI there are many applications and tangible objects that focus on how we can measure quantified data about ourselves, which encourage us to pursue desired behavior and to support our achievement of goals. Yet to what extent do many of these persuasive technologies inadvertently contribute to unhealthy anxieties?".

For the artist, Hatching Scarf is designed to help us interrogate:

• Self-identification at interaction. Hatching Scarf attempts to study how we identify and represent ourselves by experiencing the aesthetics of interaction. Evoking critical thinking via socially engaged objects, the work explores the concept of “extended self”: how one’s “colors” may by revealed depending on their personal history, philosophical differences, perception gaps, experience, interests, culture, education, and so on.
• The language of design: Interface rules and intentions are communicated using different levels of rhetorics to grab the user’s attention [31]: informative (rational and without emotional impact), entertaining (with a soft emotional impact), and affective (with high impact to persuade audiences). Hatching Scarf makes us aware that these strategies for communication are also designed, and lets us consider if the language of the interfaces we use daily are always adequate for our emotional situations.
• Personal affection of design: Designs in HCI can carry or generate unexpected personal conflicts and anxieties to its users. Are devices solving our needs or generating new personal issues? Hatching Scarf—an artificial device—puts us in an estranging situation simply by eating some pieces of chocolate. Thus, the work encourages us as interface users to reflect on our persons as both objects and subjects of knowledge which are affected by the characteristics of certain design decisions.

Hatching Scarf bring us to an embodied relationship with the interface, where technology is designed to affect us to the maximum. The artwork plays in the Illusionistic realm.

3.3. Tangible Scores: Media and Illusionistic Realism

A Tangible Score [32] by the artist Enrique Tomás (author of this essay) is a tactile interface for musical expression that incorporates a score in its physical shape, surface structure, or spatial configuration (Figure 4). Every interface is conceived from a different graphical score that still represents a musical idea, but it has also been specially designed provide a diverse palette of acoustic signals when touched. More importantly, the tactile scores define and propose specific gestural behaviors due to the different affordances and constraints of the object in front. A video of the interface in action can be watched on-line [33].

The critical aspects of Tangible Scores are:

• Non-linguistic communication: In a Tangible Score there are no explicit symbols on the interface which can be explored by the users, only a continuum of materiality. Users are confronted with a total absence of information about how they can manipulate the different parameters controlling the sonic engine. The interface makes us aware of the hyper-abundance of symbols present at interaction design, making us reflect on the issues that the great dependence on cognitive and linguistic communication has created on musical expression [34].
• Performative materiality: With a Tangible Score users are forced to “think the materiality” they find in front of themselves. In fact, the project Tangible Scores was inspired by the recent theories of New Materialism which investigate “the incessant materialization of the world” [35]. Under this theory, matter is not a stable substance. The agency of materials cannot be described until they are performed. In this context, Tangible Scores aims at discovering the performative properties every material features (our interfaces too), remarking on the importance of the physical materiality of our interfaces.

Tangible Scores unfolds for us the issue of the hyper-abundance of symbols we need to learn and employ when playing musical interfaces. The interface visualizes an alternative to the hyper-parametrization of electronic music. For this reason, it can be a work of media realism. However, it also explores new types of embodiment with the interface, ideally trying to make the instrument disappear when performers are well trained to play it. In that moment, performers would be immersed in media (in this case, in sound).

4. Discussion

In this section the critical aspects described in the previous examples are analyzed comparatively. That will allow us to inform our community about possible interaction design patterns. To this end, an analysis model based on three concepts is proposed: problematization, methodological conceptualization, and artistic inquiry.

• Problematization. The systematization of artistic research projects within HCI as problem-solving processes has been been suggested by Oulasvirta and Hornbaek [36]. For the authors, the problematization of some artistic practice can help to transform an artistic process into a process of research, even more than enveloping it together with some theoretical discourse. To illustrate this model, let us imagine the case of a photographer who problematizes how she can better approach certain critical discourse about a community of people within her practice. For her, she will solve her research problem if she finds an adequate solution to communicate specific concepts within her work through some expressive means.
• Methodological conceptualization. After having problematized the artistic research case, artists will need to choose among different methodologies for carrying out their research, or even a combination of various methodologies. Following the example of the photographer, let us imagine that she decides to make use of some ethnographic research method, studying the culture and social organization of the community on which she is focusing her work. The data obtained from this ethnographic process serves to create a theoretical analysis of her practice which will be used to transfer her conclusions again into practice.
• Artistic inquiry. The conceptual element—the theory behind the practice or the phenomena being researched—will now need material inquiry and material perception. Again, in the example of the photographer, she will now work in the field taking photographs, but having a more substantiated and developed strategy for remarking on aspects she finds interesting for her work. Undoubtedly, the new design process will likely create new theoretical questions. Thus, a permanent cycle between the conceptual and the material elements is produced, forcing the researcher to learn, and sometimes producing new approaches or new artworks.

Certainly, this three-step model of artistic research will usually be iterated a number of times. The process, not always sequential or linear, can alternate between conceptual elements (e.g., communicating some theory about practice) and material elements (e.g., an art installation) as a continuous process of problem-solving. In fact, this is not an issue specific to artistic research, but to any type of research. A notable reference of this iterative research process is Bruno Latour’s Actor–Network Theory [37] applied by the author to research in natural science. For Latour, research is characterized by “friction, surprise and the obstinate nature of objects in the environment”, suggesting a continuous cycle of research.

4.1. Problematization of the Examples

• The idea for producing Interfight came from an invitation from the research program MEMBRANA, a residency for “Artistic Interface Criticism” offered by Hangar (Barcelona, Spain) aimed at providing support to a visual artist interested in developing an artwork based on the concept of interface, and participating in the investigation of a critical interface manifesto. Thus, Escudero Andaluz problematized his project as follows: Is it possible to create an interface-artifact to criticize the actual effects of computer interfaces? Which are the aspects of interfaces which are not visible and interesting to communicate nowadays?
• Hatching Scarf comes from the problematization of a personal intuition. The fundamental theme in Young Suk Lee’s work concerns how “ecosystems, societies, and life itself form an interconnected web where the disturbance of any part affects everything. As human beings an inescapable part of life is our interaction with other creatures” [29]. For the artist, environmental influences evoke inevitable engagements with social norms. This forces one to adopt socially desired ideals in response to certain pleasures, attitudes, etc. Thus, the research in Hatching Scarf attempts to solve the problem of creating socially engaged artistic artifacts which can help to show how one becomes fragile in nervous situation under societal influence.
• Tangible Scores appears as part of the PhD of this author at the Kunstuniversität of Linz. The spark for creating it comes from a research question: Is there the possibility of creating musical interfaces incorporating the idea of “musical score” in their configuration? What could be the philosophical and practical consequences of understanding musical interfaces as musical scores? This question arose after the observation that many electronic music instruments carry equally the notion of instrument and composition [32].

A conclusion obtained from this analysis is that the questions formulated at the initial moment of problematization are precise, but still far from the critical knowledge created by each artwork (and described in Section 3). These initial questions carry quite a lot of research force present in every artistic project, but their final artifacts will be the most relevant source for extracting knowledge and new questions. The embodied and aesthetic experience will afford much more relevant aspects about each of the topics investigated.

4.2. Methodologies of the Examples

All of the examples presented above share Art-Practice as a research methodology as long as the artistic artifact created is the basis of the contribution to knowledge. Without experiencing the artistic artifact, all knowledge it contributes (it being of any type) could not be fully achieved. For example, without experiencing the possibility of wearing and touching the Hatching Scarf, it is impossible to understand the real value and type of the knowledge proposed. For instance, the sensations and emotions created while wearing it as well as the critical message which it conveys.

Additionally, the authors of these artworks manifested the influence from the following methodologies as the most relevant:

• Interfight makes use of the Media Analysis methodology. This methodology suggests the examination, interpretation, and critique of both the material content of the channels of communication media and the structure, composition, and operations of corporations that either own or control those media—in this case, the Internet. For its development, Escudero Andaluz investigated different strategies of control that companies are using on the Internet without informing their users. Among different possibilities, the artist found it suitable to work with those hidden tracking services for GUI optimization. For instance, many websites track the path users make with their pointers on screen while navigating their pages or the pixels where they usually stop more often, etc. Thus, through this methodology, the artist obtained enough data to fully understand fully the issue and be able to respond with an artistic artifact.
• Hatching Scarf shows a typical example of Critical Discourse as methodology. It asserts that human interaction and social practices are tied to specific historical contexts and are the means by which existing social relations are reproduced or contested and by which different interests are served. It deals with the questions pertaining to interests that relate discourse to relations of power. The methodology accompanying Hatching Scarf criticizes the social aspects of HCI’s language of design, thus giving enough examples of devices and clinic cases of study which inspired the final artwork.
• Tangible Scores appears as a clear case of Practice-Based research. In this case, the author’s practice (concerts, composition, etc.) with the interface is the basis of the contribution to knowledge. While the significance and context of the research claims could be described in words, a full understanding can only be obtained with direct reference to the practice outcomes. With Tangible Scores, many aspects of its contributions to research come after the direct observation of the artifact created, once it exists. Thus, following this methodology, the artifact informs the author important concepts which were in fact not formulated at the moment of design and performance: they can only be perceived at the moment of experiencing it, at the moment of practice. Performing a Tangible Score means here the production of creative enactments in the relationships created between our bodies and the materiality of the interface, be it at a live performance on stage, at an exhibition, or any other form of presentation.

We see how methodologies have been the process to unify attention, to focus the perspective of every one of these projects towards one research direction—towards the creation of a particular type of artifact. Methodologies have informed the authors about the complex reality of their research in an objective way, opening up different possibilities for developing the material part of the interface. However, they also serve to limit those cases which do not seem interesting enough for the artist.

4.3. Artistic Inquiry of the Examples

The artistic inquiry process is informed by the conclusions or data extracted from the previous two steps. This is the moment for materializing the artistic process which characterizes the artistic research project. An important facet of this process is that many of the decisions taken will have a clear subjective and idiosyncratic character. As every artist understands her artistic practice differently, only her artistic culture will help to decide which particular aesthetic direction will be taken. This decision depends on the various influences of cultures and artistic trends which have actuated over time on the artist.

• Interfight follows the aesthetics of dysfunction, understood as an artistic practice emerging from the alteration of the functional change of some of the physical elements of an electronic artifact, affecting part of the interaction between a device and its users. The dysfunctional strategy reconfigures the original “reason to exist” of devices. At the same time, dysfunctions can create new unexpected—even ironic—relationships with the objects, shifting the original intention of an artifact from the functional to the artistic or poetic.
  Transforming a tablet device into a dysfunctional artifact made it possible to communicate the author’s critical intention. Escudero Andaluz extended the interface with elements (the robots) that can interact with capacitive screens. Those small robots equipped with conductive plastics create valid human input interaction to confuse on-line tracking algorithms. If convincing a user to act randomly for hours in front of a tablet sounds difficult, here those small and simple robots show themselves as quite efficient at that task.
• Hatching Scarf: in this case, the artwork follows the aesthetics of estrangement as an expressive strategy. The wearable transforms its user into a kind of artificial bird, while the aspect and movement of the scarf creates a deep sentiment of estrangement. The inclusion of the colors red and black, and even jewelry reminiscent of the form of larvae contributes to this perception.
  The strategy of estrangement is typical of the Arts [38]. However, it has been used as a successful method for improving or suggesting novel design developments in tangible interaction [39,40]. Being confronted with a certain estrangement requires re-learning of the environment, bringing the mind and body to unfamiliar situations, therefore producing certain enactment of reflection and new embodied relationships. For Wilde, Vallgårda, and Tomico [40], estrangement creates a confusion which prolongs the moment of arriving to an understanding, hence provoking a deeper and more intimate relationship with the concepts and materialities the situation carries. These authors proposed a framework for analyzing estrangement in HCI following an iterative process of answering “what is done to disrupt”, “what is destabilized”, “what emerges”, and “what the whole embodies”. Thus, for Hatching Scarf:
  • The physical aspect of the socially engaged wearable is made to disrupt, to make the user feel estranged.
  • The self-representation of oneself during interaction and the communication with others gets destabilized.
  • It emerges a reflection about how certain types of HCI design can create personal anxieties to their users in socially constrained environments.
  • The whole process embodies knowledge about the risks of choosing inadequate languages of design (e.g., in the case of technology dedicated to children). Additionally, it embodies a full identification with other people suffering certain personal affections when using technology in certain personal, social, or political situations. Why is the simple action of interacting making me feel estranged? Is it because it visualizes my concerns about snacking and my weight?
• Tangible Scores follows an aesthetic strategy inspired by graphic scores [32]. Graphic scores appeared in the musical avant-garde as a way to release composers from the constraints of writing their music using the notation of a traditional score. Consequently, the representation of a musical idea opened to the personal and subjective selection of graphic figures that inspire new and imaginative ways of interpretation. One of its earliest examples can be seen in Figure 5, Earle Brown’s November 1952.

Thus, Tangible Scores’ appearance seems to release their original nature as inscriptions on paper: they extend themselves towards materiality and performativity, transforming the nuances of the materials employed into changes of the parameters used in interaction. It will not produce the same sonic outcome, for example, when creating the scores out of wood, rubber, or metal, as the algorithm depends on the harmonic properties of the sounds created in the objects. At the same time, looking at graphic scores helps musicians to find ways to navigate the “interface”. As many performers have experience in performing graphic scores, they already have methods and skills for exploring the graphical ideas drawn by contemporary composers. For instance, non-linear reading and random navigation are implicit characteristics assumed by its performers which do not need to be explained.

5. Conclusions: How the Arts Can Help Tangible Interaction Design

In this essay it has been shown that the role of artists in HCI has to be more with being critical with their research medium than with creating mere artistic experiences to inspire novel forms of design. With the analysis of three examples, we have informed about three different directions for developing artistic research projects in terms of three notions: problematization, methodological conceptualization, and artistic inquiry.

Far from suggesting any dogmatic solution within a field where there are not universal solutions, the following conclusions act as a theoretical solution of our own problematization inquiry in this essay:

How can the Arts help tangible interaction design?

• Adopting a critical attitude with our own research medium: tangible interaction. Artworks can act as breakdowns of the standard discourses of tangible interaction design. They should serve us to discover new interpretations and notions, often unfolding the tension among established notions of our fields of study and their real value in our societies.
• Adopting the format of artistic research. The Arts are a valid medium for research. The Arts create a type of knowledge which is not afforded by traditional HCI methodologies—especially in the case of embodied technologies. However, research in the Arts should have a clear formalized structure and a clear focus. Finally, it may rely on the problematization of its aims and the proposal of alternative methodologies.
• Avoiding the instrumentalization of its artistic process: The artist’s role deals more with a permanent questioning of the inner pillars of the research field. It must be more to do with destabilizing than with supporting what has already been fixed. Artists should not start their research process at the end of the HCI research chain.